How we were friends Neo4j and Meteor

Writing a Neo4j graph database support driver for Meteor


In Meteor, any work with data is associated with two-way reactivity. At the moment, the built-in Meteor MongoDB and Redis (both drivers are developed within the Meteor walls) have 100% reactivity, partially reactivity is implemented for MySQL and MSSQL (third-party developers).

For the above databases, reactivity is implemented through observers that report where, how, when, and what data has changed so that the driver serving the connection [data <-> view] knows what data and which Clients to update. Neo4j is deprived of any watchers and observers, but this did not stop us. How do we get out of this situation and why do we need Neo4j read under the cut.

Under node.js there is an official and homonymous npm package from Neo4j manufacturers, which is replete with functionality, but in the first two lines of its documentation it is gently hinted that only the connection to the GraphDatabase and the query method work stably .

Why do we need Neo4j


I believe that for each task there should be a tool created and designed to perform this task at the highest level. Everyone knows that one nail can be, and sometimes it is necessary (if the time spent searching or the funds to buy a hammer are unreasonably high) to hammer with a slipper. But for 100, and even more than 100,000 nails, it will be advisable to acquire a hammer, and preferably a nail gun. In our case, we need to store and receive data about the relationships between the records. We continue to store the data in denormalized form in MongoDB, but we store the relationship of this data in Neo4j.

How it all began: Connector


Initially, it was assumed that by creating a global variable containing an object of the GraphDatabase type , the functionality supplied in the npm package would be enough for the tasks: at that time we were writing / reading data to / from the database (without reactivity). This is how neo4jdriver was born - a package containing the globally accessible class Neo4j , during initialization of which a connection is created with a database launched locally or remotely. During initialization, you can pass a single url parameter .

Later there was a need for:
  • reactivity;
  • observer
  • isomorphism - the ability to fulfill requests both from the server and from the Client;
  • structuring incoming data - by default, Neo4j throws out a bunch of garbage that decently weighs on any MATCH request.

And then the fun began.

How it went: Reactivity


The neo4jreactivity package, based on the principle of pseudo-reactivity implemented through an interlayer in the form of MongoDB, was released second. Simply put, for any request in Neo4j, we return Mongo \ Cursor , which in turn is a source of reactive data or, as is commonly called in the Meteor community: REACTIVE DATA SOURCE.

Initially, everything seemed simple:
  • We make some caching collection in MongoDB containing the request, the hash of the request and the response from Neo4j;
  • For the Client, create a session in which we hold an array containing all the hashes of the requests that you need to subscribe to;
  • We pass all requests through the Meteor.neo4j.query method , which creates a hash from a request to the database, receives a response from the database, writes it to the database and sends it to all Clients subscribed to this request hash;
  • To launch requests from the Client, we make a Meteor method that eats! any request and executes it on the server.


At the time of the release of one of the first versions of the driver, on the Client you could run absolutely any request, i.e. could change, get or erase all data stored in Neo4j. This problem was solved by introducing the methods Meteor.neo4j.methods ({}) and Meteor.neo4j.call (methodName, opts, callback) , which work on the principle of the standard Meteor.methods ({}) , example:

if(Meteor.isServer){
    Meteor.neo4j.methods({
       ‘GetUser’: function(){
          return ‘MATCH (n:User {_id: {userId}}) RETURN n’
       }
    });
};
if(Meteor.isClient){
    Meteor.neo4j.call(‘GetUser’, {userId: 123}, function(error, data){
        if(!error){
            Session.set('theUser', data);
        }
    });
}

The second thing we did was property Meteor.neo4j.allowClientQuery , which takes the value true and false , and defaults to false . This will allow developers to work in the browser console during the development and testing of the application, send data and verify received data.

If for some reason you decide to leave the possibility of executing requests to Neo4j from the Client, then the following functionality is provided that allows you to limit the type of requests to Neo4j. Two methods are available to you: neo4j.set.allow and neo4j.set.deny . Both methods accept a single parameter - an array of strings. Additionally you can use arrays:Meteor.neo4j.rules.allow , Meteor.neo4j.rules.deny and neo4j.rules.write , which contain the current rules, and the latter contains an array with write operators, which allows you to make such a shortcut:

if(Meteor.isClient){
    Meteor.neo4j.set.deny(Meteor.neo4j.rules.write);
}

And prohibit all recording requests from the Client. All methods described in the paragraph above are isomorphic. Hack from the Client’s side will not pass, since the data is additionally checked for integrity on the Server side.

We follow the data: our Observer with blackjack and listener


It was later discovered that data change requests did not initiate data updates on Clients. Reactivity simply did not work until one of the Clients accessed the changed data and initiated the update in MongoDB, and as a result, on all Clients. This was due to the fact that we did not have an observer to monitor the changed data and initiate the launch of all requests related to the data that changed.

Listener

We return to our ideal package called neo4jdriver , erase the entire project and write again:
  • We leave the class structure and initialization of the class instance with the ability to pass the url to the database;
  • We create an array GraphDatabase.callbacks that stores callbacks that take two parameters - query and opts ;
  • We add the GraphDatabase.listen (func) method , which takes a function with two parameters - query and opts , all functions fall into the GraphDatabase.callbacks array ;
  • We reassign the query method built into the npm package - adding to it the launch of all the callbacks from the GraphDatabase.callbacks array .


Reactivity Observer:

First of all, we need to learn how to separate the request data from the request structure; for this, the sensitivities parameter was introduced . This parameter contains data that is subject to change. Now the entry in the Neo4jCacheCollection is as follows:

uid             // Unique hashed ID of the query
data            // Parsed data returned from Graph
query           // Original Cypher query string
sensitivities   // Sensitive data, which contains a map of parameters, and hardcoded data into query
opts            // Original map of parameters for the Cypher query
type            // Type of query ('READ'|'WRITE')
created         // Creation time


Bind observer and listener:


  • We put a wiretap on all requests to Neo4j;
  • We get sensitivities of the current request;
  • Find all read requests that include sensitivities from the current request;
  • We start re-sampling for the received matches;
  • Further necessary reactivity will be provided by the layer in the form of MongoDB.


We managed to ensure that the data is updated when they change - on all Clients, in a very simple way.

We get only the data we need


The third problem was the data that came from Neo4j. In addition to the fields we requested, we also get a bunch of empty objects that the npm package returns to us. Empty objects weigh a lot and do not contain information, we do not need to store them. To separate the useful and requested data, the parseReturn method was written , which parsed the query in the database (Cypher query) and understood what data was requested and what fields the developer wanted to receive. After that, for each requested information, an object was created containing an array of nodes with their data and metadata. If relations of nodes are requested, each node contains a relations object that contains data in the form of the following parameters:

  • extensions
  • start
  • end
  • self
  • type


We deliver updates to customers


We learned how to update data in MongoDB and monitor their changes in Neo4j, but nested objects in the returned data will not be updated by themselves. The functionality offered by the reactive-var package came to our aid. To do this, on the Client, when receiving data from the Neo4jCache collection, it is assigned and returned via ReactiveVar . On the Server, upon receipt from the collection, Neo4jCache will be returned from the Promise. On the server and the client, it is enough to call the get () method to get the data realistically. For those who need to get Mongo \ Cursor, there is a property cursor .

Example:

/* Изоморфный запрос (Клиент и Сервер) */
allUsers = Meteor.neo4j.query('MATCH (users:User) RETURN users');
/* Получаем данные из запроса */var users = allUsers.get().users;
/* Получаем Mongo\Cursor запроса */var usersCursor = allUsers.cursor;
/* Изоморфный запрос (Клиент и Сервер) через callback*/var allUsers;
Meteor.neo4j.query('MATCH (users:User) RETURN users', null, function(error, data){
  allUsers = data.user;
});

At this point, we created a test application and published it on GitHub. A week later, the community of developers helped us “finish” the driver and fix minor bugs. I will be glad to questions and suggestions for improvement and further development of the project. Thanks for attention.

References:


PS At the moment, Neo4j is actively involved in the development of the project and has recognized this driver for Meteor as official.

Also popular now: