DerbyJS News 09.2014

This article describes the latest Derby related events. I will also share thoughts and useful links.

Lever

Lever - the company that created Derby, passed the next (second, third or third?) Round of investments and secured a stable existence for at least the next two years. With what we congratulate them.
Also, after Derby 0.6 was released, Lever makes some efforts to promote the framework:

• Every Thursday at 9 am PST (8 pm Moscow time) video conferences are held. You can catch announcements and links to google hangout on the google groups mailing list (called Office Hours). We asked to be held at such a time, because there are many Russians. The conferences are usually led by Ian (Lever developer), often you can see Nate (CEO Lever, creator of Derby) and Joseph (creator of ShareJS). Silly and smart questions are asked, concepts and plans are discussed, everyone shares their thoughts and experiences. Join now. Recordings of past video conferences with a brief description of the main topics can be found on the Lever blog .
• website design has been updated . Now there is a solid FAQ (including in Russian) and links to useful resources. The only thing missing (and this is the main problem at the moment) is the documentation for 0.6. Workaround: Racer 0.5 is relevant for Racer , and a series of zag2art articles (starting with this one ) will cover much of the documentation for Derby 0.6.
• participation in conferences: Joseph was at ForwardJS (unfortunately did not find the video). Initially, Nate had to go and talk about the intricacies and problems of Agnostic Rendering (a template engine that works on both the client and the server), but for some reason could not, I hope we will still hear this report. Ian was at the JS meeting in San Francisco.

Github

Derby moved to a new group . This is due to the fact that a lot of junk has accumulated in the old one and the name Codeparty did not reflect the essence.
The active part of the community has created its own group , where the modules are slowly being sawn.
Lever also puts out his best practices . Still worth paying attention to the achievements of Decision Mapper .

Community

The community is slowly but surely arriving. Often the way it looks like this: client framework (AngularJS, Ember, etc.) -> Meteor -> Derby. The meteor helps to penetrate the concepts of realtime, full-stack, database on the client, isomorphism and so on, but does not satisfy everyone for one reason or another. In general, here Nate very interestingly talked about the main frameworks.
Recently created a chat on Gitter .

Ideas

There are a few thoughts about what you can add or change in Derby:

Using FoundationDB as a repository.

In the current implementation of livedb, document modification events are caught at the expense of Redis pub-sub (for a single process node, it is possible without Redis). And this part of the functionality is built-in. An external driver is used to store the operations log (oplog) and current versions of documents (snapshots), which can be written, in general, for any database, for example, for Mongo .

Problems in the current implementation:
- the use of two databases, instead of one.
- the use of sequential execution of single-threaded Lua scripts as an alternative to transactions leads to the fact that Redis potentially becomes a bottleneck (although no one has encountered this yet). Also, theoretically, with high collaboration (many clients serviced by different node processes change one document) Redis will need much more than one attempt to apply the operation to the data, since the current version of the data will change from the operations of other node processes, which may affect performance (here too, no one has had any problems yet).
- for cases when the client is subscribed to a sorted query to the collection (Query), there is no way to cheaply understand how a change in the next document will affect its position in this query. At the moment, this is done in a rather clumsy way: the request is applied to the collection, all documents for this request are obtained, the difference is calculated and sent to the client (for applications with a large number of such requests, this causes performance problems). In this case, the problem could be solved using events to change the index in the database. Unfortunately, there is no database with events for changing the index (if you know, share it). It is probably not so difficult to add to most databases, but there has never been a need (as an option, you can ask to add this to Mongu).

FoundationDB has a Pub-Sub and can replace both Redis and Mongo. The presence of embedded transactions will simplify the logic of application operations.
Also on top of FoundationDB, you can write your own Document Model layer, for example, fowl , in which to implement events on index changes. Such an adapter solves the problem of sorted query.

Cons FoundationDB - closed source, not free.

Joseph has already built a livedb-foundation . It's up to the adapter.

Client cache

The idea of ​​Offline web applications implies that the client stores data downloaded from the server, even when the application is not running. And uses this cache if there is no way to connect to the server. The client also stores data changes and, when connected to the server, synchronizes data in the client cache with data in the database, and conflict situations are resolved. Ideally, the user should be able to open the application without connecting to the Internet, work in it, close it and be sure that the next time he opens the application and has access to the Internet, all the results of his work will go to the server.

Derby / ShareJS seems to be a good platform for this. All you need to synchronize the data in the model with IndexedDB, teach Derby to first look for data in the client cache, and then (if a server is available) to request a difference from it (new versions). You also need to store operations offline in IndexedDB when offline, and send them to the server when a connection appears.

Access control

The first access-control module, racer-access, has vulnerabilities with query access and is currently not recommended for use. At the moment, it is recommended to implement access control at the ShareJS level manually or using the share-access module .

There is an idea to combine the validation module and access-control, adding to the circuit racer-schema similarity Firebase Security .

Afterword

To anyone who has questions about Derby / Racer / ShareJS, write to me on Skype vmakhaev.
Do not miss Open Hours in less than an hour. Good luck