June 4, 2008 at 10:50AJAX Client Storage Study - From Dojo Surgery to Native Fullajax Storage Implementation
With the intensive development of web applications, storing large amounts of data on the client side is becoming more and more popular. This does not just simplify the application - often by using a client storage system it is possible to significantly expand the functionality of the created application, allowing it to do something that would otherwise be impossible.
At the same time, this should not be taken as a “whim” of developers of some special and exotic web services - according to the latest data, even such a grand among mass Western social services as MySpace will use a Google solution - Google Gears (more details - gearsblog. blogspot.com/2008/05/myspace-message-center-is-now-searching.html) The network has a number of publications on the use of local storage, for example, the following publications that we studied during the preparation of the material can be noted:
browserpersistence.ru
abrdev.com/?tag=storage
pablotron.org/software/persist-js
If you have no idea what is local storage, or you are not familiar with this, at the above links you will find comprehensive information regarding storages and its applications.
Having examined all possible options for implementing local data warehouses, the most attractive at the moment is Flash-based storage. Note that we are talking about persistent data storages, that is, saving any data there, we expect to be able to access them on the client computer after closing the page in the browser, and updating, and even after restarting the computer. This article will analyze the implementation of Flash-based storage.
Let's look at the clear benefits of Flash Storage:
Some developers are inclined to attribute to this method the disadvantage that supposedly users block Flash because of banners, however I consider this problem more far-fetched. If so, they will turn off the graphics and scripts - which means, as they say in one saying, “themselves pinocchio” - Flash is now often an integral part of many sites and the only acceptable technology for implementing multimedia in a web environment.
It should be noted that of all the Flash stories we analyzed, the most attractive in terms of functionality was the story, which is part of the Dojo library (dojotoolkit.org). Moreover, by and large, this is the first industrial implementation of this technology.
Distinctive advantages:
Of course, this solution, with all its advantages, is not without drawbacks:
A test example download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/tests/test_storage.html?forceStorageProvider=dojox.storage.FlashStorageProvider loads 13 (!) Scripts. All of them are downloaded using AJAX, and even if you explicitly specify to use Flash Storage, a lot of additional files are loaded, which are not needed for this storage to work.
The simplest helloworld example is download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/demos/helloworld.html loads at least a number of JS scripts. Therefore, we can conclude - if your application does not use any of the Dojo Toolkit functionality, using it to implement only Storage is quite resource-intensive and not efficient.
Since the “root of the flaws” is one and the same, we decided to get rid of these flaws, or at least minimize their impact. After downloading the Dojo 1.1 distribution, I had to sit down to analyze its architecture in order to separately isolate only Flash Storage. This is where the first difficulty turned out. Subjectively, the internal architecture of Dojo is very non-trivial and complex (well, of course, because this is the most powerful framework of all, probably implemented on JS, using several programming paradigms, for example, it has recently been replenished with a module for implementing aspect-oriented programming). Therefore, it was not possible to simply and separately separate Flash Storage. But the desire to eliminate these shortcomings turned out to be more than the difficulties encountered.
The first stage of work took several hours of work, and this version (without changing the internal architecture of the code) weighed more than 100 Kb (Dojo core + Storage module). Having made an in-depth analysis, we realized that there was still a lot of excess left. The second stage, in order to squeeze this super-minimized version even more, has already taken several days and this implementation takes only 15 Kb, which came at the cost of a huge number of changes to the Dojo Toolkit source code.
Already more detailed tests of Flash Storage showed another advantage: the storage is the same for all browsers on the computer, i.e. Data saved using Firefox is easily taken from Internet Explorer and vice versa.
An additional drawback was also identified: you must wait for Flash Storage to initialize. That is, the use of storage is possible only in asynchronous mode, after the flash clip of the storage is loaded. Accordingly, web applications using Flash Storage must be designed with this in mind.
But the ability to simply store some data on the client and receive it for the application to work is not the only functionality. I would very much like to be able to control the data in such a cache and automatically update it from the server side.
When using local Storage, a very important point is the implementation of the data update algorithm on the client side. We bring to your attention three different implementation options.
The method of global versioning a site
The algorithm of this method is similar to the principle of implementing SVN. The site has a global version number. Any changes to individual content or a group change in content increases the version number. During the initial loading of the site, the version number on the client side is sent to the server. The server verifies the sent version on the client with the current version on the server and, if there are differences, a list of changed identifier keys of the content is generated in response to the difference between the client and server versions of the site. In other words, when changing content on the server, a list of content keys is sent to the client, which must be deleted in the repository. This list corresponds to the difference between the version of the client’s site and the current version on the server. At the same time, the content itself is not sent to the client.
Original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-1.jpg The
considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This method is convenient to use with a large number of content units.
Method of versioning individual units of content
The algorithm is based on identifying the uniqueness of specifically each individual unit of content. The bottom line is that when the site boots up, a list of value pairs {key: novelty identifier} comes from the server. The client script checks for matching novelty identifiers in the repository. If there are differences, old values of novelty identifiers and data corresponding to the updated content key are deleted from the repository. Immediately in the store new values of identifiers of novelty are written. Just as in the previously considered method, content is sent to the client only when the client requests specific data and, if it is successfully loaded, it enters the repository.
Original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-2.png
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This algorithm is simple to implement and its use is effective with a small number of content units. With a large number of content units, the list of value pairs {key: novelty identifier} may be too large, which reduces the effectiveness of this algorithm.
Standard Etag Usage
Method The method is based on sending the content identifier to the Etag server. If the sent identifier on the client matches the current server identifier, the server responds with 304 Not modifiedand the client takes data from the storage. If identifiers differ, the server responds with new content. The client, having successfully uploaded new content, replaces the old data with new data in the repository and writes its Etag novelty identifier.
The considered algorithm reduces only the total amount of information transmitted. The method is simple to implement, but in terms of the number of connections to the server it is less efficient compared to the previously considered algorithms.
The choice of using this or that approach depends on the specific situation.
Application examples
Using Flash Storage is convenient not only for storing configuration data and objects of a web application, but also for storing scripts, styles, pictures, and even the content itself. An example of the application of the storage is implemented on the site you already knowfullajax.ru . On this site, for updating data on the client side, the above-considered “Method of versioning individual content” is applied . Scripts and content are stored in the repository, i.e. once loading the page of this site the next time it is taken from the repository until a newer version of the page appears on the server.
It is especially effective to use the AJAX + Storage bundle. The Fullajax library includes a module for implementing Flash Storage. For users who already use the Fullajax library, it’s enough to connect an additional script and when requesting data using AJAX (dax or hax methods), the data will automatically go to the storage. It remains only to take care of the mechanism for updating data on the client side.
Of course, this implementation is not the only one. And in many cases, it will be very lacking in individual functionalities. I will not hide it - the built-in DBMS from Google Gears will really be the best option, but this solution is quite exclusive and can only be effectively used in special cases of web applications. Well, for mass use, including on regular websites, it is really best to use either the solution described above, or one of the universal ones, for example, Browserpersistense or PersistJS.
Material author: Ruslan Sinitsky (sirus, fullajax.ru ), co-author Alexander Lozovyuk (aleks_raiden, abrdev.com ).
At the same time, this should not be taken as a “whim” of developers of some special and exotic web services - according to the latest data, even such a grand among mass Western social services as MySpace will use a Google solution - Google Gears (more details - gearsblog. blogspot.com/2008/05/myspace-message-center-is-now-searching.html) The network has a number of publications on the use of local storage, for example, the following publications that we studied during the preparation of the material can be noted:
browserpersistence.ru
abrdev.com/?tag=storage
pablotron.org/software/persist-js
If you have no idea what is local storage, or you are not familiar with this, at the above links you will find comprehensive information regarding storages and its applications.
Having examined all possible options for implementing local data warehouses, the most attractive at the moment is Flash-based storage. Note that we are talking about persistent data storages, that is, saving any data there, we expect to be able to access them on the client computer after closing the page in the browser, and updating, and even after restarting the computer. This article will analyze the implementation of Flash-based storage.
Let's look at the clear benefits of Flash Storage:
- Maximum cross-browser accessibility (for obvious reasons, we do not consider exotic browser options that do not support technologies common for the web).
- Maximum prevalence: Flash is installed on more than ~ 95% of browsers.
- The ability to store an almost unlimited amount of data (although with some reservations, for example, that the user must explicitly allow storage of the specified volume or set permission to store an unlimited amount of information).
Some developers are inclined to attribute to this method the disadvantage that supposedly users block Flash because of banners, however I consider this problem more far-fetched. If so, they will turn off the graphics and scripts - which means, as they say in one saying, “themselves pinocchio” - Flash is now often an integral part of many sites and the only acceptable technology for implementing multimedia in a web environment.
It should be noted that of all the Flash stories we analyzed, the most attractive in terms of functionality was the story, which is part of the Dojo library (dojotoolkit.org). Moreover, by and large, this is the first industrial implementation of this technology.
Distinctive advantages:
- The ability to store JS-objects, automatic serialization and deserialization (of course, in the JSON format).
- Some bugs of the Flash ExternalInterface operation were taken into account.
- Powerful and universal API for working with data warehouse.
- Built-in namespace support for stored data, work with several repositories at the same time.
- PENDING status.
- If the current storage size limit is exceeded, automatically displays a dialog request for permission to increase the size.
Of course, this solution, with all its advantages, is not without drawbacks:
- A large volume of the distribution kit (meaning that if you use Dojo only for the Storage system).
- When initializing the repository, loading unnecessary code, which may or may not be needed when the application is running.
- Inoperability when running the script from the local file system file: //
A test example download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/tests/test_storage.html?forceStorageProvider=dojox.storage.FlashStorageProvider loads 13 (!) Scripts. All of them are downloaded using AJAX, and even if you explicitly specify to use Flash Storage, a lot of additional files are loaded, which are not needed for this storage to work.
The simplest helloworld example is download.dojotoolkit.org/release-1.1.0/dojo-release-1.1.0/dojox/storage/demos/helloworld.html loads at least a number of JS scripts. Therefore, we can conclude - if your application does not use any of the Dojo Toolkit functionality, using it to implement only Storage is quite resource-intensive and not efficient.
Since the “root of the flaws” is one and the same, we decided to get rid of these flaws, or at least minimize their impact. After downloading the Dojo 1.1 distribution, I had to sit down to analyze its architecture in order to separately isolate only Flash Storage. This is where the first difficulty turned out. Subjectively, the internal architecture of Dojo is very non-trivial and complex (well, of course, because this is the most powerful framework of all, probably implemented on JS, using several programming paradigms, for example, it has recently been replenished with a module for implementing aspect-oriented programming). Therefore, it was not possible to simply and separately separate Flash Storage. But the desire to eliminate these shortcomings turned out to be more than the difficulties encountered.
The first stage of work took several hours of work, and this version (without changing the internal architecture of the code) weighed more than 100 Kb (Dojo core + Storage module). Having made an in-depth analysis, we realized that there was still a lot of excess left. The second stage, in order to squeeze this super-minimized version even more, has already taken several days and this implementation takes only 15 Kb, which came at the cost of a huge number of changes to the Dojo Toolkit source code.
Already more detailed tests of Flash Storage showed another advantage: the storage is the same for all browsers on the computer, i.e. Data saved using Firefox is easily taken from Internet Explorer and vice versa.
An additional drawback was also identified: you must wait for Flash Storage to initialize. That is, the use of storage is possible only in asynchronous mode, after the flash clip of the storage is loaded. Accordingly, web applications using Flash Storage must be designed with this in mind.
But the ability to simply store some data on the client and receive it for the application to work is not the only functionality. I would very much like to be able to control the data in such a cache and automatically update it from the server side.
When using local Storage, a very important point is the implementation of the data update algorithm on the client side. We bring to your attention three different implementation options.
The method of global versioning a site
The algorithm of this method is similar to the principle of implementing SVN. The site has a global version number. Any changes to individual content or a group change in content increases the version number. During the initial loading of the site, the version number on the client side is sent to the server. The server verifies the sent version on the client with the current version on the server and, if there are differences, a list of changed identifier keys of the content is generated in response to the difference between the client and server versions of the site. In other words, when changing content on the server, a list of content keys is sent to the client, which must be deleted in the repository. This list corresponds to the difference between the version of the client’s site and the current version on the server. At the same time, the content itself is not sent to the client.
Original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-1.jpg The
considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This method is convenient to use with a large number of content units.
Method of versioning individual units of content
The algorithm is based on identifying the uniqueness of specifically each individual unit of content. The bottom line is that when the site boots up, a list of value pairs {key: novelty identifier} comes from the server. The client script checks for matching novelty identifiers in the repository. If there are differences, old values of novelty identifiers and data corresponding to the updated content key are deleted from the repository. Immediately in the store new values of identifiers of novelty are written. Just as in the previously considered method, content is sent to the client only when the client requests specific data and, if it is successfully loaded, it enters the repository.
Original image of the algorithm fullajax.ru/doc/storage/Storage-Algorithm-2.png
The considered algorithm reduces the number of connections to the server and the total amount of information transmitted. This algorithm is simple to implement and its use is effective with a small number of content units. With a large number of content units, the list of value pairs {key: novelty identifier} may be too large, which reduces the effectiveness of this algorithm.
Standard Etag Usage
Method The method is based on sending the content identifier to the Etag server. If the sent identifier on the client matches the current server identifier, the server responds with 304 Not modifiedand the client takes data from the storage. If identifiers differ, the server responds with new content. The client, having successfully uploaded new content, replaces the old data with new data in the repository and writes its Etag novelty identifier.
The considered algorithm reduces only the total amount of information transmitted. The method is simple to implement, but in terms of the number of connections to the server it is less efficient compared to the previously considered algorithms.
The choice of using this or that approach depends on the specific situation.
Application examples
Using Flash Storage is convenient not only for storing configuration data and objects of a web application, but also for storing scripts, styles, pictures, and even the content itself. An example of the application of the storage is implemented on the site you already knowfullajax.ru . On this site, for updating data on the client side, the above-considered “Method of versioning individual content” is applied . Scripts and content are stored in the repository, i.e. once loading the page of this site the next time it is taken from the repository until a newer version of the page appears on the server.
It is especially effective to use the AJAX + Storage bundle. The Fullajax library includes a module for implementing Flash Storage. For users who already use the Fullajax library, it’s enough to connect an additional script and when requesting data using AJAX (dax or hax methods), the data will automatically go to the storage. It remains only to take care of the mechanism for updating data on the client side.
Of course, this implementation is not the only one. And in many cases, it will be very lacking in individual functionalities. I will not hide it - the built-in DBMS from Google Gears will really be the best option, but this solution is quite exclusive and can only be effectively used in special cases of web applications. Well, for mass use, including on regular websites, it is really best to use either the solution described above, or one of the universal ones, for example, Browserpersistense or PersistJS.
Material author: Ruslan Sinitsky (sirus, fullajax.ru ), co-author Alexander Lozovyuk (aleks_raiden, abrdev.com ).