October 23, 2013 at 16:07

IndexedDB - unlimited data storage

Good afternoon, dear community.
For those who are not aware of what IndexedDB is and what it is eaten with, you can read here .

And we go further.

Unlimited


In the office in which I work, there was a need to use an indexed local database on the client side and the choice immediately fell on IndexedDB.

But as always there is one “BUT”, this is the very “BUT” - the database size limit on the user's machine is 5 MB, which did not suit us at all. Since this technology was planned to be used in the admin panel of our project, and all users used Google Chrome as the default browser, a decision was made to find a workaround for that restriction through the proxy extension. Having shoveled a lot of information, we came to the conclusion that the restriction on the database size can be removed by using special flags in the manifest of our extension:

"permissions": [
        "unlimitedStorage",
        "unlimited_storage"
    ],



Posting site-extension-site


We go further. We figured out unlimited data storage, but now there is a need to work with the same unlimited database directly from the site itself. For this, sending messages between the site and the extension was used (the extension acted as a proxy, between the site and the unlimited database). To do this, the following flags were added to the manifest of our extension:

"externally_connectable": {
        "matches": [
            "*://localhost/*",
	"ЗДЕСЬ_ДОБАВЛЯЕМ_РАЗРЕШЕННЫЕ_ШАБЛОНЫ_URL "
        ]
    }


It turned out that URLs of the form: *: //google.com/* and http: //*.chromium.org/*, but, http: // * / *, *: / / * are considered valid. Com / are not.
You can read more about externally_connectable here .

We go further.

The stage of writing the very “bridge” between the site and the extension to access the database has come.
Db.js was used as the main library for working with IndexedDB on the extension side, which you can find here .

In order not to reinvent the wheel, it was decided to use access syntax on the site side which is implemented in db.js.

Expansion


And so let's go, create background.js, which we will listen to incoming messages, and respond to them. Listing the code below:

var server;
chrome.runtime.onMessageExternal.addListener(
    function (request, sender, sendResponse) {
        var cmd = request.cmd,
            params = request.params;
        try {
            switch (cmd) {
                case "getUsageAndQuota":
                    navigator.webkitPersistentStorage.queryUsageAndQuota(function(u,q){
                        sendResponse({"usage": u,"quota":q});
                    });
                    break;
                case "open":
                    db.open(params).done(function (s) {
                        server = s;
                        var exclude = "add close get query remove update".split(" ");
                        var tables = new Array();
                        for(var table in server){
                            if(exclude.indexOf(table)==-1){
                                tables.push(table);
                            }
                        }
                        sendResponse(tables);
                    });
                    break;
                case "close":
                    server.close();
                    sendResponse({});
                    break;
                case "get":
                    server[request.table].get(params).done(sendResponse)
                    break;
                case "add":
                    server[request.table].add(params).done(sendResponse);
                    break;
                case "update":
                    server[request.table].update(params).done(sendResponse);
                    break;
                case "remove":
                    server[request.table].remove(params).done(sendResponse);
                    break;
                case "execute":
                    var tmp_server = server[request.table];
                    var query = tmp_server.query.apply(tmp_server, obj2arr(request.query));
                    var flt;
                    for (var i = 0; i < request.filters.length; i++) {
                        flt = request.filters[i];
                        if (flt.type == "filter") {
                            flt.args = new Function("item", flt.args[0]);
                        }
                        query = query[flt.type].apply(query, obj2arr(flt.args));
                    }
                    query.execute().done(sendResponse);
                    break;
            }
        } catch (error) {
            if (error.name != "TypeError") {
                sendResponse({RUNTIME_ERROR: error});
            }
        }
        return true;
    });


But here a surprise awaited us, namely on the execution of the code section:

flt.args = new Function("item", flt.args[0]);


We get the exception

Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' chrome-extension-resource:"..

To solve this problem, add another line to the manifest that allows the execution of user js on the extension side.

"content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'"


I also had to implement an auxiliary function of moving an object into an array, to pass it as arguments to the function.

var obj2arr = function (obj) {
    if (typeof obj == 'object') {
        var tmp_args = new Array();
        for (var k in obj) {
            tmp_args.push(obj[k]);
        }
        return tmp_args;
    } else {
        return [obj];
    }
}


Full listing manifest.json

{
    "manifest_version": 2,
    "name": "exDB",
    "description": "This extension give proxy access to indexdb from page.",
    "version": "1.0",
    "background": {
        "page": "background.html"
    },
    "content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'",
    "externally_connectable": {
        "matches": [
            "*://localhost/*"
        ]
    },
    "permissions": [
        "unlimitedStorage",
        "unlimited_storage"
    ],
    "icons": {
        "16": "icons/icon_016.png",
        "48": "icons/icon_048.png"
    }
}


Client


We’ve figured out the extension, now let's start writing a client library for working with our proxy extension.
The first thing that is necessary, when sending a message from the client, indicate to which extension we want to send it, for this, indicate its id:

chrome.runtime.sendMessage("ID_РАСШИРЕНИЯ", data, callback);


Full client library listing:

(function (window, undefined) {
    "use strict";
    function exDB() {
        var self = this;
        this.extensionId = arguments[0] || "eojllnbjkomphhmpcpafaipblnembfem";
        this.filterList = new Array();
        this._table;
        this._query;
        self.sendMessage = function sendMessage(data, callback) {
            chrome.runtime.sendMessage(self.extensionId, data, callback);
        };
        self.open = function (params, callback) {
            self.sendMessage({"cmd": "open", "params": params}, function(r){
                var tn;
                for(var i=0;i< r.length;i++)
                    tn = r[i];
                    self.__defineGetter__(tn,function(){
                        self._table = tn;
                        return this;
                    });
                callback();
            });
            return self;
        };
        self.close = function (callback) {
            self.sendMessage({"cmd": "close", "params": {}}, callback);
            return self;
        }
        self.table = function (name) {
            self._table = name;
            return self;
        };
        self.query = function () {
            self._query = arguments;
            return self;
        };
        self.execute = function (callback) {
            self.sendMessage({"cmd": "execute", "table": self._table, "query": self._query, "filters": self.filterList}, function (result) {
                if (result && result.RUNTIME_ERROR) {
                    console.error(result.RUNTIME_ERROR.message);
                    result = null;
                }
                callback(result);
            });
            self._query = null;
            self.filterList = [];
        };
        self.getUsageAndQuota = function(callback){
            self.sendMessage({"cmd": "getUsageAndQuota"},callback);
        };
        "add update remove get".split(" ").forEach(function (fn) {
            self[fn] = function (item, callback) {
                self.sendMessage({"cmd": fn, "table": self._table, "params": item}, function (result) {
                    if (result && result.RUNTIME_ERROR) {
                        console.error(result.RUNTIME_ERROR.message);
                        result = null;
                    }
                    callback(result);
                });
                return self;
            }
        });
        "all only lowerBound upperBound bound filter desc distinct keys count".split(" ").forEach(function (fn) {
            self[fn] = function () {
                self.filterList.push({type: fn, args: arguments});
                return self;
            }
        });
    }
    window.exDB = exDB;
})(window, undefined);


At this stage, our complex for working with unlimited indexDB is ready. Below are examples of use.

Connection

    var db = new exDB();
    db.open({
        server: 'my-app',
        version: 1,
        schema: {
            people: {
                key: { keyPath: 'id', autoIncrement: true },
                // Optionally add indexes
                indexes: {
                    firstName: { },
                    answer: { unique: true }
                }
            }
        }
    }, function () {});

Close connection

db.close();

Add Record

db.table("people").add({  firstName: 'Aaron', lastName: 'Powell', answer: 142},function(r){ });


Record Update 

 db.table("people").update({ id:1, firstName: 'Aaron', lastName: 'Powell', answer: 1242}, function (r) {});


Delete record by ID

db.table("people").remove(1,function(key){});


Getting record by ID

db.table("people").get(1,function(r){
	console.log(r);
 });


Fetch / Sort

db.people.query("firstName").only("Aaron2").execute(function(r){
         console.log("GETTER",r);
});
db.table("people").query("answer").all().desc().execute(function(r){
         console.log("all",r);
 });
db.table("people").query("answer").only(12642).count().execute(function(r){
         console.log("only",r);
});
db.table("people").query("answer").bound(20,45).execute(function(r){
         console.log("bound",r);
});
db.table("people").query("answer").lowerBound(50).keys().execute(function(r){
         console.log("lowerBound",r);
});
db.table("people").query("answer").upperBound(43).execute(function(r){
         console.log("upperBound",r);
});
db.table("people").query("answer").filter("return item.answer==42 && item.firstName=='Aaron'").execute(function(r){
         console.log("filter",r);
});

Getting DB size (used / maximum size)

db.getUsageAndQuota(function(r){
        console.log("used", r.usage); //bytes
        console.log("quota", r.quota); //bytes
});


Conclusion


At present, this solution is actively used on one of our projects. I will be grateful for constructive criticism and suggestions. Since this is my first article on the Haber, I ask you not to strongly judge.

You can familiarize with source codes on github .
Tags:

Also popular now: