
IndexedDB - unlimited data storage
Good afternoon, dear community.
For those who are not aware of what IndexedDB is and what it is eaten with, you can read here .
And we go further.
In the office in which I work, there was a need to use an indexed local database on the client side and the choice immediately fell on IndexedDB.
But as always there is one “BUT”, this is the very “BUT” - the database size limit on the user's machine is 5 MB, which did not suit us at all. Since this technology was planned to be used in the admin panel of our project, and all users used Google Chrome as the default browser, a decision was made to find a workaround for that restriction through the proxy extension. Having shoveled a lot of information, we came to the conclusion that the restriction on the database size can be removed by using special flags in the manifest of our extension:
We go further. We figured out unlimited data storage, but now there is a need to work with the same unlimited database directly from the site itself. For this, sending messages between the site and the extension was used (the extension acted as a proxy, between the site and the unlimited database). To do this, the following flags were added to the manifest of our extension:
It turned out that URLs of the form: *: //google.com/* and http: //*.chromium.org/*, but, http: // * / *, *: / / * are considered valid. Com / are not.
You can read more about externally_connectable here .
We go further.
The stage of writing the very “bridge” between the site and the extension to access the database has come.
Db.js was used as the main library for working with IndexedDB on the extension side, which you can find here .
In order not to reinvent the wheel, it was decided to use access syntax on the site side which is implemented in db.js.
And so let's go, create background.js, which we will listen to incoming messages, and respond to them. Listing the code below:
But here a surprise awaited us, namely on the execution of the code section:
We get the exception
To solve this problem, add another line to the manifest that allows the execution of user js on the extension side.
I also had to implement an auxiliary function of moving an object into an array, to pass it as arguments to the function.
Full listing manifest.json
We’ve figured out the extension, now let's start writing a client library for working with our proxy extension.
The first thing that is necessary, when sending a message from the client, indicate to which extension we want to send it, for this, indicate its id:
Full client library listing:
At this stage, our complex for working with unlimited indexDB is ready. Below are examples of use.
At present, this solution is actively used on one of our projects. I will be grateful for constructive criticism and suggestions. Since this is my first article on the Haber, I ask you not to strongly judge.
You can familiarize with source codes on github .
For those who are not aware of what IndexedDB is and what it is eaten with, you can read here .
And we go further.
Unlimited
In the office in which I work, there was a need to use an indexed local database on the client side and the choice immediately fell on IndexedDB.
But as always there is one “BUT”, this is the very “BUT” - the database size limit on the user's machine is 5 MB, which did not suit us at all. Since this technology was planned to be used in the admin panel of our project, and all users used Google Chrome as the default browser, a decision was made to find a workaround for that restriction through the proxy extension. Having shoveled a lot of information, we came to the conclusion that the restriction on the database size can be removed by using special flags in the manifest of our extension:
"permissions": [
"unlimitedStorage",
"unlimited_storage"
],
Posting site-extension-site
We go further. We figured out unlimited data storage, but now there is a need to work with the same unlimited database directly from the site itself. For this, sending messages between the site and the extension was used (the extension acted as a proxy, between the site and the unlimited database). To do this, the following flags were added to the manifest of our extension:
"externally_connectable": {
"matches": [
"*://localhost/*",
"ЗДЕСЬ_ДОБАВЛЯЕМ_РАЗРЕШЕННЫЕ_ШАБЛОНЫ_URL "
]
}
It turned out that URLs of the form: *: //google.com/* and http: //*.chromium.org/*, but, http: // * / *, *: / / * are considered valid. Com / are not.
You can read more about externally_connectable here .
We go further.
The stage of writing the very “bridge” between the site and the extension to access the database has come.
Db.js was used as the main library for working with IndexedDB on the extension side, which you can find here .
In order not to reinvent the wheel, it was decided to use access syntax on the site side which is implemented in db.js.
Expansion
And so let's go, create background.js, which we will listen to incoming messages, and respond to them. Listing the code below:
var server;
chrome.runtime.onMessageExternal.addListener(
function (request, sender, sendResponse) {
var cmd = request.cmd,
params = request.params;
try {
switch (cmd) {
case "getUsageAndQuota":
navigator.webkitPersistentStorage.queryUsageAndQuota(function(u,q){
sendResponse({"usage": u,"quota":q});
});
break;
case "open":
db.open(params).done(function (s) {
server = s;
var exclude = "add close get query remove update".split(" ");
var tables = new Array();
for(var table in server){
if(exclude.indexOf(table)==-1){
tables.push(table);
}
}
sendResponse(tables);
});
break;
case "close":
server.close();
sendResponse({});
break;
case "get":
server[request.table].get(params).done(sendResponse)
break;
case "add":
server[request.table].add(params).done(sendResponse);
break;
case "update":
server[request.table].update(params).done(sendResponse);
break;
case "remove":
server[request.table].remove(params).done(sendResponse);
break;
case "execute":
var tmp_server = server[request.table];
var query = tmp_server.query.apply(tmp_server, obj2arr(request.query));
var flt;
for (var i = 0; i < request.filters.length; i++) {
flt = request.filters[i];
if (flt.type == "filter") {
flt.args = new Function("item", flt.args[0]);
}
query = query[flt.type].apply(query, obj2arr(flt.args));
}
query.execute().done(sendResponse);
break;
}
} catch (error) {
if (error.name != "TypeError") {
sendResponse({RUNTIME_ERROR: error});
}
}
return true;
});
But here a surprise awaited us, namely on the execution of the code section:
flt.args = new Function("item", flt.args[0]);
We get the exception
Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' chrome-extension-resource:".
. To solve this problem, add another line to the manifest that allows the execution of user js on the extension side.
"content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'"
I also had to implement an auxiliary function of moving an object into an array, to pass it as arguments to the function.
var obj2arr = function (obj) {
if (typeof obj == 'object') {
var tmp_args = new Array();
for (var k in obj) {
tmp_args.push(obj[k]);
}
return tmp_args;
} else {
return [obj];
}
}
Full listing manifest.json
{
"manifest_version": 2,
"name": "exDB",
"description": "This extension give proxy access to indexdb from page.",
"version": "1.0",
"background": {
"page": "background.html"
},
"content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'",
"externally_connectable": {
"matches": [
"*://localhost/*"
]
},
"permissions": [
"unlimitedStorage",
"unlimited_storage"
],
"icons": {
"16": "icons/icon_016.png",
"48": "icons/icon_048.png"
}
}
Client
We’ve figured out the extension, now let's start writing a client library for working with our proxy extension.
The first thing that is necessary, when sending a message from the client, indicate to which extension we want to send it, for this, indicate its id:
chrome.runtime.sendMessage("ID_РАСШИРЕНИЯ", data, callback);
Full client library listing:
(function (window, undefined) {
"use strict";
function exDB() {
var self = this;
this.extensionId = arguments[0] || "eojllnbjkomphhmpcpafaipblnembfem";
this.filterList = new Array();
this._table;
this._query;
self.sendMessage = function sendMessage(data, callback) {
chrome.runtime.sendMessage(self.extensionId, data, callback);
};
self.open = function (params, callback) {
self.sendMessage({"cmd": "open", "params": params}, function(r){
var tn;
for(var i=0;i< r.length;i++)
tn = r[i];
self.__defineGetter__(tn,function(){
self._table = tn;
return this;
});
callback();
});
return self;
};
self.close = function (callback) {
self.sendMessage({"cmd": "close", "params": {}}, callback);
return self;
}
self.table = function (name) {
self._table = name;
return self;
};
self.query = function () {
self._query = arguments;
return self;
};
self.execute = function (callback) {
self.sendMessage({"cmd": "execute", "table": self._table, "query": self._query, "filters": self.filterList}, function (result) {
if (result && result.RUNTIME_ERROR) {
console.error(result.RUNTIME_ERROR.message);
result = null;
}
callback(result);
});
self._query = null;
self.filterList = [];
};
self.getUsageAndQuota = function(callback){
self.sendMessage({"cmd": "getUsageAndQuota"},callback);
};
"add update remove get".split(" ").forEach(function (fn) {
self[fn] = function (item, callback) {
self.sendMessage({"cmd": fn, "table": self._table, "params": item}, function (result) {
if (result && result.RUNTIME_ERROR) {
console.error(result.RUNTIME_ERROR.message);
result = null;
}
callback(result);
});
return self;
}
});
"all only lowerBound upperBound bound filter desc distinct keys count".split(" ").forEach(function (fn) {
self[fn] = function () {
self.filterList.push({type: fn, args: arguments});
return self;
}
});
}
window.exDB = exDB;
})(window, undefined);
At this stage, our complex for working with unlimited indexDB is ready. Below are examples of use.
Connection
var db = new exDB();
db.open({
server: 'my-app',
version: 1,
schema: {
people: {
key: { keyPath: 'id', autoIncrement: true },
// Optionally add indexes
indexes: {
firstName: { },
answer: { unique: true }
}
}
}
}, function () {});
Close connection
db.close();
Add Record
db.table("people").add({ firstName: 'Aaron', lastName: 'Powell', answer: 142},function(r){ });
Record Update
db.table("people").update({ id:1, firstName: 'Aaron', lastName: 'Powell', answer: 1242}, function (r) {});
Delete record by ID
db.table("people").remove(1,function(key){});
Getting record by ID
db.table("people").get(1,function(r){
console.log(r);
});
Fetch / Sort
db.people.query("firstName").only("Aaron2").execute(function(r){
console.log("GETTER",r);
});
db.table("people").query("answer").all().desc().execute(function(r){
console.log("all",r);
});
db.table("people").query("answer").only(12642).count().execute(function(r){
console.log("only",r);
});
db.table("people").query("answer").bound(20,45).execute(function(r){
console.log("bound",r);
});
db.table("people").query("answer").lowerBound(50).keys().execute(function(r){
console.log("lowerBound",r);
});
db.table("people").query("answer").upperBound(43).execute(function(r){
console.log("upperBound",r);
});
db.table("people").query("answer").filter("return item.answer==42 && item.firstName=='Aaron'").execute(function(r){
console.log("filter",r);
});
Getting DB size (used / maximum size)
db.getUsageAndQuota(function(r){
console.log("used", r.usage); //bytes
console.log("quota", r.quota); //bytes
});
Conclusion
At present, this solution is actively used on one of our projects. I will be grateful for constructive criticism and suggestions. Since this is my first article on the Haber, I ask you not to strongly judge.
You can familiarize with source codes on github .