Maps in a browser without a network: open source strikes back

    Once upon a time I wrote about how you can use cards without a network on the web and tried to do this using google maps. Unfortunately, the terms of use forbade modifying the resources, and the code I wrote only worked with localStorage, so I decided to switch to the bright side of the force, where the code is open, simple and clear.

    What do I want?


    I want to do map caching for full work without a network, for the first time you download the map, view the tiles of interest (which will be cached at the same time) and the next time the map with the watched tiles will be fully accessible without the network.

    In principle, caching on the fly is not necessary and you can do the same separately for a specific region. But I just want to show the approach.

    What do we have?


    On the modern web, for storing our data, the following may be suitable:
    Application Cache - for statics, but not for tiles.
    Local Storage - using base64 data uri, synchronously, supported everywhere, but very little space.
    Indexed DB - using base64 data uri, asynchronously, is supported in full and mobile chrome, ff, ie10.
    Web SQL - using base64 data uri, asynchronously, designated as deprecated, supported in full and mobile chrome, safari, opera, android browser.
    File Writer is chrome only.

    You can also try using blobs and blob urls to reduce the space occupied by tiles, but this can only work withThe DB the Indexed . I will leave this venture for now.

    So, if you combine Application Cache , Indexed DB and Web SQL , you can solve the problem of storing tiles sufficient for normal use in modern browsers, including mobile.

    Theory


    In theory, we need:
    1. take the API;
    2. add all the statics to Application Cache;
    3. redefine the tiles layer so that it loads data from our asynchronous storages;
    4. add logic for loading tiles to the repository.

    Storage


    First, we organize the key-value repository with the basic operations add, delete, get for Indexed DB and Web SQL. There is one magic construction emr.fire('storageLoaded', storage);that will be called after the storage has been initialized and ready to use so that the card does not fall when accessing the storage.

    Implementing Storage with Indexed DB
    var getIndexedDBStorage = function () {
        var indexedDB = window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB;
        var IndexedDBImpl = function () {
            var self = this;
            var db = null;
            var request = indexedDB.open('TileStorage');
            request.onsuccess = function() {
                db = this.result;
                emr.fire('storageLoaded', self);
            };
            request.onerror = function (error) {
                console.log(error);
            };
            request.onupgradeneeded = function () {
                var store = this.result.createObjectStore('tile', { keyPath: 'key'});
                store.createIndex('key', 'key', { unique: true });
            };
            this.add = function (key, value) {
                var transaction = db.transaction(['tile'], 'readwrite');
                var objectStore = transaction.objectStore('tile');
                objectStore.put({key: key, value: value});
            };
            this.delete = function (key) {
                var transaction = db.transaction(['tile'], 'readwrite');
                var objectStore = transaction.objectStore('tile');
                objectStore.delete(key);
            };
            this.get = function (key, successCallback, errorCallback) {
                var transaction = db.transaction(['tile'], 'readonly');
                var objectStore = transaction.objectStore('tile');
                var result = objectStore.get(key);
                result.onsuccess = function () {
                    successCallback(this.result ? this.result.value : undefined);
                };
                result.onerror = errorCallback;
            };
        };
        return indexedDB ? new IndexedDBImpl() : null;
    };
    


    Implementing storage with Web SQL
    var getWebSqlStorage = function () {
        var openDatabase = window.openDatabase;
        var WebSqlImpl = function () {
            var self = this;
            var db = openDatabase('TileStorage', '1.0', 'Tile Storage', 5 * 1024 * 1024);
            db.transaction(function (tx) {
                tx.executeSql('CREATE TABLE IF NOT EXISTS tile (key TEXT PRIMARY KEY, value TEXT)', [], function () {
                    emr.fire('storageLoaded', self);
                });
            });
            this.add = function (key, value) {
                db.transaction(function (tx) {
                    tx.executeSql('INSERT INTO tile (key, value) VALUES (?, ?)', [key, value]);
                });
            };
            this.delete = function (key) {
                db.transaction(function (tx) {
                    tx.executeSql('DELETE FROM tile WHERE key = ?', [key]);
                });
            };
            this.get = function (key, successCallback, errorCallback) {
                db.transaction(function (tx) {
                    tx.executeSql('SELECT value FROM tile WHERE key = ?', [key], function (tx, result) {
                        successCallback(result.rows.length ? result.rows.item(0).value : undefined);
                    }, errorCallback);
                });
            };
        };
        return openDatabase ? new WebSqlImpl() : null;
    };
    


    Storage creation
    var storage =  getIndexedDBStorage() || getWebSqlStorage() || null;
    if (!storage) {
        emr.fire('storageLoaded', null);
    }
    


    I propose to consider this implementation very sketchy, I think there is something to think about, for example, so as not to block the initialization of the card while the storage is initializing; remember which tiles are in the repository without directly accessing the API; try to combine several save operations into one transaction to reduce the number of writes to disk; try to use blobs where they are supported. Perhaps the implementation of Indexed DB in older browsers will fall, because the event may not be implemented in them onupgradeneeded.

    IMG to data URI & CORS


    In order to store tiles, we need to convert them to data URI, i.e. base64 representation. To do this, use canvas and its methods, toDataURLor getImageData:

    _imageToDataUri: function (image) {
        var canvas = window.document.createElement('canvas');
        canvas.width = image.width;
        canvas.height = image.height;
        var context = canvas.getContext('2d');
        context.drawImage(image, 0, 0);
        return canvas.toDataURL('image/png');
    }
    

    Since the html img element can take any available resource as a picture, including on authorized services and the local file system, the ability to send this content to a third party is a security risk, so pictures that do not allow Access-Control-Allow-Origingyour domain will not be saved. Fortunately, mapnik or tile.openstreetmap.org tiles have a header Access-Control-Allow-Origing: *, but for everything to work, you need to set the element flag img.crossOriginto value Anonymous.

    The work of CORS in this implementation is not guaranteed in all mobile browsers, therefore it is easiest to set up proxies for your site on your domain or disable CORS checking for Phoengap adherents, for example. Personally, this code did not take off in the Android browser by default (androin 4.0.4 sony xperia active), and in the opera some tiles were saved in a strange way (compare what sometimes happens and what should actually be , but it looks like an opera bug )

    Here you can try using WebWorkers+ AJAXinstead canvas.

    Leaflet


    So we will need the popular open source JS API card, one of these candidates is Leaflet .

    Having looked a bit at the source, you can find the tile layer method, which is responsible for the direct indication srcfor the tiles:

    _loadTile: function (tile, tilePoint) {
        tile._layer = this;
        tile.onload = this._tileOnLoad;
        tile.onerror = this._tileOnError;
        tile.src = this.getTileUrl(tilePoint);
    }
    

    That is, if you override this class and directly this method for loading data into srcthe repository, then we will do what we need. We also implement the addition of data to the repository if they have been downloaded from the network and get full caching.

    Implementation for Leaflet
    var StorageTileLayer = L.TileLayer.extend({
        _imageToDataUri: function (image) {
            var canvas = window.document.createElement('canvas');
            canvas.width = image.width;
            canvas.height = image.height;
            var context = canvas.getContext('2d');
            context.drawImage(image, 0, 0);
            return canvas.toDataURL('image/png');
        },
        _tileOnLoadWithCache: function () {
            var storage = this._layer.options.storage;
            if (storage) {
                storage.add(this._storageKey, this._layer._imageToDataUri(this));
            }
            L.TileLayer.prototype._tileOnLoad.apply(this, arguments);
        },
        _setUpTile: function (tile, key, value, cache) {
            tile._layer = this;
            if (cache) {
                tile._storageKey = key;
                tile.onload = this._tileOnLoadWithCache;
                tile.crossOrigin = 'Anonymous';
            } else {
                tile.onload = this._tileOnLoad;
            }
            tile.onerror = this._tileOnError;
            tile.src = value;
        },
        _loadTile: function (tile, tilePoint) {
            this._adjustTilePoint(tilePoint);
            var key = tilePoint.z + ',' + tilePoint.y + ',' + tilePoint.x;
            var self = this;
            if (this.options.storage) {
                this.options.storage.get(key, function (value) {
                    if (value) {
                        self._setUpTile(tile, key, value, false);
                    } else {
                        self._setUpTile(tile, key, self.getTileUrl(tilePoint), true);
                    }
                }, function () {
                    self._setUpTile(tile, key, self.getTileUrl(tilePoint), true);
                });
            } else {
                self._setUpTile(tile, key, self.getTileUrl(tilePoint), false);
            }
        }
    });
    


    The card itself in this case will be initialized as follows:

    var map = L.map('map').setView([53.902254, 27.561850], 13);
    new StorageTileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', {storage: storage}).addTo(map);
    

    We also add our resources to the Application Cache so that the card can fully work without a network with cached tiles:

    Application Cache manifest for Leaflet
    CACHE MANIFEST
    NETWORK:
    *
    CACHE:
    index.html
    style.css
    event.js
    storage.js
    map.js
    run.js
    leaflet.css
    leaflet.js
    images/layers.png
    images/marker-icon.png
    images/marker-icon@2x.png
    images/marker-shadow.png
    


    An example and its code on the github .

    Mapbox (modesmaps)


    Another candidate for the open JS API maps is a mapbox based on modesmaps .

    After looking at the source of the mapbox, we won’t find anything interesting for us, so let's move on to the source of modestmaps . Let's start with TemplatedLayer, which is a regular map layer with a template provider, the code we need will be in the layer class:

    MM.TemplatedLayer = function(template, subdomains, name) {
        return new MM.Layer(new MM.Template(template, subdomains), null, name);
    };
    

    Finding the use of a template provider in the map layer, you can notice that our provider can return either a tile URL or a finished DOM element, and the DOM element is immediately positioned, and the tile URL is sent to requestManager:

    if (!this.requestManager.hasRequest(tile_key)) {
        var tileToRequest = this.provider.getTile(tile_coord);
        if (typeof tileToRequest == 'string') {
            this.addTileImage(tile_key, tile_coord, tileToRequest);
        } else if (tileToRequest) {
            this.addTileElement(tile_key, tile_coord, tileToRequest);
        }
    }
    

    addTileImage: function(key, coord, url) {
        this.requestManager.requestTile(key, coord, url);
    }
    

    addTileElement: function(key, coordinate, element) {
        element.id = key;
        element.coord = coordinate.copy();
        this.positionTile(element);
    }
    

    Itself is requestManagerinitialized in the designer of a map layer. Creating a DOM element imgand setting it srcoccurs in a method processQueuethat also twitches from a map layer:

    processQueue: function(sortFunc) {
        if (sortFunc && this.requestQueue.length > 8) {
            this.requestQueue.sort(sortFunc);
        }
        while (this.openRequestCount < this.maxOpenRequests && this.requestQueue.length > 0) {
            var request = this.requestQueue.pop();
            if (request) {
                this.openRequestCount++;
                var img = document.createElement('img');
                img.id = request.id;
                img.style.position = 'absolute';
                img.coord = request.coord;
                this.loadingBay.appendChild(img);
                img.onload = img.onerror = this.getLoadComplete();
                img.src = request.url;
                request = request.id = request.coord = request.url = null;
            }
        }
    }
    

    That is, if we override this method, we will also get the desired result.

    Implementation for mapbox (modestmaps)
    var StorageRequestManager = function (storage) {
        MM.RequestManager.apply(this, []);
        this._storage = storage;
    };
    StorageRequestManager.prototype._imageToDataUri = function (image) {
        var canvas = window.document.createElement('canvas');
        canvas.width = image.width;
        canvas.height = image.height;
        var context = canvas.getContext('2d');
        context.drawImage(image, 0, 0);
        return canvas.toDataURL('image/png');
    };
    StorageRequestManager.prototype._createTileImage = function (id, coord, value, cache) {
        var img = window.document.createElement('img');
        img.id = id;
        img.style.position = 'absolute';
        img.coord = coord;
        this.loadingBay.appendChild(img);
        if (cache) {
            img.onload = this.getLoadCompleteWithCache();
            img.crossOrigin = 'Anonymous';
        } else {
            img.onload = this.getLoadComplete();
        }
        img.onerror = this.getLoadComplete();
        img.src = value;
    };
    StorageRequestManager.prototype._loadTile = function (id, coord, url) {
        var self = this;
        if (this._storage) {
            this._storage.get(id, function (value) {
                if (value) {
                    self._createTileImage(id, coord, value, false);
                } else {
                    self._createTileImage(id, coord, url, true);
                }
            }, function () {
                self._createTileImage(id, coord, url, true);
            });
        } else {
            self._createTileImage(id, coord, url, false);
        }
    };
    StorageRequestManager.prototype.processQueue = function (sortFunc) {
        if (sortFunc && this.requestQueue.length > 8) {
            this.requestQueue.sort(sortFunc);
        }
        while (this.openRequestCount < this.maxOpenRequests && this.requestQueue.length > 0) {
            var request = this.requestQueue.pop();
            if (request) {
                this.openRequestCount++;
                this._loadTile(request.id, request.coord, request.url);
                request = request.id = request.coord = request.url = null;
            }
        }
    };
    StorageRequestManager.prototype.getLoadCompleteWithCache = function () {
        if (!this._loadComplete) {
            var theManager = this;
            this._loadComplete = function(e) {
                e = e || window.event;
                var img = e.srcElement || e.target;
                img.onload = img.onerror = null;
                if (theManager._storage) {
                    theManager._storage.add(this.id, theManager._imageToDataUri(this));
                }
                theManager.loadingBay.removeChild(img);
                theManager.openRequestCount--;
                delete theManager.requestsById[img.id];
                if (e.type === 'load' && (img.complete ||
                    (img.readyState && img.readyState === 'complete'))) {
                    theManager.dispatchCallback('requestcomplete', img);
                } else {
                    theManager.dispatchCallback('requesterror', {
                        element: img,
                        url: ('' + img.src)
                    });
                    img.src = null;
                }
                setTimeout(theManager.getProcessQueue(), 0);
            };
        }
        return this._loadComplete;
    };
    MM.extend(StorageRequestManager, MM.RequestManager);
    var StorageLayer = function(provider, parent, name, storage) {
        this.parent = parent || document.createElement('div');
        this.parent.style.cssText = 'position: absolute; top: 0px; left: 0px;' +
            'width: 100%; height: 100%; margin: 0; padding: 0; z-index: 0';
        this.name = name;
        this.levels = {};
        this.requestManager = new StorageRequestManager(storage);
        this.requestManager.addCallback('requestcomplete', this.getTileComplete());
        this.requestManager.addCallback('requesterror', this.getTileError());
        if (provider) {
            this.setProvider(provider);
        }
    };
    MM.extend(StorageLayer, MM.Layer);
    var StorageTemplatedLayer = function(template, subdomains, name, storage) {
        return new StorageLayer(new MM.Template(template, subdomains), null, name, storage);
    };
    


    The card itself in this case will be initialized as follows:

    var map = mapbox.map('map');
    map.addLayer(new StorageTemplatedLayer('http://{S}.tile.osm.org/{Z}/{X}/{Y}.png', ['a', 'b', 'c'], undefined, storage));
    map.ui.zoomer.add();
    map.ui.zoombox.add();
    map.centerzoom({lat: 53.902254, lon: 27.561850}, 13);
    

    We also add our resources to the Application Cache so that the card can fully work without a network with cached tiles:

    Application Cache manifest for Mapbox (modestmaps)
    CACHE MANIFEST
    NETWORK:
    *
    CACHE:
    index.html
    style.css
    event.js
    storage.js
    map.js
    run.js
    mapbox.css
    mapbox.js
    map-controls.png
    


    An example and its code on the github .

    Openlayers


    And the latest candidate for the open JS API maps is OpenLayers .

    I had to spend some time figuring out how to start the minimal view, as a result, my build file acquired the following form:

    [first]
    [last]
    [include]
    OpenLayers/Map.js
    OpenLayers/Layer/OSM.js
    OpenLayers/Control/Zoom.js
    OpenLayers/Control/Navigation.js
    OpenLayers/Control/TouchNavigation.js
    [exclude]
    

    I will use it OpenLayers.Layer.OSM, so I'll start the search with it:

    url: [
        'http://a.tile.openstreetmap.org/${z}/${x}/${y}.png',
        'http://b.tile.openstreetmap.org/${z}/${x}/${y}.png',
        'http://c.tile.openstreetmap.org/${z}/${x}/${y}.png'
    ]
    

    OpenLayers.Layer.OSMinherited from OpenLayers.Layer.XYZwith redefined URLs. The method is interesting here getURL:

    getURL: function (bounds) {
        var xyz = this.getXYZ(bounds);
        var url = this.url;
        if (OpenLayers.Util.isArray(url)) {
            var s = '' + xyz.x + xyz.y + xyz.z;
            url = this.selectUrl(s, url);
        }
        return OpenLayers.String.format(url, xyz);
    }
    

    Also interesting is the method getXYZthat can be used to create the key:

    getXYZ: function(bounds) {
        var res = this.getServerResolution();
        var x = Math.round((bounds.left - this.maxExtent.left) /
            (res * this.tileSize.w));
        var y = Math.round((this.maxExtent.top - bounds.top) /
            (res * this.tileSize.h));
        var z = this.getServerZoom();
        if (this.wrapDateLine) {
            var limit = Math.pow(2, z);
            x = ((x % limit) + limit) % limit;
        }
        return {'x': x, 'y': y, 'z': z};
    }
    

    Itself is OpenLayers.Layer.XYZinherited from OpenLayers.Layer.Grid, which has a method addTileand which internally creates tiles with the help of tileClass, which is OpenLayers.Tile.Image:

    addTile: function(bounds, position) {
        var tile = new this.tileClass(
            this, position, bounds, null, this.tileSize, this.tileOptions
        );
        this.events.triggerEvent("addtile", {tile: tile});
        return tile;
    }
    

    In OpenLayers.Tile.Imagesrcis set in the method setImgSrc:

    setImgSrc: function(url) {
        var img = this.imgDiv;
        if (url) {
            img.style.visibility = 'hidden';
            img.style.opacity = 0;
            if (this.crossOriginKeyword) {
                if (url.substr(0, 5) !== 'data:') {
                    img.setAttribute("crossorigin", this.crossOriginKeyword);
                } else {
                    img.removeAttribute("crossorigin");
                }
            }
            img.src = url;
        } else {
            this.stopLoading();
            this.imgDiv = null;
            if (img.parentNode) {
                img.parentNode.removeChild(img);
            }
        }
    }
    

    But it does not specify handlers onloadand onerror. The method itself twitches from initImagewhere these handlers hang themselves:

    initImage: function() {
        this.events.triggerEvent('beforeload');
        this.layer.div.appendChild(this.getTile());
        this.events.triggerEvent(this._loadEvent);
        var img = this.getImage();
        if (this.url && img.getAttribute("src") == this.url) {
            this._loadTimeout = window.setTimeout(
                OpenLayers.Function.bind(this.onImageLoad, this), 0
            );
        } else {
            this.stopLoading();
            if (this.crossOriginKeyword) {
                img.removeAttribute("crossorigin");
            }
            OpenLayers.Event.observe(img, "load",
                OpenLayers.Function.bind(this.onImageLoad, this)
            );
            OpenLayers.Event.observe(img, "error",
                OpenLayers.Function.bind(this.onImageError, this)
            );
            this.imageReloadAttempts = 0;
            this.setImgSrc(this.url);
        }
    }
    

    You may notice that the layer class method getURL, as well initImage, twitch from renderTile:

    renderTile: function() {
        if (this.layer.async) {
            var id = this.asyncRequestId = (this.asyncRequestId || 0) + 1;
            this.layer.getURLasync(this.bounds, function(url) {
                if (id == this.asyncRequestId) {
                    this.url = url;
                    this.initImage();
                }
            }, this);
        } else {
            this.url = this.layer.getURL(this.bounds);
            this.initImage();
        }
    }
    

    So, if we redefine this class, we also get the desired result.

    Implementation for OpenLayers
    var StorageImageTile = OpenLayers.Class(OpenLayers.Tile.Image, {
        _imageToDataUri: function (image) {
            var canvas = window.document.createElement('canvas');
            canvas.width = image.width;
            canvas.height = image.height;
            var context = canvas.getContext('2d');
            context.drawImage(image, 0, 0);
            return canvas.toDataURL('image/png');
        },
        onImageLoadWithCache: function() {
            if (this.storage) {
                this.storage.add(this._storageKey, this._imageToDataUri(this.imgDiv));
            }
            this.onImageLoad.apply(this, arguments);
        },
        renderTile: function() {
            var self = this;
            var xyz = this.layer.getXYZ(this.bounds);
            var key = xyz.z + ',' + xyz.y + ',' + xyz.x;
            var url = this.layer.getURL(this.bounds);
            if (this.storage) {
                this.storage.get(key, function (value) {
                    if (value) {
                        self.initImage(key, value, false);
                    } else {
                        self.initImage(key, url, true);
                    }
                }, function () {
                    self.initImage(key, url, true);
                });
            } else {
                self.initImage(key, url, false);
            }
        },
        initImage: function(key, url, cache) {
            this.events.triggerEvent('beforeload');
            this.layer.div.appendChild(this.getTile());
            this.events.triggerEvent(this._loadEvent);
            var img = this.getImage();
            this.stopLoading();
            if (cache) {
                OpenLayers.Event.observe(img, 'load',
                    OpenLayers.Function.bind(this.onImageLoadWithCache, this)
                );
                this._storageKey = key;
            } else {
                OpenLayers.Event.observe(img, 'load',
                    OpenLayers.Function.bind(this.onImageLoad, this)
                );
            }
            OpenLayers.Event.observe(img, 'error',
                OpenLayers.Function.bind(this.onImageError, this)
            );
            this.imageReloadAttempts = 0;
            this.setImgSrc(url);
        }
    });
    var StorageOSMLayer = OpenLayers.Class(OpenLayers.Layer.OSM, {
        async: true,
        tileClass: StorageImageTile,
        initialize: function(name, url, options) {
            OpenLayers.Layer.OSM.prototype.initialize.apply(this, arguments);
            this.tileOptions = OpenLayers.Util.extend({
                storage: options.storage
            }, this.options && this.options.tileOptions);
        },
        clone: function (obj) {
            if (obj == null) {
                obj = new StorageOSMLayer(this.name,
                    this.url,
                    this.getOptions());
            }
            obj = OpenLayers.Layer.Grid.prototype.clone.apply(this, [obj]);
            return obj;
        }
    });
    


    The card itself in this case will be initialized as follows:

    var map = new OpenLayers.Map('map');
    map.addLayer(new StorageOSMLayer(undefined, undefined, {storage: storage}));
    var fromProjection = new OpenLayers.Projection('EPSG:4326');
    var toProjection   = new OpenLayers.Projection('EPSG:900913');
    var center = new OpenLayers.LonLat(27.561850, 53.902254).transform(fromProjection, toProjection);
    map.setCenter(center, 13);
    

    We also add our resources to the Application Cache so that the card can fully work without a network with cached tiles:

    Application Cache manifest for OpenLayers
    CACHE MANIFEST
    NETWORK:
    *
    CACHE:
    index.html
    style.css
    event.js
    storage.js
    map.js
    run.js
    theme/default/style.css
    OpenLayers.js
    


    An example and its code on the github .

    I also found a ready-made implementation of the cache OpenLayers.Control.CacheWrite, but only using it localStorage, which is not very interesting.

    Conclusion


    Actually, I got what I wanted. The examples work smartly in chrome, or if you have an SSD, there are brakes in the fox and opera when saving to disk, the donkey was not at hand. Brakes also appear in mobile browsers, even when reading, which upset me a bit, as this task is more relevant for mobile devices.

    The standard storage size is IndexedDBeither WebSQLenough to cache a city or more, which makes applying the approach more interesting than in version c localStorage.

    In my examples, you can work with asynchronous storages, but for greater comfort, you still need to work on their implementation in order to increase performance compared to what it is now.

    Только зарегистрированные пользователи могут участвовать в опросе. Войдите, пожалуйста.

    Какими JS API карт пользуетесь Вы?

    • 59%Google Maps (https://developers.google.com/maps/documentation/javascript/)333
    • 2.3%Bing Maps (http://www.microsoft.com/maps/developers/web.aspx)13
    • 2.8%Nokia (Here) Maps (http://developer.here.com/)16
    • 46.4%Yandex Maps (http://api.yandex.ru/maps/)262
    • 15.2%2gis (http://api.2gis.ru/)86
    • 17.7%Leaflet (http://leafletjs.com)100
    • 3%Mapbox (http://mapbox.com/mapbox.js)17
    • 12%OpenLayers (http://openlayers.org)68
    • 3.5% Other (in comments) 20

    Also popular now: