July 30, 2015 at 16:23How to find a place on earth and not get on the Yandex counter
... this story began a long time ago in the distant, distant country of Cracovia, whose inhabitants lived their lives carelessly and did not know ...
But I am local, and today I will tell you a terrible story about what prevented me from sleeping for many years. And these are not taxes (everything is fine with them), this is the Yandex.Map geocoder !
The geocoder will tell you where the wonderful country of Cracovia is located on the map. And he will be the main hero of this story, the plot of which was described in a completely different book - in the ancient folio Yandex.Maps API User Agreement . Legend has it that there is a limit on the number of requests to the geocoding function. It is maximum permissible to make no more than 25,000 requests to HTTP and JS geocoder per day per day. Or oatmeal, sir.
And just the other day, this rule from ordinary words turned into a threat to the website’s performance. A message has been published in the API Maps club about mass banning of various services in excess of limits. What to do? For clarity, we get a piano from the bushes - 8 years ago, a project was lit on Habré
"ESoobies" - "Map of interesting places." All these years I continue to work on it.
Some of the places that users drew on the map had only coordinates on the map - for them it was necessary to find out the address. In other places found in various sources, on the contrary, only the address was known. So, it had to be translated into coordinates for display on the map.
Moreover - the translation of coordinates to and from the address had to be repeated regularly!
In addition, tens of thousands of people visited the eSoads website daily to search for places, which again led to the use of a geocoder.
In fact, everything could not have been so scary if not for laziness and faith in a MIRACLE! See:
As a result of a simple operation, one or more points will appear on the map and in exactly the right place. Isn't that a miracle?
Blinded by the ease of use of the JS geocoder, many developers programmed their “mapping solutions” exactly in this way. It’s really easier to display ATMs, post offices, your points of sale or any other “organization” tied to an address if you work only with addresses.
The texts will be swallowed by the IPA, somewhere in the depths they will be magically digested - and marks will appear on the map in the right places. I will tell you even more - many people do not see the difference between setting the location of any label by numerical coordinates or legal address.
The trouble is that you have to pay for all the magic! And you have to go into the pen and get up on the counter. And if you have one page, there are 10 tags on it, and 2,500 people a day - then you have run into a limit.
That is what prevented me from sleeping the last days. One of the parts of “eNoshedey” is the materialization of KLADR (cities, streets, addresses) - postindex.esosedi.ru . This is a database of “about addresses” and, of course, it is described by addresses (text) and often does not contain coordinates. Even now, many villages of our great Motherland are not geocoded (remember Cracovia). So the coordinates are secondary.
Each time you enter the page of a city or street, a map is displayed. At the same time, a request is sent to translate a known address (Timur Frunze, 11k2) into coordinates. By the way, geocoding does not happen instantly, and if you have a lot of points, it can slow down the map loading.
Everything would be fine, but 40,000 post offices, 211,000 settlements, a million streets, 55,000 visitors for yesterday ... That’s enough for two daily limits with a tail. But for many years this worked just like that. And why?
We will not wait for the evil robot - we will solve the problem. Fortunately, a solution exists, and not one, but two.
Modify the backend so that there is where to save the coordinates. And this means making a normal admin panel, planting a girl who, on the basis of a visual inspection of the “response” of the geocoder, will display TTL data, move the icons, achieving centimeter accuracy of label placement ...
Pros: technological, reliable, beautiful girl in the office.
Cons: a backend is required, a database is required, it is not clear what to do with custom search; it’s not completely understood how the wife will react.
Make your search! Such that something kept in itself (ie the caching proxy), was "out of the box", to cope with simultaneous identical requests and so on.
In general, the idea is very simple: go to some own pen, check the cache status there and, if there is no necessary data, send a request to the Yandex.Maps HTTP geocoder.
For many large projects, the right solution would be to write something of their own. Maybe even deploy your own geocoder, for example, based on OSM.
But from experience I know that for most the most profitable option is to take something ready with github. By tradition, I give a link to the geocode-tool a solution to the problem from dimik . Dima once worked Yandex.Maps and can not stop at all.
In general, the geocode-tool is the server part (“handle”), a web interface with statistics, and a module for Yandex.Maps .
Cons: you need to run another service on the server, moreover, on the node. The girl will not appear in the office.
Pros: All work was reduced to a turnip jump and adding a couple of lines on the client (well, almost).
1000 people who visit 10 pages of the abstract store, where each page has a location map, 2 points (addresses) on each - this is already 20,000 requests, which is dangerously close to the limit. Suddenly, one day you decide to buy an advertisement, and a little more people will visit the site?
How did this end with us, at eNososedi:
That is - do not forget that the geocoder is not an artificial mind - can give garbage.
A geocoder is a simple database search mechanism, which means it can give out a bit of the wrong object that you want. For example, because what you are looking for is actually not there (remember Cracovia).
This is all more fun than it sounds. But! I! Finally! I can sleep peacefully. Which I advise you.
Good geocaching.
PS: Actually, I work on “eSo Neighbors” in my free time from my main job. And the main work is just in Yandex.Maps. And I warned you.
PPS: All pictures from fevrony .
But I am local, and today I will tell you a terrible story about what prevented me from sleeping for many years. And these are not taxes (everything is fine with them), this is the Yandex.Map geocoder !
A geocoder is one of Yandex.Maps HTTP services that receives a text representation of an address in a request and returns objects found on its basis in the response. Or vice versa: receiving coordinates and responding with an address.
The geocoder will tell you where the wonderful country of Cracovia is located on the map. And he will be the main hero of this story, the plot of which was described in a completely different book - in the ancient folio Yandex.Maps API User Agreement . Legend has it that there is a limit on the number of requests to the geocoding function. It is maximum permissible to make no more than 25,000 requests to HTTP and JS geocoder per day per day. Or oatmeal, sir.
And just the other day, this rule from ordinary words turned into a threat to the website’s performance. A message has been published in the API Maps club about mass banning of various services in excess of limits. What to do? For clarity, we get a piano from the bushes - 8 years ago, a project was lit on Habré
"ESoobies" - "Map of interesting places." All these years I continue to work on it.
Some of the places that users drew on the map had only coordinates on the map - for them it was necessary to find out the address. In other places found in various sources, on the contrary, only the address was known. So, it had to be translated into coordinates for display on the map.
Moreover - the translation of coordinates to and from the address had to be repeated regularly!
According to the rules of using the service, the response of the geocoder cannot be stored in the database forever, but it can be cached for up to 30 days. It's not that Yandex does not want to share data, it’s just that the quality of data and coverage is constantly changing, and it was important for me, as the creator of the service, to keep the latest information available to the user. Krakow, by the way, still does not find.
In addition, tens of thousands of people visited the eSoads website daily to search for places, which again led to the use of a geocoder.
In fact, everything could not have been so scary if not for laziness and faith in a MIRACLE! See:
ymaps.geocode('Тимура фрунзе д.12').then(function(res){myMap.geoObjects.add(res.geoObjects);});
//или даже
ymaps.geoQuery(ymaps.geocode('Арбат')).addToMap(myMap);
As a result of a simple operation, one or more points will appear on the map and in exactly the right place. Isn't that a miracle?
Blinded by the ease of use of the JS geocoder, many developers programmed their “mapping solutions” exactly in this way. It’s really easier to display ATMs, post offices, your points of sale or any other “organization” tied to an address if you work only with addresses.
The texts will be swallowed by the IPA, somewhere in the depths they will be magically digested - and marks will appear on the map in the right places. I will tell you even more - many people do not see the difference between setting the location of any label by numerical coordinates or legal address.
The trouble is that you have to pay for all the magic! And you have to go into the pen and get up on the counter. And if you have one page, there are 10 tags on it, and 2,500 people a day - then you have run into a limit.
Earlier, when crossing the border of 25,000 requests per day, a letter might come from a manager from Yandex explaining what you are doing wrong. Now an evil robot with a plusome will come - that's all.
That is what prevented me from sleeping the last days. One of the parts of “eNoshedey” is the materialization of KLADR (cities, streets, addresses) - postindex.esosedi.ru . This is a database of “about addresses” and, of course, it is described by addresses (text) and often does not contain coordinates. Even now, many villages of our great Motherland are not geocoded (remember Cracovia). So the coordinates are secondary.
Each time you enter the page of a city or street, a map is displayed. At the same time, a request is sent to translate a known address (Timur Frunze, 11k2) into coordinates. By the way, geocoding does not happen instantly, and if you have a lot of points, it can slow down the map loading.
Everything would be fine, but 40,000 post offices, 211,000 settlements, a million streets, 55,000 visitors for yesterday ... That’s enough for two daily limits with a tail. But for many years this worked just like that. And why?
We will not wait for the evil robot - we will solve the problem. Fortunately, a solution exists, and not one, but two.
The first solution is the right one
Modify the backend so that there is where to save the coordinates. And this means making a normal admin panel, planting a girl who, on the basis of a visual inspection of the “response” of the geocoder, will display TTL data, move the icons, achieving centimeter accuracy of label placement ...
Pros: technological, reliable, beautiful girl in the office.
Cons: a backend is required, a database is required, it is not clear what to do with custom search; it’s not completely understood how the wife will react.
Solution Two - Cheap and Angry
Make your search! Such that something kept in itself (ie the caching proxy), was "out of the box", to cope with simultaneous identical requests and so on.
In general, the idea is very simple: go to some own pen, check the cache status there and, if there is no necessary data, send a request to the Yandex.Maps HTTP geocoder.
For many large projects, the right solution would be to write something of their own. Maybe even deploy your own geocoder, for example, based on OSM.
But from experience I know that for most the most profitable option is to take something ready with github. By tradition, I give a link to the geocode-tool a solution to the problem from dimik . Dima once worked Yandex.Maps and can not stop at all.
In general, the geocode-tool is the server part (“handle”), a web interface with statistics, and a module for Yandex.Maps .
Cons: you need to run another service on the server, moreover, on the node. The girl will not appear in the office.
Pros: All work was reduced to a turnip jump and adding a couple of lines on the client (well, almost).
If you have a map on your site with at least a location map and you are using geocoding (check), and it seems to you that it often runs, connect a proxy server to yourself. If you have an address transformation on every page, use the server geocoder or set the coordinates directly.
1000 people who visit 10 pages of the abstract store, where each page has a location map, 2 points (addresses) on each - this is already 20,000 requests, which is dangerously close to the limit. Suddenly, one day you decide to buy an advertisement, and a little more people will visit the site?
How did this end with us, at eNososedi:
- Connecting the local cache made it possible to speed up the page by an average of 150ms - access to the geocoder in conditions of mass loading of pictures and other advertising is not fast.
When generating the page code, the geocoder cache twitches at the beginning, bypassing the original service, and only if there is no answer, a function call on the client is added. PS: the server version of this case on github is again there. - Finally, query statistics have appeared!
- After the appearance of the administrative division ( an article on the hub ) and the normal geocoder cache, it became possible to check the correctness of both of them after the fact.
That is - do not forget that the geocoder is not an artificial mind - can give garbage.
As you know, in the south of Moscow there are four ( 1 , 2 , 3 , 4 ) Svitino and all in about the same place, but two of them are in Moscow, one in the Moscow region, and one more in Kaluga.
A geocoder is a simple database search mechanism, which means it can give out a bit of the wrong object that you want. For example, because what you are looking for is actually not there (remember Cracovia).
This is all more fun than it sounds. But! I! Finally! I can sleep peacefully. Which I advise you.
Good geocaching.
PS: Actually, I work on “eSo Neighbors” in my free time from my main job. And the main work is just in Yandex.Maps. And I warned you.
PPS: All pictures from fevrony .