Ariadna. Why do we need another geocoder for OSM?

Hello!

Most recently, I finished making a geocoder for my purposes. Ariadna
Under the cat, a story about why I did it and what it can do.

The article will not have a single line of code on Go. But there will be a full description of the work of the geocoder and the problems that I encountered. You can see the code on the github.

Background

I work in one of the Bishkek taxi services in Kyrgyzstan. We decided to optimize the location of coordinates at the address for a more optimal distribution of orders.

What we do not have:

• Full Yandex, Google or 2Gis card
• Trust in GPS data

What do we have:

• Very mixed input data
• Openstreetmap
• Own accumulated address base with coordinates

What can users enter?

Users can enter addresses in different formats:

• Street house
• Crossroads
• Name of institution
• Point Name
• housing estate
• microdistrict street house

And there are a lot of such options, for example,
Kievskaya 28
Kievskaya Sovetskaya
5-42
5 Sovetskaya microdistrict 42
TSUM
cafe near Ashot's
barrier
and so on

Formulation of the problem

Make a geocoder that would be able to accept any input and give them the coordinates.
Search language is only Russian.

How to come to such a life

Before making my bike, I decided to figure out the solutions that already exist.
Were:

• Pelias
• Nominatim
• Photon

What did not suit the nominate that we had:

• Folded on its own
• Made on php + c (This is not because php is bad, but because only for this tool we have apache and php)
• Complex logic in Postgresql stored procedures

What Pelias liked:

• Can work with many geodatabases
• Search organized by ElasticSearch

As a result, I decided to abandon all three geocoders and make my tool for several reasons:

1. I want to figure out the data in OSM and import only the ones I need to search
2. I can process geodata before indexing
3. I do not like javascript and node.js, hence the desire to do a search based on Pelias

Design

The following algorithm was laid down:

1. First we get the geometry of large settlements (cities, capitals, villages, housing estates)
2. We unload all possible addresses and correlate them to the desired housing estate, city and other locality, setting the desired value
3. Unload all roads
4. We are looking for a road intersection
5. Put everything in the index
6. Are looking for

For implementation, I chose Go, given projects like pbf2json , golang-geo, and many others for processing geodata. I also wanted to rock the skill in it.

Implementation

With the receipt and parsing of data with the osm, I sort of figured it out. For residential areas, we use the tags place = city, place = village, place = suburb, place = town, place = neighborhood for filtering. To get addresses, buildings addr: street + addr: housenumber, amenity, shop, addr: housenumber
All roads can be obtained using the highway tag. There are

difficulties with finding English names in Russian. How I tried to solve this:

1. Simple automatic transliteration into Russian. The result was absurd and incorrect. An example of data conversion was this: City House -> Citi House
2. Let's try to convert like this. Receive the transcription of the word and transliterate it already. It turned out something like Adrenaline rush -> Erdenaline Rash. Passable, but need a Russian accent, such as adrenaline rush.
3. Such a mechanism came up. Automatically transliterate all data using a dictionary of replacements. Still, simple and dumb transliteration works reasonably well. The dictionary was filled in principle quickly through several runs on the data.

With this sorted out by this moment we are already receiving data that:

1. Normalized and reduced to the Russian language
2. addresses are given in the format Country, city, village or town, microdistrict or housing estate, street, house

The next part of the quest is to find the road intersections. I made it fast and got a very slow implementation, with complexity O (n ^ 2). As a temporary exit, I use Postgres + postgis to find intersections until I find a good algorithm for finding intersections.

The result is a good osm data parser that puts data into ElasticSearch. Which got the simple name importer

Automate it

Considering the fact that constantly pumping and creating indexes in an elastixerch was soon tired of it, the updater component appeared. There was also an automatic configuration in JSON format.

The process of downloading a file and importing it into elastic search was automated. Plus, it became possible to update data in the elasticixer without downtime, thanks to aliases.

How it works:

1. Updater downloads file
2. Recognizes the current version of the index from the config
3. Increments the version and creates a new index
4. Fills it with data
5. Change aliases
6. Deletes an old index

Received such benefits from this:

1. We write a config
2. Launch ./ariadna update
3. Let's go drink some coffee
4. We get a ready-made customized index.

Also, for convenience, I screwed a simple web interface with a map and the ability to search.

Automatic data replenishment

In addition to OSM, we still have many drivers and operators who fill orders.
Accordingly, we have a name and coordinates. The
following scheme is made:

1. Tracks of drivers are stored in the drivers_data index
2. OSM data is stored in the osm_data index
3. They are combined through the alias addresses at which addresses are searched

Data from the drivers is entered if we have an error in certain coordinates greater than 200 meters.

Total

The result is a geocoder that can:

1. Search for coordinates by synonyms. e.g. ShVK - ChampagneVinKombinat
2. Able to search for addresses in a certain radius (for example, for myself I did a search for addresses 30 km from the city center)
3. Search by institution name (cafe near Ashot for example)
4. Search for intersections
5. Search for addresses in housing estates and housing estates
6. Reverse geocoding
7. Automatically replenished with new data from drivers

Consists of three components:

1. Data importer
2. Data updater
3. Web interface

Minuses

1. Tested only for Kyrgyzstan
2. No demos
3. No support for all addressing schemes

Therefore, I hope someone helps to finish it for a good search in other countries and cities.

If someone thought the project was interesting, then I am not against any criticism, the pool of requests, issues on the github and feedback in general.