GeoPuzzle - collect the world in pieces

I want to talk about a project that has developed over the last couple of years. It is called GeoPuzzle and is a puzzle game on the political map of the world. The goal - to put the pieces of the country into place. The idea was spied on in the article “Mercator Puzzle for Geography Experts” , also played Tetris from countries (still under DOS) as a child, but I don’t remember the name of the program. I was so inspired by the idea that I wanted to make a complete product, interesting not only for schoolchildren, but also for geography connoisseurs. The development of the project can be observed on GitHub .

Prototype

The article “Mercator Puzzle for Geography Experts” was published on February 8, 2013, but after 4 months I had a prototype in which polygons of all countries of the world were collected. A little later, I added regions of Russia and US states and made a choice of an initial position on the map for polygons random. The development process described in his blog , the source posted on GitHub. And that's all - stuck. I had significantly less free time, my motivation was gone (no decision was made), and the complexity increased exponentially. It was a pet-project, and the main task was to learn something new, so I was a little distorted with technology. On the client, of course, javascript (with which he worked a little then), the script was responsible for preparing the data in ruby ​​(again, a new language for me), and on the server there was an erlang (I wanted to try something purely functional). Full exit from the comfort zone: it was difficult to work directly with PostGIS objects, pain when converting strings to erlang, setting YAWS is a separate topic ... At the next stage, I realized that I simply couldn’t cope with all this zoo to make a full-fledged product and left think for a couple of years. 

Geopuzzle

All this time the site worked, even people came in there, but I really wanted to add one more little detail: to show at least some information about the newly solved landfill. Thus, I planned to spend the New Year holidays in 2017 with benefit. Due to problems in the prototype, I decided to rewrite everything and make the product on something familiar - Django. There are things out of the box that have significantly simplified my life, for example, the admin panel and working with PostGIS through ORM. But first it was necessary to recreate the functionality that was already working. It took only a couple of evenings, and most of the time was taken away by loading data from KML files. Not so much the import process itself, as their preparation and restoration of my knowledge about how to work with them. By the way, at that time I took polygons from gadm.org. This worked perfectly for countries, but with accuracy for the regions there were certain problems, so I took time out for this problem.

Project development

At that time, the application was simply a collection of HTML pages with a minimum of CSS. I wanted to quickly check the idea, and not bother with the design. The idea turned out to be realizable, and it was time to make a beautiful shell for it. Because I don’t have feelings of beauty in the UI, then Bootstrap will help me. The interface is no, but appeared, and even adapted for mobile devices. But this was only the first small step in bringing the front end in order.

What is it to learn javascript in 2016?! When the code is compiled into another dialect, it is template-edged, glued together to be cut into pieces. As a back-up person, it was scary, but the complexity of the client part was planned to be rather large, which necessitated the use of a framework or library. I stopped at React for two reasons: it was not a SPA, but a set of components for different pages, and I wanted to quickly see the result. But before you start programming, you had to set up the environment. Now I understand a familiar front-end vendor who said that he set up a Webpack for 2 days. It turns out that it was not a joke. 

At that time, I succumbed to the entreaties and implemented the logic of the application using Redux. Perhaps it was not a mistake, because allowed to quickly enter the topic. Formal rules allowed me to write code and make sure that it was working, without looking under the hood. Redux, with the help of its middleware, abstracted me from network interaction, which allowed me to check the answers to the server. Yes, up to this point, the client worked on its own - ajax pulled out all the necessary data with a request and checked the answers on its own. The user could cheat by looking at the data that comes from the server. In addition, when loading, the data arrived only necessary after the correct answer. After the implementation of the verification through web sockets, the process became ideologically more correct - the answer checks the code that is not available to the client.



Having learned all the power of javascript, it is difficult to stop. Immediately there were ideas where to add animation, folding blocks, blinking and new versions of games. One of them is “Quiz”, in which you need to guess the country by name, flag, emblem or capital. However, in the process of testing it turned out that some regions do not have flags, while others do not indicate the capital, so some countries had to hide from the list of available. At the same time, the game mode appeared on the physical map of the world - without borders of countries, for real pros.

Open data sources

Now there are ~ 50,000 polygons in the game, and I want to say a big thank you to such great projects as Wikipedia and Open Street Map, without which filling the base would be simply impossible. The principal requirement was to receive and update data from public sources, that is, without manual editing, because I do not want to do complex synchronization logic. As a result, I got 2 scripts that can update infoboxes and polygons.

Wikipedia and SPARQL

What is the largest database of countries and regions? Wikipedia! Initially, I wanted to show users the whole infobox, but soon refused this idea. Yes, there were important things like names, flags, capitals and other things, but there was also a lot of garbage (phone code, form of government, GDP ...). I tried to parse already collected, but found that they have a different structure. This turned out to be a catastrophe: labor costs for implementation have increased many times. It was time to stop and think. The very next day I found out about the existence of a special query language - SPARQL. On the surface it looks like SQL - too declarative, keywords SELECT, WHERE, ORDER BYbut operates completely differently. A small example that returns a list of states with their capitals in English and Russian languages:

SELECTDISTINCT ?country ?capital ?rowWHERE
{
?country wdt:P31 wd:Q3624078 .
FILTERNOTEXISTS {?country wdt:P31 wd:Q3024240}
OPTIONAL { ?country wdt:P36/rdfs:label ?capital } .
BIND(lang(?capital) as ?row)
filter (?row = 'ru' || ?row = 'en')
}
ORDERBY ?capital

Looks crazy, is not it? Check here . I even wrote a small note in my blog to somehow structure my experience and help to enter the topic, because Any detailed materials on the Internet a bit. Reading such requests can be learned quite quickly, but it took me a whole weekend to write something meaningful. A lot of "magic" of wd:Q3624078other attributes. You need to know that wdt:P31this is the "essence", and wd:Q3624078- "sovereign state." Unknowns begin with a question mark, and the execution of a query is precisely the search for such triples of facts that would satisfy the conditions. For example,?country wdt:P31 wd:Q3024240- “find all objects that are historical states”; and then the same object participates in other threes ?country wdt:P36/rdfs:label ?capital- where the capital is taken from it.

Somewhere in a week I had the first version of the script ready, which uploaded information about the regions from Wikipedia. And then another problem emerged - this time with the data. Some svg did not start with <?xml version="1.0" encoding="UTF-8"?>or were recognized by the browser as valid images. Fortunately, the source file can be edited. Registration on wikipedia.org is not a big deal, but you immediately find yourself in a sauna for a day. Here they have such protection from robots. So the very next night, I ruled XML and was happy about how simple it turned out, and flags and emblems appeared on the map.

Polygons

If for the facts we go to Wikipedia, then for geo-data - in the Open Street Map. It would be cool to pick up a local copy, learn the language of requests to overpass , but I can’t even imagine how long it would take. It’s also necessary to take a hierarchy from somewhere ... Fortunately, one kind person has already solved this problem for me. The service even provides an API for getting information. I managed to download all the polygons from there to level 6 inclusive (district) and upload everything to Postgres, a little more than 2 Gb came out. It was not without incident - some of the polygons were so large (for example, Canada weighs more than 100 Mb in a GeoJSON zipped) that the server either fell or did not respond. It was necessary to bypass such moments manually. I downloaded all the children and merged them into QGIS. By the way, this is another example of an open source project that helped me a lot. 

Big Data Problems

So, I have a database with data, I launch the game and ... and wait ... wait again ... appear! I tried to drag the polygon - He's dead, Jim! Chrome could not handle such a volume of points and fell. The strategy "in the forehead" no longer works, it's time to think. The most obvious is to reduce polygon detailing. Empirically derived a formula that depends on the area of ​​the figure - it became better. On the working computer, the algorithm worked on the fly, while the server is strictly limited in resources. Connected redis, it became better already on the server. But the trimmed polygons are good for Drag'n'Dropʻa, when installed on the right place, the borders do not coincide with those that draw Google Maps. Well, this can be circumvented by applying a less aggressive formula to reduce detail. Since there are already 2 caches, why not try to cache everything that is even possible ?! In Redis, infoboxes (in two languages) flew, the boundaries for which the answer is calculated, the center of the polygon, as well as the static pages of the site. As a result, the game began to work much faster, and a lot of work was removed from Postgres, which theoretically could be the bottleneck. Minus - the application does not work without Redis at all. 

First patch

So it's time to show the project to your friends for feedback. Only a little remains: generate sitemap.xml, add robots.txt, connect metrics, add social buttons. networks and ... fuck up! I chose AWS as the hosting. I expected to fit into the free resources. And this is a very good stack for a beginner project:

• application server (t2.micro: 1xCPU, 1 Gb RAM, 20Gb SSD)
• database (db.t2.micro: 1xCPU, 1 Gb RAM, 20Gb SSD)
• file storage with CDN (5 Gb S3, 50 Gb traffic)
• cache server (cache.t2.micro: 1xCPU, 0.5 Gb RAM)
• Elasticsearch + Kibana (t2.small.elasticsearch: 1xCPU, 2 Gb RAM)

This is just a list of what I managed to use. Along the way, I decided to draw up my rake in the form of articles , but quickly died. Time is running out decently, but it is not clear whether someone needs this. 

As a result, for the year of service, I paid something of the order of $ 10, and that was by stupidity. But the trial period came to an end, and I had to move, because The cost of owning all this farming has come close to a few hundred dollars. Equalized tariffs, and stopped at DigitalOcean. So far I have enough machines with 2 Gb RAM for everything (application server, database and cache), but I left statics and CDN on AWS. Now I learned that DO also got a CDN and a vault for $ 5 / mo, so it makes sense to think about moving this part as well.

Transfer to Open Source

January evening of this year I received a letter from the Danish school. The essence of it was that they had $ 100, and they want to give them to me. But there is a condition - the source of the project should be open. Up to this point, I did not even think about Open Source. A couple of evenings went to think about and select a license. As a result, I posted the source code on Githubunder the GPLv3 license and received the promised $ 100. This greatly increased motivation - my project really turned out to be useful! And I rushed to the next goal - the game editor. So that everyone can create their own jigsaw puzzles. For example, “Countries participating in the Second World War”, “Okrugs of the Krasnodar Territory”, “Countries without access to the sea” ... But for this, registration and a primitive personal account was necessary. As a result, the development was delayed for a long 3 months. During this time, I wrote a tree of regions that would pull up the data through ajax, connect localization, learn how to save the Google Map as a picture for generating thumbnails, and cut down redux. Yes, he helped me deal with the data at the very beginning, but now I was rather in the way. It would be necessary to drag the reducers to draw polygons on the map along with the code that would handle their movement. Luckily, removing the binding to the global state took only a couple of days, and even transferring the code to the local one simplified the application. And of course this is a good experience :)

Connecting services

It turns out that many paid services provide their services for free for open source projects. I will list only those that are connected to yours.

- Sentry . I think this error catching service is familiar to everyone. When I just checked the project, the logging was to send the frame to the post office. This worked only for the backend, but I also wanted to follow the bugs on the front end. And for good reason - I have almost reached the free limit of messages in just 2 weeks. Most of the errors were in the depths of the google map library, which at first glance is very strange. During the investigation, it turned out that I was to blame. Corrections lasted more than a month, but it was a very useful practice of working with errors in javascript.

- crowdin.com- localization. I plan to make the project accessible to everyone. Including that infoboxes are displayed in his native language. Filling them from Wikipedia is not a problem, but for the consistency I would also like to have an interface in the same language, but it is only translated into Russian and English.



- CircleCI . No modern project can do without CI / CD, tests and automatic deployment. I chose CircleCI solely because I already worked with TravisCI when I wrote the library for working with Yandex.Disk . I got the impression that it is more suitable for testing libraries, since it is easy to set the matrix of environments in which the code should be tested. But with the tests themselves I have trouble - there are not as many of them as we would like, although the infrastructure is already ready.



- Coveralls . Service code coverage visualization. It also knows how to give a label to be inserted into the project's README.md.

- SonarQube . The combine for quality control of a code. It checks the code according to a set of rules, considers cyclomatic complexity, monitors test coverage, and even recognizes duplication of code! Very interesting service, which I did not have time to fully understand.



- Github bots. So far only Dependabot is connected , which updates dependencies. 



I suggest in the comments to share a list of services and bots on their projects. 

Bugs

Analysis of bugs and problems deserves a separate article. There were funny, complicated and difficult to fix (therefore, Chukotka always stands in its place). Currently, there is one that really bothers users. When a response is received, the polygons are deleted and re-created (in the react-google-maps library), and if at that moment the user dragged some, the google map continues to assume that the process is not over yet. It looks like the polygon disappears in the process of drag'n'drop, and you can no longer grab any other. You can, of course, put a lock on handling the response in the process of drag'n'drop, but this is guaranteed to kill the multiplayer game, on the implementation of which I am currently working. I tried to find a way to programmatically interrupt dragging, but in the end I started a question on StackOverflowand a bug on Google Maps in the hope that it will pay attention. Until then, he added a button “the game is broken!”, Which reinitializes the entire card, but does not reset the result.

What's next?

1. Design. I admit that this is all bad. We need to hire a designer and coder, because I am not friends with the layout and layouts.
2. Monetization. Initially, I did not plan anything. The project is dedicated to basic education, which in my opinion should be accessible to all. I was very inspired by a letter from the Danish school, but almost a year passed, and during this time there was only one transfer for $ 5. Well, I did not believe that it could at least pay for the server. However, I still started a campaign on Patreon . At the same time, perhaps, you can think about the introduction of paid opportunities for teachers or organizations. For example, I have experience in integrating with Learning Management Systems - a set of platforms that allow you to create courses that are very popular in Europe and the USA. As far as I understand, even though the sources are also under the GPL on Github, this does not prevent me, as an author, from developing in parallel the commercial version.
3. Mobile phones. I want to release an application for iOS / Android. Judging by the Yandex-metric, a quarter of users try to play from a phone or tablet, but it turns out that they have difficulty.
4. Development. All work is done on GitHub . I want to continue to develop the project, but alone it is hard to do. Plans to add multiplayer, make tags, ratings and filters in the Workshop, add polygons for the physical map (mountains, seas, peninsulas). There is still a lot of interesting things ahead, so one of the goals of the article is to find like-minded people. Another option is to go to the foundation, for example, the Python Software Foundation and get a grant.

That is what is at the moment. Thank you for reading to the end! You can play here - geopuzzle.org , and the source to look at GitHub .

Minute of care from UFO

This material could cause conflicting feelings, so before writing a comment, refresh something important in your memory:

How to write a comment and survive

• Не пишите оскорбительных комментариев, не переходите на личности.
• Воздержитесь от нецензурной лексики и токсичного поведения (даже в завуалированной форме).
• Для сообщения о комментариях, нарушающих правила сайта, используйте кнопку «Пожаловаться» (если доступна) или форму обратной связи.

Что делать, если: минусуют карму | заблокировали аккаунт

→ Кодекс авторов Хабра и хабраэтикет
→ Полная версия правил сайта