Add-on to Avito. Startup or architecture example

Background

Honestly, I have never bought used items on bulletin boards before. But when the next economic crisis happened and the need made me, I had to turn my attention to Avito.

When examining the proposals, it immediately struck me that some of the ads looked dubious for a number of signs: a low price, an inaccurate description, etc., which made it seem like they were selling not what they were offering, or they themselves did not know what they were selling. An image from the past immediately appeared in my memory - a flea market of the 90s, from which you could return both with purchases, and with empty pockets or a cut bag.

As you know, Avito does not provide information about other ads of the seller, therefore, the text of the ad often uses keywords or hashtags that the buyer can use to search. But it should be noted that the Avito administration does not welcome the presence in the text of the announcement of any information not related to the subject of sale, as stated in the placement rules. In case of violation of the placement rules Avito may block both the ad and the seller’s account.

The search engine gave me several sites that parasitize on Avito and other message boards, mainly related to used cars. Mostly they offer search ads by phone number of the seller in the database of pre-downloaded ads. All of them have shortcomings, important for me: an intricate interface, paid services, updating data with a certain lag.

Sharing the opinion of Stephen Levy about free access to information, it was decided to analyze Avito for the development of its own sravnito.ru service with blackjack and all things, namely: with a simple interface and free access.

Based on the analysis, the main attributes of the ad were identified:

• title
• publication date
• price
• location
• phone number (either the image of the number, or a hash from it, or an OCR value recognized)

Architecture

Now, directly about the architecture of the software complex, on which the search service of all seller’s ads is based:

Storage

MySQL (Maria DB) with the InnoDB engine is used as a storage for storing ads and their screenshots.

DB1 stores declarations with basic attributes. To reduce the amount of memory occupied by data, the VARCHAR type is used for text fields, since their length varies from ad to ad. About half a million lines are added daily, among which are both the ads themselves, and the logs and service information. With such dynamics, the storage can rightly be attributed to Big Data. The following parameters can be distinguished from the configuration features:

max_heap_table_size = 512M
innodb_buffer_pool_size = 3G

Tables on the heap are used to optimize queries from several large tables when data is first selected in a temporary table:

CREATE TEMPORARY TABLE _temp_table ENGINE=MEMORY AS (
SELECT field
FROM table
WHERE key = i_key
LIMIT i_limit);

which then connects to another:

SELECT table.*
FROM table
JOIN _temp_table
ON table.field = _temp_table.field;

DB2 stores screenshots of the ads as they appear to the browser user. Before recording, the screenshots are compressed in JPEG with quality = 5, which provides a file size with an image equal to approximately 20Kb. It is generally accepted that with a BLOB size of not more than 200Kb, the performance of file storage in MySQL is in no way inferior to NoSQL storage, which allows you to remain in the comfort zone of a relational database management system with all its advantages. Such a compressed screenshot, despite the minimum image quality, allows the user to make sure that the specified ad really existed and to see at least a schematic image of the product.

All logic is implemented in stored procedures to encapsulate code dependent on the data structure in the DBMS itself. Thus, DBMS clients have authority only to access stored procedures that are idempotent. As an additional plus, we get the inability to make SQL injections.

Storage API

The M1 server is implemented as a golang microservice and provides a RESTful API for saving ads, screenshots, as well as for reading data displayed on the page of the search engine for ads. There is no reason to use any frameworks or external libraries to implement RESTful on golang, therefore only standard libraries are used, except for one:

import (
"database/sql"
"encoding/json"
"net/http"
_ "github.com/go-sql-driver/mysql"
)

GET and POST requests from clients are processed and the corresponding DB DB, DB2 database stored procedures are called.

Ad Loaders

Ads are downloaded from the Avito site by S (1) -S (N) downloaders written in Java using the Selenium WebDriver library. After receiving the advertisement attributes and a screenshot from Avito, the loader contacts the M1 server for data transfer. Also, feedback has been implemented to control boot loaders that periodically poll the M1 server for commands, such as “stop”, “start”.

Captcha

To solve the captcha that Avito sometimes asks for, there is a C1 server, similar to the M1 server, implemented on golang and providing a RESTful API. There are two ways to solve captcha:

• using rucaptcha.com
• manually in an android application

Their API is used to contact rucaptcha.com. For manual guessing, a Java application written in Java is used that displays the image and receives the answer. The C1 server decides to redirect the captcha to rucaptcha.com or to the Android application, depending on the number of requests that have accumulated in the queue. Having received the solution of the captcha, server C1 sends a response to the requesting loader.

Monitoring

Similar to the Android app for solving captcha, there is a monitoring application. The Android monitoring application accesses the M1 server, which in turn accesses the database where logs are aggregated, based on which you can judge the number of loaded ads, malfunctions, etc.

Conclusion

Further development of the service may be as follows:

• Creating downloaders for other bulletin boards, which will allow you to cross-search all the ads of one seller
• Using the complex to process other data, for example, downloading from social networks to search for all posts by account name

In both cases, only boot loaders are subject to development that can easily integrate with the M1 and C1 servers. Refinement of other parts of the system is not required.

If you make the API of the M1, C1 servers public with authorization and add a key field for dividing by clients into the database structure, you can provide a service for storing data and processing captcha as SaaS. Client data can be stored in BLOB in the form of JSON, while finalizing the database with stored procedures and the API.

What can be said about the selected technologies:

• MySQL is a classic of the genre, no comment
• Java - loaders can be run on coffee makers and refrigerators
• golang - microservices are developed very quickly, easily deployed by copying a single binary to the server

I would appreciate comments and discussion.