Fetchee Product API: Automatically Parsing Online Store Items / Fetchee Blog

UPD: Today, API access keys were sent to everyone who submitted a request for https://fetch.ee/ru/developers/ .
UPD2: Placed tutorial on the use of Product API in a new note to Habré.

From the beginning of our work on Fetchee, we had a plan to offer a solution for parsing data on products of online stores to corporate clients and the startup community. The reviews that we collected during the work talked about its potential demand and led to the launch of a new product that will help developers quickly create their own applications, based on one of the main functions of Fetchee - collecting eCommerce data and tracking prices in any online store in the world .

At the start of the project, we wanted to show the finished product to users as quickly as possible, so we tried to simplify the application in all ways. One of the options was to refuse to develop your own parser for online stores and implement a turnkey solution.

In order to reduce the volume of the story, I’ll miss a comparison of popular applications for extracting data from sites (if you want to see this comparison, leave your vote in the survey at the end of the article). Suffice it to say that diffbot turned out to be the most suitable for our task - an extremely expensive service, popular among large companies and startups with decent funding. Many bookmarking applications use it, because high-quality parsing is difficult.

Firstly, we could not afford a solution, and secondly, diffbot did not work well with Russian online stores. I had to come up with my own parsing technology, which should be able to:

• Automatically determine the main parameters of the product;
• Work with any currency and language;
• Take into account the geographical location of users and the features of the stores with this region (language, currency and country);
• Identify shops with discriminatory pricing (version of the operating system and city within the country).

We managed to implement all the principles described above. The system learns from new data and grows smarter before our eyes. The more requests, the more accurate it becomes. Our desire to make the selection more diverse was one of the reasons for providing third-party developers access to the Product API, as the rate of appearance of new data on the part of users has slowed down - 90% buy in the same online stores.

So, meet the Fetchee Product API . With it, you can, by specifying any product URL in the online store, get basic data about the item, including name, image, price and currency. In the case of our API, you do not have to configure the parser for each site, the system independently analyzes the page and determines the necessary data.



For example, here are the parsing resultsbike with Avito .

{
   "id":"58234b6cfd920b507bfd6b1f",
   "url":"https://www.avito.ru/sankt-peterburg/velosipedy/velosiped_author_basic_rama_19_i_21_835103333",
   "title":"Велосипед Author Basic, рама 19\" и 21\"",
   "price":14900,
   "currency":"RUB",
   "img_url":"https://fetch.ee/assets/item-images/5823/4b78475d39467b4b25eb.jpg",
   "created_at":"2016-11-09T16:14:36.542Z",
   "last_track_at":"2016-11-09T16:14:48.061Z"
}

Fashionable bags with wildberries .

{
   "id":"5824212c65ef60477b38b890",
   "url":"https://www.wildberries.ru/catalog/3095060/detail.aspx?targetUrl=GP",
   "title":"Сумка, GUESS",
   "price":13560,
   "currency":"RUB",
   "img_url":"https://fetch.ee/assets/item-images/5824/2144475d39467b4b26de.jpg",
   "created_at":"2016-11-10T07:26:36.368Z",
   "last_track_at":"2016-11-10T07:27:00.544Z"
}

Or an SUV from a car site .

{
   "id":"5824219b65ef60477b38b8be",
   "url":"https://auto.ru/cars/new/sale/bmw/x6_m/1044423007-94d1a/",
   "title":"Продажа BMW X6 M II (F86) в Москве",
   "price":8099000,
   "currency":"RUB",
   "img_url":"https://fetch.ee/assets/item-images/5824/21ad475d39467b4b26df.jpg",
   "created_at":"2016-11-10T07:28:27.038Z",
   "last_track_at":"2016-11-10T07:28:45.516Z"
}

On any page where there is a price, image and product name - Product API will find this data and return it to you in JSON for further processing.

You can try the API directly on the Beta program page . If you like the result of the test drive, register as a developer - the more requests there are, the sooner we will provide full access to the API.

We are very interested to see how your ideas come to life using the Product API.