How I fought theft ... with php


When we pay daily for services - this is the purchase of services.
When we pay daily for nothing (sometimes without even knowing it), it is theft.

Good afternoon, readers of Habr!

How it all started


I wanted to steal less, and let's fight it! But manually it was very tiring, long and ineffective, then the thought came to somehow automate this business.

Which one of the “thieves” am I? About where we, walking on the Internet, click on the "watch video" button, some page is loaded, the video for some reason does not play, we leave and walk further, but in fact we "voluntarily" hooked up our service to receive that Something that no one has ever seen for a nominal fee of 30 rubles a day from the account of his mobile. In humans, this is called wap-click or mobile subscriptions, and mobile operators come up with a variety of beautiful names. Still, not to include in the list of services “theft on a video button”.

Here is a little more detail. And here is a story about a good way to "earn."

Described cases is not entirely voluntary subscriptions lot, this , for example. Undescribed - much more.

Wrestlers also have:


What and why was automated


Search and block ads in the Google AdSense publisher panel.
The goal is to increase the blocking efficiency and free up the time spent manually cleaning.

The essence of the problem and the existing solutions
Долгие годы (первое упоминание о подобном, что я нашёл было летом 2014-го) издатели вручную отлавливали потоки «смертей Якубовича», «каменных стояков», «смотреть видео смотреть, жми смотреть» и прочей нечисти (начало, продолжение), сей процесс почти никак не автоматизировался1 и это казалось практически невозможным.

1 Есть (по крайней мере когда-то было) два решения, но у них довольно серьёзные требования, которые не каждый может себе позволить.
Эти самые решения:

  1. AdSense Cleaner. Требуется много доп. ПО.
  2. AdsAutomation. Сценарий для управления браузером Google Chrome (как я понял, на ZennoPoster). Необходим отдельный ПК. И в данный момент с GitHub проект удалён.

If you make software that replaces the person blocking ads, then it should be done taking into account a number of requirements:

  • should work on the hardware and software that almost all site owners have;
  • do not require additional software and change the settings available;
  • Easy to install and configure, so that an ordinary user can deliver.
.

In general, php (with cURL) will be what you need. You can throw right on your site and work without additional computers and other difficulties.

And one clarification to the requirements.
Так как решение подразумевалось автоматизированным на php, следовательно, запуск через cron, то хранение пользовательских настроек и временных данных должно быть на диске (не в cookie). В Cookie-файлах будет хранится только ключ для доступа к панели управления. Для избранных, кто не имеет возможности настроить cron, но может на ПК/планшете/смартфоне держать одну вкладку открытой будет добавлена возможность периодического запуска по таймеру на Javascript.

What foreshadowed the beginning or Google API


And for AdSense there is an API, I somehow saw it out of the corner of my eye and did not go deep. And now - it's time to understand. There are many possibilities, but it turned out that neither here , well, there is nothing described about the API for CSP . Want to watch ads that are spinning on the site, please - manually.

Start


The Google AdSense interface is built on AngularDart, everything looks beautiful and quite difficult from the point of view of the device.

First of all, I looked into the Google Chrome developer’s tools on the “Network” tab to “listen in” how this clever interface communicates with the server. There was a lot of requests there, the most interesting for me were in the section “XHR and Fetch”, where I found something that looked completely unraveled, if you think well. For example, one of the post requests:

String to be passed.
{"method":"searchArcApprovals","params":"{\"1\":\"ca-pub-8958890276790964\",\"2\":{\"1\":0,\"2\":1,\"3\":0,\"4\":{\"1\":{\"1\":\"AClZvXKL6S3HChRty5YBa81BLWDBQkb3FYDsifZ9V/mBTKbOGlj3gMWVpzTtXggA1880Le9NyVZIicNm/4pz724e/MO8fyLfjOReF205cyjLV9C8OCCeKe7VvZHyvyKpXh8x9smTQ0n8qIIqzuIXle5UK0hD4VBkZDvy//qoSPRCr94UtWYqqi//Rot22LJ2JFNjWEGb4n1YQbAw0cKWPR3LAugPBajInWXEFGWJRTnmY2TkI5VzUzIkcXpJ/bkajn3c8GnecCfFNvNhGLS10VXdRwiykngG3xfoMTRhQOR5GXbm4kwdIhzQUM/d6xP0Xda3FOIZGGk9bymneg+9oDY+rMFiRfDFCb66g50t9J9r++oHXjek09Ci1rqC7LOw2pvkqp3hjG6RyVmsiT/eWGq+OsfjE7CgRk43QIRMSa+jlZBQhARUPlpUXzyZyoTiIPTRZ5ND/4MnIMqaUWSRoDGffiE/XkHJPEkNZtLX2XR5gZ3x5/K+ejU/fqxfZIjI6A3kueJybNA46wSLbmflhDCGDJEE2aeYemLFGqNzFG43B80LzU3yuwgZhrLu/jaMvBJozi0nq+gXEz6r+8tic4fvsQ9lWDA+IXzXw6MKzamgfWV0ORGDW0+966KIY6IkjtIlNRKGyp3pSAd2Po+br4Dl4WNwSkMdmuV60wOrkb5BpnKZKIhDtpjWF7q6ly3FFhwo8Ktdq5ddVJ8ijJ9Y9tQhs2O0idA9N0yV86khV1IQ72OgbMv15qAswnbqF9WCo3qpfJNjJqMCHBRTohPCxhRp0cWz2thszZTmDDADPxU46sclnurd/JxHFO7lJZVdrsFB4vdLIx9kObV3bP1gOpU66kdcmom2tiedknugj7s0jLcgf1EfXnp+SUUAQyoqwS+kdhhQtGqSXgI2TopsuaLVzj+EtAuPwWeLvtI9CFPSe4o2x+gjCRPl8wVvWKV5FIrZavUVOAHZIL4nKyJjHxZi3jPfVnAia/hq1gW6XKoCg1eWGg/cAWZY4mZYQ6W4XnC0MY0uMC6fhPQdXnIS5iLZNhan80jbr/leBr4fO22+tXc6oZpZsDkXd0r3ilBJFPS2I/zAhotuzZgNA+nF2N86pyiSrdeEYFDhKWKadcKAVc3BMxxlrqZYcAXnlus9GW7R9F/ImXQ/fjRfSjVRUaJuQ0EnFejNAwdGcS6STYMa1G0wnNMAKcZ52xcHgil1SZ6N9BQ7A27z6eViOxw0LHBqNJIRZwQml2KjPd5b00D9XvohDr6jBqYXLGS/HMVvpGDJZLDI2LRlmkqBqx7YEgDZqvspeoMLHIJP22SkQDnaJtsOLGVBSi20ZD5nRyjAgS6MmcgFCvfJVWjCIL1RPHqmUU90eK4WXve0ayH9cJnpbtWrkXYCibhVPCMmYowMROw7rI4bPir0\"}}}}","xsrf":"ABOvogKvrE9fIqAKh0w02RIsB4OJ4hsB_g:1535467885347"}

In the request, the publisher ID is immediately visible, under the second item, a set of parameters, the essence of which can be determined experimentally and the XSRF token.

And in response, he receives detailed information about the ad, but not all of it, even without the ad itself (hereinafter, the images stretched out in base64, cropped).

Sheet on several pages.
{"result":{"1":[{"1":0,"3":0,"4":{"1":"AClZvXJ2t4wiEZ/VZ0i54m0Qtqpi2DTqkI1kaPMTRi4LnsQn0iR5K1xBlFpS1xmJV7ko4a6qx5RcTkp7CzVjwoy5UDSWZ5jOCPLGRcoQdDt+wOk46bdr0yA\u003d"},"5":{"1":82,"2":0,"3":0,"4":"\u003cdiv id\u003d\"ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60\"\u003e\u003c/div\u003e","5":"\u003cdiv id\u003d\"ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60\"\u003e\u003c/div\u003e","6":"\u003cdiv\u003e\u041c\u043d\u043e\u0433\u043e\u0444\u043e\u0440\u043c\u0430\u0442\u043d\u044b\u0435\u003cspan id\u003d'multi-format-tooltip'\u003e\u003c/span\u003e\u003c/div\u003e\u003ca class\u003d'arc-url-link-ellipsis' target\u003d'_blank' href\u003d'https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/' title\u003d'https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/'\u003ehttps://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\u003c/a\u003e","7":"\u003cdiv class\u003d'arc-one-by-one-legend'\u003e\u0422\u0438\u043f \u043e\u0431\u044a\u044f\u0432\u043b\u0435\u043d\u0438\u044f\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003e\u041c\u043d\u043e\u0433\u043e\u0444\u043e\u0440\u043c\u0430\u0442\u043d\u044b\u0435\u003cspan id\u003d'multi-format-tooltip'\u003e\u003c/span\u003e\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-legend'\u003e\u0426\u0435\u043b\u0435\u0432\u043e\u0439 URL\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003e\u003ca class\u003d'arc-url-link-ellipsis' target\u003d'_blank' href\u003d'https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/' title\u003d'https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/'\u003ehttps://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\u003c/a\u003e\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-legend'\u003e\u0414\u043e\u043c\u0435\u043d\u044b \u0438\u0437\u0434\u0430\u0442\u0435\u043b\u0435\u0439\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003e4aynikam.ru\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003eandroidphone.su\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003eandroidphones.ru\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003efull-repair.com\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003ehowgadget.com\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-legend'\u003e\u041e\u0431\u043d\u0430\u0440\u0443\u0436\u0435\u043d\u043d\u044b\u0439 \u0440\u0435\u043a\u043b\u0430\u043c\u043e\u0434\u0430\u0442\u0435\u043b\u044c\u003cspan id\u003d'adx-advertiser-tooltip'\u003e\u003c/span\u003e\u003c/div\u003e\u003cdiv class\u003d'arc-one-by-one-data'\u003eDNS Shop\u003c/div\u003e","8":"\u003cdiv\u003e\u003cspan class\u003d'arc-impression-score high'\u003e\u0412\u042b\u0421\u041e\u041a\u041e\u0415\u003c/span\u003e \u0447\u0438\u0441\u043b\u043e \u043f\u043e\u043a\u0430\u0437\u043e\u0432\u003c/div\u003e","9":{"1":"\u003ca href\u003d\"https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\" target\u003d\"_blank\"\u003e\u003cimg onerror\u003d\"this.src\u003d'data:image/gif;base64,RA7'\" src\u003d\"https://www.google.com/webpagethumbnail?c\u003d58\u0026s\u003d400:400\u0026r\u003d4\u0026d\u003dhttps://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\u0026a\u003dAIYkKU9ZGGjFTOWtm771MQwgDYxqtlBLCw\" border\u003d0 alt\u003d\"\"\u003e\u003c/a\u003e","2":"\u003ca href\u003d\"https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\" target\u003d\"_blank\"\u003e\u003cimg onerror\u003d\"this.src\u003d'data:image/gif;base64,R0AA7'\" src\u003d\"https://www.google.com/webpagethumbnail?c\u003d58\u0026s\u003d400:400\u0026r\u003d3\u0026d\u003dhttps://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\u0026a\u003dAIYkKU_CQ2K6v5f11Nk1RXtc87FtmG2B1w\" border\u003d0 alt\u003d\"\"\u003e\u003c/a\u003e","3":"\u003ca href\u003d\"https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\" target\u003d\"_blank\"\u003e\u003cimg onerror\u003d\"this.src\u003d'data:image/gif;base64,R0lAA7'\" src\u003d\"https://www.google.com/webpagethumbnail?c\u003d58\u0026s\u003d400:400\u0026r\u003d6\u0026d\u003dhttps://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/\u0026a\u003dAIYkKU_My0a48LAsW-ZKpQX-ATXkMoPEVg\" border\u003d0 alt\u003d\"\"\u003e\u003c/a\u003e"},"10":"https://adwords-displayads.googleusercontent.com/da/b/preview.js?client\u003dasfe-arc-external-preview\u0026obfuscatedCustomerId\u003d5240877441\u0026creativeId\u003d288930210411\u0026htmlParentId\u003dad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60\u0026sig\u003dACiVB_yMUjLwDjRO2T-0VAaVuRPt8uLHGQ","13":"https://adwords-displayads.googleusercontent.com/da/b/preview.js?client\u003dasfe-arc-external-preview\u0026obfuscatedCustomerId\u003d5240877441\u0026creativeId\u003d288930210411\u0026htmlParentId\u003dad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60\u0026showVariations\u003dtrue\u0026sig\u003dACiVB_yMUjLwDjRO2T-0VAaVuRPt8uLHGQ","14":"https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/","15":"","17":"","18":"DNS Shop","20":"adv-5594449542310820","21":["site1.ru","site2.com","site3.com","site4.ru"]},"6":{"5":"-6668648012302470727","7":["DNS"],"9":0},"7":1,"9":{"3":[{"1":{"1":"AClZvXLE9HJbFYq9TrAsXFgV4YkXsQt9lXp1xWjSB5aT5bFBpe4VNgo\u003d"},"2":"\u0418\u043d\u0442\u0435\u0440\u043d\u0435\u0442 \u0438 \u0442\u0435\u043b\u0435\u043a\u043e\u043c\u043c\u0443\u043d\u0438\u043a\u0430\u0446\u0438\u0438","3":"\u0422\u043e\u0432\u0430\u0440\u044b \u0438 \u0443\u0441\u043b\u0443\u0433\u0438, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0435 \u0441 \u0442\u0435\u043b\u0435\u043a\u043e\u043c\u043c\u0443\u043d\u0438\u043a\u0430\u0446\u0438\u044f\u043c\u0438, \u0432 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435 \u043a\u0430\u0431\u0435\u043b\u044c\u043d\u043e\u0435 \u0438 \u0441\u043f\u0443\u0442\u043d\u0438\u043a\u043e\u0432\u043e\u0435 \u043e\u0431\u0441\u043b\u0443\u0436\u0438\u0432\u0430\u043d\u0438\u0435 \u0438 \u0434\u043e\u0441\u0442\u0443\u043f \u0432 \u0418\u043d\u0442\u0435\u0440\u043d\u0435\u0442."},{"1":{"1":"AClZvXKrUJJ3kKBen2scP56BynOtGhf160i1F1LLmtBj3b/oh2dUFg8\u003d"},"2":"\u041c\u043e\u0431\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u044b","3":"\u041c\u043e\u0431\u0438\u043b\u044c\u043d\u044b\u0435 \u0438 \u0441\u043e\u0442\u043e\u0432\u044b\u0435 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u044b, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0441\u043e\u043f\u0443\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0430\u044f \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044f, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440 \u0442\u0435\u0445\u043d\u0438\u0447\u0435\u0441\u043a\u0438\u0435 \u0445\u0430\u0440\u0430\u043a\u0442\u0435\u0440\u0438\u0441\u0442\u0438\u043a\u0438 \u0438 \u0441\u0440\u0430\u0432\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u0430\u043d\u0430\u043b\u0438\u0437 \u0442\u043e\u0432\u0430\u0440\u043e\u0432. \u0412 \u044d\u0442\u0443 \u043a\u0430\u0442\u0435\u0433\u043e\u0440\u0438\u044e \u043d\u0435 \u0432\u0445\u043e\u0434\u044f\u0442 \u0430\u043a\u0441\u0435\u0441\u0441\u0443\u0430\u0440\u044b \u0434\u043b\u044f \u043c\u043e\u0431\u0438\u043b\u044c\u043d\u044b\u0445 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u043e\u0432."},{"1":{"1":"AClZvXL4W+khZ4O9SJiu97cTbTs2+0Wecf1IVNju8ffd4ysIT9PJ7XY\u003d"},"2":"\u041c\u043e\u0431\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u044b \u0438 \u0430\u043a\u0441\u0435\u0441\u0441\u0443\u0430\u0440\u044b \u0434\u043b\u044f \u043d\u0438\u0445","3":"\u041c\u043e\u0431\u0438\u043b\u044c\u043d\u044b\u0435 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u044b, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0441\u043e\u043f\u0443\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0435 \u0430\u043a\u0441\u0435\u0441\u0441\u0443\u0430\u0440\u044b \u0438 \u0430\u043f\u043f\u0430\u0440\u0430\u0442\u043d\u043e\u0435 \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u0435, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440 \u0447\u0435\u0445\u043b\u044b, \u043c\u043e\u043d\u043e\u043f\u043e\u0434\u044b \u0434\u043b\u044f \u0441\u0435\u043b\u0444\u0438, \u0437\u0430\u0449\u0438\u0442\u043d\u044b\u0435 \u044d\u043a\u0440\u0430\u043d\u044b \u0438 \u0437\u0430\u0440\u044f\u0434\u043d\u044b\u0435 \u0443\u0441\u0442\u0440\u043e\u0439\u0441\u0442\u0432\u0430."},{"1":{"1":"AClZvXLQ3gPoVwjQbokDpB3+nni4xURwH5+YlnwkqjYtUowjhiKvk8Q\u003d"},"2":"\u041f\u041a \u0438 \u0431\u044b\u0442\u043e\u0432\u0430\u044f \u044d\u043b\u0435\u043a\u0442\u0440\u043e\u043d\u0438\u043a\u0430","3":"\u0422\u043e\u0432\u0430\u0440\u044b, \u0443\u0441\u043b\u0443\u0433\u0438 \u0438 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044f, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0435 \u0441 \u043a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u0430\u043c\u0438 \u0438 \u0431\u044b\u0442\u043e\u0432\u043e\u0439 \u044d\u043b\u0435\u043a\u0442\u0440\u043e\u043d\u0438\u043a\u043e\u0439."},{"1":{"1":"AClZvXLKYOGgOROaa32IUxU15jP89AtTM4dV24WKS+daMhqJMTNmeSY\u003d"},"2":"\u0422\u0435\u043b\u0435\u0444\u043e\u043d\u0438\u044f","3":"\u0422\u043e\u0432\u0430\u0440\u044b, \u0443\u0441\u043b\u0443\u0433\u0438, \u0430 \u0442\u0430\u043a\u0436\u0435 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0435 \u0438 \u0434\u0440\u0443\u0433\u0438\u0435 \u0440\u0435\u0441\u0443\u0440\u0441\u044b, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0435 \u0441 \u0442\u0435\u043b\u0435\u0444\u043e\u043d\u0438\u0435\u0439 \u0438 \u0433\u043e\u043b\u043e\u0441\u043e\u0432\u043e\u0439 \u0441\u0432\u044f\u0437\u044c\u044e."}]},"10":{"1":"AClZvXLdGOShgJo+BM3apOUAFzQkE41z1/hiZhIY8eUlC7p7xXPm82P3dq7yXhbEI+tN/YHgdH4P"}}],"2":0.0,"3":"60609","4":1,"5":"","6":"ClD3Z2nP2P/////1/ff99fXV98nMyMrJz8rH9fHV883Hx8bMz83Oz8vOzv8A/v/+9f33/fX11ffJzMjKyc/Kx/Xx1fPNx8fGzM/Nzs/Lzs7//hABIWxUk293Pm+qOQAAAAAnMJaYSAFQAFoLCS8wxxaTatL1EAJgp7737gY\u003d","7":"3639","9":0},"xsrf":"ABOvogKaRsVZECZZJU-gDWrOqoP0CSqf7Q:1535467886413"}

After json_decode, it looks like this:

Object from json-string (carefully, 175 lines).
object(stdClass)#19 (2) {
  ["result"]=>
  object(stdClass)#18 (8) {
    ["1"]=>
    array(1) {
      [0]=>
      object(stdClass)#1 (8) {
        ["1"]=>
        int(0)
        ["3"]=>
        int(0)
        ["4"]=>
        object(stdClass)#2 (1) {
          ["1"]=>
          string(120) "AClZvXJ2t4wiEZ/VZ0i54m0Qtqpi2DTqkI1kaPMTRi4LnsQn0iR5K1xBlFpS1xmJV7ko4a6qx5RcTkp7CzVjwoy5UDSWZ5jOCPLGRcoQdDt+wOk46bdr0yA="
        }
        ["5"]=>
        object(stdClass)#3 (17) {
          ["1"]=>
          int(82)
          ["2"]=>
          int(0)
          ["3"]=>
          int(0)
          ["4"]=>
          string(102) "<div id="ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60"></div>"
          ["5"]=>
          string(102) "<div id="ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60"></div>"
          ["6"]=>
          string(355) "<div>Многоформатные<span id='multi-format-tooltip'></span></div><a class='arc-url-link-ellipsis' target='_blank' href='https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/' title='https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/'>https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/</a>"
          ["7"]=>
          string(1066) "<div class='arc-one-by-one-legend'>Тип объявления</div><div class='arc-one-by-one-data'>Многоформатные<span id='multi-format-tooltip'></span></div><div class='arc-one-by-one-legend'>Целевой URL</div><div class='arc-one-by-one-data'><a class='arc-url-link-ellipsis' target='_blank' href='https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/' title='https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/'>https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/</a></div><div class='arc-one-by-one-legend'>Домены издателей</div><div class='arc-one-by-one-data'>4aynikam.ru</div><div class='arc-one-by-one-data'>androidphone.su</div><div class='arc-one-by-one-data'>androidphones.ru</div><div class='arc-one-by-one-data'>full-repair.com</div><div class='arc-one-by-one-data'>howgadget.com</div><div class='arc-one-by-one-legend'>Обнаруженный рекламодатель<span id='adx-advertiser-tooltip'></span></div><div class='arc-one-by-one-data'>DNS Shop</div>"
          ["8"]=>
          string(98) "<div><span class='arc-impression-score high'>ВЫСОКОЕ</span> число показов</div>"
          ["9"]=>
          object(stdClass)#4 (3) {
            ["1"]=>
            string(4191) "<a href="https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/" target="_blank"><img onerror="this.src='data:image/gif;base64,RCw" border=0 alt=""></a>"
            ["2"]=>
            string(4191) "<a href="https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/" target="_blank"><img onerror="this.src='data:image/gif;base64,R1w" border=0 alt=""></a>"
            ["3"]=>
            string(4191) "<a href="https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/" target="_blank"><img onerror="this.src='data:image/gif;base64,Rg" border=0 alt=""></a>"
          }
          ["10"]=>
          string(291) "https://adwords-displayads.googleusercontent.com/da/b/preview.js?client=asfe-arc-external-preview&obfuscatedCustomerId=5240877441&creativeId=288930210411&htmlParentId=ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60&sig=ACiVB_yMUjLwDjRO2T-0VAaVuRPt8uLHGQ"
          ["13"]=>
          string(311) "https://adwords-displayads.googleusercontent.com/da/b/preview.js?client=asfe-arc-external-preview&obfuscatedCustomerId=5240877441&creativeId=288930210411&htmlParentId=ad-parent-id-6A2DE3D206234468F53C743C0EEACD67A59E6C5B62C0371F770419826258CB1AD9591F60&showVariations=true&sig=ACiVB_yMUjLwDjRO2T-0VAaVuRPt8uLHGQ"
          ["14"]=>
          string(69) "https://www.dns-shop.ru/actions/c09a061b-a048-11e8-9547-00155d03330d/"
          ["15"]=>
          string(0) ""
          ["17"]=>
          string(0) ""
          ["18"]=>
          string(8) "DNS Shop"
          ["20"]=>
          string(20) "adv-5594449542310820"
          ["21"]=>
          array(4) {
            [0]=>
            string(8) "site1.ru"
            [1]=>
            string(9) "site2.com"
            [2]=>
            string(9) "site3.com"
            [3]=>
            string(8) "site4.ru"
          }
        }
        ["6"]=>
        object(stdClass)#5 (3) {
          ["5"]=>
          string(20) "-6668648012302470727"
          ["7"]=>
          array(1) {
            [0]=>
            string(3) "DNS"
          }
          ["9"]=>
          int(0)
        }
        ["7"]=>
        int(1)
        ["9"]=>
        object(stdClass)#16 (1) {
          ["3"]=>
          array(5) {
            [0]=>
            object(stdClass)#7 (3) {
              ["1"]=>
              object(stdClass)#6 (1) {
                ["1"]=>
                string(56) "AClZvXLE9HJbFYq9TrAsXFgV4YkXsQt9lXp1xWjSB5aT5bFBpe4VNgo="
              }
              ["2"]=>
              string(52) "Интернет и телекоммуникации"
              ["3"]=>
              string(217) "Товары и услуги, связанные с телекоммуникациями, в том числе кабельное и спутниковое обслуживание и доступ в Интернет."
            }
            [1]=>
            object(stdClass)#9 (3) {
              ["1"]=>
              object(stdClass)#8 (1) {
                ["1"]=>
                string(56) "AClZvXKrUJJ3kKBen2scP56BynOtGhf160i1F1LLmtBj3b/oh2dUFg8="
              }
              ["2"]=>
              string(35) "Мобильные телефоны"
              ["3"]=>
              string(359) "Мобильные и сотовые телефоны, а также сопутствующая информация, например технические характеристики и сравнительный анализ товаров. В эту категорию не входят аксессуары для мобильных телефонов."
            }
            [2]=>
            object(stdClass)#11 (3) {
              ["1"]=>
              object(stdClass)#10 (1) {
                ["1"]=>
                string(56) "AClZvXL4W+khZ4O9SJiu97cTbTs2+0Wecf1IVNju8ffd4ysIT9PJ7XY="
              }
              ["2"]=>
              string(73) "Мобильные телефоны и аксессуары для них"
              ["3"]=>
              string(283) "Мобильные телефоны, а также сопутствующие аксессуары и аппаратное обеспечение, например чехлы, моноподы для селфи, защитные экраны и зарядные устройства."
            }
            [3]=>
            object(stdClass)#13 (3) {
              ["1"]=>
              object(stdClass)#12 (1) {
                ["1"]=>
                string(56) "AClZvXLQ3gPoVwjQbokDpB3+nni4xURwH5+YlnwkqjYtUowjhiKvk8Q="
              }
              ["2"]=>
              string(45) "ПК и бытовая электроника"
              ["3"]=>
              string(142) "Товары, услуги и информация, связанные с компьютерами и бытовой электроникой."
            }
            [4]=>
            object(stdClass)#15 (3) {
              ["1"]=>
              object(stdClass)#14 (1) {
                ["1"]=>
                string(56) "AClZvXLKYOGgOROaa32IUxU15jP89AtTM4dV24WKS+daMhqJMTNmeSY="
              }
              ["2"]=>
              string(18) "Телефония"
              ["3"]=>
              string(181) "Товары, услуги, а также информационные и другие ресурсы, связанные с телефонией и голосовой связью."
            }
          }
        }
        ["10"]=>
        object(stdClass)#17 (1) {
          ["1"]=>
          string(76) "AClZvXLdGOShgJo+BM3apOUAFzQkE41z1/hiZhIY8eUlC7p7xXPm82P3dq7yXhbEI+tN/YHgdH4P"
        }
      }
    }
    ["2"]=>
    float(0)
    ["3"]=>
    string(5) "60609"
    ["4"]=>
    int(1)
    ["5"]=>
    string(0) ""
    ["6"]=>
    string(168) "ClD3Z2nP2P/////1/ff99fXV98nMyMrJz8rH9fHV883Hx8bMz83Oz8vOzv8A/v/+9f33/fX11ffJzMjKyc/Kx/Xx1fPNx8fGzM/Nzs/Lzs7//hABIWxUk293Pm+qOQAAAAAnMJaYSAFQAFoLCS8wxxaTatL1EAJgp7737gY="
    ["7"]=>
    string(4) "3639"
    ["9"]=>
    int(0)
  }
  ["xsrf"]=>
  string(48) "ABOvogKaRsVZECZZJU-gDWrOqoP0CSqf7Q:1535467886413"
}


This was an example response, containing only one announcement. Understand what you need.
Yes, and other requests methods are quite humanly called. A few examples:

  • getWebPropertyMetricsToken
  • getAdDisplayLanguages
  • getArcSettings
  • getAdNetworkApprovals
  • getPubControlsCapabilities

Theoretically possible. To battle?


Okay, to solve their communication is possible (theoretically), but all this will be useless, but the theory will remain, if you do not make authorization in Google.

Authorization Or how to log in to Google on php + cURL


Again, developer tools, logout and look at the exchange of data. I do not remember in detail, because I could not understand anything there. A huge amount of JS, it seems that some calculations are made right on the client, the results are sent to the server. In general, it is almost impossible for a non-human to enter.

We think further. A bunch of js. And if JS disable? Cann’t Google users without JS be able to log in? Well, try without JS. Externally, the authorization window already looks much simpler. As before, we first enter the login, and the password on the next page. Most importantly, in terms of HTML is also much easier! The usual tag "form" with the usual fields "input", though not without a heap of security or system hidden fields. But hidden fields are not a problem, because what they received at the entrance was transferred to the next script. And so it came to log in to Google. And two-step authorization? More on that later. First you need to make sure that you manage to pull out ads to inspect them, otherwise it does not make all the sense.

Is theoretical possible in practice?


Google logged on - it's time to test the theory of solving communication protocols in practice. I had to tinker with the experiments and observations, carefully observe and record what user actions lead to which requests, identify common and changing elements of the request, match the long incomprehensible values ​​received from the server and the same long sent back in the next request. It was a dense forest, which eventually became clearer and more transparent.

What had to be done to understand that continuation makes sense?

  1. Sign in to cpo .
  2. Get a list of ads.
  3. Get a specific ad (for a start text).

The entrance to the center is the simplest, roughly speaking, just follow the link. Happened.

Details
Мы просто как бы переходим по ссылке, получаем ответ (который в данном случае не используем). Ещё нам нужно запросить и сохранить цифровой жетон для дальнейших запросов.

В AdSense на момент написания статьи есть два ЦПО. Назову их условно старый и новый.

Для старого ЦПО.

Post-запрос «без нагрузки»:

https://www.google.com/adsense/gwt-properties?pid=pub-8958890276790964&authuser=0&tpid=pub-8958890276790964&ov=3&hl=en

Ответ:

<metaname="gwt:property"content="usePropertyService=true"><metaname="gwt:property"content="applicationType=ASFE3"><metaname="gwt:property"content="syn.token=ABOvogJ1yQyL9pgHcGYM-J3OLj_9VSh31w:1535115071772"><metaname="gwt:property"content="syn.token.pb=ABOvogKJ6-xmsNWK4Mbe_H5bT1xXhyj8SQ:1535115071772"><metaname="gwt:property"content="syn.login=XXXXXX@gmail.com"><metaname="gwt:property"content="syn.csi.backendUrl="><metaname="gwt:property"content="syn.helpCenterUrl=//support.google.com/adsense/"><metaname="gwt:property"content="syn.helpHost=//support.google.com"><metaname="gwt:property"content="syn.helpCenterUri=/adsense"><metaname="gwt:property"content="syn.newHelpHost=https://clients6.google.com"><metaname="gwt:property"content="syn.newHelpCenterUri=/adsense"><metaname="gwt:property"content="syn.helpCenterGaiaAuthDisabled=false"><metaname="gwt:property"content="syn.billing3BaseUri=https://bpui0.google.com"><metaname="gwt:property"content="syn.contextPath=/adsense"><metaname="gwt:property"content="syn.userLanguage=en-US"><metaname="gwt:property"content="syn.bruschettaContextPath=/adsense/new"><metaname="gwt:property"content="userProfileImageUrl=https://lh5.googleusercontent.com/-v7nuoAI4eEQ/AAAAAAAAAAI/AAAAAAAAAAA/AT3-yjmKyg8/s96/photo.jpg"><metaname="gwt:property"content="userDisplayName="Имя Фамилия"><metaname="gwt:property"content="userSettingsUrl=https://www.google.com/settings"><metaname="gwt:property"content="googlePlusProfileUrl=https://plus.google.com/me"><metaname="gwt:property"content="googlePrivacyUrl=http://www.google.com/intl/en_US/policies/privacy/"><metaname="gwt:property"content="syn.features=562,465,612,604,616,618"><metaname="gwt:property"content="analyticsHomePageUrl=https://www.google.com/analytics/web/"><metaname="gwt:property"content="disableDebugIds=true"><metaname="gwt:property"content="syn.pubControlsCapabilitiesLoadTimeout=5000"><metaname

Also popular now: