Dreams of a perfect API or how to overcome difficulties in the AdHands project

    Users are accustomed to the principle of one device and one interface. This trend is also reflected in business: multifunctional corporate systems and portals, common trading and procurement platforms, etc. Online advertising was not left behind - services are being developed aimed at significantly optimizing the work of advertisers and advertising agencies. And one of the serious technical aspects when creating dashboards and aggregators of statistics remains working with the API - a software interface through which data is exchanged between systems. Is there a dream API, how to create it and why is it needed?



    Advertising on the Internet passes through several channels at once and is placed on many platforms to reach the largest possible audience. Campaign management is carried out through the interfaces of various advertising systems (Yandex.Direct, Google AdWords, Vkontakte placement system, etc.). Each time to view statistics you need to log in, switch, and in case you need to compare statistics of different sites, you also have to aggregate data in Excel tables or manually import data into Google Analytics.

    Realweb like no one knows about the need for convenient and efficient work with online advertising data: we run hundreds of customers with tens of thousands of campaigns, so the visual presentation of statistics is a critical factor for us. For this we use a system of collecting advertising statisticsAdhands . This is a kind of dashboard: information on all advertising campaigns is aggregated in a single interface. This approach provides convenience, broad analytical capabilities and an accurate understanding of which advertising is effective and which is not.

    In the case of advertising on the Internet to assess the "exhaust" from it is extremely difficult. In the courtyard of the XXI century, gentlemen! And for measuring the performance of advertising on the Internet, there are interesting functional tools that are ready to work through the API with all services.

    AdHands - Long Online Advertising Hands


    AdHands is one such solution. This is a system that connects to online advertising sites (Yandex.Direct, Yandex.Market, Google Adwords, Vkontakte, Target@Mail.ru, DoubleClick Bid Manager, Yandex.Auction, etc.) through the API.



    We thought for a long time what a unified system of managing advertising campaigns on the Internet should be. As a result, it was decided to develop a system that can overcome several problems at once. The first and main feature of AdHands was a single interface for accessing all statistics collected - this solution allowed us to abandon the need to log in every time, to work in different GUIs. The result was a single convenient panel, dashboard, on which all the necessary indicators of advertising are available.

    Of course, not a single statistic has practical value if it is not aggregated and properly processed. The system has been added the ability to automatically calculate the cost of the target, order, call, ROI calculation. Media planning was introduced - a valuable function for those who manage other people's budgets or have a strict business planning system in the company, within which it is necessary to calculate marketing and advertising budgets in advance.

    AdHands works for the end user, making his life a lot easier. However, when creating the system itself, there are a number of difficulties associated with the fact that you have to interact with various external systems.

    Work with API - build bridges


    Everyone knows how complicated the world of programming and development is - despite the creation of constant descriptions of standards and attempts to agree, each developer brings something different to the code. In the case of the API, this sometimes puzzles developers of external systems, for example, AdHands. We will tell you more about what we encountered using other people's APIs when creating our online advertising management system.

    Working with third-party systems, we have developed a number of criteria by which we determine how convenient the system that we connect to AdHands is. Broadly speaking, such a system should be primarily convenient from the point of view of the developer. And if you consider in particular, it is important that the connected system meets several parameters.

    • The stability of the system is its availability at 99.9% of the time in a year. It is probably unnecessary to explain how important this is for a system that collects and transmits data from online advertising campaigns that are broadcast every second across the Internet.
    • Having documentation is an absolutely critical requirement. Understanding the API without documentation is extremely difficult, especially if you have to handle errors and emergency situations. You need to understand that the API is actually someone else’s code, and looking at the desire of the largest online platforms for universality, a happy future is still far away and the API must be accompanied by detailed developer documentation, for example, as implemented by Google AdWords and Yandex.Direct.
    • Authorization support OAuth 2.0 , a convenient and secure open authorization protocol.
    • Responsive technical support of the system, of course, cannot be a requirement, but this wish - unfortunately, it is not always possible to quickly get a qualified answer. And solving problems in systems like AdHands requires maximum efficiency in order to avoid loss of customer loyalty.
    • Understandability - API methods and functionality should correspond in semantics to what they perform. For example, the “add user” method should add a user without solving a number of tasks.

    Of course, ideally, I would like all connected systems to have a uniform universal API and one code could be used for all systems. Certain steps have already been taken in this direction, but it is so far from complete unification that for now it seems impossible.

    Working with the API - let's talk about the problems


    Discrepancy in the terminology of the API sites - entities of different systems may lack or be called differently. For example, recently Yandex.Direct introduced the entity “ad group”, and for a long time only AdWords had such an entity. A similar situation arises, for example, with the term “creative”, which is used by BaYang and DBM, but in fact for the developer, the creative in these two systems are different entities. Bridging such discrepancies takes time and therefore you have to look for a solution using the special AdHands architecture. However, this problem is not as significant as the rest.

    Differences in data structure- a problem arising from differences in the structure of the sites themselves (again, for example, there may not be ad groups). This is an important issue for AdHands, because we do not copy system interfaces in one application, but collect data for presentation to the user. I don’t want to change the page structure every time, adapting to the systems. We circumvent the problem by bringing data from various sources to a single structure. In fact, we create hierarchies close in structure and hybrid solutions, when the entity remains available when choosing one system and closed to others. AdHands achieves uniformity and convenience of the interface by abstracting entities.

    All developers relate to documentation in different ways: someone thoroughly documents each variable, someone describes in general terms, and some completely refuse any documentation. When working with the API of an external system, the availability of detailed documentation is the key to success. However, even with detailed manuals and a community of developers, problems arise, among which the main one is the mismatch of documentation and the real situation: Sometimes it happens that the documentation, program interface and graphical user interface diverge. As a rule, this is due to the fact that the API changes and is updated quite quickly, and the documentation is updated more slowly, so you can find outdated functions in it or not find any of the new features. Ideally, if the release of a new version of the API occurs simultaneously with the release of the documentation.

    From the experience of Realweb, we can say that there is practically no documentation that does not correspond to reality. However, the human factor comes into play here, and sometimes due to the lack of notifications and newsletters about changes, it is not possible to find out right away.

    For each system, the API is periodically changed and refined.Constantly changing the API itself is a necessary phenomenon - the program interface is developing along with the system. It’s bad if changes come out, but let's say the method signature has not changed. It’s good if during the update of the API all the libraries were updated on a schedule.

    The divergence of API and interface is not a very common problem. Of course, the API provider understands that the entities must be the same for both the software interface and the user interface. It happens that the API does not provide methods for some GUI elements. If some entities are missing in the program interface, you have to create pseudo-entities - a bunch in order to fit into the universal structure of AdHands.

    It is extremely rare, but still it happens that the API is completely absent. For example, we encountered such a situation while working with the Yandex.MMS banner placement system. In such cases, you have to parse the interface using a crawler and log in as a user through the login / password link. Here the main stumbling block arises if the graphical user interface changes - the layout changes, the parser returns errors. But, basically, all modern systems provide access through the API. But those who are not ready to provide adequate libraries and documentation are at risk of remaining beyond the competition.

    The lack of a convenient and universal OAuth 2.0 authorization is a problem that continues to occur. As is known, the authorization protocol OAuth 2.0 is being finalized specificationsHowever, he is already quite ready to replace OAuth. This is a convenient protocol, which is distinguished by ease of use, the absence of a long chain of requests and complex signature schemes. Some platforms and advertising aggregators do not use it yet, apparently in anticipation of a stable specification. This greatly complicates the process of authorization and authentication when requesting data from external systems and transferring data. For example, Yandex's Bayan works on tokenization by login / password. From the developer's point of view, this is not entirely safe, since it can cause significant damage if the database in which the usernames and passwords for Bayan are stored is compromised.
    Now, most large systems have already switched to OAuth 2.0. Besides the fact that it is safe, it is still a universal standard and a guarantee of quick connection of the system to AdHands.

    The lack of a single universal data exchange protocol is another problem. If it existed, AdHands would simply forward the system data as a single control panel. However, if such a protocol is created, many companies will appear on the market that are ready to build a business for innovation, as is usually the case. And this can lead to the appearance of ill-conceived decisions that compromise serious systems in the eyes of the end user. In addition, the creation of a single protocol is not very beneficial for large sites that are not ready and are not going to abandon their graphical interfaces.

    Another issue we can have little influence on is API limits.- A limited number of requests per day, which are set by advertising systems. Each method of the API has its own limit of calls, which is determined by the resource consumption and the "severity" of the method. In principle, Google AdWords and Yandex.Direct have such a large number of requests that it is difficult to feel the restriction, but there are also problematic services, for example, Yandex.Market, the use of which sometimes resembles trying to get a ticket to the right doctor. Venues often meet and expand limits, however, attention is also paid to the structure of aggregator systems themselves, which may not make optimal use of requests.

    Of course, this is not a complete list of problems and small obstacles that AdHands developers have to deal with every day. We are always looking for compromises and are not afraid to use solutions in the code that may not seem very elegant to someone. This is due to the large number of connecting systems, the diversity of their GUI and API. We write parsers for systems without APIs, create redundant entities, add pieces of code with checks, implement new functionality that is not built into AdHands. All these decisions can affect the manageability and convenience of system administration, and its performance. But at the same time, we are constantly optimizing the code, adapting to the system of limits and doing everything for the convenience of our users.

    Recently, there has been a tendency towards unification of the API of large sites. For example, the prototype Yandex.Metrica API in the Core reporting part copies the Google Analytics API. We hope that other API providers will not live their own lives, but will look at the best practices and follow not a fashionable, but a very useful trend. And with joint efforts, we will do better for those for whom everything was started, for end users, advertising agencies and advertisers. Most of them do not know about the API and its problems, they need fast, convenient and clear. Our business is to provide it.

    Also popular now: