October 4, 2017 at 14:45Feed Manager Development for Automated Traffic Purchasing
Mobile marketing involves the interaction of two or three key figures.
A client is an advertiser or mobile application developer who wants to buy good-quality traffic.
Publisher - arbitrator, group administrator in social. network, application developer with in-app advertising.
And also often the intermediary is a mobile marketing agency that selects offers from customers and optimizes partners' traffic.
Since Mobio is an agency, in the article we will consider how, from our point of view, traffic is handled.
2 types of work with traffic are mainly used:
Manual(manual) - each offer is manually created in the tracker, after which traffic sources are also manually connected to it, payments are configured. Changes are also controlled by the manager independently.
API-feed (automatic) - the automated collection of a large number (several thousand) of offers, their processing, loading into the system and connecting sources. Including the automatic introduction of all changes received from the advertiser in real time.
When we are in mobiojust starting out with API-feed, we decided to study the existing solutions on the market. The task was to find something that can be adapted to the needs of the company. But we faced a number of difficulties that impeded the team’s effectiveness. Therefore, it was decided to go their own way, creating their own system of working with API-feed (feed-management).
Axonite is one of the largest players in the API feed market. It was originally imprisoned under the HasOffers tracker. Mobio successfully set up integration with Affise . The system has ready integration with most major advertisers, just insert the key and additional data for the desired client, after which the offers are automatically loaded into the system.
Choosing a connection in Axonite. The list is large, in the screenshot only a small part of it.
Then they are filtered by configurable parameters. We select which partners you want to connect to each of the flows (the flow is the selection of offers from one / several advertisers according to the parameters specified in the filter). The system exports data to the tracker and maintains its relevance.
But after the launch, we encountered many difficulties:
1. Inconvenient stream formation
Each stream should be separate for each advertiser. To connect a partner, you also need to create a separate “Stream-source” connection. In practice, this turns out to be necessary to create (manually!) 40 connections to connect 20 sources to 2 streams. It takes a lot of time. In addition, the cost of the tariff plan limits the number of such connections. This makes it time-consuming to remove connections when there are too many of them. Or pay more money.
Create a stream-source connection. Enter all the data, save, open a new form - repeat N times
2. Limit requests from the IP address
Under load, the server cannot cope with the prompt sending of information. Axonite employees explained this with the limits of the tracker, but later, when developing our system, we were convinced that this was not entirely true.
Of course, we expanded the instance, but this was not done as quickly as required for a comfortable expansion of the business.
3. Outdated connections
Many connections are very outdated. The large list of clients to which the connection was configured was not as relevant as we expected. Often I had to wait until the connection was reconfigured, because Axonite itself does not update the information until requested by the user.
4. Inability to make a copy of the connection as a separate category
For example, an advertiser Advertiser provides 2 lists of offers: one without complex KPIs, and the second - highly sensitive offers with difficult KPIs. The first can be connected to all sources, the second - only to the best. However, Axonite will mix all the offers in one stream, which spoils the whole picture.
As a result, after working with Axonite for several months, we realized that it would be easier and more useful to make our own system. For all the time with the work of Axonite, we have accumulated enough cases with nuances that we could focus on during development.
CPAPI is an Api integration solution implemented by Affise. We tested them before Axonite, but we did not take them into permanent work. At that moment, when we were just starting our activity in setting up API integrations, the system was pretty damp. There were few ready-made connections, we could not influence the queue of their creation. CPAPI had its own priority list. Now the list has expanded, but there are still some flaws:
1. The list is small, many large customers are not, especially Chinese.
List of available connections
2. Partner connection is not automated.
After sending offers to the tracker, you must manually connect partners to offers. Given the volume of sources that appear over time, this is extremely suboptimal. Especially when it comes to a system designed to get rid of such troubles.
But you can choose which fields to synchronize constantly and which ones not. For example, if you want to change something, and not leave it “as is”.
In addition to specific disadvantages, there are also general “Wishlist”, not implemented by default in systems on the market. This is not a drawback of the system, but at the same time, the implementation of something like this will require considerable time, costs and coordination. For example, you cannot block individual sub-id traffic sources, and this is an integral part of the business. Or, analytics of the level of conversion of offers with disabling non-converting ones - for such an analysis, access to tracker statistics that API systems do not have is required.
As a result, we realized that outsourcing can be done only at the initial stage of work, and since our growth rate required the expansion of technical restrictions, we sat down to think over and develop our own system of API offers.
The following requirements were imposed on the system, which were included in the terms of reference for developers:
In our area, many advertisers use their own trackers. All have different APIs and data storage formats. Categories of offers (payment for installation or for an event) are transmitted in different ways. Ultimately, all this information must not only be collected, but also brought to a single view. Moreover, this view should adequately pass into our own tracker, given its capabilities.
It was decided to write a separate collector for GoLang for each client tracker. This made it possible to quickly collect and update raw data.
2. Matching fields
Further, the developer created a pool request for the system, helping her to understand which field in the collector corresponds to what. If necessary, simple calculations were added, for example, if the advertiser sends a monthly cap (the maximum number of installations per month), it had to be divided by 30, because In the Affise tracker there is only a daily cap (per day) and a total cap (for all time).
3. Checking the activity of the offer
The check was carried out in this way: if at the next request for data an offer that was already downloaded earlier was not found, it was transferred to the inactive status.
When the data was first loaded into the system, only active records from the collector were affected. Further, the system also compared the statuses of already loaded offers.
4. Offer filtering
After the first data download, filtering was performed according to the specified rules. Offers with an uninteresting GEO business, low payments, inappropriate currency, OS, type of device and offer, motive or non motive were removed (we work exclusively with unmotivated offers).
In the basic version, filtering was the only level of verification before the offer got into our tracker. However, this was not optimal - many offers came to us “broken”, i.e. the link didn’t lead where it needed to. Some offers had so many redirects on the way to the final link that working with them was completely uninteresting.
The solution came from our business colleagues. Mobrand gave us the opportunity to use our Offertest- This is a third-party system that checks that a click from the desired GEO and OS leads exactly where it is needed, and also reports how many redirects you had to go through. Working conditions with this system are much more favorable than with the popular Affilitest analogue, while the functionality is no less. Therefore, after filtering the offers, those that passed the filter were tested for Offertest. The basic rule for filtering is to match the final page declared in the offer. In addition, the maximum permissible number of redirects was individually set.
We are satisfied with the system and can recommend this product for use. It allows you to significantly improve the quality of the received sets of offers and costs several times cheaper than analogues. The stability of the work was personally tested by us at the maximum possible capacities - no drops were noticed. With all this, colleagues regularly improve their system, observing versioning, i.e. a fresh update will not break the old settings.
But that's not all! Even after all the checks, there were offers duplicating each other. For example, there were 5 offers of the conditional Yandex.Browser for Android and RU-geo, but with different payouts. Our system analyzed them, creating a top-group - a group of offers with a unique application identifier and GEO, after which we selected 2 with the highest payout. If one or both offers stopped, the next ones on the list came in their place.
All such checks are carried out in real time for all active or new offers in order to maximize the existing flow of offers.
Finally, all remaining offers are loaded via API into our Affise tracker. The work of traffic sources with offers takes place exclusively in it, leaving the feed system closed to a third-party user.
5. Work with traffic sources
In addition to processing and loading offers, it was also necessary to conveniently work with traffic sources, quickly connecting and disconnecting them from different advertisers. This was also done in the feed system interface. It was not necessary to form any flows, as it was in Axonite, the connection was made intuitively. In addition, it was possible to configure connections separately for each offer to enable the most detailed traffic flow settings.
In addition, a CR evaluation system (conversion ratio relative to clicks) was implemented. If it is too low (the offer does not convert traffic from a specific source), the source is disconnected from the specific offer in order to switch to those that will have a good CR and give a positive profit.
Plans to add integration with the FraudScore system . In order to block this source when it gets into the source of a random fraud. At the moment, this is done manually, but otherwise the level of manual work is as low as possible, especially in comparison with those solutions that are available on the market.
Difficulties
1. Unification
As already mentioned earlier, all the data received in the collectors was necessary to bring to a general view. Sometimes the response format from the advertiser was difficult to understand and unify. For example, the FuseClick tracker considers CPI offers a subspecies of CPA - which is a gross error from the point of view of our system. Or sometimes there is no preview-link in the response - you had to take package_name (application identifier) and create a link to the application store yourself.
Such aspects often arise when writing a collector, so it was extremely important both to understand what is meant in one field or another and to maintain constant contact with the advertiser during development to clarify unclear points.
2. Limits on the number of queries
Like any system, the Affise tracker has a limit of requests that can be sent from IP. In this regard, the speed of the feed system is limited by this limit - we had to take this factor into account. With current capacities, it would be possible to achieve greater speed, but it is worth saying thanks to colleagues who increased the limit for the system by 2 times. At the moment, the average update speed is 10 minutes for the entire stream, which is a very good market indicator.
In the process of constant refinement of different systems, you often have to think over and rewrite those modules that were responsible for the logic of work - as it can repeatedly change. It all depends on the developer - our employee perfectly thought out the modularity and extensibility of the components. It is important not to sacrifice this for the sake of low development time - because if you make a monolithic system, the improvements will take a lot of time (I dare to assume that the solutions presented on the market sin exactly this - because our thoughts of adding the module really needed by the customers often ran into failure due to “technical difficulties”)
4. Tracker limitations
Another problem is the limited capabilities of the tracker into which offers are exported. Despite the fact that Affise has no special problems with this, some of the things that are familiar in the feed market were initially absent. For example, the tracker does not have a separate field for package_name, and its transmission is critical for sources. Of course, you can come up with a “crutch” using another text field for this parameter, which is present in the tracker, but not really needed for feeds - but it would be nice to do everything beautifully. Well, we are sure that our colleagues will not disappoint and will remove those restrictions that are currently present
Conclusion
In the process of constant refinement of different systems, you often have to think over and rewrite those modules that were responsible for the logic of work, since it can repeatedly change.
It all depends on the developer. Our employee has well thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time, because if you make a monolithic system, then improvements will take a lot of time.
Despite all the difficulties that arose in the development of their own solutions, the team never regretted that they had decided to spend their own resources and resources on this project. The time that was ultimately saved, and the unique advantages that we received, repeatedly paid for all development costs. Perhaps we will think about preparing a public version of the product so that anyone can appreciate our work.
In the meantime, if you want to get excellent traffic from trusted sources, or you yourself have a good inventory of traffic and are looking for a suitable set of offers for it, we will be glad to talk about potential cooperation.
Subscribe and read interesting news, analytics and other insights about mobile marketing on the Mobio blog .
A client is an advertiser or mobile application developer who wants to buy good-quality traffic.
Publisher - arbitrator, group administrator in social. network, application developer with in-app advertising.
And also often the intermediary is a mobile marketing agency that selects offers from customers and optimizes partners' traffic.
Since Mobio is an agency, in the article we will consider how, from our point of view, traffic is handled.
2 types of work with traffic are mainly used:
Manual(manual) - each offer is manually created in the tracker, after which traffic sources are also manually connected to it, payments are configured. Changes are also controlled by the manager independently.
API-feed (automatic) - the automated collection of a large number (several thousand) of offers, their processing, loading into the system and connecting sources. Including the automatic introduction of all changes received from the advertiser in real time.
When we are in mobiojust starting out with API-feed, we decided to study the existing solutions on the market. The task was to find something that can be adapted to the needs of the company. But we faced a number of difficulties that impeded the team’s effectiveness. Therefore, it was decided to go their own way, creating their own system of working with API-feed (feed-management).
Third-Party Market Solutions
Axonite
Axonite is one of the largest players in the API feed market. It was originally imprisoned under the HasOffers tracker. Mobio successfully set up integration with Affise . The system has ready integration with most major advertisers, just insert the key and additional data for the desired client, after which the offers are automatically loaded into the system.
Choosing a connection in Axonite. The list is large, in the screenshot only a small part of it.
Then they are filtered by configurable parameters. We select which partners you want to connect to each of the flows (the flow is the selection of offers from one / several advertisers according to the parameters specified in the filter). The system exports data to the tracker and maintains its relevance.
But after the launch, we encountered many difficulties:
1. Inconvenient stream formation
Each stream should be separate for each advertiser. To connect a partner, you also need to create a separate “Stream-source” connection. In practice, this turns out to be necessary to create (manually!) 40 connections to connect 20 sources to 2 streams. It takes a lot of time. In addition, the cost of the tariff plan limits the number of such connections. This makes it time-consuming to remove connections when there are too many of them. Or pay more money.
Create a stream-source connection. Enter all the data, save, open a new form - repeat N times
2. Limit requests from the IP address
Under load, the server cannot cope with the prompt sending of information. Axonite employees explained this with the limits of the tracker, but later, when developing our system, we were convinced that this was not entirely true.
Of course, we expanded the instance, but this was not done as quickly as required for a comfortable expansion of the business.
3. Outdated connections
Many connections are very outdated. The large list of clients to which the connection was configured was not as relevant as we expected. Often I had to wait until the connection was reconfigured, because Axonite itself does not update the information until requested by the user.
4. Inability to make a copy of the connection as a separate category
For example, an advertiser Advertiser provides 2 lists of offers: one without complex KPIs, and the second - highly sensitive offers with difficult KPIs. The first can be connected to all sources, the second - only to the best. However, Axonite will mix all the offers in one stream, which spoils the whole picture.
As a result, after working with Axonite for several months, we realized that it would be easier and more useful to make our own system. For all the time with the work of Axonite, we have accumulated enough cases with nuances that we could focus on during development.
CPAPI
CPAPI is an Api integration solution implemented by Affise. We tested them before Axonite, but we did not take them into permanent work. At that moment, when we were just starting our activity in setting up API integrations, the system was pretty damp. There were few ready-made connections, we could not influence the queue of their creation. CPAPI had its own priority list. Now the list has expanded, but there are still some flaws:
1. The list is small, many large customers are not, especially Chinese.
List of available connections
2. Partner connection is not automated.
After sending offers to the tracker, you must manually connect partners to offers. Given the volume of sources that appear over time, this is extremely suboptimal. Especially when it comes to a system designed to get rid of such troubles.
But you can choose which fields to synchronize constantly and which ones not. For example, if you want to change something, and not leave it “as is”.
In addition to specific disadvantages, there are also general “Wishlist”, not implemented by default in systems on the market. This is not a drawback of the system, but at the same time, the implementation of something like this will require considerable time, costs and coordination. For example, you cannot block individual sub-id traffic sources, and this is an integral part of the business. Or, analytics of the level of conversion of offers with disabling non-converting ones - for such an analysis, access to tracker statistics that API systems do not have is required.
As a result, we realized that outsourcing can be done only at the initial stage of work, and since our growth rate required the expansion of technical restrictions, we sat down to think over and develop our own system of API offers.
Building your own solution: the general scheme of work and additional chips
The following requirements were imposed on the system, which were included in the terms of reference for developers:
1. The uniformity of information
In our area, many advertisers use their own trackers. All have different APIs and data storage formats. Categories of offers (payment for installation or for an event) are transmitted in different ways. Ultimately, all this information must not only be collected, but also brought to a single view. Moreover, this view should adequately pass into our own tracker, given its capabilities.
It was decided to write a separate collector for GoLang for each client tracker. This made it possible to quickly collect and update raw data.
2. Matching fields
Further, the developer created a pool request for the system, helping her to understand which field in the collector corresponds to what. If necessary, simple calculations were added, for example, if the advertiser sends a monthly cap (the maximum number of installations per month), it had to be divided by 30, because In the Affise tracker there is only a daily cap (per day) and a total cap (for all time).
3. Checking the activity of the offer
The check was carried out in this way: if at the next request for data an offer that was already downloaded earlier was not found, it was transferred to the inactive status.
When the data was first loaded into the system, only active records from the collector were affected. Further, the system also compared the statuses of already loaded offers.
4. Offer filtering
After the first data download, filtering was performed according to the specified rules. Offers with an uninteresting GEO business, low payments, inappropriate currency, OS, type of device and offer, motive or non motive were removed (we work exclusively with unmotivated offers).
In the basic version, filtering was the only level of verification before the offer got into our tracker. However, this was not optimal - many offers came to us “broken”, i.e. the link didn’t lead where it needed to. Some offers had so many redirects on the way to the final link that working with them was completely uninteresting.
The solution came from our business colleagues. Mobrand gave us the opportunity to use our Offertest- This is a third-party system that checks that a click from the desired GEO and OS leads exactly where it is needed, and also reports how many redirects you had to go through. Working conditions with this system are much more favorable than with the popular Affilitest analogue, while the functionality is no less. Therefore, after filtering the offers, those that passed the filter were tested for Offertest. The basic rule for filtering is to match the final page declared in the offer. In addition, the maximum permissible number of redirects was individually set.
We are satisfied with the system and can recommend this product for use. It allows you to significantly improve the quality of the received sets of offers and costs several times cheaper than analogues. The stability of the work was personally tested by us at the maximum possible capacities - no drops were noticed. With all this, colleagues regularly improve their system, observing versioning, i.e. a fresh update will not break the old settings.
But that's not all! Even after all the checks, there were offers duplicating each other. For example, there were 5 offers of the conditional Yandex.Browser for Android and RU-geo, but with different payouts. Our system analyzed them, creating a top-group - a group of offers with a unique application identifier and GEO, after which we selected 2 with the highest payout. If one or both offers stopped, the next ones on the list came in their place.
All such checks are carried out in real time for all active or new offers in order to maximize the existing flow of offers.
Finally, all remaining offers are loaded via API into our Affise tracker. The work of traffic sources with offers takes place exclusively in it, leaving the feed system closed to a third-party user.
5. Work with traffic sources
In addition to processing and loading offers, it was also necessary to conveniently work with traffic sources, quickly connecting and disconnecting them from different advertisers. This was also done in the feed system interface. It was not necessary to form any flows, as it was in Axonite, the connection was made intuitively. In addition, it was possible to configure connections separately for each offer to enable the most detailed traffic flow settings.
In addition, a CR evaluation system (conversion ratio relative to clicks) was implemented. If it is too low (the offer does not convert traffic from a specific source), the source is disconnected from the specific offer in order to switch to those that will have a good CR and give a positive profit.
Plans to add integration with the FraudScore system . In order to block this source when it gets into the source of a random fraud. At the moment, this is done manually, but otherwise the level of manual work is as low as possible, especially in comparison with those solutions that are available on the market.
Difficulties
1. Unification
As already mentioned earlier, all the data received in the collectors was necessary to bring to a general view. Sometimes the response format from the advertiser was difficult to understand and unify. For example, the FuseClick tracker considers CPI offers a subspecies of CPA - which is a gross error from the point of view of our system. Or sometimes there is no preview-link in the response - you had to take package_name (application identifier) and create a link to the application store yourself.
Such aspects often arise when writing a collector, so it was extremely important both to understand what is meant in one field or another and to maintain constant contact with the advertiser during development to clarify unclear points.
2. Limits on the number of queries
Like any system, the Affise tracker has a limit of requests that can be sent from IP. In this regard, the speed of the feed system is limited by this limit - we had to take this factor into account. With current capacities, it would be possible to achieve greater speed, but it is worth saying thanks to colleagues who increased the limit for the system by 2 times. At the moment, the average update speed is 10 minutes for the entire stream, which is a very good market indicator.
3. Change of logic
In the process of constant refinement of different systems, you often have to think over and rewrite those modules that were responsible for the logic of work - as it can repeatedly change. It all depends on the developer - our employee perfectly thought out the modularity and extensibility of the components. It is important not to sacrifice this for the sake of low development time - because if you make a monolithic system, the improvements will take a lot of time (I dare to assume that the solutions presented on the market sin exactly this - because our thoughts of adding the module really needed by the customers often ran into failure due to “technical difficulties”)
4. Tracker limitations
Another problem is the limited capabilities of the tracker into which offers are exported. Despite the fact that Affise has no special problems with this, some of the things that are familiar in the feed market were initially absent. For example, the tracker does not have a separate field for package_name, and its transmission is critical for sources. Of course, you can come up with a “crutch” using another text field for this parameter, which is present in the tracker, but not really needed for feeds - but it would be nice to do everything beautifully. Well, we are sure that our colleagues will not disappoint and will remove those restrictions that are currently present
Conclusion
In the process of constant refinement of different systems, you often have to think over and rewrite those modules that were responsible for the logic of work, since it can repeatedly change.
It all depends on the developer. Our employee has well thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time, because if you make a monolithic system, then improvements will take a lot of time.
Despite all the difficulties that arose in the development of their own solutions, the team never regretted that they had decided to spend their own resources and resources on this project. The time that was ultimately saved, and the unique advantages that we received, repeatedly paid for all development costs. Perhaps we will think about preparing a public version of the product so that anyone can appreciate our work.
In the meantime, if you want to get excellent traffic from trusted sources, or you yourself have a good inventory of traffic and are looking for a suitable set of offers for it, we will be glad to talk about potential cooperation.
Subscribe and read interesting news, analytics and other insights about mobile marketing on the Mobio blog .