We make our firmware for the IP camera on Rust and fight with fraud
This is a continuation of the story about the best ( according to the participants ) reports of HighLoad ++ 2017. In the previous article I wrote about the two leaders of the rating. Move on. In this article - a report by Vadim Antonyuk about frauds and Max Lapshin - about firmware for an IP camera on Rust.
Honorable third place. This is not the first time Vadim talks about antifraud in Highload ++. A year earlier, he talked about antifraud in RTB systems in general, this time "went" to mobile applications. Looking ahead, I’ll say that there is still debate about what to consider fraud in mobile applications, unlike the web, where everything is more or less clear. Nevertheless, the segment itself is growing, and with it the problem. So, it's time to start solving it - "not yet begun."
Meanwhile, if you don’t know or still doubt that RTB is a highlod, how do you like the figure of 400 billion requests per day? Moreover, according to statistics, mobile applications account for about 40% - more than desktop web. IPONWEB (one of the largest players in the RTB advertising market) has been detecting fraud in the web segment for a long time and successfully, but it’s just about to get to the applications.
So. Classical methods (bots, AdStackng, Domains Spoofing, Ghost sites), which have long been used on the classical web, do not work explicitly in mobile applications. As a result, methods of dealing with them are not suitable, and something new needs to be invented.
By the way, the industry has not yet agreed on what to consider fraud in mobile applications. It would seem that stories with advertising in cell phones are already decent, but the fact remains, and Vadim speaks about this.
The report discusses two (and actually almost three) fraud detection approaches:
If this topic is of interest, then you can look through the links under the spoiler or just google the terms that Vadim mentions. The only thing that is a little annoying is the lion's share of information on this topic is semi-advertising articles in exchange for mail from teams that are engaged in both advertising and fighting fraud.
Max is a regular participant in IT events, a member of the Highload ++ program committee, the author of very fast and productive video streaming solutions that are used all over the planet, an erlang expert and just a great speaker. His story was not included in the top three, but you must definitely listen to it, if only because his company, one of the few in Russia, is successfully promoting its software product on the world market.
Max spoke directly about streaming and its nuances at past conferences. This time it was about how to change software on IP cameras, as Rust appeared in this story, and what came of it. The report has two parts: the first is directly about the specifics of working with IP cameras, the second is devoted to Rust and its features.
Despite the fact that IP cameras are widespread, cheap, and the hardware platform is becoming more reliable every year, the software for IP cameras leaves much to be desired. That is, along with the camera, you get 10-12 year old software with all the defects, lack of features, and most importantly - the inability to make changes. As a result, it is most reasonable to write your own software in order to get a set of necessary features and reduce the complexity of working with the device. By the way, there was also an option to make your own camera - but it was abandoned due to the high cost and duration, as well as the need to work with existing cameras, of which there are millions.
In fact, the IP camera is a computer with a processor, memory and other nodes, which means that you can work with it, for example, by analogy with mobile phones. Just like in mobile phones, there are a bunch of options for executing internal software: somewhere you can write something, somewhere you just read, there is a different U-boot (a powerful bootloader with a bunch of functions you need to fill in) with different software For dubbing, there is a large set of various cables, tongs and soldering irons. Well, and as a result, all the charms of flashing your favorite android come out: the power jumped - the camera turned into a brick, the firmware was wrong - again a brick. Fortunately, the cost of such a "brick" at the moment is around ten dollars apiece.
To make life easier for you, you can, as soon as you manage to get the documentation for the camera and crack it, put software on it that allows you to make further updates in a more familiar way.
After assembly, you can safely proceed to the firmware. It turns out that what you used to think of as a “zoo” of technology is babble compared to when it comes to cameras. Different versions of the kernel, libraries, SDKs are the absolute norm for them. The norm should be considered the complete lack of the ability to run tests without uploading to the camera, a severe lack of space (8 megabytes, yeah), as well as constant work with hardware and precise memory management. To summarize - the camera is a very interesting and at the same time cheap device (you can safely experiment and turn cameras into bricks), with which you can interact very well, reaching your goals.
The second part of the story is entirely devoted to Rust and its features. An attentive listener will definitely notice for himself the reasons why Rust was chosen, and not the good old S.
The next two slides show the results that were achieved using this not-so-common language for working with iron.
I won’t dwell in detail, I’ll only note that Max very clearly explains to hype lovers that you have to pay for everything, and Rust is no exception. In conclusion, it is noted that, despite the external complexity, it is possible to use Rust for semi-embedded systems, and it is fresh and fun, although it will have to reformat your brain a little.
On my own, I note that many of the things Max talks about are characteristic of the development of most embedded systems. If you are aiming for this direction, be sure to listen.
Soon you will have an overview of four more reports. If you yourself are ready to share experience or interesting cases in the field of hi-tech, management and team leadership, programming or mobile development, we invite you to become the speaker of the corresponding section of RIT ++ 2018. The acceptance of applications can be left in the program committee, which can be left here . Perhaps your report will be the best this year.
Fraud in mobile applications: how to define and detect, Vadim Antonyuk (4.73 points)
Honorable third place. This is not the first time Vadim talks about antifraud in Highload ++. A year earlier, he talked about antifraud in RTB systems in general, this time "went" to mobile applications. Looking ahead, I’ll say that there is still debate about what to consider fraud in mobile applications, unlike the web, where everything is more or less clear. Nevertheless, the segment itself is growing, and with it the problem. So, it's time to start solving it - "not yet begun."
Meanwhile, if you don’t know or still doubt that RTB is a highlod, how do you like the figure of 400 billion requests per day? Moreover, according to statistics, mobile applications account for about 40% - more than desktop web. IPONWEB (one of the largest players in the RTB advertising market) has been detecting fraud in the web segment for a long time and successfully, but it’s just about to get to the applications.
So. Classical methods (bots, AdStackng, Domains Spoofing, Ghost sites), which have long been used on the classical web, do not work explicitly in mobile applications. As a result, methods of dealing with them are not suitable, and something new needs to be invented.
By the way, the industry has not yet agreed on what to consider fraud in mobile applications. It would seem that stories with advertising in cell phones are already decent, but the fact remains, and Vadim speaks about this.
The report discusses two (and actually almost three) fraud detection approaches:
- Backgroundness - display ads in the background. The essence of this type of fraud is in its name. That is, there is advertising, but the user does not see it. How to look for her? Everything is not very complicated, but there are nuances. The essence of the original method is to search all incoming requests from the application + user pair in the incoming request stream, which send requests continuously. Such intervals are calculated and a realistic analysis is done.
For example, on the slide below, the stream of T1 signals with an interval between t1 signals can be quite long, 10 hours, which almost certainly indicates that it is a fraud. After all, the user can not look at the screen for 10 hours without a break. If you collect statistics on this application, you can with high probability calculate the villains. But, as usual, there is a nuance - the method depends on at least two parameters, and both of them are calculated empirically.
Can I make it easier? It turns out that you can only rely on the value of T (the period of continuous emission of the signal) and analyze the intervals between adjacent time intervals (in our example, the desired interval between T1 and T2. If there is no realistic interval during the day (say, 6 hours), then, most likely something went wrong.As a result, with a large number of measurements from one application, the accuracy of the method increases. - The second method is Entropy Score. Search for villains through abnormal entropy. To make it clearer, I recommend pre- reading the wiki . The method is based on a sequential numerical analysis of user and application behavior. The Entropy Score is calculated, due to the large number of measurements, the admissible interval of the value of this parameter is determined - when the behavior of the application and the user is real with a high degree of probability - and all other applications are analyzed.
A slightly more complicated method is used in IPONWEB as an addition to the first. It allows you to catch applications that have a small number of users, even though there is no continuous stream of requests. As an example, applications for artificial viewing of advertising are given. - The third method is not explicitly highlighted. It analyzes a large number of requests from a single device. In fact, this is an analogue of the bot on the web. So if you go back to the beginning of the conversation, then Vadim was a little disingenuous when he said that fraud analysis methods on the web are not suitable.
As a result, through the use of a combination of fraud identification methods, the guys from IPONWEB filter out more than 15% of the traffic. A very solid indicator, taking into account the total number of requests. All methods are probabilistic, and there is always the probability of error.
A dialogue on this topic is constantly conducted when good applications are cut as a result of the analysis, and therefore the system is constantly tuned and adjusted. However, one must understand that bad traffic still remains. Returning to the question of the application of certain methods in the web and mobile applications, Vadim noted that in a number of cases nothing prevents combining them and getting good results.
If this topic is of interest, then you can look through the links under the spoiler or just google the terms that Vadim mentions. The only thing that is a little annoying is the lion's share of information on this topic is semi-advertising articles in exchange for mail from teams that are engaged in both advertising and fighting fraud.
Fraud Links
https://www.youtube.com/watch?v=CIfhrpN5DF4 - report with Highload ++ 2016 about fraud on the web.
https://en.wikipedia.org/wiki/Click_farm
http://blog.ezanga.com/blog/ad-fraud-101-types-of-ad-fraud-that-plague-all-marketers
https: // www.kochava.com/detecting-fraud-counting-clicks/ (in the comments to the articles there are links to the following articles in the series)
https://performancein.com/news/2017/02/17/seven-things-you-should -know-about-app-fraud /
http://telecoms.com/opinion/mobile-app-fraud-how-to-fight-back-to-protect-against-risk/
https://en.wikipedia.org/wiki/Click_farm
http://blog.ezanga.com/blog/ad-fraud-101-types-of-ad-fraud-that-plague-all-marketers
https: // www.kochava.com/detecting-fraud-counting-clicks/ (in the comments to the articles there are links to the following articles in the series)
https://performancein.com/news/2017/02/17/seven-things-you-should -know-about-app-fraud /
http://telecoms.com/opinion/mobile-app-fraud-how-to-fight-back-to-protect-against-risk/
We make our firmware for the IP camera on Rust, Max Lapshin (4.70 points)
Max is a regular participant in IT events, a member of the Highload ++ program committee, the author of very fast and productive video streaming solutions that are used all over the planet, an erlang expert and just a great speaker. His story was not included in the top three, but you must definitely listen to it, if only because his company, one of the few in Russia, is successfully promoting its software product on the world market.
Max spoke directly about streaming and its nuances at past conferences. This time it was about how to change software on IP cameras, as Rust appeared in this story, and what came of it. The report has two parts: the first is directly about the specifics of working with IP cameras, the second is devoted to Rust and its features.
Despite the fact that IP cameras are widespread, cheap, and the hardware platform is becoming more reliable every year, the software for IP cameras leaves much to be desired. That is, along with the camera, you get 10-12 year old software with all the defects, lack of features, and most importantly - the inability to make changes. As a result, it is most reasonable to write your own software in order to get a set of necessary features and reduce the complexity of working with the device. By the way, there was also an option to make your own camera - but it was abandoned due to the high cost and duration, as well as the need to work with existing cameras, of which there are millions.
In fact, the IP camera is a computer with a processor, memory and other nodes, which means that you can work with it, for example, by analogy with mobile phones. Just like in mobile phones, there are a bunch of options for executing internal software: somewhere you can write something, somewhere you just read, there is a different U-boot (a powerful bootloader with a bunch of functions you need to fill in) with different software For dubbing, there is a large set of various cables, tongs and soldering irons. Well, and as a result, all the charms of flashing your favorite android come out: the power jumped - the camera turned into a brick, the firmware was wrong - again a brick. Fortunately, the cost of such a "brick" at the moment is around ten dollars apiece.
To make life easier for you, you can, as soon as you manage to get the documentation for the camera and crack it, put software on it that allows you to make further updates in a more familiar way.
After assembly, you can safely proceed to the firmware. It turns out that what you used to think of as a “zoo” of technology is babble compared to when it comes to cameras. Different versions of the kernel, libraries, SDKs are the absolute norm for them. The norm should be considered the complete lack of the ability to run tests without uploading to the camera, a severe lack of space (8 megabytes, yeah), as well as constant work with hardware and precise memory management. To summarize - the camera is a very interesting and at the same time cheap device (you can safely experiment and turn cameras into bricks), with which you can interact very well, reaching your goals.
The second part of the story is entirely devoted to Rust and its features. An attentive listener will definitely notice for himself the reasons why Rust was chosen, and not the good old S.
The next two slides show the results that were achieved using this not-so-common language for working with iron.
I won’t dwell in detail, I’ll only note that Max very clearly explains to hype lovers that you have to pay for everything, and Rust is no exception. In conclusion, it is noted that, despite the external complexity, it is possible to use Rust for semi-embedded systems, and it is fresh and fun, although it will have to reformat your brain a little.
On my own, I note that many of the things Max talks about are characteristic of the development of most embedded systems. If you are aiming for this direction, be sure to listen.
Several Max management reports about his product and company
- https://www.youtube.com/watch?v=xKsj3Av1ITE - a story about how to make a small profitable company in the field of video streaming.
- https://www.youtube.com/watch?v=8qdo-HnXBJ8 - interview with Max at RIT 2017
- https://www.youtube.com/watch?v=Q29TKg_3xvo - how can a programmer grow his company.
Soon you will have an overview of four more reports. If you yourself are ready to share experience or interesting cases in the field of hi-tech, management and team leadership, programming or mobile development, we invite you to become the speaker of the corresponding section of RIT ++ 2018. The acceptance of applications can be left in the program committee, which can be left here . Perhaps your report will be the best this year.