Experience blocking online advertising in the company's local network
Hello, habrovchane! Having received from the management the task of blocking advertisements on Internet sites for employees, I decided to approach the process creatively: to figure out how this or that advertisement is technically broadcast, what methods of blocking exist, their pros and cons. I also cite as an example the implementation of blocking at the gateway level using Traffic Inspector Next Generation, as well as using the ad blocking option in Kaspersky Internet Security (hereafter KIS) and AdBlock and Adblock Plus browser applications for this purpose. The article will be interesting to sysadmins and other specialists working with local networks.

Why block ads, in my opinion, is clear: to increase the speed of loading pages and the consumption of traffic, to combat plugin-container memory consuming, and spyware scripts that monitor user actions. And most importantly, why management has raised this issue at all, is a distraction: advertising shifts the employee’s attention from the task being carried out to a tempting offer, often specially tailored for it by clever algorithms. By the way, every two years the number of computers with blocked advertising in the world triples .
First, in a nutshell, how can you block ads .
Firstly, there are applications for browsers that use dozens of different algorithms and schemes for each type of content and even for some separately taken platforms (Facebook, for example). The most popular cutters are AdBlock and Adblock Plus (hereinafter referred to as ADP). In my opinion, ADP is more effective of them, but sometimes useful content gets under its powerful skating rink (at the end). For me, the lack of all such solutions is that they are put on each computer and, moreover, on each browser. When you have hundreds of computers in LAN, it becomes a problem.
Secondly, many antivirus programs (for example, we use Kaspersky) have an in-built ad blocking option. The advantage of this method is that once you use an antivirus, you don’t need to add anything to your computer - just activate the function in the menu. But since this function is not the main one, it works worse than the applications that were specially sharpened for this business.
Third, you can cut advertising by the network gateway.. Blocking occurs by redirecting HTTP requests from advertising sites to the address 0.0.0.0. For me, a huge plus is that the lock works immediately for the entire local network. The main drawback - you need to monitor the relevance of the database of advertising addresses and, if necessary, add them manually. The second drawback is that only one blocking mechanism is used - via HTTP requests, and yet not all types of ads use them.
Looking ahead, I will say that in my LAN I use all three methods - they complement each other well. Details - at the end of the article, and now let's see what types of advertising are .
SEO and all sorts of emails there are not interesting to us now. We are interested in advertising, which is broadcast along with useful content. According to this principle, I divided it into the following types:
1. Embedded ad units of services such as the Yandex Advertising Network and Google.Adsense.
These are text, text-graphic modules and simply banners that are placed by the owners of the sites for earnings. In my opinion, this is the most harmful type of advertising, because it: a) often mimics useful content and mixes with it to increase clickability; b) shows the user what he is most likely to be interested in (we all know about big data, behavioral factors, etc., etc.); c) shows that the user (in our case, the employee) is not needed at the moment, that is, distracts from work.
Take a few resources with a similar type of advertising:
1.1. sharpologist.com/2018/08/whats-menthol-shaving-stuff.html The
Google banner was neatly cut out by KIS tools - the layout did not suffer, no voids were formed:

1.2. There are also blocks that use user requests, which he entered earlier in a search engine (for example, on zmoe.ru ):

1.3. www.bestfree.ru is a great example of a re-optimized SEO site, expectedly filled to capacity with advertising. Red boxes are marked with ad units (see picture on the left). All of them were blocked (see right):



1.4. A bad example of ad blocking is Mail.ru mail. On the inbox page, the Yandex Advertising System manifests itself twice: in the information bar above the list of letters and the ad unit on the left. Not a single insertion was blocked by any of the methods used by me:

2. Banner exchange, teaser, RTB-networks like Rotaban, advmaker.net, Marketgid, etc.
They can’t show personalized ads like Google or Yandex, but take on their intrusiveness.
2.1. The banner banner on adindex.ru (Adriver ad network) was cut out by the AdBlock browser application:

Note that AdBlock cuts out the whole div and iframe, so that even a white rectangle does not remain.
2.2. Here is an example of another block of the AdIndex network that mimics content, which was also coolly cut out by the browser plugin:

3. Pop-ups
The most perplexing advertising format, especially when a pop-up window prevents you from entering the site or closes some of the content. From the sites with pop-ups, I did not find anything decent for the test, so I used treshovy megashara.com. Browser applications coped with the task perfectly. But with the network gateway is not so simple. On the one hand, “megashara” sends HTTP requests by which pop-ups can theoretically be blocked, but, on the other hand, it constantly gives out mirrors with different URLs, so in practice it is impossible to do this. And if pop-up windows work through JavaScript, then at the gateway level it’s generally unrealistic to block them. (However, megashara.com in my lokalka blocked completely.)
4. Affiliate programs
Usually they are distributed by large online stores and lead generation services. For example, if you are the owner of a portal about books, then you can use the Labyrinth affiliate program and place their code on your website - the online store will pay you money from each transition. For example, entered the site marketopedia.ru . This is what the partner module looks like:

This seemingly simple thing couldn’t be overcome by the fact that this JavaScript farm is working. However, this type of affiliate program is quite rare and, as a rule, looks non-aggressive - you can give up on it.
But the affiliate program from Amazon at sharpologist.com was successfully blocked by browser applications:

5. Advertising on YouTube.
First of all, these are clips that anticipate watching the main video, clips that interrupt viewing, and advertisements that pop up over the video.
I took the first video I got - an advertising video first popped up: The

advertising video was going to be broadcast for a full minute; it's good that you could skip it in 10 seconds. Then I saw the full stuffing: both the banner and the built-in commercials: I

launch the ADP - and voila, this is nothing:

6. Built-in banners
From the point of view of blocking, this is the most difficult case, because the blocker, in fact, needs to ban the picture. How among all the pictures on the page to find exactly advertising, without affecting the rest? In browser applications there is an analysis of their sizes, because banners, as a rule, have standard sizes. Obviously, this method will not work if the banner of non-standard proportions.
6.1. For a start, I took cnews.ru, and this is how it is with him:

As we see, AdBlock successfully cut all the banners, including the ad of cnews.ru :)
6.2. But a less successful example is the affiliate banner on the website lezvie.info:

None of our filters blocked it. We look at the code:

Dimensions 300x416 - this is a non-standard format (at least in the list of standard Google sizes. Adsenseit is not), so the banner predictably remained in its place.
7. Contextual advertising in search
In my opinion, this is the most innocuous type of advertising that I would not block at all. If a person is looking for something in a search engine and relevant ads are shown to him, there is nothing wrong with that. On the contrary, they help to find what you need, especially if the requests are commercial.
For the sake of interest, I looked to see if contextual ads were blocking browser applications:

As we see, ads and banners were blocked, but there were suggestions from Yandex.Market.
8. Advertising in social networks
I will not develop this topic in this article, because I believe (and I agree with the management) that social networks should be blocked at all in the workplace.
What is the best blocking?
The most effective way was to block the browser application, and ADP proved to be better than AdBlock. But there are three points that confuse me greatly.
ADP Problem # 1 Cuts everything, including useful content. Take, for example, the site vedomosti.ru - it has not only purely advertising, but also semi-advertising banners with announcements of events that may be useful to the reader. And there are completely harmless banner links. Everything is cut to zero.

The ADP icon in the browser header shows how many banners were cut out on the plugin on this page, just what is the use of it? The feeling of not seeing something important, blocked by mistake, does not leave.
ADP Problem # 2Collided with sites that ask to disable ad blocking before showing content. And these are not some sludge resources, but sites like forbes.com. WordPress even has the Block Adblock plugin, which blocks content for those who use Edblock.

Of course, there is no such problem with the gateway blocker.
ADP Problem 3 Plug-ins must be installed on each computer and on each browser. And it is not enough to install - then you will have to run for each request: someone has blocked something useful, someone does not open the site. Do I need it?
Antivirus blocking works worse than ADP, but better than the redirection of HTTP requests at the gateway level. There is nothing more to say here.
Network Gateway Lock, as I have already said, occurs due to the redirection of requests directed to websites with advertising to the address 0.0.0.0. I used the Traffic Inspector Next Generation S100 piece of hardware running on OPNsense.
I will describe in a nutshell how I set up the lock (instructions here >> ). Using ssh, I went to the hardware and installed the curl package (https://curl.haxx.se/) with the command: pkg install curl.

Then I ran the script to initialize the list of blocked web resources update-hosts.sh. After forming the list, I launched the Unbound DNS function for Traffic Inspector Next Generation, selected “Services” -> “Unbound DNS” -> “General Settings” in the web interface. Enabled DNS resolver, DNS forwarding, DHCP registration. In the user settings indicated the file list of blocked web resources /var/unbound/ad-blacklist.conf. Saved settings. Now the piece of iron has become a DNS server.

Then I installed a new DNS server as a DHCP server: “Services” -> “DHCP v4” -> “LAN” -> “Enable DHCP server on the LAN interface”. On the users' computers, DHCP servers must be specified by Traffic Inspector Next Generation, then the piece of hardware will automatically be registered as a DNS server:

Now, if it is redirected to the advertisement pages, it will be blocked.
The list of blocked resources by default is downloaded from the http://winhelp2002.mvps.org/hosts.txt file . But to this list I have a lot of questions. For example, why in it, by default, there are no addresses that use Yandex and Google in their advertising networks? This means that the most annoying type of advertising (see point 1) will not be blocked. I had to prescribe them manually. In order to add a new address for blocking, you need to add the address in the following file format: server: local-data: “<address of the resource being blocked> A 0.0.0.0” and restart the Unbound DNS service.
Instead of a summary
My recipe for blocking ads on the local network is simple:
1. Activate the blocking at the gateway level.
2. We add Yandex and Google advertising networks to blockable addresses (or make sure they are there), as well as other addresses you want.
3. We inform employees that they can install an ad blocker on an individual request, and install it for those who really need it.
4. We block social networks, job search sites and other non-targeted resources at the gateway level, and then the problem of blocking ads on them will disappear by itself :)
PS As I wrote above, the list of blocked HTTP requests in Traffic Inspector Next Generation is pulled from the winhelp2002.mvps file .org / hosts.txtbut this list is not as complete as it could be. If among the readers will be prosharennyh specialists who know where to get more relevant lists (especially taking into account the specifics of the Runet), please write in the comments. If there are such lists, I will register them in the script. By the way, technical support informed me that OPNsense discusses the addition of a special plug-in for managing such lists, but for now this is not in production.

Why block ads, in my opinion, is clear: to increase the speed of loading pages and the consumption of traffic, to combat plugin-container memory consuming, and spyware scripts that monitor user actions. And most importantly, why management has raised this issue at all, is a distraction: advertising shifts the employee’s attention from the task being carried out to a tempting offer, often specially tailored for it by clever algorithms. By the way, every two years the number of computers with blocked advertising in the world triples .
First, in a nutshell, how can you block ads .
Firstly, there are applications for browsers that use dozens of different algorithms and schemes for each type of content and even for some separately taken platforms (Facebook, for example). The most popular cutters are AdBlock and Adblock Plus (hereinafter referred to as ADP). In my opinion, ADP is more effective of them, but sometimes useful content gets under its powerful skating rink (at the end). For me, the lack of all such solutions is that they are put on each computer and, moreover, on each browser. When you have hundreds of computers in LAN, it becomes a problem.
Secondly, many antivirus programs (for example, we use Kaspersky) have an in-built ad blocking option. The advantage of this method is that once you use an antivirus, you don’t need to add anything to your computer - just activate the function in the menu. But since this function is not the main one, it works worse than the applications that were specially sharpened for this business.
Third, you can cut advertising by the network gateway.. Blocking occurs by redirecting HTTP requests from advertising sites to the address 0.0.0.0. For me, a huge plus is that the lock works immediately for the entire local network. The main drawback - you need to monitor the relevance of the database of advertising addresses and, if necessary, add them manually. The second drawback is that only one blocking mechanism is used - via HTTP requests, and yet not all types of ads use them.
Looking ahead, I will say that in my LAN I use all three methods - they complement each other well. Details - at the end of the article, and now let's see what types of advertising are .
SEO and all sorts of emails there are not interesting to us now. We are interested in advertising, which is broadcast along with useful content. According to this principle, I divided it into the following types:
1. Embedded ad units of services such as the Yandex Advertising Network and Google.Adsense.
These are text, text-graphic modules and simply banners that are placed by the owners of the sites for earnings. In my opinion, this is the most harmful type of advertising, because it: a) often mimics useful content and mixes with it to increase clickability; b) shows the user what he is most likely to be interested in (we all know about big data, behavioral factors, etc., etc.); c) shows that the user (in our case, the employee) is not needed at the moment, that is, distracts from work.
Take a few resources with a similar type of advertising:
1.1. sharpologist.com/2018/08/whats-menthol-shaving-stuff.html The
Google banner was neatly cut out by KIS tools - the layout did not suffer, no voids were formed:

1.2. There are also blocks that use user requests, which he entered earlier in a search engine (for example, on zmoe.ru ):

1.3. www.bestfree.ru is a great example of a re-optimized SEO site, expectedly filled to capacity with advertising. Red boxes are marked with ad units (see picture on the left). All of them were blocked (see right):



1.4. A bad example of ad blocking is Mail.ru mail. On the inbox page, the Yandex Advertising System manifests itself twice: in the information bar above the list of letters and the ad unit on the left. Not a single insertion was blocked by any of the methods used by me:

2. Banner exchange, teaser, RTB-networks like Rotaban, advmaker.net, Marketgid, etc.
They can’t show personalized ads like Google or Yandex, but take on their intrusiveness.
2.1. The banner banner on adindex.ru (Adriver ad network) was cut out by the AdBlock browser application:

Note that AdBlock cuts out the whole div and iframe, so that even a white rectangle does not remain.
2.2. Here is an example of another block of the AdIndex network that mimics content, which was also coolly cut out by the browser plugin:

3. Pop-ups
The most perplexing advertising format, especially when a pop-up window prevents you from entering the site or closes some of the content. From the sites with pop-ups, I did not find anything decent for the test, so I used treshovy megashara.com. Browser applications coped with the task perfectly. But with the network gateway is not so simple. On the one hand, “megashara” sends HTTP requests by which pop-ups can theoretically be blocked, but, on the other hand, it constantly gives out mirrors with different URLs, so in practice it is impossible to do this. And if pop-up windows work through JavaScript, then at the gateway level it’s generally unrealistic to block them. (However, megashara.com in my lokalka blocked completely.)
4. Affiliate programs
Usually they are distributed by large online stores and lead generation services. For example, if you are the owner of a portal about books, then you can use the Labyrinth affiliate program and place their code on your website - the online store will pay you money from each transition. For example, entered the site marketopedia.ru . This is what the partner module looks like:

This seemingly simple thing couldn’t be overcome by the fact that this JavaScript farm is working. However, this type of affiliate program is quite rare and, as a rule, looks non-aggressive - you can give up on it.
But the affiliate program from Amazon at sharpologist.com was successfully blocked by browser applications:

5. Advertising on YouTube.
First of all, these are clips that anticipate watching the main video, clips that interrupt viewing, and advertisements that pop up over the video.
I took the first video I got - an advertising video first popped up: The

advertising video was going to be broadcast for a full minute; it's good that you could skip it in 10 seconds. Then I saw the full stuffing: both the banner and the built-in commercials: I

launch the ADP - and voila, this is nothing:

6. Built-in banners
From the point of view of blocking, this is the most difficult case, because the blocker, in fact, needs to ban the picture. How among all the pictures on the page to find exactly advertising, without affecting the rest? In browser applications there is an analysis of their sizes, because banners, as a rule, have standard sizes. Obviously, this method will not work if the banner of non-standard proportions.
6.1. For a start, I took cnews.ru, and this is how it is with him:

As we see, AdBlock successfully cut all the banners, including the ad of cnews.ru :)
6.2. But a less successful example is the affiliate banner on the website lezvie.info:

None of our filters blocked it. We look at the code:

Dimensions 300x416 - this is a non-standard format (at least in the list of standard Google sizes. Adsenseit is not), so the banner predictably remained in its place.
7. Contextual advertising in search
In my opinion, this is the most innocuous type of advertising that I would not block at all. If a person is looking for something in a search engine and relevant ads are shown to him, there is nothing wrong with that. On the contrary, they help to find what you need, especially if the requests are commercial.
For the sake of interest, I looked to see if contextual ads were blocking browser applications:

As we see, ads and banners were blocked, but there were suggestions from Yandex.Market.
8. Advertising in social networks
I will not develop this topic in this article, because I believe (and I agree with the management) that social networks should be blocked at all in the workplace.
What is the best blocking?
The most effective way was to block the browser application, and ADP proved to be better than AdBlock. But there are three points that confuse me greatly.
ADP Problem # 1 Cuts everything, including useful content. Take, for example, the site vedomosti.ru - it has not only purely advertising, but also semi-advertising banners with announcements of events that may be useful to the reader. And there are completely harmless banner links. Everything is cut to zero.

The ADP icon in the browser header shows how many banners were cut out on the plugin on this page, just what is the use of it? The feeling of not seeing something important, blocked by mistake, does not leave.
ADP Problem # 2Collided with sites that ask to disable ad blocking before showing content. And these are not some sludge resources, but sites like forbes.com. WordPress even has the Block Adblock plugin, which blocks content for those who use Edblock.

Of course, there is no such problem with the gateway blocker.
ADP Problem 3 Plug-ins must be installed on each computer and on each browser. And it is not enough to install - then you will have to run for each request: someone has blocked something useful, someone does not open the site. Do I need it?
Antivirus blocking works worse than ADP, but better than the redirection of HTTP requests at the gateway level. There is nothing more to say here.
Network Gateway Lock, as I have already said, occurs due to the redirection of requests directed to websites with advertising to the address 0.0.0.0. I used the Traffic Inspector Next Generation S100 piece of hardware running on OPNsense.
I will describe in a nutshell how I set up the lock (instructions here >> ). Using ssh, I went to the hardware and installed the curl package (https://curl.haxx.se/) with the command: pkg install curl.

Then I ran the script to initialize the list of blocked web resources update-hosts.sh. After forming the list, I launched the Unbound DNS function for Traffic Inspector Next Generation, selected “Services” -> “Unbound DNS” -> “General Settings” in the web interface. Enabled DNS resolver, DNS forwarding, DHCP registration. In the user settings indicated the file list of blocked web resources /var/unbound/ad-blacklist.conf. Saved settings. Now the piece of iron has become a DNS server.

Then I installed a new DNS server as a DHCP server: “Services” -> “DHCP v4” -> “LAN” -> “Enable DHCP server on the LAN interface”. On the users' computers, DHCP servers must be specified by Traffic Inspector Next Generation, then the piece of hardware will automatically be registered as a DNS server:

Now, if it is redirected to the advertisement pages, it will be blocked.
The list of blocked resources by default is downloaded from the http://winhelp2002.mvps.org/hosts.txt file . But to this list I have a lot of questions. For example, why in it, by default, there are no addresses that use Yandex and Google in their advertising networks? This means that the most annoying type of advertising (see point 1) will not be blocked. I had to prescribe them manually. In order to add a new address for blocking, you need to add the address in the following file format: server: local-data: “<address of the resource being blocked> A 0.0.0.0” and restart the Unbound DNS service.
Instead of a summary
My recipe for blocking ads on the local network is simple:
1. Activate the blocking at the gateway level.
2. We add Yandex and Google advertising networks to blockable addresses (or make sure they are there), as well as other addresses you want.
3. We inform employees that they can install an ad blocker on an individual request, and install it for those who really need it.
4. We block social networks, job search sites and other non-targeted resources at the gateway level, and then the problem of blocking ads on them will disappear by itself :)
PS As I wrote above, the list of blocked HTTP requests in Traffic Inspector Next Generation is pulled from the winhelp2002.mvps file .org / hosts.txtbut this list is not as complete as it could be. If among the readers will be prosharennyh specialists who know where to get more relevant lists (especially taking into account the specifics of the Runet), please write in the comments. If there are such lists, I will register them in the script. By the way, technical support informed me that OPNsense discusses the addition of a special plug-in for managing such lists, but for now this is not in production.