February 5, 2009 at 15:40Automatic shooting of grabers or how to avoid automatic robbery of a site
I’m the administrator of the bulletin board and apparently the efforts to maintain unique content (TBD - a topic of a separate typical) are not in vain because I noticed a bad tendency to grab ads via RSS feed, parsing html-pages of a site i.e. gray people like the content.
As SEO-readers know, such duplication of content can adversely affect the ranking of a site because someone else’s site can be re-indexed first and therefore will be the only one relevant in the search results.
The first and effective method of struggle is to look at the ping IP of the site stealing content and block it, which helped only the first time. But sometimes I just didn’t have enough time to identify thieves, and some sites began to add the use of proxy servers.
As a real programmer, I started looking for an automated solution and quickly found it. Suffice it to recall that one of the most important differences between a web server and a user's computer on the Internet is that the server has open ports served by HTTP / SMTP / POP servers. Those. if you do not allow copying content from IP with ports open to the outside, then this will be a good obstacle.
To begin to identify gray individuals and their sites, this PHP code was enough: It should be noted that the HTTP_X_FORWARDED_FOR field is set if the user goes through a normal proxy. But this field can be a fake, so you need to check the IP as specified in HTTP_X_FORWARDED_FOR, and in REMOTE_ADDR.
Since a session for each visitor is set up on the bulletin board, the check was performed only on the 2nd hit for each unique IP.
A test run of the script for 1 day showed high efficiency - 4 robber sites were identified and neutralized. Caught IP with open 80-m ports - more than 2000! Therefore, for the analysis of IP, we used the WHOIS service kindly provided by nic.ru and another script. Well, they don’t have protection from automatic scanners, and that’s good;)
Now everyone on my ban list receives the text of the announcement like “This announcement was illegally copied from xxxx, which is a violation of copyright and related rights. Please do not use sites that steal content. "such sites may be used to spread hidden malware, and may also collect confidential information about you." ;)
However, you should not just block all IPs that have ports open to the outside. It turns out there are home networks where proxy service statistics are opened on port 80 (yes, for each guest!), Which lists in detail the list of favorite sites of this audience with the number of hits / Mb and who pulled how much ...
Additional information:
I ask for your ideas and comments in the thread!
As SEO-readers know, such duplication of content can adversely affect the ranking of a site because someone else’s site can be re-indexed first and therefore will be the only one relevant in the search results.
The first and effective method of struggle is to look at the ping IP of the site stealing content and block it, which helped only the first time. But sometimes I just didn’t have enough time to identify thieves, and some sites began to add the use of proxy servers.
As a real programmer, I started looking for an automated solution and quickly found it. Suffice it to recall that one of the most important differences between a web server and a user's computer on the Internet is that the server has open ports served by HTTP / SMTP / POP servers. Those. if you do not allow copying content from IP with ports open to the outside, then this will be a good obstacle.
To begin to identify gray individuals and their sites, this PHP code was enough: It should be noted that the HTTP_X_FORWARDED_FOR field is set if the user goes through a normal proxy. But this field can be a fake, so you need to check the IP as specified in HTTP_X_FORWARDED_FOR, and in REMOTE_ADDR.
//$ip – это может быть $_SERVER["REMOTE_ADDR"] и $_SERVER["HTTP_X_FORWARDED_FOR"]
$fp = @fsockopen($ip, 80, $errno, $errstr, 1);
if( $fp !== FALSE )
{
//тут логируем, что данный IP имеет открытый HTTP-порт т.е. вероятно это сервер
}Since a session for each visitor is set up on the bulletin board, the check was performed only on the 2nd hit for each unique IP.
A test run of the script for 1 day showed high efficiency - 4 robber sites were identified and neutralized. Caught IP with open 80-m ports - more than 2000! Therefore, for the analysis of IP, we used the WHOIS service kindly provided by nic.ru and another script. Well, they don’t have protection from automatic scanners, and that’s good;)
Now everyone on my ban list receives the text of the announcement like “This announcement was illegally copied from xxxx, which is a violation of copyright and related rights. Please do not use sites that steal content. "such sites may be used to spread hidden malware, and may also collect confidential information about you." ;)
However, you should not just block all IPs that have ports open to the outside. It turns out there are home networks where proxy service statistics are opened on port 80 (yes, for each guest!), Which lists in detail the list of favorite sites of this audience with the number of hits / Mb and who pulled how much ...
Additional information:
- To analyze the walks through a proxy server, you should analyze the ports 8080, 1080, 3126.
- Search for duplicates (I have a premium, personal opinion - the service is often wrong) - http://www.copyscape.com/
I ask for your ideas and comments in the thread!
