Googlebot started making POST requests through Ajax

    Google's search crawler is constantly improving to gain access to relatively closed parts of sites. In 2008, Googlebot started submitting GET forms, and this summer it started executing JavaScript . Now it comes to transferring data to the server using the POST method.

    The webmaster thumbtack.com shows examples of such requests in Apache logs for September-October 2011.

    Requests come from IP addresses that really belong to the Google Crawler: 66.249.71.47, 66.249.72.198, 66.249.72.207.

    66.249.71.47 - - [04 / Sep / 2011: 04: 53: 52 +0000] "POST / act / site / clienterror HTTP / 1.1" 200 36 "http://www.thumbtack.com/ma/malden/dog -walking / dog-walking-and-pet-care-services "" Mozilla / 5.0 (compatible; Googlebot / 2.1; + http: //www.google.com/bot.html) "

    66.249.72.198 - - [25/Sep/2011:04:27:50 +0000] "POST /act/site/clienterror HTTP/1.1" 200 36 "http://www.thumbtack.com/ca/solana-beach/wedding-photographers/photography-cary-pennington-photography" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

    66.249.72.207 - - [04/Oct/2011:09:53:08 +0000] "POST /act/site/clienterror HTTP/1.1" 200 36 "http://www.thumbtack.com/tx/san-antonio/painting/residential-commercial-construction-services" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

    On thumbtack.com, these POST requests are generated by an error monitoring script that attempts to report them to the server. Requests from Google also contain such information, apparently, the robot was able to analyze the script. This behavior of the crawler does no harm to the site, so it can be considered normal. However, the intelligent behavior of the bot is surprising.

    Also popular now: