Google Safe Browsing - trouble came from where they did not wait

    Google Safe Browsing suddenly found a virus on my site. [WNC-611600] Malicious or unwanted software detected on site ... (Which was not there, as it turned out later).



    Visitors to the site see a full-screen, red window with the text that the site contains malware, and the authors of the site are trying to trick them into installing these programs on your computer in order to change the home page or display additional advertising on the sites (all this is a form of slander).

    And users leave, run from the site, because they believe everything that is written on the red screen. And I do not blame them. Anyone in their place would do the same.

    False positive Antivirus Error!

    This happens. We’ll write to Google now, they’ll figure it out and justice will triumph in an instant.

    In fact, everything turns out to be more complicated and the path to justice is thorny and fraught with losses of time and money due to incorrect actions.

    But this is not the worst.

    The trouble is that the error is not an isolated one. Not only I suffer from it. And Google is not able to fix it completely. At least for now.

    So, the more people find out about it, the better.

    And maybe someone important on Google will read, call and say:
    - Sorry, Dima! Our cant. (just my dreams, which I myself do not believe in)

    In parentheses is my inner voice, which does not agree with the main one in everything and always wants to use foul language. For which I apologize in advance.

    Let's figure it out in order.

    What is Google Safe Browsing?


    Google Safe Browsing (GSB) has been protecting about 3 billion devices (computers, phones) around the world for 12 years, since 2007.

    Many do not know about this, since they never installed such a program on their computer. And this is not required, since GSB is automatically installed in Chrome and its clones, in Firefox and Safari. From here came the impressive numbers of 3 billion.

    GSB is not a browser add-on, but hides in its settings, which is not so easy to get to.

    In Firefox, you won’t find Google Safe Browsing. Although it is installed and active by default.
    In the settings of Firefox, Security there is an option “Block dangerous and deceptive content”. Behind this inscription it is hiding.

    It looks like Firefox itself is protecting something, although it is not. Firefox developers can change their defender at any time, for example, on Yandex Safe Browsing.

    The browser has built-in code that exchanges data with the Safe Browsing system and checks the hashes of URLs and files with a table received from GSB. Depending on the results, the browser blocks access and displays a red picture.

    GSB controls wholly and wholly who to block and who not. And the browser does everything they say.

    Microsoft has its own counterpart - Smart Screen, which did not find anything bad on my site and programs.

    Smart Screen is built into Internet Explorer and Edge, whose share is much smaller than Chrome.

    Now detective!


    The story began in the early morning of November 30, 2018.

    I received a letter from the Google Search Console

    [WNC-611600] Malicious or unwanted software detected on site ...

    This is a mistake! I thought.

    Because it was my site and my files identified as malicious.

    While I was having breakfast, the same letters came about my second site and the number of “malicious” files found increased.

    Run to work.

    Checked the files. All are digitally signed and by downloading them I was convinced that the signature is valid. The signature is not new, issued more than a year ago.

    All found files with different dates, but not very old - ranging from a week to a month.
    Uploaded them to Virustotal with double-checking.

    Clean!

    I have been creating programs for Windows for 20 years and during this time there have been false antivirus detections.

    “Well, not in the first,” I thought.

    In a letter from Google they suggested sending an appeal, which I did immediately.
    Two hours later, the answer came that the appeal was dismissed .

    And no comments on the answer.

    In the next letter, Google announced that it had blocked the entire domain of the site and all the pages from which links to files were located.

    The logic of his actions is apparently the following: if the page contains a link to a malicious file, then this page is considered malicious. If the links are on the main page of the site, then the entire domain.

    When they entered the site, users saw a terrible full-screen red window: “The site ahead contains harmful programs”.

    When downloading a file, it is marked as malicious and you can open it in principle, confirming several times that I’m sure that I want to open this file. I do not think that at least one of the regular users of the site will do this.

    Signature stolen?

    I conducted several experiments: I signed the file using another certificate (EVO, signature on the token), created an empty project in C ++ Builder, assembled it, signed the file, uploaded it to the site.

    Google considered it a virus.

    From which I concluded that now he considers all files from this domain to be malicious, created after a certain time .

    Google considered the old file a month ago to be completely clean.

    I know that nothing fundamentally changed in it (I did not add viruses there).
    It was also embarrassing that the detection was somehow selective. For some reason, Google considered the standard version of the program to be viral, and the gold one to be pure ( probably gold protects against viruses ).

    It all looked strange.

    In a letter from Google, they suggested creating a topic on the Google forum.

    I did it.

    In response, I received a message from a secret moderator that he was already seeing the third such case in a day.

    I checked the forum and found many answers from the moderator. Comrade tried to help as much as possible ( offered useless advice), but it didn’t work on Google and really couldn’t help. But other victims began to register in the topic.

    It turns out I'm not alone!

    Link to Google forum .

    I began to search the Google Webmasters forum for similar cases with a happy ending and found one last year. It even had a response from a “likely Google employee” that suggested sending all false positives directly from Chrome through the Report an issue function .

    I

    give his answer: Sergey_Semenov:
    If it doesn't help after a review in console, send Chrome issue report from browser (Alt + Shift + I) saying you're a good white-hat company and your files is absolutely clean. It clearly helped us because we had all issues in google console disappeared after Chrome issue report without requesting another review.
    It’s a little strange to report an error in the form of a notification about unwanted programs and sites. But there is no separate False Positive form either in Chrome or on the GSB website.
    Probably, Google considers them to have no mistakes ( no words ).

    The night was alarming. ( Actually, it’s much worse. I thought what to do. How to live on. Well, at least the child has already grown ) The

    number of victims has increased. All were united by the fact that they were manufacturers of programs for Windows and, to one degree or another, had a connection with the Delphi programming environment .

    Delphi bug?
    ( I don’t think ... )

    Incomplete list of victims: Greatis Software (RegRun, UnHackMe, BootRacer), Scooter Software (Beyond Compare), IBE Software (HelpNDoc), Blumentals Software (HTMLPad, WeBuilder, RapidPHP), Balanced Scorecard Software (BSC Designer), SpamBully, Gillmeister Software ( Rename Expert), Autorun Organizer (Chemtable) ...

    9 out of 10 used the Innosetup installer, which is written in Delphi. One used Nullsoft. All files were signed with digital signatures of companies.

    The line of business is different for everyone, but peaceful: PHP editor, archiver, file comparison program, Power Point add-on, startup manager, help file creation program.
    I do not advertise myself, but I have one program that removes viruses ( it is easy to mix up a virus and an antivirus ).

    And the other works in some large companies around the world (European Parliament, Western Digital, Metropolitan Police, banks, etc.). What could result in big trouble.

    Options with falls on Delphi and Inno Setup were previously known.

    It was embarrassing that the number of victims, although increasing, was not global. There are many companies in the world using Delphi Inno Setup installers and programs.

    Why don't they suffer?

    Thinking about this issue, I started to “clean” the sites, removing links to supposedly malicious files. There were several Download archives from which you could download the same files and where Google did not get.

    Where the GSB got was bad. They were also tagged. My program page on Fileforum.com met red screens. Download.com simply blocked my company’s account and removed all programs from its website.

    Name.com (a division of IBM) has denied access to its DNS servers for the domain. This is damage that is difficult to repair immediately.

    I cleared the sites of links. Sent program links to Google.

    And lo and behold!

    A day after all the file was sent to GSB, the sites were sent for review, Google backtracked and removed all its claims.

    All files became clean as a tear. Both new and old!

    And all the other victims too!

    We returned to life, to work ( and began to live happily and did not die in one day ).

    And everything would be fine if ...

    After a week I posted a new version of the program and after 2 hours it was detected as malicious!

    It was a heavy blow ( I sat down and then lay down ).

    I quickly restored the old version. Submitted for review. In the morning, everything was cleared.

    Since then I constantly do the procedure before uploading new versions:

    1. I put the new file in a new directory on the site.
    2. Send via Chrome, Report an Issue link.
    3. I’m waiting a couple of hours.
    4. I post the new version on a new site.

    I had no more trips.

    Relapse occurred again in one of the victims on January 30 .

    He wrote to me in the mail and together we solved the problem in about a day. In February, another author of the programs had the same problem. I saw on RSDN.

    The problem of false positives is not solved (and nobody is going to solve it).

    What now fear the rest of your life?




    If you're out of luck too, try this.
    The methodology of actions based on personal experience:

    1. Do not attempt to dispute immediately through the Google Search Console. You may lose time and increase your damage.
    2. Clean up files found by Google from sites. If you leave, the damage may be greater.
    3. If there are old versions of files, restore them. If not, look where your files are hosted on the Internet and Google will not ban them. Send traffic there.
    4. Submit your cleaned site to Review through the Google Search Console.
    5. Upload your suspicious files (a new directory, another site) to a new location and send links to incorrectly defined files through Report Issue. The new Search Console has the same link. So Chrome is optional.
    6. Wait for excuses within 2 hours from GSB and within 24 hours from the Google Search Console.
    7. There will be no response from GSB.

    You can check the status of links by searching the site.

    First conclusion: the appeal system is built very badly


    ( you are to blame for everything, and why and why, we won’t tell you! )
    There is no Report False Positive form .

    For example, Microsoft has one. And the answer comes from them. Not unsubscribing, but the test results and the actions taken by them. No answers from GSB.

    I found that in the United States people dial up to Google support (it's not easy) and get the same advice to write on the forum. Nobody at Google is going to help and answer.
    The forum is attended by Google employees and is likely to read it (verified to read).
    But they rarely answer. Lately - never.
    ( Talking to yourself and to the wall is not the best feeling )

    The reaction time to the appeal is too long.

    It may take a day or two.

    The GSB verdict is meekly believed by everyone (especially Google) .
    ( Then you won’t get washed away! )

    Google will block everything for you: the site, Youtube, mail, etc., if there is a diagnosis from GSB.
    GSB reads and Gmail. And it is not clear what and how it can react ( it blocks the letters I find if they have the names of viruses. That is, it finds a virus in the text. This is a high achievement! )

    Therefore, I think that they refuse to review. If there is a GSB verdict, everything you say is ignored. It’s also bad that you will never know the reasons for the refusal, because Google does not report it. ( Google sends standard replies that you are guilty of violating anything and everything )

    Let’s figure out what the GSB verdict is based on.


    It turns out to be a mystery!



    The results of Virustotal, also owned by Google, may completely contradict GSB's decisions.

    Do you know GSB antivirus? No? Me neither. And there is no Google Safe Browsing antivirus!

    GSB is a check of the URL or hash of a file against a database of its own.

    No one analyzes the site in real time, just before you get there.
    No one is viewing JS scripts or files on this site.

    Just a browser from time to time downloads the database on your computer and checks locally.

    The GSB is more likely to respond to behavior.
    A new file appeared, unknown to him, and the system strained.
    They began to download it more than usual from this site - it is already dangerous.
    And if the same file is on a "bad" site, for example, pirates?
    ( Well, for sure - the virus )

    But the reason is commonplace. Just a new version of the program. And they begin to download it more.

    For GSB, it’s normal that if you download a file using one link, it is infected. Download the same file from another link - clean.
    Surprised?
    So do I.
    This suggests that the link is placed in the database without checking its contents.
    ( Based on the assumptions of the robot )

    This usually happens at the initial stage of turning on the GSB bulldog. Then the bulldog is turned on fully and adds the hash of the file to the database. After that, it doesn’t matter which URL the file is downloaded from. It is marked everywhere.

    At the same time, with 99% probability, not a single person analyzed the file.

    Then the bulldog proceeds to the next stage. He begins to tag all the similar files on the site. It marks files by digitally signing the file, if any.

    I tested this process by creating an empty project and compiling it into an exe file. The signed file is detected as malicious. The last stage: all files from the domain are detected as dangerous.

    The logic of the bulldog’s actions is clear: grab onto and stop the distribution of the file as soon as possible.

    Judging by the history of the detections, it is clear that the problem started with the URL detection (the hash of the file was not detected). Then the detection grew to the point that any new files from the site were considered malicious.

    The detection is exclusively machine based on "unknown principles."
    In any case, all this is called “heuristics” with a certain degree of probability.
    And the verdict of such a system should not be VIRUS, but POSSIBLE suspicious.

    Why did the trips appear at the same time for many, but had never appeared before?

    My guess is that something came up in the GSB, for example, it trained a neural network.
    The neural network finds similar files. Unfortunately, our files turned out to be similar with some kind of virus.

    And the GSB considers it reason enough to blame the crime.
    Then silently remove their mistakes and happily report how many viruses they found.

    ( If the court works according to such a scheme, then anyone can end up in jail tomorrow )

    Little ones suffer


    All the victims are small companies that find it hard to sue Google. Apparently, they have large ones in the white list. ( and you can simply ignore the small ones )

    As I understand it, at the moment, the only thing that can help Google is to remove URLs and hashes from a bad list by request.

    Maybe the GSB is so good that it defeated the malware around the world?
    Does not look like it.
    I have been meeting the same sites with malware for many years and they feel great. The number of dects that the GSB boasts cannot convince me either. Malvar breeds easily and quickly.

    The very idea of ​​checking against a certain database, which is also downloaded locally, implies a certain lag in time.
    ( Already caught, and then the base downloaded )

    Second conclusion: do not rely on the GSB to protect and save


    Here, completely different means are needed.

    Why does Google need this fight against viruses?


    What is all these GSB team efforts for?
    Make the world a better place?
    Or get information about sites visited by users?

    Google assures that it checks the URLs of sites only by hash and locally on the client, and not on the server. And sends requests to Google servers using only hashes, not full URLs.
    www.chromium.org/developers/design-documents/safebrowsing
    How it works in Firefox:
    support.mozilla.org/en-US/kb/how-does-phishing-and-malware-protection-work

    But what good is GSB from hashes?

    Maybe because Google is also a company that scans the entire Internet and has a database in which it can easily find a hash and thus receive the URLs of visited sites and analyze them.

    What Google discloses.

    As a result, Google can receive the behavioral characteristics of sites, even if Google Analytics is not installed on the site. All data is provided by GSB absolutely free.

    Despite its own rules for “good” applications, when installing Chrome there is no “Enable Safe Browsing” checkbox and it’s not obvious to anyone that Chrome sends information to the GSB.
    ( My personal conclusion is that GSB is not at all disinterested concern for others )

    But I would like GSB to follow at least generally accepted standards in terms of responsibility for its actions, be open and understandable to all participants in the process: users, site owners, software manufacturers.
    ( don't be angry, please! )

    Also popular now: