February 28, 2019 at 09:32ReCaptcha bypass in Selenium tests
ReCaptcha (it’s the popularly beloved “captcha” ) is one of the most painful things that a testing automaton may encounter on its way. Thousands of various videos recorded by immigrants from sunny India are walking on the Web regarding what dances with a tambourine it is possible to deceive this beast. Indeed, it’s quite difficult to try to interact with the programmed scripts with a thing whose main purpose is to make sure that “you are not a robot”.
A very important disclaimer: it is impossible to deceive a captcha.
If you have already encountered this problem, and are reading this article trying to google a recipe for a panacea, then know that it does not exist. Moreover, in your head most likely innovative ideas have arisen about simulating realistic user behavior using WebDriver, through random mouse overing of elements, clicks on inputs, and carefully placed Thread.sleep (). It is absolutely known that this approach will not work, do not waste your time in vain.
It turns out there is no way out?
Not everything is so pessimistic. Sometimes it’s enough to try to give yourself the most accurate answer to the question “What is the challenge facing me?” and look at the situation broader. In most cases, you will understand that your goal is not to deceive the captcha, but to bypass it in order to test the functionality hidden behind it. On the example of my case, I will share with you the options I have found for solving the task.
Context: we integrated part of our product into a third-party service, and wanted to monitor whether everything is OK on their side, because they do not cover third-party parts of their platform. To get access to our functionality, you first had to log in. It was then that I met with the captcha face to face. Next, I give all the options I have tried to circumvent this problem.
In addition to classic authentication, there were canonical Login with Google / Facebook. Of course, their "captcha" were also present there, so this option did not help solve the problem.
Yes, I tried it too. It was funny, but too naive.
Let's talk about the first "live" option. The drivers for these browsers (chromedriver / geckodriver) have the ability to boot under a predefined User Profile. It stores all stored passwords, cookies, sessions, and even browser history and bookmarks. Those. thus, we simply missed the login step that is absolutely unimportant for our task, and thus we got directly to the page with the test object. It is implemented as follows:
After that, we need to tell the driver that it should be loaded from the specified profile:
Firefox:
Chrome:
This approach proved to be good when testing on a local machine with a browser installed and the usual gecko- / cromedriver'ami, but there were problems when running on Jenkins. We are raising the Selenium hub and nodes inside the Kubernetes cluster, so we ran into troubles in the form of a directory that was too long to mount the directory inside the container (a clean profile weighs about 25 MB on average, which is quite a lot), as well as some problems with the CRUD rights of the browser, which could not make changes to the profile in runtime, and fell with an “unknown error: failed to write prefs file” execution. In addition, updating the profile after cookies and sessions have reached their Expiration Dates is quite inconvenient, and I did not want to keep a huge folder with the profile internals in the project, so the following option was ultimately chosen.
“And the box just opened” - this is how it was possible to describe the situation, after we just added the manually received cookies to the driver. The algorithm of actions is as simple as possible and does not depend on the selected browser:
Add them to our tests as follows:
The obvious minus of this approach is the need to manually change cookies after their validity period has expired. But, since this period is 3 months on the tested platform, we chose this solution.
But what about the situation when it is not about authorization and sessions, but about the commission of a one-time action (eg placing an order from the basket, registering a new user, etc.)? Here the situation is even worse. Two options that I could find are:
As you can see, there are no hopeless situations. However, it would be foolish to deny that absolutely all of the above options have their own, quite significant, disadvantages, so the choice is yours.
Thanks for attention.
PS If you know any other solutions working in real life - please describe them in the comments, it will be very interesting to read.
A very important disclaimer: it is impossible to deceive a captcha.
If you have already encountered this problem, and are reading this article trying to google a recipe for a panacea, then know that it does not exist. Moreover, in your head most likely innovative ideas have arisen about simulating realistic user behavior using WebDriver, through random mouse overing of elements, clicks on inputs, and carefully placed Thread.sleep (). It is absolutely known that this approach will not work, do not waste your time in vain.
It turns out there is no way out?
Not everything is so pessimistic. Sometimes it’s enough to try to give yourself the most accurate answer to the question “What is the challenge facing me?” and look at the situation broader. In most cases, you will understand that your goal is not to deceive the captcha, but to bypass it in order to test the functionality hidden behind it. On the example of my case, I will share with you the options I have found for solving the task.
Context: we integrated part of our product into a third-party service, and wanted to monitor whether everything is OK on their side, because they do not cover third-party parts of their platform. To get access to our functionality, you first had to log in. It was then that I met with the captcha face to face. Next, I give all the options I have tried to circumvent this problem.
Non-working
Login via Google or Facebook
In addition to classic authentication, there were canonical Login with Google / Facebook. Of course, their "captcha" were also present there, so this option did not help solve the problem.
Simulate user behavior
Yes, I tried it too. It was funny, but too naive.
Workers
Chrome / Firefox Profiles
Let's talk about the first "live" option. The drivers for these browsers (chromedriver / geckodriver) have the ability to boot under a predefined User Profile. It stores all stored passwords, cookies, sessions, and even browser history and bookmarks. Those. thus, we simply missed the login step that is absolutely unimportant for our task, and thus we got directly to the page with the test object. It is implemented as follows:
- Create a “clean” browser profile
- Manually enter the captcha and log in to the desired resource
- Copy the required profile to our project (HOWTO for Firefox and Chrome )
After that, we need to tell the driver that it should be loaded from the specified profile:
Firefox:
// Инициализируем профиль
FirefoxProfile profile = new FirefoxProfile(new File("/путь/к/вашему/профилю"));
// Указываем профиль в передаваемых опциях
FirefoxOptions options = new FirefoxOptions();
options.setProfile(profile);
// Создаем браузер
WebDriver driver = new FirefoxDriver(options);
Chrome:
// Указываем профиль в передаваемых опциях
ChromeOptions options = new ChromeOptions();
options.addArgument("--user-data-dir=/путь/к/папке/с/профилями");
options.addArgument("--profile-directory=Название_папки_с_нужным_профилем");
// Создаем браузер
WebDriver driver = new ChromeDriver(options);
This approach proved to be good when testing on a local machine with a browser installed and the usual gecko- / cromedriver'ami, but there were problems when running on Jenkins. We are raising the Selenium hub and nodes inside the Kubernetes cluster, so we ran into troubles in the form of a directory that was too long to mount the directory inside the container (a clean profile weighs about 25 MB on average, which is quite a lot), as well as some problems with the CRUD rights of the browser, which could not make changes to the profile in runtime, and fell with an “unknown error: failed to write prefs file” execution. In addition, updating the profile after cookies and sessions have reached their Expiration Dates is quite inconvenient, and I did not want to keep a huge folder with the profile internals in the project, so the following option was ultimately chosen.
Cookies
“And the box just opened” - this is how it was possible to describe the situation, after we just added the manually received cookies to the driver. The algorithm of actions is as simple as possible and does not depend on the selected browser:
- Log in manually
- Through Network we look Request Headers -> Cookies which our browser sends
Add them to our tests as follows:
// Создаем cookie
private static final Cookie COOKIE = new Cookie("имя", "содержимое", "домен", "путь", new Date("дата"));
// Создаем браузер
WebDriver driver = new ChromeDriver(options);
// Добавляем cookie в браузер
driver.manage().addCookie(COOKIE);
The obvious minus of this approach is the need to manually change cookies after their validity period has expired. But, since this period is 3 months on the tested platform, we chose this solution.
And if I do not need to login?
But what about the situation when it is not about authorization and sessions, but about the commission of a one-time action (eg placing an order from the basket, registering a new user, etc.)? Here the situation is even worse. Two options that I could find are:
- Agree with your developers to provide you with some kind of workaround. Google provides such an opportunity , but remember that you consciously make a small hole in security.
- Take advantage of third-party paid services that take a captcha screenshot on your part, try to decode it, and send you a decrypted value. I myself have not tried this method and cannot fully recommend it.
To summarize
As you can see, there are no hopeless situations. However, it would be foolish to deny that absolutely all of the above options have their own, quite significant, disadvantages, so the choice is yours.
Thanks for attention.
PS If you know any other solutions working in real life - please describe them in the comments, it will be very interesting to read.
