March 19, 2018 at 18:25What else do we check with Selenium, except for the interface logic
Hello, Habr!
My name is Vitaliy Kotov, I work in the Badoo testing department. Most of the time I work with Selenium. We use this wonderful tool to solve a variety of tasks: from testing functionality to simplifying work with error logs and verifying interaction with the API.
What tasks Selenium helps us solve will be discussed in this article. Go! :)
The first thing Google will tell us for “Selenium” is:
Selenium allows us to do almost the same thing that we could do with our hands: open pages and interact with them. However, he does it faster and more reliably, since his eyes cannot “get blurred” and he cannot mistakenly click in the wrong direction.
In test automation, the first thing Selenium is used for is checking the functionality of the site.
What do users usually do on a site? They open it, click back and forth, see some results of their clicks: redirects, popups, highlighting elements, and so on. I want (and need) to test this process. It is necessary to make sure that the user sees exactly that response to his action, which is embedded in our business logic.
To do this, scripts are created that describe the actions of users. For instance:
If the authorization is broken on the site (for example, the programmer messed up something - and the getUser method now always returns false), Selenium will not be able to pass step 6. The test will fail and let us know. In fact, somewhere in the code an Exception will be thrown, and the process will end with the corresponding error code.
This sequence of actions is called functional testing. This is perhaps the most common way to use Selenium. But there are more interesting ones.
A regular user does not see a JS error. This is right - he does not need. But it’s important for developers to understand whether their code “shoots” in some cases. In this context, I would separate JS errors into two types: those noticeable to the user and those that do not interfere with the user.
With the first type of error, everything is clear. If the site does not respond to user actions due to a JS error, the Selenium test will notice this.
With the second type of errors, everything is more complicated. If they do not interfere with using the application, is it worth fixing them? In Badoo, we try to correct such errors along with errors of the first kind. Firstly, any error in one way or another signals a problem. Secondly, if it does not interfere with the user in one case, this does not mean that it will not interfere with him in another.
Badoo developers use a proprietary solution to collect JS errors. When an error occurs on the client, the code collects all possible data and sends it to a special store. It stores information about the time the error occurred, user data and the trace. But even this information is sometimes not enough to reproduce the error. This is where Selenium helps us.
Our Selenium tests check all the most popular actions that users perform on the site. If there is an error, it will most likely occur during the passage of these tests. Just teach Selenium to pay attention to such errors.
I solved this problem as follows: added a template that was connected only for test servers. In this template, there was a piece of JS code that collected JS errors into a specific object. Something like this:
The same thing could be done in the Selenium test itself using the execute command . It allows you to execute JS-code on the page as if the user opened the browser console and executed the code in it. But then we could miss some of the mistakes.
The fact is that you can execute this code in the Selenium test only after the page is fully loaded. Therefore, all errors that appear during the download itself will go unnoticed. There is no such problem with the template, since the code in it is executed before the main JS code.
After adding the template, all errors began to be collected in the errorObj object, accessible globally. And now it was possible to use the execute command. I added a method to our Selenium framework that executed errorObj.getErrors (), that is, received all the errors that got into errorObj and saved them on the side of the Selenium test itself.
Let me remind you that we write Selenium tests in PHP. The error collection code is something like this:
We get errors from the errorObj JS object and process each one.
Some errors are already known to us. We know that they are on the project and are either already in the process of correction, or reproduced only in “laboratory” conditions. For example, errors that appear only for test users and which are related to the way we prepared them for the test. We ignore such errors in the test.
If the same error occurs the second time during the test, we also ignore it - we are only interested in unique ones.
For each new error, we add a URL and save all the information in an array.
We call the collectJsErrors () method after each action on the site: opening a page, waiting and clicking on an element, entering some data, and so on. No matter how we interact with the interface, we are sure to make sure that this action did not cause an error.
An array with collected errors we check only at the end of the test in tearDown (). After all, if the error somehow affects the user, the test will fall. And if it does not, then dropping the test right away is bad, first you need to check the script to the end. Suddenly there are more serious problems behind this error.
Thus, we were able to teach Selenium tests to catch client errors. And the errors themselves became easier to reproduce, knowing which test catches them.
Selenium tests themselves are not designed to test layout. Of course, if the element with which you have to interact is invisible or hidden by another element, the test will “tell us” about it. This Selenium can do out of the box. But if the button has gone somewhere down and looks bad, then the Selenium test as a whole doesn’t care - its task is to click on it ...
Nevertheless, check the layout manually before each release (and for the Desktop Web, for example, we have two of them per day) on every page, on every browser - it's a long time. And again, the human factor can intervene: the eye “gets washed up”, you can forget about something ... It is necessary to somehow automate this process.
First, we created a simple storage for pictures. By a POST request, it was possible to send a screenshot to it, indicating an additional version of the release, and by a GET request indicating the version of a release, one could get all screenshots related to this release.
Next, we wrote Selenium tests that raised all the browsers of interest to us, on which we opened all the pages of interest to us, pop-ups, overlays, and so on. In each place, they took a screenshot of the page and sent it to this store.
The storage interface allows you to quickly scroll through all the screenshots manually and make sure that on all browsers all pages look decent. A minor bug in this way, of course, is difficult to catch. But if the layout broke significantly, it immediately caught my eye. And in any case, flipping screenshots is faster and easier than clicking on a site in all browsers with your hands.
At first, we did not have many screenshots, and we could live with it. However, after some time we wanted to test the letters, the options of which we have much more than the pages on the site.
To get screenshots of letters I had to resort to some trick. The fact is that getting HTML of the generated letter in the test is much easier than going to the real mail service (and we are still talking about Selenium tests), find the letter you need there, open it and take a screenshot. Just because HTML letters can be received on our side using the backdoor (we have a QaAPI tool that allows us to receive data about a specific user and manipulate them in a test environment for test users using a simple cURL request). But for a third-party service, I would have to write, stabilize and support Page Objects: locators, methods, and so on.
It is clear that almost all email clients display the letter in a slightly different way than we will see on the “bare” HTML. But some serious errors in the layout, breaking the appearance of the letter even out of context, are thus successfully caught.
But what to do with the resulting HTML? How to make sure that it displays correctly in all browsers? It turned out pretty simple. Our collector has an empty page where the Selenium test can go and perform a simple execute:
If you do this on the right browser, you can see how the HTML will look in it. It remains only to take a screenshot and send it to the same collector.
After this was done, there were a lot of screenshots. And checking them manually twice a day has become really difficult.
To solve this problem, you can resort to a simple comparison of pictures by pixels. In this case, you will have to take into account all the variables that are on your project. In the example with a letter, for example, it can be the username and photo of him. In this case, before you take a screenshot, you have to slightly correct the HTML, replacing the pictures and names with default ones. Or generate letters for users with the same name and photo. :)
For less persistent content, such manipulations can be much more.
Next, you can get the number of mismatching pixels and conclude how critical the changes are. You can build a diff by marking on the checked image those zones that do not coincide with the reference ones.
In PHP, the ImageMagick library is great for these purposes . Also, such tests are written by JavaScript developers, they use Resemble.js .
As an example, consider the Badoo authorization page:
Suppose, for some reason, the “remember me” checkbox is lost. Because of this, the submit button was slightly higher. Then the diff will look like this:
In the picture, the violet marks the discrepancies between the reference screenshot and the one on which there is not enough element. It’s immediately clear what exactly is wrong.
The comparison system can be set to a critical number of pixels, and if not a single screenshot exceeds this number, you can send a notification that everything is fine. If some screenshots began to differ significantly - to sound the alarm ...
So you can permanently forget about checking the screenshots manually, looking only at what really deserves attention.
In Badoo, all clients (Desktop Web, Mobile Web, Android and iOS applications) communicate with the server using a specific HTTP-based API. This approach has several advantages. For example, it is much easier to develop clients and a server in parallel.
However, there is room for errors. It may happen that the client accesses the server more often than it should, due to a bug or due to the generally incorrectly thought out architecture. In this case, an extra load will be created, not to mention that, perhaps, something will not work correctly.
To monitor such things even at the stage of development and testing, we also use Selenium tests. At the beginning of these tests, we generate a cookie that we call “device id”. We pass its value to the server, which for this “device id” begins to record all the API requests that come to it.
After that, the tests themselves are run, which execute certain scenarios: open pages, write to each other, like and so on. All this time the server counts requests.
At the end of the tests, we send the server a signal to stop recording. The server generates a special log, by which you can understand how many types of requests were made. If this amount deviates from the reference by more than N percent, a special notification is sent to the auto testers, and we begin to understand the situation. Sometimes this behavior is expected (for example, when a new request appears that is associated with new functionality on the site) - then we adjust the reference values. And sometimes it turns out that the client mistakenly creates unnecessary requests, and this needs to be fixed.
We run such tests on staging before the new version of the client will be at the user. Visually, comparing the number of requests from release to release looks something like this:
Each color indicates a specific type of request. If a request begins to be repeated more often, visually it will be noticeable. In this case, we also have a special notification, so you do not need to go in and look at the statistics before each release.
We resort to similar tricks in order to quickly deal with errors in server logs. When a new error appears on the staging, it is not always clear how to reproduce it and because of what ticket it appeared.
At the very beginning of any Selenium test, a special cookie “testname” is created, into which we pass the name of the current test. Accordingly, if an error on the server was generated during the autotest, it is easier to reproduce it in most cases, since we know the playback scenario. To find the task from which the error came to the release, you can run the Selenium test on all the tasks that are being released and see their logs.
We have automated this process, I wrote about this in a previous article .
In general, this approach allows you to quickly deal with server errors in the release and localize them.
And why, in fact, no? :)
Selenium is a tool for manipulating the browser. And exactly how and why, it is up to those who use this tool to decide. By being smart, you can significantly make your life easier.
Of course, for some types of checks there are more suitable tools. For example, Selenium will not be able to fully replace vulnerability scanners or fast parsers written, say, in cURL libraries. It is also rather strange to use it for load testing - in this case, it will be more likely to check the fault tolerance of the Selenium farm than the combat application. :)
However, there is a certain set of tasks for which Selenium is great. I talked about how he helps us. I will be glad to hear about interesting ways to use Selenium in your companies.
Thanks for attention.
My name is Vitaliy Kotov, I work in the Badoo testing department. Most of the time I work with Selenium. We use this wonderful tool to solve a variety of tasks: from testing functionality to simplifying work with error logs and verifying interaction with the API.
What tasks Selenium helps us solve will be discussed in this article. Go! :)
A bit about Selenium
The first thing Google will tell us for “Selenium” is:
Selenium is a web browser automation tool. In most cases, it is used to test Web applications, but is not limited to this.
Selenium allows us to do almost the same thing that we could do with our hands: open pages and interact with them. However, he does it faster and more reliably, since his eyes cannot “get blurred” and he cannot mistakenly click in the wrong direction.
In test automation, the first thing Selenium is used for is checking the functionality of the site.
Functional Check
What do users usually do on a site? They open it, click back and forth, see some results of their clicks: redirects, popups, highlighting elements, and so on. I want (and need) to test this process. It is necessary to make sure that the user sees exactly that response to his action, which is embedded in our business logic.
To do this, scripts are created that describe the actions of users. For instance:
- Open a site.
- Click on the authorization button.
- Wait for the login form to load.
- Enter username and password.
- Click on the submit button.
- Wait for the redirect to the authorized page.
If the authorization is broken on the site (for example, the programmer messed up something - and the getUser method now always returns false), Selenium will not be able to pass step 6. The test will fail and let us know. In fact, somewhere in the code an Exception will be thrown, and the process will end with the corresponding error code.
This sequence of actions is called functional testing. This is perhaps the most common way to use Selenium. But there are more interesting ones.
JS error collection
A regular user does not see a JS error. This is right - he does not need. But it’s important for developers to understand whether their code “shoots” in some cases. In this context, I would separate JS errors into two types: those noticeable to the user and those that do not interfere with the user.
With the first type of error, everything is clear. If the site does not respond to user actions due to a JS error, the Selenium test will notice this.
With the second type of errors, everything is more complicated. If they do not interfere with using the application, is it worth fixing them? In Badoo, we try to correct such errors along with errors of the first kind. Firstly, any error in one way or another signals a problem. Secondly, if it does not interfere with the user in one case, this does not mean that it will not interfere with him in another.
Badoo developers use a proprietary solution to collect JS errors. When an error occurs on the client, the code collects all possible data and sends it to a special store. It stores information about the time the error occurred, user data and the trace. But even this information is sometimes not enough to reproduce the error. This is where Selenium helps us.
Our Selenium tests check all the most popular actions that users perform on the site. If there is an error, it will most likely occur during the passage of these tests. Just teach Selenium to pay attention to such errors.
I solved this problem as follows: added a template that was connected only for test servers. In this template, there was a piece of JS code that collected JS errors into a specific object. Something like this:
var errorObj = {
_errors : [],
addError : function(message, source, lineno) {
this._errors.push(
"Message: " + message + "\n" +
"Source: " + source + "\n" +
"Line: " + lineno
);
},
getErrors : function() {
return this._errors;
}
}
window.onerror = function(message, source, line_no) {
errorObj.addError(message, source, line_no);
}
The same thing could be done in the Selenium test itself using the execute command . It allows you to execute JS-code on the page as if the user opened the browser console and executed the code in it. But then we could miss some of the mistakes.
The fact is that you can execute this code in the Selenium test only after the page is fully loaded. Therefore, all errors that appear during the download itself will go unnoticed. There is no such problem with the template, since the code in it is executed before the main JS code.
After adding the template, all errors began to be collected in the errorObj object, accessible globally. And now it was possible to use the execute command. I added a method to our Selenium framework that executed errorObj.getErrors (), that is, received all the errors that got into errorObj and saved them on the side of the Selenium test itself.
Let me remind you that we write Selenium tests in PHP. The error collection code is something like this:
public function collectJsErrors()
{
$return_errorObj = /** @lang JavaScript */
'if (typeof errorObj !== "undefined") {return errorObj.getErrors();} else {return null;}';
$js_result = $this->execute($return_errorObj)->sync();
foreach ($js_result as $result) {
$current_stacktrace = $result;
//check if an error is known
foreach (self::$known_js_errors as $known_error) {
if (strpos($current_stacktrace, $known_error) !== false) {
continue 2;
}
}
//check if the error already caught
foreach ($this->getJsErrors() as $error) {
$existed_stacktrace = $error['error'];
if ($current_stacktrace == $existed_stacktrace) {
continue 2;
}
}
//collect an error
$this->addJsError([
'error' => $result,
'location' => $this->getLocation(),
]);
}
}
We get errors from the errorObj JS object and process each one.
Some errors are already known to us. We know that they are on the project and are either already in the process of correction, or reproduced only in “laboratory” conditions. For example, errors that appear only for test users and which are related to the way we prepared them for the test. We ignore such errors in the test.
If the same error occurs the second time during the test, we also ignore it - we are only interested in unique ones.
For each new error, we add a URL and save all the information in an array.
We call the collectJsErrors () method after each action on the site: opening a page, waiting and clicking on an element, entering some data, and so on. No matter how we interact with the interface, we are sure to make sure that this action did not cause an error.
An array with collected errors we check only at the end of the test in tearDown (). After all, if the error somehow affects the user, the test will fall. And if it does not, then dropping the test right away is bad, first you need to check the script to the end. Suddenly there are more serious problems behind this error.
Thus, we were able to teach Selenium tests to catch client errors. And the errors themselves became easier to reproduce, knowing which test catches them.
Layout check
Selenium tests themselves are not designed to test layout. Of course, if the element with which you have to interact is invisible or hidden by another element, the test will “tell us” about it. This Selenium can do out of the box. But if the button has gone somewhere down and looks bad, then the Selenium test as a whole doesn’t care - its task is to click on it ...
Nevertheless, check the layout manually before each release (and for the Desktop Web, for example, we have two of them per day) on every page, on every browser - it's a long time. And again, the human factor can intervene: the eye “gets washed up”, you can forget about something ... It is necessary to somehow automate this process.
First, we created a simple storage for pictures. By a POST request, it was possible to send a screenshot to it, indicating an additional version of the release, and by a GET request indicating the version of a release, one could get all screenshots related to this release.
Next, we wrote Selenium tests that raised all the browsers of interest to us, on which we opened all the pages of interest to us, pop-ups, overlays, and so on. In each place, they took a screenshot of the page and sent it to this store.
The storage interface allows you to quickly scroll through all the screenshots manually and make sure that on all browsers all pages look decent. A minor bug in this way, of course, is difficult to catch. But if the layout broke significantly, it immediately caught my eye. And in any case, flipping screenshots is faster and easier than clicking on a site in all browsers with your hands.
At first, we did not have many screenshots, and we could live with it. However, after some time we wanted to test the letters, the options of which we have much more than the pages on the site.
To get screenshots of letters I had to resort to some trick. The fact is that getting HTML of the generated letter in the test is much easier than going to the real mail service (and we are still talking about Selenium tests), find the letter you need there, open it and take a screenshot. Just because HTML letters can be received on our side using the backdoor (we have a QaAPI tool that allows us to receive data about a specific user and manipulate them in a test environment for test users using a simple cURL request). But for a third-party service, I would have to write, stabilize and support Page Objects: locators, methods, and so on.
It is clear that almost all email clients display the letter in a slightly different way than we will see on the “bare” HTML. But some serious errors in the layout, breaking the appearance of the letter even out of context, are thus successfully caught.
But what to do with the resulting HTML? How to make sure that it displays correctly in all browsers? It turned out pretty simple. Our collector has an empty page where the Selenium test can go and perform a simple execute:
public function drawMailHTML(string $html)
{
$js = /** @lang JavaScript */
'document.write("{HTML}")';
$js = str_replace('{HTML}', $html, $js);
$this->execute($js)->sync();
}
If you do this on the right browser, you can see how the HTML will look in it. It remains only to take a screenshot and send it to the same collector.
After this was done, there were a lot of screenshots. And checking them manually twice a day has become really difficult.
To solve this problem, you can resort to a simple comparison of pictures by pixels. In this case, you will have to take into account all the variables that are on your project. In the example with a letter, for example, it can be the username and photo of him. In this case, before you take a screenshot, you have to slightly correct the HTML, replacing the pictures and names with default ones. Or generate letters for users with the same name and photo. :)
For less persistent content, such manipulations can be much more.
Next, you can get the number of mismatching pixels and conclude how critical the changes are. You can build a diff by marking on the checked image those zones that do not coincide with the reference ones.
In PHP, the ImageMagick library is great for these purposes . Also, such tests are written by JavaScript developers, they use Resemble.js .
As an example, consider the Badoo authorization page:
Suppose, for some reason, the “remember me” checkbox is lost. Because of this, the submit button was slightly higher. Then the diff will look like this:
In the picture, the violet marks the discrepancies between the reference screenshot and the one on which there is not enough element. It’s immediately clear what exactly is wrong.
The comparison system can be set to a critical number of pixels, and if not a single screenshot exceeds this number, you can send a notification that everything is fine. If some screenshots began to differ significantly - to sound the alarm ...
So you can permanently forget about checking the screenshots manually, looking only at what really deserves attention.
Verify client interaction with the API
In Badoo, all clients (Desktop Web, Mobile Web, Android and iOS applications) communicate with the server using a specific HTTP-based API. This approach has several advantages. For example, it is much easier to develop clients and a server in parallel.
However, there is room for errors. It may happen that the client accesses the server more often than it should, due to a bug or due to the generally incorrectly thought out architecture. In this case, an extra load will be created, not to mention that, perhaps, something will not work correctly.
To monitor such things even at the stage of development and testing, we also use Selenium tests. At the beginning of these tests, we generate a cookie that we call “device id”. We pass its value to the server, which for this “device id” begins to record all the API requests that come to it.
After that, the tests themselves are run, which execute certain scenarios: open pages, write to each other, like and so on. All this time the server counts requests.
At the end of the tests, we send the server a signal to stop recording. The server generates a special log, by which you can understand how many types of requests were made. If this amount deviates from the reference by more than N percent, a special notification is sent to the auto testers, and we begin to understand the situation. Sometimes this behavior is expected (for example, when a new request appears that is associated with new functionality on the site) - then we adjust the reference values. And sometimes it turns out that the client mistakenly creates unnecessary requests, and this needs to be fixed.
We run such tests on staging before the new version of the client will be at the user. Visually, comparing the number of requests from release to release looks something like this:
Each color indicates a specific type of request. If a request begins to be repeated more often, visually it will be noticeable. In this case, we also have a special notification, so you do not need to go in and look at the statistics before each release.
Server Logs
We resort to similar tricks in order to quickly deal with errors in server logs. When a new error appears on the staging, it is not always clear how to reproduce it and because of what ticket it appeared.
At the very beginning of any Selenium test, a special cookie “testname” is created, into which we pass the name of the current test. Accordingly, if an error on the server was generated during the autotest, it is easier to reproduce it in most cases, since we know the playback scenario. To find the task from which the error came to the release, you can run the Selenium test on all the tasks that are being released and see their logs.
We have automated this process, I wrote about this in a previous article .
In general, this approach allows you to quickly deal with server errors in the release and localize them.
Automatic pizza order on Friday night
And why, in fact, no? :)
Summary
Selenium is a tool for manipulating the browser. And exactly how and why, it is up to those who use this tool to decide. By being smart, you can significantly make your life easier.
Of course, for some types of checks there are more suitable tools. For example, Selenium will not be able to fully replace vulnerability scanners or fast parsers written, say, in cURL libraries. It is also rather strange to use it for load testing - in this case, it will be more likely to check the fault tolerance of the Selenium farm than the combat application. :)
However, there is a certain set of tasks for which Selenium is great. I talked about how he helps us. I will be glad to hear about interesting ways to use Selenium in your companies.
Thanks for attention.