Selenium and Node.js: writing reliable browser tests

There are many good articles on how to start writing automated browser tests using the version of Selenium designed for Node.js. Some materials talk about how to wrap tests in Mocha or Jasmine, some automate everything using npm, Grunt, or Gulp. In all such publications, you can find information on how to install and configure everything you need. There you can see simple examples of working code. All this is very useful, since, for a beginner, to build a working testing environment, consisting of many components, may not be so simple. However, as far as Selenium pitfalls are concerned, as far as the analysis of the best practical methods of test development is concerned, these articles usually do not meet expectations.







Today we will start with what other materials for automating browser tests with Selenium for Node.js usually end with. Namely, we’ll talk about how to increase the reliability of tests and “untie” them from the unpredictable phenomena that are full of browsers and web applications.

Sleep is evil

The Selenium method driver.sleepis the worst enemy of a test developer. However, despite this, it is used everywhere. Perhaps this is due to the brevity of the documentation for the Node version of Selenium , and because it covers only the API syntax. She lacks real life examples.

Perhaps the reason is that this method is used in many code examples on blogs and on Q & A sites like StackOverflow.

In order to understand the features of the method driver.sleep, consider an example. Suppose we have an animated panel that, when it appears on the screen, changes its size and position. Take a look at her.


Panel Animation

This happens so quickly that you may not notice that the buttons and controls inside the panel also resize and change position.
Here is a slow-motion version of the same process. Notice how the green button Closechanges with the panel:


Slowed panel animation

It is unlikely that such a panel behavior can interfere with the normal operation of real users, since the animation is very fast. If it is slow enough, as in the second example, and you try to click on the button Closeduring the animation, you may very well not get on it.

Typically, these animations occur so quickly that the user does not have the desire to “catch” the changing buttons. People just wait for the animation to complete. However, this does not apply to Selenium. He is so fast that he can try to click on an element that is still animating. As a result, you may encounter something like this error message:

System.InvalidOperationException : Element is not clickable at point (326, 792.5)

In such a situation, many programmers will say: “Yeah, I need to wait for the animation to complete, so I just use driver.sleep(1000)it so that the panel returns to normal.” It seems that the problem is solved? However, not all so simple.

Driver.sleep problems

The team driver.sleep(1000)does exactly what you can expect from it. It stops the test for 1000 milliseconds and allows the browser to continue to work: load pages, place fragments of documents on them, animate or smoothly display elements on the screen, or do anything else.

Returning to our example, if we assume that the panel reaches its normal state in 800 milliseconds, the command driver.sleep(1000)usually helps to achieve what it is called for. So why not take advantage of it?

The main reason is that this behavior is non-deterministic. This refers to the fact that sometimes such code will work, and sometimes not. Since this does not always work, we come to unreliable tests, which, under certain conditions, fail. Hence the bad reputation of automated browser tests.

Why are designs with driver.sleepnot always functional? In other words, why is this a non-deterministic mechanism?

A web page is much more than what you can see. And element animation is a great example. However, while everything is working as it should, nothing special needs to be seen.

It is worth mentioning that web pages are designed with the expectation that people will work with them. During testing using Selenium, a program that is much faster than a person will interact with the pages. For example, if you command Selenium to first find an item and then click on it, only a few milliseconds can elapse between these operations.

When a person is working with a website, he waits until the element appears completely before clicking on it. And when the appearance of an element takes less than a second, we probably will not even notice this “expectation”. Selenium is not only faster and more demanding than the average user. Tests, in the course of working with pages, are forced to face various unpredictable factors. Let's consider some of them:

1. The designer can change the animation time from 800 milliseconds to 1200 milliseconds. As a result, test c driver.sleep(1000)will fail.
   
2. Browsers do not always do exactly what is required of them. Due to the load on the system, the animation may slow down and take more than 800 milliseconds. Perhaps even more than the wait time set to 1000 milliseconds. As a result, the test fails again.
   
3. Different browsers have different data visualization mechanisms, assign different priorities to the operations of placing elements on the screen. Add a new browser to the test suite and the test will crash again.
   
4. Browsers that control pages, JavaScript calls that change their contents, are inherently asynchronous. If the animation in our example is applied to a block that needs information from the server, then before starting the animation you will have to wait for something like the result of an AJAX call. Now, among other things, we are dealing with network delays. As a result, it is impossible to accurately estimate the time required to display the panel on the screen. The test will fail again.
   
5. Of course, there are other reasons for test failures that I don't know about. Even the browsers themselves, without taking into account external factors, are complex systems in which, in addition, there are errors. Different errors in different browsers. As a result, trying to write reliable tests, we strive to ensure that they work in different browsers of different versions and in several operating systems of different releases. Non-deterministic tests in such conditions sooner or later fail. Given all this, it becomes clear why programmers refuse automated tests and complain about how unreliable they are.
   

What will the programmer do in order to fix one of the above problems? He will begin to search for the source of the failure, find out that the whole thing is in the time that the animation takes and will come to the obvious decision - to increase the waiting time in the call driver.sleep. Then, relying on luck, the programmer will hope that this improvement will work in all possible test scenarios, that it will help to cope with various loads on the system, smooth out differences in the visualization systems of various browsers, and so on. But we still have a non-deterministic approach. Therefore, one cannot do this.

If you have not yet been convinced that driver.sleep- in many situations, this is a harmful team, think about this. Withoutdriver.sleeptests will work much faster. For example, we hope that the animation from our example takes only 800 milliseconds. In a real test suite, such an assumption will lead to the use of something like driver.sleep(2000), again, in the hope that 2 seconds is enough for the animation to complete successfully, whatever the additional factors that affect the browser and page.

This is more than a second lost in just one step of an automated test. If there will be many similar steps, a lot of such “spare” seconds will come very quickly. For example, a recently redesigned test for just one of our web pages, which took several minutes due to overuse driver.sleep, now runs in less than fifteen seconds.

I bring to your attention specific examples of getting rid ofdriver.sleep and converting tests to reliable, fully deterministic designs.

A few words about promises

The JavaScript API for Selenium makes heavy use of promises. Moreover, the details are hidden from the programmer due to the use of the built-in manager of promises. It is expected that this functionality will be removed , so in the future you will either have to figure out how to independently combine the promises into chains, or how to use the new JavaScript async / await mechanism.

In this material, the examples still use the traditional built-in Selenium Promise Manager and the ability to chain Promises. If you understand how promises work, this will be a big plus when analyzing the code examples below. However, you will also benefit from this material if you have not yet figured out the promises properly.

Writing tests

Let's continue the example with the button located on the animated panel. We want to click on this button. Let's take a look at a few specific features that may break our tests.

What about an element that is dynamically added to the page and does not exist immediately after the page finishes loading?

Waiting for an element to appear in the DOM

The following code will not work if an element with a CSS id my-buttonwas added to the DOM after loading the page:

// Код инициализации Selenium опущен для ясности
// Загрузка страницы.
driver.get('https:/foobar.baz');
// Найти элемент.
const button = driver.findElement(By.id('my-button'));
button.click();

The method driver.findElementexpects the element is already present in the DOM. It will throw an error if the item cannot be found immediately. In this case, “immediately”, due to a call driver.get, means: “after the page has finished loading.”

Keep in mind that the current version of Selenium for JavaScript independently manages promises. Therefore, each expression will be completely completed before moving on to the next expression.

Please note that the above scenario is not always desirable. The call itself driver.findElementcan be convenient if you are sure that the element is already in the DOM.

First, take a look at how you should not correct this error. Suppose we know that adding an element to the DOM can take several seconds:

driver.get('https:/foobar.baz');
// Страница загружается, засыпаем на несколько секунд
driver.sleep(3000);
// Надеемся, что три секунды достаточно для того, чтобы по прошествии этого времени элемент можно было бы найти на странице.
const button = driver.findElement(By.id('my-button'));
button.click();

For the reasons mentioned above, such a design can lead to failure, and is likely to lead. We need to figure out how to wait for an element to appear in the DOM. It’s quite simple, this is often found in examples that can be found on the Internet. We will use the well-documented method driver.waitin order to wait for the moment when the element appears in the DOM, no more than twenty seconds.

const button = driver.wait(
  until.elementLocated(By.id('my-button')), 
  20000
);
button.click();

This approach will immediately give us a lot of advantages. For example, if an element is added to the DOM within one second, the method driver.waitexits in one second. He will not wait for all twenty seconds that are allotted to him.

Due to this behavior, we can set timeouts with a large margin without worrying that they will slow down the tests. This model of behavior compares favorably with driver.sleepthat which will always wait all the given time.

This works in many situations. But the only case in which this approach does not help us is to try to click on an element that is present in the DOM but is not yet visible on the screen.

Selenium is smart enough to not try to click on an invisible element. This is good, since the user cannot click on such an element, but it complicates the work of creating reliable automated tests.

Waiting for an item to appear on the screen

We will be based on the previous example, because before you wait for the element to become visible, it makes sense to wait until it is added to the DOM. In addition, in the code below you can see the first example of using a chain of promises:

const button = driver.wait(
  until.elementLocated(By.id('my-button')), 
  20000
)
.then(element => {
   return driver.wait(
     until.elementIsVisible(element),
     20000
   );
});
button.click();

In general, we could stop at this, since we have already considered enough to significantly improve the tests. Using this code, you can avoid many situations that, otherwise, would cause the test to fail due to the element not being in the DOM immediately after the page has finished loading. Or due to the fact that it is invisible immediately after loading the page due to something like animation. Or even for both of these reasons.

If you master the above approach, you should have no reason to write non-deterministic code for Selenium. However, writing such code is far from always the case.

When the complexity of the task grows, developers often lose ground and turn to driver.sleep. Let's look at a few more examples that can do withoutdriver.sleep in more difficult circumstances.

Description of own conditions

Thanks to the untilJavaScript method, the Selenium API already has a number of helper methods that you can use with driver.wait. In addition, you can wait until an item no longer exists, wait for an item to contain specific text, wait for a notification to appear, or use many other conditions.

If you cannot find what you need among the standard methods, you will need to write your own conditions. This is actually quite simple. The main problem here is that it is difficult to find examples of such conditions. Here is another pitfall we need to deal with.

According to the documentation , methoddriver.waitYou can provide a function that returns trueor false.

Let's say we need to wait for the property of opacitya certain element to become equal to unity:

// Получить элемент.
const element = driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
);
// driver.wait всего лишь нужна функция, которая возвращает true или false.
driver.wait(() => { 
  return element.getCssValue('opacity')      
    .then(opacity => opacity === '1');
});

This design seems useful and suitable for reuse, so put it in a function:

const waitForOpacity = function(element) {
  return driver.wait(element => element.getCssValue('opacity')      
    .then(opacity => opacity === '1');
  );
};

Now use this function:

driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
)
.then(waitForOpacity);

And here we are faced with a problem. What if you want to click on an element when it becomes completely opaque? If we try to use the value returned by the above code, nothing good will come of it:

const element = driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
)
.then(waitForOpacity);
// Вот незадача. Переменная element может быть true или false, это не элемент, у которого есть метод click().
element.click();

For the same reason, we cannot use the chaining of promises in a similar construction.

const element = driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
)
.then(waitForOpacity)
.then(element => {
  // Так тоже не пойдёт, element и здесь является логическим значением.
  element.click();
}); 

This is all, however, easy to fix. Here is an improved method:

const waitForOpacity = function(element) {
  return driver.wait(element => element.getCssValue('opacity')      
    .then(opacity => {
      if (opacity === '1') {
        return element;
      } else {
        return false;
    });
  );
};

This code fragment returns an element if the condition is true, otherwise it returns false. Such a template is suitable for reuse, it can be used when writing your own conditions.

Here's how to apply this along with chaining promises:

driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
)
.then(waitForOpacity)
.then(element => element.click());

Or even like this:

const element = driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
)
.then(waitForOpacity);
element.click();

Creating your own conditions allows you to expand the capabilities of the tests and make them deterministic. However, this is not always enough.

We are going to minus

This is true, sometimes you need to go negative, and not strive to go plus. I mean checking something that no longer exists, or something that is no longer visible on the screen.

Suppose an element is already present in the DOM, but you should not interact with it before some data is loaded via AJAX. An element can be blocked by a panel with the inscription "Loading ...".

If you pay close attention to the conditions that the method offers until, you might notice methods like elementIsNotVisibleor elementIsDisabledor a less obvious method stalenessOf.

Thanks to one of these methods, you can organize a check for hiding a panel with a loading indicator:

// Элемент уже добавлен в DOM, отсюда сразу же произойдёт возврат.
const desiredElement = driver.wait(
  until.elementLocated(By.id('some-id')),
  20000
);
// Но с элементом нельзя взаимодействовать до тех пор, пока панель с индикатором загрузки
// не исчезнет.
driver.wait(
  until.elementIsNotVisible(By.id('loading-panel')),
  20000
);
// Панель с информацией о загрузке больше не видна, с элементом теперь можно взаимодействовать, не опасаясь ошибок.
desiredElement.click();

Examining the methods described above, I found that the method is stalenessOfespecially useful. It waits until the item has been removed from the DOM, which, among other reasons, could be due to a page refresh.

Here is an example of pending content updates iframeto continue working:

let iframeElem = driver.wait(
  until.elementLocated(By.className('result-iframe')),
  20000  
);
// Выполняем некое действие, которое приводит к обновлению iframe.
someElement.click();
// Ожидаем пока предыдущий iframe перестанет существовать:
driver.wait(
  until.stalenessOf(iframeElem),
  20000
);
// Переключаемся на новый iframe. 
driver.wait(
  until.ableToSwitchToFrame(By.className('result-iframe')),
  20000
);
// Всё, что будет написано здесь, относится уже к новому iframe.

Summary

The main recommendation that can be given to those who seek to write reliable tests on Selenium is that you should always strive for determinism and abandon the method sleep. Relying on a method sleepis based on arbitrary assumptions. And this, sooner or later, leads to failures.

I hope that the examples given here will help you move towards creating quality tests on Selenium.

Dear readers! Do you use Selenium to automate tests?