November 25, 2015 at 14:24Testing layouts for visual regression using PhantomCSS
- Transfer
Working with someone else's code is one of the common and complex problems that I have encountered in my work. In almost every case, the previous developer did not write the code the way I would like it to.
And such situations arise quite often. Not every client has a need, desire or budget to rewrite an entire project from scratch.
Recently, our team received a code from a new client, and we were instructed, after a little refactoring, to quickly switch to implementing new functionality. We understood that we could improve the code by transferring client styles to Sass, and this would simplify our support in the future.
On the other hand, we could just rename the files and include them in one precompiled css (without any refactoring). But to improve the code, it would be nice to refactor the styles. This work is more costly, but would pay for itself in the future. And, most importantly, it would allow us to work faster, with more confidence that we will not break something.
I used to see such changes as big risks . After all, C in CSS is cascading where order is absolutely important. Restructuring several styles means changing the order, which naturally leads to a big risk of breaking something.
As a result, we had to either additionally test our changes manually, or issue such an invoice to the customer, who would immediately frighten him off with his cost.
This time it was decided to build a visually regression test suite.
This type of test has recently begun to gain popularity, and it is justified. In fact, this is a series of tests that take place throughout your site and take screenshots of various components, compare them with basic screenshots and warn you about changes.
This may seem a little illogical. We are changing CSS because we ourselves want our interface to look different. Simply put, why do we need to build some kind of process that will notify us every time that we break something when we change something in our styles?
If you change your client styles or work in a team, it’s pretty easy to make changes to the CSS, which, as you think, affected only 1 component. And only later do you discover that your changes have broken something on another page.
To understand the usefulness of visual regression tests, it’s important to understand what a person is doing poorly.
In fact, a person really does not very well distinguish between changes in visual images. This is due to its physiological and psychological characteristics.
There is even a game based on this. Do you remember the pictures from the “find some differences” series?
There are a number of real problems that psychologists are trying to solve. On the other hand, scientists have already gained a lot of knowledge in this area, which applies to web development.
One important phenomenon to consider here is blindness to change.
Research in this area began back in 1970. In 1996, George McConkie and Christopher Currie at Urbana-Champaign University of Illinois conducted a series of studies that generated considerable interest in this area.
Blindness to change is a lack of perception. Experiments have shown that serious changes in the field of view often go unnoticed, regardless of whether they occur gradually, in sharp short flashes, or appear unexpectedly at different time intervals. It is not connected with any visual defect, it is pure psychology.
A McConkie & Currie study found that in some cases, one fifth of all changes can go unnoticed. This Video This is a great example of how many changes can be skipped if you do not focus on them.
There is a wide range of tools for creating tests. I always recommend comparing tools and determining which ones are best for solving specific problems.
I chose PhantomCSS as a visual regression testing tool. There were several reasons for this.
First, it has a relatively large and active community on GitHub. When it comes to open source, I always check that the tool or library is actively developing. Working with frozen open source projects can quickly become a burden.
Secondly, PhantomCSS has a convenient Grunt plugin , which allows it to easily integrate into my existing development process.
The PhantomCSS core is a combination of three key components:
And, as I mentioned, we will use Grunt to run our tests.
Now that we’ve figured out what's what, let's go through all the steps of implementing visual regression testing.
First, we need Grunt to run the tests, so make sure it is installed. To do this, at the command prompt, type
Then open the project Gruntfile, load the PhantomCSS task, and add it to grunt.initConfig ():
I like to use Sass MQ to work with various devices. This approach has the added benefit of returning a list of all my devices that I can easily use for my tests.
With PhantomCSS, you can control the width of the browser within your test, but I prefer to ignore my tests to get more flexibility for the tests. Therefore, I delegate this task to Grunt.
With grunt-phantomcss, we can define a set of tests to run on various devices and, as an added bonus, configure them to be saved in different folders.
To maintain semantics and order, I try to name the subtasks according to the devices in Sass MQ.
Example:
We carried out the same set of tests that works on different devices and saves the results to the appropriate directories.
In the Grunt definition, you can see that we are starting the process from the tests / phantomcss / start.js file. This file launches Casper (which then runs the test scripts and our browser), and it looks like this:
Now back to our grunt. You probably already noticed that we run all the files in the `tests / phantomcss /` directory, which end in `-test.js`. Grunt loops through each of these files in alphabetical order.
You can decide how to organize the test files yourself. Personally, I create a test file for each component in my application.
Once you have implemented your `start.js` file, it is time to write your first test. We will call this file `header-test.js`.
At the beginning of the file, we tell Casper to open the root URL, then take a screenshot of the entire .site-header element. The second parameter is the name of the screenshot. I prefer to name the screenshots as well as the component that they capture. It is much easier to understand and share the results with teammates.
For the simplest example, this test will suffice. Nevertheless, we can build a more reliable test, covering the actual state of the element, page and the application as a whole.
Casper allows you to automate the interaction that occurs during the operation of the PhantomCSS browser. For example, we could write a check on the state of a button on hover as follows:
You can also test login / logout states. In the file `start.js` we can write a function that fills in the login form in WordPress as soon as we launch the Casper instance.
You may notice that we do this in casper.start () instead of doing it inside each individual test. This session setting inside casper.start () in the file `start.js` makes the session accessible from other files with your tests, since casper.start () is always started first.
For more information, I recommend taking a look at the Casper documentation.
Now that we have implemented the test suite, it's time to run them. At the command prompt, run `$ grunt phantomcss`.
PhantomCSS will automatically take screenshots on first run, and set them as basic ones for comparison with future results.
If the test crashes, PhantomCSS saves three different screenshots to the configured folder, and names the files as `.diff.png` and` .fail.png`.
For example, we changed the font size of text on one page, but accidentally reduced the font size to another. PhantomCSS will take screenshots for comparison:
Building a set of visual regression tests, of course, is not without problems. The two main problems that you have to deal with are dynamic content and the distribution of tests within the team.
The first question is how to handle dynamic content. A set of tests goes through each page, takes screenshots, and compares them. If the page content has dynamically changed, the test will not be able to detect this.
If you work in a team, changes will always be tested in your own local environment. Testing on one common environment does not always fix the problem, because the content there can still change, for example, if the page has a random set of related articles.
To solve this problem, there are two approaches that I like.
The first approach, my favorite, is to use JavaScript to replace the content inside the elements being tested.
Since these tests will not be deployed to the production server, you do not have to worry about the XSS vulnerability. Thus, before taking a screenshot, I use `.html ()` in my tests to replace the dynamic contact with a static one, which is taken from the JSON object that I included in my repository.
The second approach is to use a tool called Hologram or mdcss , which allows you to use CSS comments to create auto-generated styles. This approach makes big changes to the workflow, which requires more time and money, but has the additional advantage of creating excellent documentation for front-end web components.
The second major issue that I encountered during regression testing was how best to distribute these tests among the engineering team. Until now, in our tests, we have rigidly prescribed our test URL, and this will cause problems when working in a team, because the same URL cannot be used by all programmers for local testing.
To solve this problem, we registered our `$ grunt test` task, which takes the` --url` parameter, which is then saved to the file locally using grunt.log.
Then, at the beginning of the test file, we write:
At startup, your tests will look at the `.local_url` file, but if the file does not exist, it will use the default value of` http: // local.wordpress.dev`.
There are many benefits that visual regression testing can bring to your projects. Sprints and continuous integration are increasingly being used by developers.
Visual regression testing is also great for working with people on open source projects. In fact, the WordPress project is working on a template library with a built-in regression test suite. This test suite provides the foundation that allows a WordPress project to move forward confidently and implement its style refactoring plans.
PhantomCSS is not the only tool. I chose it simply because I considered it to be the best fit for the purposes of our team. If you are interested in visual regression testing, but you didn’t like PhantomCSS, I advise you to take a look at the following tools:
And such situations arise quite often. Not every client has a need, desire or budget to rewrite an entire project from scratch.
Recently, our team received a code from a new client, and we were instructed, after a little refactoring, to quickly switch to implementing new functionality. We understood that we could improve the code by transferring client styles to Sass, and this would simplify our support in the future.
On the other hand, we could just rename the files and include them in one precompiled css (without any refactoring). But to improve the code, it would be nice to refactor the styles. This work is more costly, but would pay for itself in the future. And, most importantly, it would allow us to work faster, with more confidence that we will not break something.
I used to see such changes as big risks . After all, C in CSS is cascading where order is absolutely important. Restructuring several styles means changing the order, which naturally leads to a big risk of breaking something.
As a result, we had to either additionally test our changes manually, or issue such an invoice to the customer, who would immediately frighten him off with his cost.
This time it was decided to build a visually regression test suite.
This type of test has recently begun to gain popularity, and it is justified. In fact, this is a series of tests that take place throughout your site and take screenshots of various components, compare them with basic screenshots and warn you about changes.
This may seem a little illogical. We are changing CSS because we ourselves want our interface to look different. Simply put, why do we need to build some kind of process that will notify us every time that we break something when we change something in our styles?
If you change your client styles or work in a team, it’s pretty easy to make changes to the CSS, which, as you think, affected only 1 component. And only later do you discover that your changes have broken something on another page.
To understand the usefulness of visual regression tests, it’s important to understand what a person is doing poorly.
Man vs Machine
In fact, a person really does not very well distinguish between changes in visual images. This is due to its physiological and psychological characteristics.
There is even a game based on this. Do you remember the pictures from the “find some differences” series?
There are a number of real problems that psychologists are trying to solve. On the other hand, scientists have already gained a lot of knowledge in this area, which applies to web development.
One important phenomenon to consider here is blindness to change.
Blindness to Change
Research in this area began back in 1970. In 1996, George McConkie and Christopher Currie at Urbana-Champaign University of Illinois conducted a series of studies that generated considerable interest in this area.
Blindness to change is a lack of perception. Experiments have shown that serious changes in the field of view often go unnoticed, regardless of whether they occur gradually, in sharp short flashes, or appear unexpectedly at different time intervals. It is not connected with any visual defect, it is pure psychology.
A McConkie & Currie study found that in some cases, one fifth of all changes can go unnoticed. This Video This is a great example of how many changes can be skipped if you do not focus on them.
Instruments
There is a wide range of tools for creating tests. I always recommend comparing tools and determining which ones are best for solving specific problems.
I chose PhantomCSS as a visual regression testing tool. There were several reasons for this.
First, it has a relatively large and active community on GitHub. When it comes to open source, I always check that the tool or library is actively developing. Working with frozen open source projects can quickly become a burden.
Secondly, PhantomCSS has a convenient Grunt plugin , which allows it to easily integrate into my existing development process.
The PhantomCSS core is a combination of three key components:
- PhantomJS or SlimerJS is a primitive browser. PhantomJS is a primitive version of WebKit, while Slimer is the Gecko engine used in Firefox.
- CasperJS - Casper is a JavaScript utility for navigation and testing. It allows us to define a set of actions that occur inside our primitive browser.
- ResembleJS is a JavaScript / HTML5 library for image comparisons. She will look for the difference between the results of our tests and the base results, and will warn about the differences between them.
And, as I mentioned, we will use Grunt to run our tests.
Implementation
Now that we’ve figured out what's what, let's go through all the steps of implementing visual regression testing.
Grunt setup
First, we need Grunt to run the tests, so make sure it is installed. To do this, at the command prompt, type
$ cd /path/to/your-siteThen open the project Gruntfile, load the PhantomCSS task, and add it to grunt.initConfig ():
grunt.loadNpmTasks('@micahgodbolt/grunt-phantomcss');
grunt.initConfig({
phantomcss: {
desktop: {
options: {
screenshots: 'baselines/desktop',
results: 'results/desktop',
viewportSize: [1280, 800]
},
src: [
'tests/phantomcss/start.js',
'tests/phantomcss/*-test.js'
]
}
}
});
Testing on various devices
I like to use Sass MQ to work with various devices. This approach has the added benefit of returning a list of all my devices that I can easily use for my tests.
With PhantomCSS, you can control the width of the browser within your test, but I prefer to ignore my tests to get more flexibility for the tests. Therefore, I delegate this task to Grunt.
With grunt-phantomcss, we can define a set of tests to run on various devices and, as an added bonus, configure them to be saved in different folders.
To maintain semantics and order, I try to name the subtasks according to the devices in Sass MQ.
Example:
grunt.initConfig( {
pkg: grunt.file.readJSON('package.json'),
phantomcss: {
desktop: {
options: {
screenshots: 'baselines/desktop',
results: 'results/desktop',
viewportSize: [1024, 768]
},
src: [
'tests/phantomcss/start.js',
'tests/phantomcss/*-test.js'
]
},
mobile: {
options: {
screenshots: 'baselines/mobile',
results: 'results/mobile',
viewportSize: [320, 480]
},
src: [
'tests/phantomcss/start.js',
'test/phantomcss/*-test.js'
]
}
}
});
We carried out the same set of tests that works on different devices and saves the results to the appropriate directories.
Test setup
In the Grunt definition, you can see that we are starting the process from the tests / phantomcss / start.js file. This file launches Casper (which then runs the test scripts and our browser), and it looks like this:
phantom.casperTest = true;
casper.start();
Now back to our grunt. You probably already noticed that we run all the files in the `tests / phantomcss /` directory, which end in `-test.js`. Grunt loops through each of these files in alphabetical order.
You can decide how to organize the test files yourself. Personally, I create a test file for each component in my application.
Writing the first test
Once you have implemented your `start.js` file, it is time to write your first test. We will call this file `header-test.js`.
casper.thenOpen('http://mysite.dev/')
.then(function() {
phantomcss.screenshot('.site-header', 'site-header');
});
At the beginning of the file, we tell Casper to open the root URL, then take a screenshot of the entire .site-header element. The second parameter is the name of the screenshot. I prefer to name the screenshots as well as the component that they capture. It is much easier to understand and share the results with teammates.
For the simplest example, this test will suffice. Nevertheless, we can build a more reliable test, covering the actual state of the element, page and the application as a whole.
Script interactions
Casper allows you to automate the interaction that occurs during the operation of the PhantomCSS browser. For example, we could write a check on the state of a button on hover as follows:
casper.then(function() {
this.mouse.move('.button');
phantomcss.screenshot('.button');
});
You can also test login / logout states. In the file `start.js` we can write a function that fills in the login form in WordPress as soon as we launch the Casper instance.
casper.start('http://default.wordpress.dev/wp-admin/', function() {
this.fill('form#loginform', {
'log': 'admin',
'pwd': 'password'
}, true);
this.click('#wp-submit');
console.log('Logging in...');
});
You may notice that we do this in casper.start () instead of doing it inside each individual test. This session setting inside casper.start () in the file `start.js` makes the session accessible from other files with your tests, since casper.start () is always started first.
For more information, I recommend taking a look at the Casper documentation.
Running tests
Now that we have implemented the test suite, it's time to run them. At the command prompt, run `$ grunt phantomcss`.
PhantomCSS will automatically take screenshots on first run, and set them as basic ones for comparison with future results.
If the test crashes, PhantomCSS saves three different screenshots to the configured folder, and names the files as `.diff.png` and` .fail.png`.
For example, we changed the font size of text on one page, but accidentally reduced the font size to another. PhantomCSS will take screenshots for comparison:
Problems
Building a set of visual regression tests, of course, is not without problems. The two main problems that you have to deal with are dynamic content and the distribution of tests within the team.
Dynamic content
The first question is how to handle dynamic content. A set of tests goes through each page, takes screenshots, and compares them. If the page content has dynamically changed, the test will not be able to detect this.
If you work in a team, changes will always be tested in your own local environment. Testing on one common environment does not always fix the problem, because the content there can still change, for example, if the page has a random set of related articles.
To solve this problem, there are two approaches that I like.
The first approach, my favorite, is to use JavaScript to replace the content inside the elements being tested.
Since these tests will not be deployed to the production server, you do not have to worry about the XSS vulnerability. Thus, before taking a screenshot, I use `.html ()` in my tests to replace the dynamic contact with a static one, which is taken from the JSON object that I included in my repository.
The second approach is to use a tool called Hologram or mdcss , which allows you to use CSS comments to create auto-generated styles. This approach makes big changes to the workflow, which requires more time and money, but has the additional advantage of creating excellent documentation for front-end web components.
Distribution of tests within the team
The second major issue that I encountered during regression testing was how best to distribute these tests among the engineering team. Until now, in our tests, we have rigidly prescribed our test URL, and this will cause problems when working in a team, because the same URL cannot be used by all programmers for local testing.
To solve this problem, we registered our `$ grunt test` task, which takes the` --url` parameter, which is then saved to the file locally using grunt.log.
// All a variable to be passed, eg. --url=http://test.dev
var localURL = grunt.option( 'url' );
/**
* Register a custom task to save the local URL, which is then read by the PhantomCSS test file.
* This file is saved so that "grunt test" can then be run in the future without passing your local URL each time.
*
* Note: Make sure test/visual/.local_url is added to your .gitignore
*
* Props to Zack Rothauser for this approach.
*/
grunt.registerTask('test', 'Runs PhantomCSS and stores the --url parameter', function() {
if (localURL) {
grunt.log.writeln( 'Local URL: ' + localURL );
grunt.file.write( 'test/visual/.local_url', localURL );
}
grunt.task.run(['phantomcss']);
});
Then, at the beginning of the test file, we write:
var fs = require('fs'), siteURL;
try {
siteURL = fs.read( 'test/visual/.local_url' );
} catch(err) {
siteURL = (typeof siteURL === 'undefined') ? 'http://local.wordpress.dev' : siteURL;
}
casper.thenOpen(siteURL + '/path/to/template');
At startup, your tests will look at the `.local_url` file, but if the file does not exist, it will use the default value of` http: // local.wordpress.dev`.
Finally
There are many benefits that visual regression testing can bring to your projects. Sprints and continuous integration are increasingly being used by developers.
Visual regression testing is also great for working with people on open source projects. In fact, the WordPress project is working on a template library with a built-in regression test suite. This test suite provides the foundation that allows a WordPress project to move forward confidently and implement its style refactoring plans.
Alternatives
PhantomCSS is not the only tool. I chose it simply because I considered it to be the best fit for the purposes of our team. If you are interested in visual regression testing, but you didn’t like PhantomCSS, I advise you to take a look at the following tools: