September 4, 2014 at 15:49Case Crocs: one hundred million sales optimization. Part 2
In the last post, we started the story about our optimization of the checkout process for Crocs and got acquainted with the most widespread variations of testing online resources: A / B and multivariate testing. It is worth noting that in the case of e-commerce, the basis for success in this part is the ability to effectively test the entire site and its individual elements. In turn, the ability to test and improve test results directly depends on the technical capabilities of a single taken web resource. In projects with a large turnover, the testing functionality becomes archived. Today we will talk about the organization of a framework for automated regression testing, which was introduced for Crocs, about the key technologies used in this process, and about the implementation of the code itself.
Organization of a framework for automated regression testing:
The Thucydides + JUnit framework was used to implement the project . It was this solution that allowed us to provide maximum configuration flexibility, implement many auxiliary methods for working with web elements, provide logging and handling of exceptional situations, and, of course, create beautiful reports.
At this stage, we were able to cover the following localizations of Crocs sites: USA, Canada, Europe, Australia, Taiwan, Great Britain, Poland, Germany, Korea, Japan, Singapore, Hong Kong, Finland, France and Holland.
Since the functionality of individual site localizations is different, the number of tests for sites also varies. In this case, the minimum amount was 70, the maximum - 101 tests.
Continuous Integration Environment: Jenkins .
Key technologies that are used in the framework:
1. As a rule, end-to-end automated testing is written separately for one specific site. Multiple differences in the design and logic of the site require a large number of forks in the code, which leads to poor readability. Therefore, to implement the idea “one test should be performed on different locales and versions of the site”, we decided to use reflection (the mechanism for examining data about the program during its execution is carried out usingJava Reflection API , which consists of the package classes java.lang and java.lang.reflect ). Here's how to do it:
Suppose there is a test. At a high level, a test is a sequence of steps that must be performed regardless of the site being tested. The site will determine the implementation features of each step.
The site on which it will run depends on the value of the site system property. This property in Java is passed as a command line parameter. To access it, use the line of code System.getProperty (“site”); Thus, it is necessary to implement the switching of the steps goHome, goToSite, login and complexStep depending on the site. Let all the steps be methods of the AllSteps class.
This class is the Thucydides step library. First of all, it contains links to specific implementation steps. Suppose there are three specific sites: Korea, Taiwan, and the United States. Then the
public class AllSteps extends ScenarioSteps {
} The
heart of the AllSteps class is a method called reflector(reflector). Its task is to redirect the execution of a step from a common library of steps to one of three specific ones. In a normal situation, a method through a point is called on an object. However, Java implements another way to do the same thing, that is, to call an object through a point in the method. This action is implemented in java.lang.reflect.Method . The name of the called method turns into a string, which allows different manipulations with this name. Suppose that a method that implements a step for a certain site is called the same as the step itself, but with the addition of the site name at the end of the method name. That is, login will turn into loginUS , goHome into goHomeUS and so on.
For the case of three sites, the reflector can be implemented as follows:
The main feature (and inconvenience) is that for such an “inverted” method call, it is necessary to create a special data structure that will correspond to the method signature. In the given example, this is the types variable passed from the outside , which is formed by the wrapper method called from the test located at the highest level.
The types parameter will always be, even if the signature of the called method is empty (in this case, it needs to be assigned null or new Class [0] ). The values of the step parameters are transferred into the reflector without changes. Since there can be a variable number of parameters, the reflector is also declared as a method that accepts a variable number of Object parameters .
All wrapper steps that implement a call to the reflector must be annotated
To output the reflector to the error log, use the error method, which is also a Thucydides step. It does nothing, but its parameters (error message) will be included in the report.
The reflector redirects the call to the library of steps of the corresponding site. A typical step is to redirect the call to PageObject , which will work with the page through a web driver. The libraries of PageObjects for the three sites tested in the example will also be separate. We can say that the library of Korean steps is needed only for these steps to instantiate PageObjects from the Korean library of PageObjects.
Although the given example of the step is very simple, you can see that it involves the possibility of creating complex steps with many actions (if the programmer considers it necessary to use them).
What can I get from Thucydides multisite tester?
2. An important point is the storage of data. Given the number of sites and the potential for increasing this number, the process of filling out XML files will be very time-consuming and, even worse, data duplication will reach 60% for some sites.
As a result, it’s difficult to come up with anything other than organizing all the data that is stored in a single file with a hierarchical structure from Apache. The advantage is that with this type of structure we avoid excessive duplication of products, credit cards and the like for sites where they coincide. By using the principles of inheritance, it is possible to take all the general data up, leaving only unique parameters for each site.
3. Open Commerce API (OCAPI) is the REST API for sites on the Demandware platform , which allows customers, partners and developers to receive product information for easy integration. Shop APIs provide easy integration with store data for use by other systems, applications, and so on, with the ultimate goal of completing a transaction.
OCAPI allows you to work with the following resources:
Before using OCAPI, you must configure Demandware in the business manager. Settings are saved in JSON format and are unique for each site. Especially for the test development of goals, the OCAPI Java library was written, which automatically embeds ocapi.js in the client page in the browser and processes all its calls. All communication between the OCAPI java library and ocapi.js passes through the WebDriver JavascriptExecutor interface. This means that before making a request to ocapi, you need to open the store page. The integration of the OCAPI module in Thucydides significantly speeds up the execution of tests and, thereby, obtaining results after the build.
4. Flexible configuration of the Jenkins job and raising the Selenium Grid server allows you to run tests in multi-threaded mode, speeding up the regression testing procedure.
Results:
After implementing the above functionality, we were able to ensure timely detection and elimination of problems, which in turn allowed us to implement the process of optimization and testing of the sales funnel, on the basis of which Crocs achieved significant financial success. As a result, after conducting the necessary testing and optimization, in 2012 alone, the Crocs marketing team achieved $ 100 million in sales in its online stores, followed by an annual 19 percent increase. Speaking of this, we, of course, remember that the success of an e-commerce site lies not only in high-quality testing - naturally, this is a delicate combination of high-quality design solutions, good marketing, technical know-how and, most importantly, popular products, without which such high sales would be elusive.
Authors:
Konstantin Tishchenko
Vitaliy Taradaiko
Alexander Mamalyga
Organization of a framework for automated regression testing:
The Thucydides + JUnit framework was used to implement the project . It was this solution that allowed us to provide maximum configuration flexibility, implement many auxiliary methods for working with web elements, provide logging and handling of exceptional situations, and, of course, create beautiful reports.
At this stage, we were able to cover the following localizations of Crocs sites: USA, Canada, Europe, Australia, Taiwan, Great Britain, Poland, Germany, Korea, Japan, Singapore, Hong Kong, Finland, France and Holland.
Since the functionality of individual site localizations is different, the number of tests for sites also varies. In this case, the minimum amount was 70, the maximum - 101 tests.
Continuous Integration Environment: Jenkins .
Key technologies that are used in the framework:
1. As a rule, end-to-end automated testing is written separately for one specific site. Multiple differences in the design and logic of the site require a large number of forks in the code, which leads to poor readability. Therefore, to implement the idea “one test should be performed on different locales and versions of the site”, we decided to use reflection (the mechanism for examining data about the program during its execution is carried out usingJava Reflection API , which consists of the package classes java.lang and java.lang.reflect ). Here's how to do it:
Suppose there is a test. At a high level, a test is a sequence of steps that must be performed regardless of the site being tested. The site will determine the implementation features of each step.
@Testpublic void testSomething () {
// parameters
Hashmap x = new HashMap();
Hashmap y = new HashMap();
x.put ("x1", "12");
x.put ("x2", "13");
y.put ("y1", "22");
y.put ("y2", "23");
// test steps
all.goHome ();
all.goToSite ();
all.login ("j.smith", "123456q");
all.complexStep (x, 123, "aaa", "bbb", y);
} The site on which it will run depends on the value of the site system property. This property in Java is passed as a command line parameter. To access it, use the line of code System.getProperty (“site”); Thus, it is necessary to implement the switching of the steps goHome, goToSite, login and complexStep depending on the site. Let all the steps be methods of the AllSteps class.
@Steps public AllSteps all;This class is the Thucydides step library. First of all, it contains links to specific implementation steps. Suppose there are three specific sites: Korea, Taiwan, and the United States. Then the
public class AllSteps extends ScenarioSteps {
@Stepspublic KoreaSteps korea; @Stepspublic TaiwanSteps taiwan; @Stepspublic USSteps us; } The
heart of the AllSteps class is a method called reflector(reflector). Its task is to redirect the execution of a step from a common library of steps to one of three specific ones. In a normal situation, a method through a point is called on an object. However, Java implements another way to do the same thing, that is, to call an object through a point in the method. This action is implemented in java.lang.reflect.Method . The name of the called method turns into a string, which allows different manipulations with this name. Suppose that a method that implements a step for a certain site is called the same as the step itself, but with the addition of the site name at the end of the method name. That is, login will turn into loginUS , goHome into goHomeUS and so on.
For the case of three sites, the reflector can be implemented as follows:
@Steppublic void reflector (String methodName, Class [] types, Object ... args) {
if (System.getProperty ("site"). equals ("US")) {
try {
Method m = us.getClass (). GetMethod (
methodName + "US", types
);
m.invoke (us, args);
}
catch (Exception e) {
error ("US reflection error" + e.getMessage ());
}
}
if (System.getProperty ("site"). equals ("Taiwan")) {
try {
Method m = taiwan.getClass (). GetMethod (
methodName + "Taiwan", types
);
m.invoke (taiwan, args);
}
catch (Exception e) {
error ("Taiwan reflection error" + e.getMessage ());
}
}
}
The main feature (and inconvenience) is that for such an “inverted” method call, it is necessary to create a special data structure that will correspond to the method signature. In the given example, this is the types variable passed from the outside , which is formed by the wrapper method called from the test located at the highest level.
@Steppublic void login (String login, String password) {
// method signature
Class [] types = new Class [2];
types [0] = String.class;
types [1] = String.class;
reflector ("login", types, login, password);
}The types parameter will always be, even if the signature of the called method is empty (in this case, it needs to be assigned null or new Class [0] ). The values of the step parameters are transferred into the reflector without changes. Since there can be a variable number of parameters, the reflector is also declared as a method that accepts a variable number of Object parameters .
All wrapper steps that implement a call to the reflector must be annotated
@Step, which allows them to appear in the Thucydides report. In the above example, the reflector is also marked with an annotation @Step. This will lead to the fact that in the report the reflection will also be displayed as a test step, and the steps themselves will become nested. To prevent this from happening, annotation@Stepcan be removed. To output the reflector to the error log, use the error method, which is also a Thucydides step. It does nothing, but its parameters (error message) will be included in the report.
@Steppublic void error (String message) {}The reflector redirects the call to the library of steps of the corresponding site. A typical step is to redirect the call to PageObject , which will work with the page through a web driver. The libraries of PageObjects for the three sites tested in the example will also be separate. We can say that the library of Korean steps is needed only for these steps to instantiate PageObjects from the Korean library of PageObjects.
@Steppublic void loginKorea (String login, String password) {
KoreaStorefrontPage sp = getPages (). Get (KoreaStorefrontPage.class); sp.clickOnLoginLink ();
}Although the given example of the step is very simple, you can see that it involves the possibility of creating complex steps with many actions (if the programmer considers it necessary to use them).
What can I get from Thucydides multisite tester?
- A multisite tester will allow you to have independent step libraries for each of the tested sites. Each of these libraries can be developed by a separate programmer, minimally intersecting (and conflicting) with the author of a neighboring library for another site.
- A multisite tester allows you to have independent libraries of PageObjects for each of the tested sites. Each of these libraries can be developed by a separate programmer, minimally intersecting (and conflicting) with the author of a neighboring library for another site.
- In addition to the code specific to each site, the tester will also have a code that applies to all tested sites. Since several testers are physically located inside the same program, the question of sharing shared libraries and developments is removed.
- In addition to the technical benefits, there are also psychological ones: sometimes the customer wants several sites to be tested, and the program should be one, as in our case.
2. An important point is the storage of data. Given the number of sites and the potential for increasing this number, the process of filling out XML files will be very time-consuming and, even worse, data duplication will reach 60% for some sites.
As a result, it’s difficult to come up with anything other than organizing all the data that is stored in a single file with a hierarchical structure from Apache. The advantage is that with this type of structure we avoid excessive duplication of products, credit cards and the like for sites where they coincide. By using the principles of inheritance, it is possible to take all the general data up, leaving only unique parameters for each site.
- Sections - the file is divided into sections, each section begins with a specific declaration. The syntax of the declaration begins with the symbol “[” and ends accordingly with “]". The site names fall into this section.
- Parameters - have a typical format: key = value
- Comments are two options: lines that begin with “#” or “;”
3. Open Commerce API (OCAPI) is the REST API for sites on the Demandware platform , which allows customers, partners and developers to receive product information for easy integration. Shop APIs provide easy integration with store data for use by other systems, applications, and so on, with the ultimate goal of completing a transaction.
OCAPI allows you to work with the following resources:
- Account
- Basket
- Category
- Content
- Contentsearch
- Folder
- Product
- Productsearch
- Promotion
- Site
- Store (13.4+)
- Customer (Data API, 14.2+)
- CustomerSearch (Data API, 14.2+)
Before using OCAPI, you must configure Demandware in the business manager. Settings are saved in JSON format and are unique for each site. Especially for the test development of goals, the OCAPI Java library was written, which automatically embeds ocapi.js in the client page in the browser and processes all its calls. All communication between the OCAPI java library and ocapi.js passes through the WebDriver JavascriptExecutor interface. This means that before making a request to ocapi, you need to open the store page. The integration of the OCAPI module in Thucydides significantly speeds up the execution of tests and, thereby, obtaining results after the build.
4. Flexible configuration of the Jenkins job and raising the Selenium Grid server allows you to run tests in multi-threaded mode, speeding up the regression testing procedure.
Results:
After implementing the above functionality, we were able to ensure timely detection and elimination of problems, which in turn allowed us to implement the process of optimization and testing of the sales funnel, on the basis of which Crocs achieved significant financial success. As a result, after conducting the necessary testing and optimization, in 2012 alone, the Crocs marketing team achieved $ 100 million in sales in its online stores, followed by an annual 19 percent increase. Speaking of this, we, of course, remember that the success of an e-commerce site lies not only in high-quality testing - naturally, this is a delicate combination of high-quality design solutions, good marketing, technical know-how and, most importantly, popular products, without which such high sales would be elusive.
Authors:
Konstantin Tishchenko
Vitaliy Taradaiko
Alexander Mamalyga