Nine rounds of automated testing

What?

Who!

What for?

What do we already have?

How it works?

Circle one - test scenario.

Javadoc test method, which we then collect and show to manual testers and PMAs.

1. The title of the test is a short and clear description of what we are testing.
2. Link to the statement on which the test is written; if it is not there, then the case is just a game of the tester’s inflamed imagination, we have corporate software, we’re not joking.
3. The test script in Russian, describing the three phases of the test - preparation, execution of actions and verification.

Note: The script is not a translation from java into Russian, on the contrary, the script is primary, and writing a high-quality case takes more time than coding it. In our case, it is usually not written by the person who encodes the test.

Catalog catalog = DAOCatalog.create(true, false);
catalog.set(Catalog.TITLE, “Я хочу чтоб в этом тесте у объекта было такое название”)
DSLCatalog.add(catalog);
CatalogItem item = DAOCatalogItem.create(catalog);
DSLCatalogItem.add(item);

Note: The rule is to write a DSL for each encoded test action so that it can later be executed through the API.

DSLCatalog.goToCard(tester, item.get(CatalogItem.PARENT_CODE));
tester.clickElement(ADD_ELEMEMT);
DSLForm.assertFormAppear(tester, DSLForm.DIALOG);
tester.sendKeys(DSLForm.TITLE_INPUT, item.get(CatalogItem.TITLE));
tester.sendKeys(DSLForm.CODE_INPUT, item.get(CatalogItem.CODE));
tester.clickElement(DSLForm.SAVE_ON_FORM);
DSLForm.assertFormDisappear(tester, DSLForm.DIALOG);

Note: Also, to ensure testability, each new interface element in the system has its own unique id within the page. Therefore, our xpath looks like this:

 "//*[@id='description-value']"

or

 "//*[@id='description-input']"

Not like that:

"/html/body/table/tbody/tr/td[6]/table/tbody/tr[4]/td/div" 

The class of problems with layout changes is solved.

Assert.assertEquals("Название  отображается неверно в карточке шаблона отчета",  template.get(ReportTemplate.TITLE), tester.getTextElement(DSLForm.TITLE_VALUE));

ModelMap map = SdDataUtils.getObject(sc.get(Bo.METACLASS_FQN), Bo.UUID, sc.getUUID());
String message = String.format("Атрибут БО с uuid-ом '%s' имеет другое значение.", sc.getUUID());
Assert.assertEquals(message, team.get(Bo.TITLE), SdDataUtils.getMapValue(map, "responsibleTeam").get("title"));

Note: Our application has an asynchronous interface; the automation community writes a lot about problems with waiting for the appearance of elements, but we ordered the programmer asynchronous request counter. Significantly speeds up the testing process.

Note: Someone should have the thought: is there too much attention paid to testing through the UI and on the raised application?
I will answer: a feature of our software is the high complexity of not individual functions, but business logic. Almost every test requires a database and a large context. In this case, unit-tests do not much win in time and significantly lose in writing complexity.

The second circle. The environment of the test.

• check the availability of the system and the ability to log into it
• check the global settings of the system and if necessary set the necessary values

• restart the browser
• check system availability

• delete all the objects created by him - this is necessary to maintain the speed of the system and isolate the tests. When created, objects are queued and deleted in the reverse order.

The third circle is the hierarchy of tests.

• One test - one action. Exceptions no more than 5%
• Tests are combined in classes of 3-10 pieces, they test a business requirement
• 5-15 classes are combined in packages. The package tests the essence or aspect of the application.
• Packages 15

Note: Javadoc , assembled by the team

mvn -f sdng-inttest/pom.xml javadoc:test-javadoc  -e 

It helps manual testers and PMs to find out what tests are, and allows you not to search for what is needed in the git web interface and not to install an IDE.

The root looks like this:

The fourth circle is the organization of the design of the testing system.

• Test code
• Utility domain methods: DSL code (work with the system under test) and DAO (work with models)
• Utility methods not tied to the application under test - working with strings, files, JSON, etc.
• The core of the testing system: logs, screenshots, cleaning, interface to webdriver, exception mechanism.

Note: This is how the browser settings for tests look, I hope someone will find it useful:

Firefoxprofile

 /**
     * Открыть Firefox браузер.
     * http://code.google.com/p/selenium/wiki/FirefoxDriver
     * @return возвращает экземпляр класса FirefoxDriver, реализующий интерфейс WebDriver.
     * @exception WebDriverException если невозможно открыть браузер.
     */
    private WebDriver openFirefox()
    {
        FirefoxProfile firefoxProfile = new FirefoxProfile();
        //Память на вкладки
        firefoxProfile.setPreference("browser.sessionhistory.max_total_viewer", "1");
        //Значение 3 не просто так, иначе не работает авторефреш
        firefoxProfile.setPreference("browser.sessionhistory.max_entries", 3);
        firefoxProfile.setPreference("browser.sessionhistory.max_total_viewers", 1);
        firefoxProfile.setPreference("browser.sessionstore.max_tabs_undo", 0);
        //Асинхронные запросы к серверу
        firefoxProfile.setPreference("network.http.pipelining", true);
        firefoxProfile.setPreference("network.http.pipelining.maxrequests", 8);
        //Задержка отрисовки
        firefoxProfile.setPreference("nglayout.initialpaint.delay", "0");
        //Сканирование внутренним сканером загнрузок
        firefoxProfile.setPreference("browser.download.manager.scanWhenDone", false);
        //Анимация переключения вкладок
        firefoxProfile.setPreference("browser.tabs.animate", false);
        //Автоподстановка
        firefoxProfile.setPreference("browser.search.suggest.enabled", false);
        //Анимация гифок
        firefoxProfile.setPreference("image.animation_mode", "none");
        //Резервные копии вкладок
        firefoxProfile.setPreference("browser.bookmarks.max_backups", 0);
        //Автодополнение
        firefoxProfile.setPreference("browser.formfill.enable", false);
        //Убрал дисковый кеш и кеш в памяти
        //firefoxProfile.setPreference("browser.cache.memory.enable", false);
        firefoxProfile.setPreference("browser.cache.disk.enable", false);
        //Сохранять файлы без подтверждения в tslogs
        firefoxProfile.setPreference("browser.download.folderList", 2);
        firefoxProfile.setPreference("browser.download.manager.showWhenStarting", false);
        firefoxProfile.setPreference("browser.download.dir", new File("").getAbsolutePath());
        firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk",
                "application/xml,application/pdf,application/zip,text/plain,application/vnd.ms-excel");
        return new FirefoxDriver(firefoxProfile);
    }

Note: The testing system, has been living for a year and a half, has survived the move from svn to git, 300 commits, so 100 unit tests are vital

1. Testing system settings passed from maven - the highest priority. We write them in the config. If there is no config, then create.
2. We take the rest of the settings from the config. If there is no config, then by default.
3. Thus, the developers of the application and tests configure their personal config, and in CI we pass all the necessary parameters through maven.

allowscreenshot=true
needinit=true
superlogin=naumen
allowsaveconfig=true
clickertime=3600000
testerpassword=atpassword
testerlogin=atlogin
superpassword=тутявсежезаменилтекст
needdelete=true
webaddress=http://localhost:9090/sd/
browsertype=firefox

Note: TS is all adult - it has its own settings and usage scenarios.
“As an autotester, I want to be able to run a testing application on a local computer, to perform certain tests on a local bench or client, for further analysis of the data received, or for the development of new tests.”
“As a developer, I want the TS to check the project after each change , for timely identification of problems and errors.
Etc.

smoke-snap2-branch
 ...
                            integration-testsall**/selenium/**/advlist/*Test.java**/selenium/**/bo/*Test.java**/selenium/**/advimport/*Test.java**/selenium/**/user/*Test.java**/selenium/**/rights/*Test.java
...
        

The fifth circle. Source code management.

Note: Tests are in the same place as the application code. Thus, by detaching a branch, the developer also detaches the test code, this ensures the synchronization of tests and applications. We did not immediately come to this obvious decision.

The circle of the sixth. Separate assembly in CI Jenkins.

Look

• Clone Workspace , which avoids the desync when someone made a commit between compiling the code in the parent assembly and running the tests in the child.
• Xvfb is a virtual screen.

export DISPLAY='localhost:'$DISPLAY_NUM'.0';
mvn verify -P war-deploy,$TEST_PROFILE -Dext.prop.dir=$WORKSPACE/$PROP_DIR -Dwebaddress=http://localhost:$DEPLOY_PORT/sd -Dselenium.deploy.port=$DEPLOY_PORT -Dcargo.tomcat.ajp.port=$ARJ_PORT -Dcargo.rmi.port=$RMI_PORT

• Archive artifacts, save fingerprints
• Jabber and email notification
• Blame Upstream Committers allows you to send letters to the committers of the parent assembly.

Circle of the seventh. Assembly hierarchy.

Typical Third Branch

Note: The migration testing assembly is a simple and effective test in which we take a backup of an ancient database with data and climb to the last war file. Allows you to make sure that all data migrations pass and the application rises to a non-empty database.

Eighth circle, continuous integration system level

The loading schedule looks like this:

PMs are meditating on this picture on release day.

The result looks something like this:

Note: A few more useful things:

• Parameterized Trigger Plugin - if you have a complex assembly hierarchy.
• JobConfigHistory Plugin - works wonders with LDAP.
• Configuration Slicing Plugin - if you have more than 50 almost different assemblies.
• Rebuild Plugin is just a useful thing.
• The Continuous Integration Game plugin — моя группа всегда выигрывает, так как мы только и делаем, что пишем тесты.

Девятый круг, люди и процессы.

Simple coverage schedule

Afterword