Testing in Yandex. What we learned about the Appium framework, and whether it can be used for serious tasks

    In the world of software testing, a very young direction is gaining momentum - automation of testing mobile applications. And it is expected that like mushrooms after the rain, the corresponding tools began to appear: Calabash , iOS Driver , Robotium , Selendroid , Appium . And it is about our experiments with the latest in mobile testing that I want to tell.

    Recently Appium is often mentioned at conferences and here, on Habré, there were already several posts about it. This is an open source framework written in JavaScript and designed to automate testing of mobile applications. In fact, this is Selenium WebDriver, but for mobile applications. Appium allows you to control Safari and Chrome on their respective devices, which means you can test websites under them, but an overview of these features and the nuances associated with them is a separate topic.

    To protect you from those bumps that we ourselves have accumulated while working with Appium, I want to tell you about what features of the framework we encountered, what difficulties you may have and how to deal with them.

    First, a little about why we chose Appium for ourselves. In Yandex, the automated testing infrastructure is mainly designed to use the Java language, and I do not want to fence something new. The same Calabash, for example, focuses on Ruby. Appium can be used with Selenium Grid out of the box. And we have a lot of applications, so I want to conveniently run tests in many threads. It is important to write them for all platforms using the same tool, because we have applications for both iOS and Android. In addition, Appium does not require additional modification of the application with which we deal in tests. Of great importance is the fact that there is a large community of third-party developers interested in the further development of this framework.

    In addition, I want to highlight the main features of the tool:

    • Included with the server is the test developer is offered an inspector of application elements.
    • The framework allows you to manage applications written for iOS and Android.
    • The interaction between the test and the Appium server is based on the WebDriver API.

    But all is not limited to the choice of the tool, the field from a rake begins at its direct use. So I will dwell on these points in more detail.

    Application Item Inspector


    The process of writing tests for a mobile application using Appium is very similar to writing tests for websites. You find some application elements (buttons, input fields, lists) by their selectors, interact with them and analyze their status. In the case of websites, in order to understand which element the selector is, we use various web inspectors. In the case of mobile applications, each operating system offers its own solution. Appium also made a single application for all platforms, which allows you to look at the tree of elements and the current screenshot, as well as at the values ​​of some attributes of the selected element.



    But not everything is so rosy, and here's why:
    • At the time of writing this post, the inspector worked only under MacOS, but a version for Windows is being actively developed.
    • The inspector does not show all the attributes of the elements. For Android, for example, you will not see the beautiful resource id attribute in the inspector, by which Appium can search for elements.
    • The tool is developed separately from the server. Because of this, the inspector sometimes stops working. For example, starting with version 0.14, Appium requires the device field to be specified when starting the test. The inspector was not designed for such a requirement, but he could automatically update the Appium server inside himself.
    • Sometimes not all the elements that you see with your eyes on the phone screen are shown. The situation is rare, and its nature could not be understood.
    • In the case of an Android application, the inspector may show an incorrect xpath element selector. If you try to use an automatically generated selector, be prepared for the fact that Appium may return a completely different element. Below I will tell you more about this problem.

    Manage iOS applications


    The principle of operation is to convert your WebDriver API format requests into commands of the testing framework from Apple - UI Automation . To run the tests on the simulator, the app file of your application is enough for you - no integration of third-party libraries is required. If you want to run tests on a real device, you’ll have to bother a bit . In general, everything works out of the box: just install Appium from npm and install Xcode. But let's look at the pitfalls.

    One test session - one application. While the test is running, you cannot exit the application specified when it was launched without losing the session. If you want to log into Safari first and then launch your application, then you won’t succeed. Below I will talk about a possible solution to the problem.

    On big data, Appium runs at low speed and Appium just dies by timeout . This causes an unpleasant problem. The way out of this situation is the competent thinking out of test data. Avoid lists with lots of items in the application under test.

    Xcode does not allow running multiple simulators. So you can forget about running multiple tests on the same machine in parallel. There is one single legal way to brighten up the situation a little. The server version of MacOS allows the user to create two virtual machines, each of which can be operated on. The solution is expensive, since with one Mac mini you get only 2 iOS simulators, but we stopped on it due to the lack of other options.

    Port for browser control - immutable. This item is about testing websites on the iOS simulator, which I promised not to touch, but his understanding better explains the impossibility of running tests in parallel. The fact is that when we tried to configure the launch of many emulators on the same iron machine, we did not stop at the option with virtual machines. It turns out that if you open several accounts under the same operating system, then each of the users will be able to run the simulator. But here the described problem arises. Appium controls the browser of the simulator through the Web Inspector, which in the case of the simulator listens on port 27753 on the local host. This port cannot be set, so when starting from different users, each of the simulators will try to occupy this port and nothing good will come of it.

    You cannot select the desired version of iOS on the simulator . When launched through the console, the simulator starts by default with the latest version of iOS available in Xcode, and you cannot select it. Accordingly, in order to test the application on versions of iOS 6 and iOS 7, you need to download Xcode 5 and Xcode 4 and before starting the tests, switch to the desired version with the sudo xcode-select -switch command .

    The tool does not see transparent elements. In the Yandex.Maps mobile application, a transparent element lies on top of the entire card, capturing the actions of your fingers. His eyes are not visible, but he is. And there are, for example, the zoom and delete buttons. And we see them with our eyes, but if you try using Appium to find the elements responsible for these buttons and try to tap on them, then nothing will work. UI Automation begins to think that this element is not on the screen, and you need to fix it by scrolling the screen, but there is nowhere to scroll. As a result, a mistake. The solution to the problem is tap on the coordinates. We read the position and dimensions of the required object. We add half the width and height to the position and tap. Works.

    Unstable Simulator in Xcode 5+. Yes, problems are not always related to Appium itself. Developers from large corporations also throw sticks into the wheels. So the iOS simulator, starting with Xcode version 5, began to crash with enviable regularity at startup. As a result, the Appium server has an argument that is responsible for the number of attempts to run the simulator.

    There is no way to prepare a simulator. Sometimes an empty operating system is not enough to test an application. For example, when testing the Yandex.Disk application, we needed to check the loading of unique files. I wanted to have a fresh image in the photo album when starting the test. If you run tests on one single machine, then there is no problem: run the test, which in Safari downloads the picture for you, and then run the test, which already works with the application that uses the picture. But when working with Selenium GRID, different tests will fall on different simulators. To solve the problem, I had to slightly modify Appium. A test was written in JavaScript that launched Safari, pulled a pen that returns the current timestamp in the form of qr code, and saved the image in a photo album. In turn, Appium began to monitor another parameter - DesiredCapabilities, and if it was set to true, then before returning the current session to the tests, our test to create the file was run locally. Accordingly, when the main test began, the picture was already in the albums. Downloading files is required not only in our tasks, it will be possible soondo without crutches .

    Manage Android applications


    With the Android operating system, the framework discussed in this article is also pretty good overall. This time, calls to the WebDriver API are converted to calls to the test framework methods from Google - uiautomator . The application can be tested on a simulator, and on a real device. It’s even more convenient to work with Android places than with iOS. For example, Android does not limit the test developer to one application within one session, you can enter and exit any application an unlimited number of times per session. And running tests on a live phone will only require you to connect it to a computer, Appium will do the rest for you. Let's go over the features of the work of Appium with applications for Android.

    Difficulties

    In the tree of elements there are only those of them that fall at the time of constructing the tree into the screen area. In the case of iOS, if an element is outside the scope of the screen, then iOS will try to scroll to it, but Android does not know anything about such an element, because it is not in the current tree. Accordingly, if you need to tap on a button that you don’t see right now, you will have to scroll to it first.

    Android images for Intel and for ARM. ARM is the most common, but the simulator has to emulate the processor architecture, and as a result, everything works very slowly. Intel is already closer to reality (at least the processor architecture does not differ from that presented on the system that launched the simulator). In addition, Intel themselves release the accelerator HAXMwhich makes testing even more comfortable - everything works very fast. But there are situations when it turns out that the third-party library used in the application is compiled only under ARM. For such cases, you can run tests for functionality using this library on slow ARMs, and the rest on fast Intel.

    Fake xpath. The inspector from Appium has an automatically generated selector for the selected element in the form of xpath. But be careful: for Android, it may be incorrect. The fact is that at the moment, Appium does not distinguish // from /. He does not care at what depth the descendant is relative to the parent. And very often the wrong element is expected to be returned. They promise to fix the situation in Appium 1.0. In the meantime, I can recommend using other types of selectors - by class or id - or try to check the xpath that Appium offers you.

    If your application accesses the API via https, then your root certificate may not be on the list of trusted certificates. So after the next certificate update on the test servers, the application stopped working for us. Visually, it looks like the lack of the Internet, but you will not see any error messages. To fix the problem, you will have to manually fill in the certificate in the simulator, and then import it in the settings. The system will ask you to set a PIN code for the phone. Bet, Appium can unlock the phone.

    OpenGL ES can slow down tests.Do not forget that in addition to acceleration in the form of HAXM, Android also uses graphics acceleration using OpenGL. So, locally, it will most likely work fine for you, and the tests will run relatively fast even on ARM. But on serious servers, where the video card is far from the most important component, OpenGL is likely to refuse to perform its task. Therefore, the tests will go slower than on the local computer. This must be taken into account.

    In the tree of elements there may not even be those objects that are now visible on the screen.This is a very rare situation that we have encountered only once. In the Yandex.Music application, it was not possible to find a point of contact with the search suggestion. How to get around the problem without modifying the application is not clear. Only interaction with the estimated coordinates of the elements will help.

    Problems on the Android SDK side.If Apple in the iOS 7 simulator became all bad with stability, then Google has misses in the Android SDK itself. One of the latest examples: text erasing stopped working in Android 4.2. Even passing backspace as a code symbol did not help. As a result, all tests for renaming the text ordered a long life. In addition, some time ago it was impossible to get a screenshot, which is also unpleasant. So if something doesn’t work for you, it’s not a fact that the problem is with Appium.

    Distortion of the input audio signal.The simulator distorts the sound coming into the microphone, so your application will not be able to recognize the voice. Theoretically, you can make your own patched image of the Android image. So in the Bluestack simulator there are no problems with sound, but there is no uiautomator in it, so you won’t be able to work with it through Appium.

    Useful features

    Running tests in parallel. Unlike iOS, the Android simulator has great flexibility in terms of settings and can run an unlimited number of its instances on one machine. So you can tell the simulator the port on which adb will communicate with it. In addition to the fact that thanks to this, the simulators will work separately, you will still know the identifier of the simulator, which is obtained from the templateemulator-{port}. Then, when you start Appium, you will need to scatter the server on different ports and set a parameter that points to another port for communication between Appium and the simulator (using adb, only a small part of the commands is executed). As a result, the command to run the simulator will look something like thisemulator-x86 -avd API18_X86_1 -port 5554 -sdcard 200MB-1.img -no-boot-anim -qemu -m 2047 -enable-kvm, and Appium itself will be launched with a line of the formappium -a {server address} -p 4721 -U emulator-5554 -dp 6721 --full-reset --nodeconfig register-x86-port-4721.json. You just have to change the ports for new instances of simulators and Appium servers. Register them all with Selenium GRID and get the infrastructure ready for parallel test runs.

    The freedom of action. In the case of Android, you can do anything: exit the application specified when the test was launched, enter another application, exit it and enter the third one. All this will be one session. Android allows you to work with your entire system as a single application in a single session. So it will be much easier to set up a simulator from the test for your test. Save files, perform some actions in the settings of the system itself.

    Proxy support.When can this come in handy? For example, to test the functionality of your application, which depends on the geographical region. By specifying proxies from another country, you can beautifully test this functionality. This attribute was useful to us in a different situation. Let's say you have server production for the application and testing. But your developers did not take into account the possibility of switching the environment in the application, and you urgently needed to check the application on the test API. This is where the proxy will help. We raise nginx, just write a redirector to perl, which will replace the address of the combat API with the tested one, and make the simulator go through our proxy with a redirector. So - without modifying the application - you can check the correctness of working with the API on various environments.

    Interaction between test and Appium server based on WebDriver API


    There can only be one problem: if it turns out that the method you need is not yet implemented. You can see the missing methods in the routing file . Otherwise, using the WebDriver API is a huge plus for Appium. After all, this allows you to use various third-party solutions intended for testing websites, for testing mobile applications. So here I will list only the advantages of using Appium.

    Support for Selenium GRID . Raise the Selenium GRID server, put the Appium servers on the machines, create json configuration files , run Appium with them. All. The infrastructure for running tests in parallel is ready.

    No need to connect extra libraries .A classic artifact (if we're talking about Java) selenium-java will help you. For other languages, it is also sufficient to connect the library containing the WebDriver client.

    Ability to use various add-ons to WebDriver . For example, our guys have developed an excellent HtmlElements library designed to conveniently store element selectors and organize a beautiful architecture of your tests according to the type of PageObject template. This approach allows the use of practices already tested on website testing in testing mobile applications.

    Conclusion


    What do we have in the end? Appium is still a raw but fast-paced product. The topic of testing automation of mobile applications is still gaining momentum, so Appium is more likely an old Selenium than a modern WebDriver. The server may crash at the wrong time, the test session may freeze in Selenium GRID (after our pull request, starting with version 0.17.7, the error has been fixed). The documentation is not always relevant and sometimes after the next update that some attribute has suddenly become mandatory, you will find out only after studying the source codes of the product. Errors are not fixed as quickly as we would like. For example, Appium has not worked under ubuntu for a long time. The error has been fixed.recently. But, as I said earlier, the product is developing rapidly, it (at the time of writing the post) has almost 1000 forks and almost 1200 likes on the github. It can be seen that the developers follow the wishes of users and take them into account. Previously, most of the development was carried out by several people from Sauce Labs, but now, thanks to open source codes, third-party developers participate in the appearance of new features and bug fixes. At the moment, we see the use of Appium as an experiment, but more and more it looks like a serious solution to practical problems.

    PS While this post was being prepared, new versions of Appium were released: 1.0 and 1.1. While we are just looking at them and are in no hurry to update. On the one hand, in the new versions some of the problems described here have been fixed. For example, "earned the correct xpath." But in version 1.0, it “earned” so much that when it asked for an “image”, it could return an “input field”. In 1.1, this problem was fixed. In general, not everything is still perfect, and if something was fixed in one place, it is not a fact that they did not break in other places. We are waiting for stability.

    Also popular now: