Monkeyrunner. Pixel-perfect web page testing on Android

    Since Google released the monkeyrunner testing automation tool, a lot of time has passed and no improvements are visible. However, for the task of regularly checking web pages for the correct layout of the best tool was not found. Those who just need a ready-made script to compare screenshots of pages on an android with scroll support can immediately download it from the link . Under the cut, it will be told what problems the mannequiner is fraught with and how to overcome them.

    Task

    Briefly, the task was this: to go through the pages of the list and make sure that they look the same as before (or almost the same). Of course, you need to scroll through the long pages to the end.

    Decision algorithm

    By itself, it looks quite simple:
    1. Open page from list
    2. Take a screenshot, compare it with the reference
    3. If the screenshot is not very different, scroll the page one screen and repeat step 2
    4. If the page has ended, or the result is very different from the original, go to the next page

    For taking screenshots, monkeyrunner has a ready-made takeSnapshot function , and for their comparison there is an absolutely wonderful ImageMagick , which allows you to evaluate how similar the pictures are, and not just require 100% match. ImageMagic has many comparison metrics, I use -metric RMSE.
    To determine when it is time to finish scrolling the page down, it is enough to compare the last two screens, if they coincide, then the end is reached. Python-junit-xml is
    used to export the results to CI or your favorite IDE

    Underwater rocks

    Now the most interesting thing: if you do all this “head on”, then nothing will work. There are several reasons.

    1. Windows
    monkeyrunner has an interesting feature, monkeyrunner.bat under Windows replaces the current directory with the one where the mannequiner itself is located, because it is so convenient for it. As a result, all relative paths and import directives stop working in our script.
    To overcome this behavior, the script itself must determine its location and continue to act only on absolute paths. This is done like this:
    filepath = path.split(os.path.realpath(__file__))[0]
    BASE_PATH = path.split(filepath)[0].encode(FILENAMES_ENCODING)
    try:
        import config
        from junit_xmls import TestSuite, TestCase
    except ImportError:
        #dirty hack that loads 3rd party modules from script's dir not from working dir, which is always changed by windows monkeyrynner
        import imp
        config = imp.load_source('config', BASE_PATH+'/src/config.py')
        junit_xml = imp.load_source('junit_xml', BASE_PATH+'/src/junit_xml/__init__.py')
        TestSuite = junit_xml.TestSuite
        TestCase = junit_xml.TestCase
    


    2. Russian symbols
    The very first attempt to test Russian Wikipedia failed because the mannequiner is based on the second python and inherits all known unicode problems and national symbols. I had to explicitly specify the encoding wherever possible, as well as slightly modify junit_xml, the author of which was unaware of national characters.
    For example, to correctly create files, you need to translate the name into unicode explicitly
    filePath.decode("utf-8")

    In addition, on Russian versions of Windows, the script will simply crash if the wrong path to ImageMagick is specified, because Popen just does not expect to receive Russian characters in the error message, and Windows localizes its messages.

    3. Scrolling
    If you scroll the same page 10 times in the same browser on the same device using MonkeyDevice.drag ()we will get 10 different results. Drag simply does not guarantee that it will scroll the page the same way. To solve this problem, I had to apply the following trick: comparing a new page with an old one, I cut off a few pixels from the top and bottom of the page and look for its place in the original (fortunately ImageMagick can even do this), how much the page is lower or higher than expected position, and there is an error of scroll, you need to subtract it at the next scroll. Such feedback allows the script not to fall off at the third iteration and safely live to the end of the page.

    4. Memory consumption
    If you open 10-20 pages in a row, any browser simply creates 10-20 tabs, over time they will gobble up all the memory and the browser simply stops responding normally to commands. In the best case, page loading will slow down, but usually this results in monkeyrunner just falling off in timeout. To avoid this, I went for a small hack: before opening a new page, the current browser process is simply killed via adb, something like this
    device.shell('am force-stop ' + BROWSER_PACKAGE_NAME)

    This hack helps a good standard AOSP browser in the emulator, but it doesn’t really work on Chrome, Opera and FF, so in the end I just had to write my wrapper on web-view, in fact a lightweight browser without tabs. Along the way, two other problems were resolved: the self-written browser does not ask for confirmation of access to the user's location and does not spoil screenshots, and besides, it can be included in the script and it will be installed automatically through MonkeyDevice.installPackage ()

    5. Failed connections
    If the script terminates and then restart it, most likely MonkeyRunner.waitForConnection ()it just fails with an error, and the repeated call usually passes without any problems. Therefore, the final version of the script always tries to connect to the device twice.

    6. Screen lock
    If the device was locked at the time of launch, the script will simply not work correctly. This can be avoided by simulating clicking on the hardware menu button (for some unknown reason, this will unlock Android devices)
    MonkeyDevice.press("KEYCODE_MENU", MonkeyDevice.DOWN_AND_UP)

    But since on the unlocked device this will lead to a menu, it is better to first make sure the device is locked, you can do this via dumpsys
       lockScreenRegexp = re.compile('mShowingLockscreen=(true|false)')
       result = lockScreenRegexp.search(device.shell('dumpsys window policy'))
       if result:
           return (result.group(1) == 'true')
       raise RuntimeError("Couldn't determine screen lock state")
    


    Unlocking by itself will not turn on the screen on the "sleeping" device, so before all these operations it is worth making a forced call to MonkeyRunner.wake ()
    After the device is successfully unlocked, you can rely on the fact that other functions will start working correctly.

    Total

    With all the edits, you can already check the correctness of the page display without fear of crashes every 5 minutes, but for the best result, you should choose either a test browser or AOSP Browser from a vanilla android.

    P.S. The code lies on the bitpack and is open for copying and modifications.

    Also popular now: