About speed testing or how not to write tests


    Recently I saw a post Test browser performance with HTML5 Canvas . In the results, IE9 started showing crazy numbers - 350+ fps.

    This, of course, is good, but for some reason, browsers that in other tests of javascript and canvas showed no less performance, in this test they showed many times (and sometimes tens of times) lower fps (on condition of launching on Windows platform, but more on that later).

    Under the cat, I’ll show why the test does not show the speed of HTML5-Canvas at all, but at the very end there will be a screen with 470 fps for FF4 without any photoshop, first we’ll analyze what exactly is wrong with this test.


    Well, for starters, let's see what 200 fps is. This is 200 drawings per second. Those. the draw event should occur approximately every 5 ms. If you look at the test code, then there really is a line:
    	setInterval(moveIt,1);
    

    Indeed, “moveIt” should be executed every 1ms judging by the specification ( WhatWG , W3C ). But there is no such thing as the minimum time after which a handler must be called.

    And then the implementation features of the event queue in browsers takes effect. There is such an interesting test that should show the minimum accuracy of setTimeout. I had 4ms in Chrome, but in FF4 it was clearly at 10ms. Perhaps this is due to the granularity of working with a timer for Windows systems (without using high-performance counters or multimedia extensions) (at the end there is a sign with the results).

    So, as the goal of this topic is not to discuss the features of the setTimeout implementation, but to discuss how you can really measure the performance of a particular operation. As we have seen, setTimeout is not suitable (maybe only “not yet suitable”?) - it does not allow you to fully load the browser with any task.

    It is also impossible without a timeout (i.e., just to perform an operation in a loop), since the user will not see anything - you need to give the browser time to display the result of the operation.

    A possible solution to this problem was published by Davin Baron : use window.postMessage ( WhatWG , W3C , MSDN) Code that implements an analog of setTimeout but with a minimum (preferably zero) delay (I modified David's code by adding an analog of setInterval (correct me if fn.apply is needed there):
    // Only add setZeroTimeout to the window object, and hide everything
        // else in a closure.
        (function() {
            var timeouts = [];
            var messageName = "zero-timeout-message";
            // Like setTimeout, but only takes a function argument.  There's
            // no time argument (always zero) and no arguments (you have to
            // use a closure).
            function setZeroTimeout(fn) {
                timeouts.push(fn);
                window.postMessage(messageName, "*");
            }
    	function intervalHelper(fn) {
    	    var h = function() {
    		fn();
    		setZeroTimeout(h);
    	    };
    	    return h;
    	}
    	function setZeroInterval(fn) {
    	    setZeroTimeout(intervalHelper(fn));
    	}
            function handleMessage(event) {
                if (event.source == window && event.data == messageName) {
                    event.stopPropagation();
                    if (timeouts.length> 0) {
                        var fn = timeouts.shift();
                        fn();
                    }
                }
            }
            window.addEventListener("message", handleMessage, true);
            // Add the one thing we want added to the window object.
            window.setZeroTimeout = setZeroTimeout;
    	window.setZeroInterval = setZeroInterval;
        })();
    

    The difference is just huge ( demo ):
    100 iterations of setZeroTimeout took 19 milliseconds.
    100 iterations of setTimeout(0) took 393 milliseconds.
    


    With this minimal change, we have a screenshot in the header in the case of Firefox4 and Direct2D / DirectWrite enabled (see about: support )

    This is the HTML5-Canvas "test" with the change applied.

    A few words about Google Chrome: under non-Windows platforms, it often shows 200+ fps in this test on even not very strong machines (provided that hardware output acceleration is enabled). On Windows, it rests on the VSync limit of 60 fps. There is Issue about it, for which you can vote by putting an asterisk.

    Well, do not forget to watch about: flags - there is a GPU Accelerated Compositing / GPU Accelerated Canvas 2D.

    Conclusion


    When writing a test / benchmark, if some browser showed an inexplicably big difference, do not start to mistakenly think that this browser is mega-super optimized. Quite possibly, just a test is testing not what it seems. For example, a test, which in theory should show the speed of HTML5-Canvas, actually showed the implementation features of setTimeout and enable / disable VSync.

    PPS To show the complete curvature of this test, open the test on FF4 with my patch and tab into another tab (almost immediately), then return after 15-20 seconds - I had about 400 fps and slowly fell down. Proof

    In general, before publishing a test, check to see if it tests well, and if it tests something sensible, and not just draws a beautiful picture with numbers.
    Minimum setTimeout delays

    After collecting the comments, I compiled a table of test results :
    Opera 11 (Win)2
    Chrome 10-11 (Win)4
    Firefox (Win)10
    Safari, OS X 10.6.610
    iPhone10
    IE 815.6

    As comments become available I will replenish.

    In this regard, it is worth noting that, according to the recommendation of HTML5 from WhatWG / W3C, setTimeout / setInterval should work like this:
    The setIntime (() method must run the following steps:
    5. If timeout is less than 10, then increase timeout to 10

    The setTimeout () method must run the following steps:
    5. If the currently running task is a task that was created by the setTimeout () method, and timeout is less than 4, then increase timeout to 4.

    Those. less than 4 ms for setTimeout and 10 for setInterval should not be. This shows a clear error in any browser where the "original" test gives more than 100 fps (since it uses setInterval). 250 fps - maximum when using setTimeout.

    UPDATE:
    I made a copy of the test in which I increased the number of polygons (in the sphere up to 72 segments and letters 30 times).
    Now I have the results (Windows7 SP1 x64, nVidia gtx 260, Core Quad 2.4ghz):
    Chrome 12.0.703.0 canary build33 fps
    IE 9.0.8112.1642126 fps
    Firefox 4.058 fps

    No “sharpening”, etc. did not (except setZeroTimeout, which also increases the speed of the original "test" in IE9). That mega-huge difference as in the original test, I do not see.

    As a result, I will say one thing: a crookedly written test can “ trick to embellish” the result tens or even hundreds of times.

    Also popular now: