Scanning with JavaScript / Ajax / DomMutation or SlimerJS + CasperJS + Magic = Profit

Today, the topic of automating web application security testing using PhantomJS in conjunction with BurpSuite, ModSecurity, Garmr, etc. is developing very actively again. I was no exception, I would like to share with you my development experience regarding a working prototype scanner with support for Javascript, Ajax and DomMutation. Maybe this will help someone develop their own solution, which will be much better. I ask all interested under cat :-)

I hope no one will argue that existing web applications are an infernal brew from various technologies and the further the more logic moves from the server to the client. Of course, this could not but affect the departments of information security, which are already often not so large, and here the familiar automated testing methods simply break off. At one point, it became clear that for the initial analysis of solutions, static analysis and the usual html-parser + fuzzer scheme were no longer enough. So it was decided to investigate this issue and try to do something about it.

Why is Javascript, Ajax and mutation support so important?

Because there is no other way :-) History knows a lot of examples when the absence of even the simplest Javascript support in security scanners nullified their effectiveness. For example, CSRF in Yandex.Mail - if you transfer the user settings change in the GET parameters, the backend will honestly respond with an error about an invalid CSRF token and then the JavaScript engine forwards the same request (!), But already supplementing the request with a valid token :) It turns out, with the backend’s position is more than correct, but from the client’s position ... Or XSS when translating a letter into any of the languages ​​- a similar story, a primitive vulnerability that w3af and others like it could not detect. The lack of JavaScript support at the current time seems simply unacceptable and it is necessary to try hard to fix it.

How to be

I decided to take advantage of existing ideas and take my favorite SlimerJS and CasperJS as a basis and try to get a crawler with the ability to recursively traverse events of DOM elements + mutation monitoring to identify “pathologies”, XSS vulnerabilities, and more. Why not PhantomJS? Because it does not yet have the support of MutationObserver, which I needed for the analysis of mutations. And so, I imagined an integrated system consisting of four large blocks:
* Kroler, which, like a monkey, bypasses all the events we need and tracks mutations based on formal rules
* A proxy that could collect data for further fuzzing and launching the rest of the checks depending on context
* Fixtures in the web application, with pre-prepared labels that we can focus on during the testing process.
* A diff-based report with previous scan results.
Most of them were implemented earlier and therefore I concentrated on the first part (if there is interest, I will tell about the rest). The general algorithm of work that I see is divided into two large sections - page processing and event processing, in a simplified form looks something like this:

As you can see, everything is quite simple, although not yet perfect, for simplicity of perception, I drew only the main blocks. Imagine a web application that displays users and clicks to display additional information received by an Ajax request:

Available here: http://crawl-test.mmmkay.info/.
Initial move: https://github.com/dharrya/monkey-crawler/tree/master/tests
For example, if you run w3af with the webSpider plugin, then no Ajax requests will be detected, but meanwhile they are often vulnerable in a number obvious reasons. Skipfish will have roughly the same result.
I prepared a small prototype to confirm the algorithm is working, available on github . At the root of the project there is a test script "test.js":

(function start(require) {
    "use strict";
    var Spider = require('lib/spider').Spider;
    var utils = require('utils');
    var startTime = new Date().getTime();
    var spider = new Spider();
    var url = 'https://github.com/dharrya';
    if (spider.cli.has(0))
        url = spider.cli.get(0);
    spider.initialize({
        targetUri: url,
        eventContainer: undefined
    });
    spider.start(url);
    spider.then(spider.process);
    spider.run(function() {
        this.echo('\n<---------- COMPLETED ---------->\n');
        var deltaTime = new Date().getTime() - startTime;
        deltaTime = (deltaTime / 1000).toFixed(2);
        this.echo('time: ' + deltaTime + 'sec');
        this.echo('Processed pages:' + this.pagesQueue.length);
        utils.dump(this.pages);
        spider.exit();
    });
})(require);

The essence of which is to start the process and output a “raw” result, we will test using my example:

$ ./test.sh http://crawl-test.mmmkay.info
<---------- COMPLETED ---------->
time: 3.61sec
Processed pages:1
[
    {
        "url": "http://crawl-test.mmmkay.info/",
        "opened": true,
        "processed": true,
        "reloadCount": 0,
        "status_code": 200,
        "jsErrors": [],
        "xss": [],
        "xssHashMap": [],
        "pages": [],
        "events": [
            {
                "eventType": "click",
                "path": "id(\"user-Lisa\")/DIV[3]/BUTTON[1]",
                "parentEvent": null,
                "depth": 0,
                "status": "completed",
                "completed": true,
                "deleted": false,
                "xss": [],
                "xssHashMap": [],
                "events": [
                    {
                        "eventType": "click",
                        "path": "//DIV[4]",
                        "parentEvent": null,
                        "depth": 1,
                        "status": "completed",
                        "completed": true,
                        "deleted": false,
                        "xss": [],
                        "xssHashMap": [],
                        "events": [],
                        "resourses": []
                    }
                ],
                "resourses": [
                    "http://crawl-test.mmmkay.info/user/Lisa.json"
                ]
            },
            {
                "eventType": "click",
                "path": "id(\"user-Jimmy\")/DIV[3]/BUTTON[1]",
                "parentEvent": null,
                "depth": 0,
                "status": "completed",
                "completed": true,
                "deleted": false,
                "xss": [
                    {
                        "innerHtml": "Another XSS...",
                        "path": "id(\"userInfoDescription\")/XSSMARK[1]",
                        "initiator": null,
                        "dbRecord": null
                    }
                ],
                "xssHashMap": [
                    0
                ],
                "events": [],
                "resourses": [
                    "http://crawl-test.mmmkay.info/user/Jimmy.json"
                ]
            },
            {
                "eventType": "click",
                "path": "id(\"user-Mark\")/DIV[3]/BUTTON[1]",
                "parentEvent": null,
                "depth": 0,
                "status": "completed",
                "completed": true,
                "deleted": false,
                "xss": [],
                "xssHashMap": [],
                "events": [],
                "resourses": [
                    "http://crawl-test.mmmkay.info/user/Mark.json"
                ]
            },
            {
                "eventType": "click",
                "path": "id(\"user-Tommy\")/DIV[3]/BUTTON[1]",
                "parentEvent": null,
                "depth": 0,
                "status": "completed",
                "completed": true,
                "deleted": false,
                "xss": [],
                "xssHashMap": [],
                "events": [],
                "resourses": [
                    "http://crawl-test.mmmkay.info/user/Tommy.json"
                ]
            }
        ],
        "deferredEvents": [],
        "startTime": 1391266464847,
        "endTime": 1391266466787,
        "resourses": [
            "http://crawl-test.mmmkay.info/"
        ]
    }
]

See Ajax events for user data requests? This is just what we need! I agree, the current debug output is a bit redundant, but informative. It can be seen from it that he successfully discovered additional requests when processing the events of the “More info” buttons, which can be refocused in the future. In addition, thanks to fixtures in the web application, he was able to immediately detect XSS in mutations, which he kindly reported. So far it’s not working very fast, but I am actively working on it in my free time. Another great example is linkedin, here is the result of work for 5 of its pages (starting from the main one):

Green nodes - processed events of elements
Blue - resources that were requested during their processing
As you can see, in similar web applications with a lot of “chain” events this approach can be effective!

Total

I think in the future to develop this idea into a full-fledged web application security scanner and add it to the rest of the bindings (if the manual allows), maybe in the form of a plug-in for W3af or Minion. But this still leaves a lot of unresolved issues related to performance and critical features.
I hope I’m not tired of you very much and my attempts will be useful to someone if I couldn’t get something to convey clearly - say, I’ll try to break the cover.