Analysis of HTTP traffic with Mitmproxy

In the practice of a web developer, situations often arise when it is necessary to track and analyze the traffic of applications that communicate with the server via HTTP (for example, testing applications for mobile devices or the HTTP API).

The tools traditionally used to listen to traffic (tshark, which we already wrote about , as well as ngrep and tcpdump) are not suitable for this purpose: they have limited functionality for working with the HTTP protocol.

There is a more specialized, simple and effective solution for analyzing HTTP traffic. Meet: mitmproxy . In Russian, there are almost no detailed publications about him. In this article, we will share our experience with mitmproxy and hope that you will find it useful.

general information

Mitmproxy is a whole set of software tools that includes:

• mitmproxy itself is an interactive console program that intercepts traffic "on the fly";
• mitmdump  is a utility that can be described as an analogue of tcpdump for the HTTP protocol: it intercepts traffic and saves all information about it to a text file;
• libmproxy  is a library for Python that implements all mitmproxy functionality.

The name mitmproxy comes from the abbreviation MITM, which means man in the middle, or "man in the middle." This is the method of compromising a communication channel in which an attacker connects to a transmission channel between two counterparties and intervenes in the transmission protocol, viewing, deleting and distorting information. mitmproxy works in a similar way: it is used as a proxy server, registering all HTTP traffic. Like any proxy server, in some cases mitmproxy can modify both user requests and responses to them.
Consider the principles and features of mitmproxy in more detail.

How it works

In the case of unencrypted HTTP connections, everything is simple: mitmproxy accepts a connection from a client (for example, from a browser on a mobile device), displays information about it on the console (or saves it to a text file), and then returns a response from the recipient of the request to the client.

Mitmproxy can also be used to intercept secure HTTPS traffic. When mitmproxy or mitmdump is launched, a set of CA files are created in the ~ / .mitmproxy directory, based on which substitute certificates are generated. Naturally, the browser will identify these certificates as suspicious, displaying a corresponding warning with each attempt to establish an SSL connection using mitmproxy.

To prevent this warning, you can add a certificate from mitmproxy to the list of certificates used by the browser (detailed instructions can be found here ).
When these two conditions are met, the client concludes that the connection being established is secure.

Mitmproxy can also intercept secure HTTPS traffic. The interception procedure consists of the following steps:

1. The client establishes a connection with mitmproxy.
2. Mitmproxy sends a 200 response to the client (connection established).
3. The client interacts with mitmproxy in the same way as with the remote server, and establishes an SSL connection. To specify the host name, it uses SNI.
4. Mitmproxy connects to the server and establishes an SSL connection using the host name specified by the client.
5. In response, the server sends an SSL certificate containing the values ​​of the CN and SAN parameters, on the basis of which a replacement certificate will then be created.
6. Mitmproxy generates a fake certificate and continues the SSL dialog with the client, suspended at step 3.
7. The client sends the request through the established SSL connection.
8. Mitmproxy sends the request to the server through the SSL connection established in step 4.

More clearly, the process of intercepting protected traffic can be represented in the form of the following graphic diagram:

Why is it necessary

The name mitmproxy comes from the name of one of the most common types of attacks. Even the official product documentation is replete with such words as “attack”, “interception” and the like. All this suggests that this tool can act as a hacking tool. Of course, mitmproxy (like all products with a similar set of functions - the so-called sniffers) can well be used for illegal purposes, but for obvious reasons we will not discuss this and will focus on legal use cases.

Mitmproxy can be used, firstly, for testing and debugging web applications. With it, you can get detailed information about what requests the application makes and what answers it receives. Also mitproxy can help in studying the functioning of some REST APIs, in particular poorly documented and using closed (and often very suspicious) technologies.

Secondly, Mitmproxy can operate in a transparent proxy mode with traffic interception , which means that it can be used to analyze the network activity of suspicious applications.

Testing Mitmproxy

Installation

Today mitmproxy is included in the linux repository and can be installed using the standard package manager:

$ sudo aptitude install mitmproxy

You can also install it in other ways:

$ pip install mitmproxy

or

$ easy_install mitmproxy

First start

Let's look at specific examples of how mitmproxy works. We’ll open the browser (in our case, it’s Firefox) and in the settings (menu “Settings” → “Network” → “Connection”) and in the section “Manual settings of the proxy service”, we will indicate the machine on which mitmproxy is installed as a proxy server.

Now connect to the server where mitmproxy is installed via ssh and run the following command:

$ mitmproxy

After that, the console will look like this:



To exit this mode, press q. Help can be obtained by pressing the key combination denoting a question mark (?).

Now open in the browser any site - for example, ya.ru. All requests to this site will be displayed on the console:



You can navigate through the list of requests by pressing the arrow keys. You can return to the main window by pressing the q key. To view detailed information about a certain request, you need to move the cursor to it and press the Enter key:



The Request field displays detailed information about the request (the requested host, the software with which the request was made, transmitted headers), and the Response field displays information about response received.

You can switch between these fields using the Tab key. You can return to the list of queries by pressing the q key.

Requests and responses to them can be changed. To do this, use the so-called interception filters. To enter a filter, press the i key. Let’s introduce as a filter, for example, ya.ru All requests containing this address will be intercepted. Intercepted requests in the list will be highlighted in orange:



Such requests will not be processed if we do not accept them. To accept the request, you need to move the cursor to it and press the key a, and to accept all intercepted requests - the key A.

You can view more detailed information about the request by moving the cursor to it and pressing the E key (E is the first letter in the English word event - “event”). The event log that took place during the processing of this request will be opened:



Both requests and responses can be edited. The editing function may be useful during testing: for example, you can simulate a certain situation and see how the application will behave, having received a definite answer from the server.

We will bring the cursor to the query that interests us, bring the cursor to it and press the Enter key. Then move the cursor to the Request field and press the E key (the first letter in the word edit is edit). The edit menu appears at the bottom of the console:



You can change both the entire request (Q key) and its individual parameters: path (P key), URL (U), title (H), form (F), body ® and method (M).

Similarly, editing the response. You can edit its code (key C), message (M), headers (H) and body ®.

Additional functions

Proxy Authentication

In mitmproxy, you can activate user authentication mode before using a proxy. Authentication headers are removed from requests and transferred to higher servers. Currently, only basic HTTP authentication is supported. Authentication is configured using the following options:

# forbid anonymous users to use proxies
$ mitmproxy --nonanonymous 
# allow proxy use only to specified user
$ mitmrpoxy --singleuser <username> 
# allow proxies to be used only when entering one of the passwords specified in the file;
$ mitmproxy —htpasswd <path to password file>

Cookie binding

The cookie binding function (sticky cookies) is useful when working with services that require authorization. It is enough to log in to such a service once, and mitmproxy will automatically add the appropriate cookie to each request. After that, all requests will be transmitted to the server without re-authorization.

The cookie binding mode is activated as follows:

$ mitmproxy -t <filter>

You can also add authorization headers to all proxied requests. The -u option is used for this.

Reverse proxy mode

In this mode, all requests are sent to the upstream server. Mitmproxy in this case can be used as a temporary layer by observing and intercepting requests.
Reverse proxy mode is activated using the command:

$ mitmproxy -P http [s]: // hostname [: port]

Anticache function

Mitmproxy can remove if-modified-since and if-none-match headers from the request. Thanks to this, you can always view the full response from the server, even if the browser reports that the requested document is in the cache.

This function is activated using the following command:

$ mitmproxy --anticache 

Play client requests

The client side replay function allows you to play requests from previously saved HTTP dialogs. Requests are executed one after another: sending one request, mitmproxy waits for a response from the server, and only then proceeds to the next one. Therefore, the client’s behavior may differ from that recorded in the saved dialog, in which some requests could be executed simultaneously.

You can play client requests using the command:

$ mitmproxy -c <path to saved dialog>

Mitmdump

As mentioned above, Mitmdump is a utility that works in exactly the same way as tcpdump, only for the HTTP protocol. It intercepts all HTTP traffic and writes information about it in a separate text file.

To get started with mitmdump, you need to run mitmproxy in proxy mode, and then run the following command:

$ mitmdump -w <file name>

The saved information can be filtered using regular expressions, and then saved to a new file:

$ mitmdump -nr <file1> -w <file2> "~ m post"

In the above example, mitmdump selects requests from file 1 that meet a certain criterion (in our case, POST requests) and writes them to file 2.

Mitmdump can read already saved information about client requests from a file, play these requests again, and save the results to a new file:

$ mitmdump -nc <file 1> -w <file 2>

This feature may be useful when testing some web applications.

Conclusion

In this article, we gave a brief overview of mitmproxy features. For those who want to know more, here are a few links:

• official mitmproxy documentation ;
• article about tracking the traffic of mobile applications ;
• Description of the principles of Mitmproxy from the developer - Aldo Cortesi .

Readers who for one reason or another are not able to leave comments here are welcome to our blog .