Developing a VPN Client for Android (Part 1)

    Hello! The reason for writing this article was the realization that, in the presence of a large number of articles and reviews about VPN client applications for Android, there is not a single normal article describing development problems using the VpnService API. Moreover, in most cases, you, as an application developer, cannot do anything with these problems.

    Let's start from the very beginning. Some time ago, our company, based on research in the field of mobile device security, decided to release a small application ( WebGuard) under Android to block all already annoying ads, as well as protection from snooping and viruses when working from any browser. To implement this functionality, we needed to solve many problems, the most time-consuming of which was the task of intercepting and processing application traffic. To intercept and filter browser connections, it was decided to use the VpnService API, which appeared in Android with version 4.0.3 and provides all the necessary functionality (although at times this functionality just doesn’t work for a bunch of different reasons, but it turned out a bit later).

    A bit about the technologies used

    Here it’s worth telling in more detail why this API was chosen and what generally exist for intercepting the network traffic of applications with another application on an “unrouted” android device. Actually, there are only three ways (apart from various vulnerabilities) and it will be useful for application developers with various in-game purchase mechanisms to know about them. Since many of them believe that replacing the network traffic of an application is possible only on a “rooted” device, they do not “bother” with the protection of transmitted data, although there is a probability (and rather high) of the appearance of “cheating” applications that make changes to the transmitted data.

    So, the first way is to install a local proxy server. Most antiviruses for Android use this method. To intercept traffic, it is enough to implement the HTTP proxy support described in the RFC 2068 standard and set the proxy server or APN in the settings of the mobile network in the WiFi network settings . In this case, the data transfer will occur as in the diagram below.

    But this method has a bunch of serious problems:

    • Installing a WiFi proxy or APN for a mobile network programmatically is possible only using a hidden API (for example, hidden methods in WifiManager);

    • you need to monitor the emergence of new WiFi or mobile networks by installing proxies (APN) for them;

    • This method only works for connections created using the Http (s) URLConnection class and then, unless you explicitly specify not to use a proxy (for example, url.openConnection (Proxy.NO_PROXY)). There are also applications that check for a proxy server and use it (or vice versa do not use it);

    • if the user removes or simply does not start the application with the proxy server, then the rest of the applications will not be able to connect to any server since the proxy will still be specified in the network settings and the user will need to remove it from the settings ( for example );

    • You cannot build a chain of several proxies, respectively, and an application that intercepts traffic can be only one.

    All these problems, except the last one, are successfully solved in the second way - by writing a VPN client application using the VpnService and VpnService.Builder classes. It is enough for the application to call a couple of functions (mandatory checks are omitted in the code):

            // показываем Activity для запроса прав у пользователя
            Intent intent = VpnService.prepare(PromptActivity.this);
            startActivityForResult(intent, VPN_REQUEST_CODE); // запрос прав

            // и в onActivityResult если нам выдали права
            // requestCode == VPN_REQUEST_CODE && resultCode == RESULT_OK
            VpnService.Builder vpnBuilder = new VpnService.Builder();
            vpnBuilder.addAddress(options.address, options.maskBits);
            vpnBuilder.addRoute("", 0);
            ParcelFileDescriptor pfd = vpnBuilder.establish();
            FileInputStream in = new FileInputStream(parcelFileDescriptor.getFileDescriptor());
            FileOutputStream out = new FileOutputStream(parcelFileDescriptor.getFileDescriptor());

    , and all TCP / IP traffic of all applications (even those running as root) will be redirected to the TUN interface that Android will create, and your application will have read / write device file / dev / tun (or / dev / tun0, / dev / tun1, etc.), from where you can read outgoing network packets, send them for processing to a remote VPN server (usually via an encrypted connection), and then record incoming network packets. In order to prevent the VPN client’s connections from “wrapping up” in TUN, the VpnService.protect method is used on the TCP or UDP sockets created by the application.

    The scheme presented above, in this case, will look as follows:

    This method has two features:

    1. An application needs to obtain rights to use VpnService through a call to startActivityForResult, and the system will show the user this dialog:

      Moreover, Android remembers that it issued the rights to the application, only until the device is rebooted, therefore, after rebooting, the rights must be requested again. As it turned out (we did not even expect this) there are users (“yes there are hundreds of them” ©) who like to restart their phone every 10 minutes and this window bothers them, for which they can rate the application 1 point in the market;

    2. "Sadness" is a little larger than the previous one. After turning on the VPN, Android, without fail, displays the notification icon in the form of a key and the notification itself, by clicking on which you can see some statistics:

              Notification in the "curtain". Notification Icon and Status Dialog

      Here, user dissatisfaction was expected, because many current applications just love to "hang up" notifications (sometimes several at once) and the status bar turns into a New Year's garland:

      Also, if the user clicks the “Disconnect” button, Android will disconnect the VPN and take away the rights from the application, after which, to activate the VPN, you will have to re-request the rights through a call to startActivityForResult.

    There is also a small additional problem in this method (not counting the “features” about which below) - for it to work, you need a remote server.

    The problem with the presence of the server is solved in the third way (slightly modified method No. 2) - the VPN client application also contains the TCP / IP stack (you can take it ready or write it yourself because there are flaws in the ready ones) to parse traffic from applications and process connections as a proxy server. Then the application traffic processing scheme will change somewhat and will look like this:

    This is the method we use in WebGuard. Of the drawbacks, compared with the previous method, only one can be noted - the impossibility of normal processing of protocols other than TCP or UDP (or protocols "on top" of them), because the application will need to create "raw" sockets, which usually require root privileges a. To make it clear what we are talking about, let's take a simple example: a user launches a shell via ADB and executes the “ping www.ya.rucommand , which sends ICMPecho request. Further, the VPN client application reads the IP packet from / dev / tun, parses it, and finds out that the packet contains an ICMP echo request to a certain server. And since the application cannot send the request further to the network, it has only two options: ignore the packet or emulate ping trying to establish a connection to the desired server and, if successful, record a fake ICMP echo reply in / dev / tun.

    Features VpnService API

    In the process of developing the application, testing and using the first versions by users, we encountered a lot of errors or shortcomings related to the VpnService API. Some of them were fixed, because in essence these were flaws in our programmers (which we honestly wrote about and got to, and with the rest, it is rather difficult or impossible to do:

    • There is no support for the TUN interface in the linux kernel , so the VPN will not work. Basically, this problem occurs on self-assembled firmware based on the projects CyanogenMod, AOSP, etc. The authors of assemblies either remove support from the kernel altogether or forget to put the tun.ko module in the firmware. It seems that the authors are guided by the principle of "I don’t know what it is, therefore, it is not necessary";

    • There is no VpnDialogs.apk file that contains dialogs for requesting rights and statistics; VPN will not work again. Oddly enough, this error most often happens on official assemblies from phone manufacturers ( including Google );

    • GUI freezes on some Samsung phones with official firmware if an unhandled exception occurs in the VPN client (at the same time, the shell through ADB is quite working, but reconnects every 1-2 minutes). In general, manufacturers of Android phones sometimes collect “wonderful” firmware for them, forgetting to check part of the “rarely used” functionality. Our development team in this regard simply “loves” Samsung and believes that for testing applications for Android it is necessary to have several phones from different manufacturers and several different phones from Samsung;

    • Thanks to this change, the user will not be able to check the "I trust this application" checkbox in the VPN dialog if it is blocked by another activity that does not allow user clicks. This is done so that it would be impossible to draw another dialog from above and trick the user into checking the box. The problem is that various utilities, launchers and some programs (for example, “Read! Free”) usually do this (and for a long time), creating a transparent activity. As a result, some users may think that they are being bullied;

    • If the user turns on traffic restriction for background processes, then a working VPN client will also fall under the distribution (when sending data through the socket, there will be a SecurityException with Permission denied exception), accordingly, there will be no Internet kina ;

    • Linux has such a wonderful mechanism as OOM Killer , and if the user suddenly launches an application that will “devour” a large amount of the device’s RAM (you can use Firefox for experiments, since there’s always little memory available), OOM Killer will start to “dump” everything in a row user processes except, as usual, the hero of the occasion (more precisely, he will be killed last). Of course, there are no exceptions for the necessary processes (launcher or VPN client), so you need to be prepared to constantly restart your service;

    • After the VPN client shuts down incorrectly (even if it later restarts), in rare cases, it may be necessary to reboot the device because Android will incorrectly reconfigure packet transfer routes and packets from applications will go to / dev / null ;

    • If the user enables the distribution of mobile Internet via WiFi simultaneously with the VPN client, then on most firmware, as in the case above, Android will incorrectly configure packet routing;

    • Despite a decent list of “features”, developers from Google “suddenly” can decide to redo to improve the functionality in the new version, and developers for Android usually do not warn about all changes, so you need to carefully monitor the commits to the Android repository and edit your code. The result of this approach is most often felt by users of Nexus phones, who receive the update faster than anyone else (read the heartbreaking requests in the style of "PLIZZZZZZZZZ !!!!!!!!!!!!! VPN !!!!!!!!!!!! !!! ”,“ Please fix this issue ”for over 230 days here , here and here ). So, be careful not to update your Nexus during your business trip.

    So you’ll read this list and think about it - maybe this VPN is it? Unfortunately, there is no other way to intercept all traffic on an “unrouted” Android phone.

    A bit about NinePatch

    The list above did not include another problem, the consequences of which were very unexpected for us (and not only for us). Therefore, we decided to talk about the consequences in more detail. It all started with the fact that at some point it was necessary to make the most important part of the application - a boring icon. No sooner said than done. A beautiful round icon is drawn and we are joyfully testing the pre-release version, when suddenly:

    It turned out that on some phones the application icon in the VPN notification can be very strange displayed (incorrect color, size or something else). After experimentation and a short but heated discussion

    , it was ordered it was decided to make the application icon in nine-patch format, the benefit of the official documentation on this subject does not say anything (that is, does not prohibit) and the egg-shaped problem is solved. But, after the release of the release, “victims” appeared very quickly. These were both Android applications and various online services working with apk files, and not expecting to receive the application icon in nine-patch format. We decided to place the most noteworthy on a pedestal of three places:

        3. Various launchers that crashed when trying to display an icon (for example LauncherPro).

        2. Stores Android applications from Samsung and Yandex. When I tried to load the application into Yandex.Store, a completely understandable error description was issued: “Failed to extract the application icon from the APK”. C Samsung Apps was more fun. Since the company is high-tech, the test results when adding the application come in the appropriate form - an email with a link to the video in which the application testing process is recorded and an error should be seen (in theory). It turned out, however, everything was as usual , a letter came with a link by which the video was not.

        1. Well, the honorable first place, by right, is occupied by Sony with an Xperia phone. Some time after the release of the release, the owners of Xperia L began to send messages that WebGuard “killed” their phone ( example) It turned out that PackageManagerService crashes when trying to process the application icon during installation, after which the phone automatically reboots and downloads endlessly:

    E / AndroidRuntime (790): *** FATAL EXCEPTION IN SYSTEM PROCESS: Thread-123 
    E / AndroidRuntime (790): java.lang.ClassCastException: cannot be cast to
    E / AndroidRuntime (790): at $ (PackageManagerService.javahaps672)
    E / AndroidRuntime (2981): FATAL EXCEPTION: ApplicationsProviderUpdater
    E / AndroidRuntime (2981): java .lang.RuntimeException: Package manager has died
    E / AndroidRuntime (2981): at (
    E / AndroidRuntime (2981): AT (
    E / AndroidRuntime (2981): AT (
    E / AndroidRuntime (2981): at $ 300 (
    E / AndroidRuntime (2981): at $ UpdateHandler.handleMessage ( : 206)
    E / AndroidRuntime (2981): at android.os.Handler.dispatchMessage (
    E / AndroidRuntime (2981): at android.os.Looper.loop (
    E / AndroidRuntime (2981): at (
    E / AndroidRuntime (2981): Caused by: android.os.DeadObjectException
    E / AndroidRuntime (2981): at android.os.BinderProxy.transact (Native Method)
    E / AndroidRuntime (2981): at $ Stub $ Proxy.queryIntentActivities (
    E / AndroidRuntime (2981): at (
    E / AndroidRuntime (2981): ... 7 more

    As a result, from version 1.3 it was decided to use the icon without nine-patch, especially since its use of all the problems with displaying the icon did not solve:

            VPN Enable Notification on Ainol Novo10 Hero


    We did not want to make too big a post, so it was decided to split the article into several parts. In the next part, we will explain why Android applications that filter traffic consume so much battery power (according to the android) and recall, using one performance test as an example, that DalvikVM! = JavaVM.

    We hope our article will help someone in writing an interesting application for Android. Have a good development!

    Also popular now: