Cross-browser web extension for custom scripts Part 1

    In this series of articles, I want to talk about my experience writing a web browser extension. I already had the experience of creating a web extension, which was installed by about 100,000 Chrome users who worked autonomously, but in this series of articles I decided to delve into the process of developing a web extension by tightly integrating it with the server part.

    imageimageimageimageimage


    Part 2 , Part 3 , Part 4

    Idea for web extension


    Each developer is faced with the tasks of automating processes in the browser. Once I had the task of gathering participants in groups of a specific subject on LinkedIn and Facebook.

    Before that, I had the experience of writing web-parser solutions for php, but these sites use many dynamic elements and the focus of the solution for this task shifted precisely to interaction with web resources through the browser.

    Of course, I was not going to violate the terms and conditions, therefore, what is described below is pure fruit of my imagination and is not recommended for implementation.

    It was possible to quickly open the developer console and write a javascript script that would imitate the user's actions, after which I could collect data. This approach partially worked with Facebook, where group members and their information can be gathered on one page. But it did not work for LinkedIn, where you need to open the page of each participant.

    After searching for third-party solutions, I chose iMacros for this task. This extension has its own language for writing macros. Somehow I adapted it to solve the problem for LinkedIn. I had to download the old version of Mozilla Firefox, since the extension did not work for a multithreaded implementation of this browser.

    During the search for third-party solutions, I ran across many variations of web-parser, web-crawler, web-scraper, etc. Many were sharpened on sites with a page-by-page transition, and not on dynamic content. Some solutions offered to install a software solution for the operating system, and then use a web extension for such a complex. There were also very highly specialized items, for example, to collect participants only on Facebook.

    After all the torment, I had the idea of ​​“inventing my bike” to automate tasks in the browser. Among the mandatory requirements for my web extension, I highlighted:

    1. Work in the maximum number of web browsers, including mobile.
    2. Using the standard scripting language for browsers is javascript.
    3. Using your own data files in scripts.
    4. The ability to save data received from the script to a file.
    5. Scripts must be either public or private. If the script is public, then the security team needs to check this script.

    Further from this list I noted more specific things on each item.

    1. It is necessary to use the framework for writing web extensions to minimize the efforts to develop cross-browser solutions.
    2. Javascript is supported by all browsers, but to interact with the web extension you need to write your own library. This library should have functions for storing data, retrieving data from downloaded files, etc.
    3. It is often necessary to use input data for script operation. For example, data for authorization, keys for the API, a list of pages for crawling, etc. Such data should be downloaded directly to the web extension and stored in the cloud.
    4. Saving data from a script is one of the most necessary functions for automation. Saved data should be uploaded to csv directly from the web extension, or sent to the user's mail when the data limit for unloading is exceeded. For example, if uploading group members is more than 10,000, then downloading from a web extension can take a long time. This should be avoided by generating a file on the server.
    5. You must have an administrative web interface of the security team to check the user scripts that they have marked as public.

    From time to time I found more and more new solutions to the same problem. - running custom scripts in the browser. All these decisions did not fit my requirements, since I wanted to have a web extension that would be able to remove ads on every page; convert the page and show me only its content, without unnecessary markup; collect data from any page and convert to a convenient format for further use; collect data after a certain period of time; and so on.

    Therefore, I violently “caught on fire” by creating an extension to run custom scripts for the maximum range of tasks.

    Choosing a framework for web extensions


    Since I initially aimed at the maximum number of web browsers, I needed a framework for building cross-browser extensions. We can immediately note the fact that such a framework simply does not exist. Because many browsers, though they work in a similar way and provide web extensions with a similar API for interaction, differ fundamentally.

    I was forced to give up the initial version of support for mobile browsers almost immediately. Since no one such browser does not provide the ability to use web extensions. I came across only one mention of web extension support in the Dolphin browser, but I could not find detailed information on the official website. It was decided to abandon this bright idea.

    When writing my old extension, I used the kango framework. In 2013, he was as comfortable as possible. Albeit without Internet Explorer support.

    Since my extension could work as a bookmarklet, I did not pay attention to this fact and chose this framework, which for its time was just the perfect solution for cross-browser development.

    Since then, other projects have emerged that have set themselves the goal of cross-browser development of web extensions. I found all these projects at an early stage of development. Their task has since been simplified, since Mozilla Firefox began to use the WebExtensions API, which made it possible with little effort to turn extensions for Chrome into extensions for Firefox.

    Extensions for the Opera browser already in 2013 were compatible with extensions for Chrome. Extensions for Safari now work only for MacOs, because support for the Windows platform of the Safari browser itself long ago stopped.

    The Tor browser runs on the old Mozilla Firefox engine and therefore supports web extensions in xpi format, which the Mozilla Foundation declined in favor of Web Extension technology.

    In fact, the kango framework almost copes with its task to this day, since it generates web extensions for all browsers, with the exception of Internet Explorer. But since a lot of time has passed and Firefox now works on a mechanism similar to Chrome, kango generates the same package for two browsers. I had to slightly modify the generation for Firefox and add the generation for Tor.

    Since the status of the kango framework project is not clear, as well as the license for the code, I cannot share my changes with the public domain. If copyright holders are allowed to release open source version 1.9.0, then I will be happy to help in this endeavor.

    All web extensions have two data points. The content.js file allows you to manipulate the DOM, while background.html allows you to work with background data and server interaction. Communication between these two points occurs through the message mechanism.

    Thus, to modify the DOM data from the server side, we need to get them from background.html and use the message passing mechanism to use them in content.js.

    Such an ideal mechanism did not work for my case for a number of reasons. Therefore, I left in the background.js only the logic of working with pop-up and the web extension button in the browser.
    The logic of any web extension with the pop-up window is the same. By clicking on the button, we simply show the pop-up, on the second click, we close it.

    The Kango framework offers the developer a unified interface for interaction, and then translates the code written in the web extension into a web extension for a specific web browser, and therefore saves a lot of time on developing cross-browser web extensions.

    In the next article I will discuss the choice of “Framework for the server part of the web extension and the interface of the web extension”

    Also popular now: