Studying the Lorenz attractor, and then editing the code in Iodide

In the last ten years there has been a real explosion of interest in “scientific computing” and “data science”, that is, the use of computational methods to find answers to questions, analyze data in the natural and social sciences. We see the flowering of specialized PLs, tools, and methods that help scientists explore and understand data and concepts, as well as communicate their findings.

But today, very few scientific tools use the full communication potential of modern browsers. The results of data mining are not very convenient to view in a browser. So today Mozilla presents Iodide - An experimental tool that helps scientists create beautiful interactive documents using web technologies, all within the framework of an iterative workflow that is familiar to many.

It is not just a programming environment for creating interactive documents in a browser. Iodide tries to help with the workflow by linking the editor and the preview. This differs from the IDE style, which produces presentation documents of the pdf type (they are then separated from the source code). And differs from the style of notebooks with cells that mix code and presentation elements. In Iodide, you see both a document that looks the way you want and easy access to the base code and editing environment.

Iodide is still in the alpha version, but in the Internet industry it is customary to say:"If the first version of your product does not bother you, you are late with the launch . " Therefore, we decided to make a very early launch in the hope of receiving feedback from the community. We have a demo that you can try right now , but there are still many shortcomings (please do not use this alpha for important work!). We hope that you will close your eyes to the doorposts and understand the value of the concept itself, and your feedback will help you to understand in which direction we should move further.

How we came to Iodide

Data Science at Mozilla

Mozilla's data science is almost entirely communications-based. Although we sometimes deploy data mining models directly in front of users, such as the mechanism for recommending extensions for the browser, most of the time our experts analyze the data to identify patterns and share information with engineers, product managers, and management.

Data science involves writing a lot of code, but unlike traditional software development, our goal is to answer questions, not create software. This usually comes down to creating some kind of report - a document, diagrams or interactive data visualization. Like everyone else, we at Mozilla study our data with fantastic modern tools like Jupyter and R-Studio. But when it comes time to share the results, we usually can’t just pass the Jupyter notebook or R script to the “customer”, so we often have to copy the key numbers and summary statistics into a Google document.

As it turned out, it’s quite difficult to switch from studying data in code to a digestible explanation and vice versa. Studies show that this is a common problem . When one scientist reads someone else's report and wants to see the corresponding code, many problems arise: sometimes it is easy to track the code, sometimes not. If a specialist wants to experiment by changing the code, obviously, it’s still getting complicated. Another scientist may have your code, but the configuration on the machine is different, and the setup takes time.


Useful Duty Cycle in Data Science

Why is there so little web in science?

Against this background, at the end of 2017, I started a project on interactive data visualization in Mozilla. Nowadays you can create such visualizations using the excellent libraries in Python, R and Julia, but for my project it was necessary to switch to Javascript. This meant a way out of the familiar environment of data science. Modern web development tools are incredibly powerful, but extremely sophisticated . Against my desire, I had to create a full-fledged chain of Javascript tools for building with a hot reload of modules, but it was still impossible to find a normal editor that generates clean, readable web documents in a lively, iterative workflow.

I began to wonder why there is no such tool — why there is no Jupyter analogue for interactive web documents — and I wondered why almost no one uses Javascript for scientific computing. There seem to be three important reasons for this:

1. Javascript itself in the scientific community has a controversial reputation as a slow and uncomfortable language.
   
2. Not many scientific computing libraries work in a browser or support Javascript.
   
3. I found a shortage of scientific programming tools with support for fast iterations and direct access to the presentation capabilities of the browser.

These are very big problems. But working in a browser has some real advantages for such a “communicative” data science that we study at Mozilla. Of course, the biggest advantage is that the browser has the most advanced and well-supported set of data visualization technologies: from DOM to WebGL , Canvas and WebVR .

Reflecting on the communication difficulties mentioned above, I came up with another potential advantage: in the browser, the final document does not have to be separated from the tool that created it. I wanted to make a tool for iterative scientific work with web documents (a web application with specific functionality) ... and many of our tools were essentially web applications. For these small web document applications, why not link the document to the editor?

Thus, readers without technical training can view a beautiful document, and the scientist instantly switches to source code mode. Moreover, since the calculations take place in the browser JS engine, the scientist can immediately begin experimenting with the code. And all this without connecting to remote computing resources or installing any software.

The advent of Iodide

I began to discuss with colleagues the potential pros and cons of scientific computing in the browser, and during the conversations we noted other interesting trends.

Inside Mozilla, many interesting demos came out on WebAssembly , a new platform for launching code in a browser written in languages ​​other than Javascript. WebAssembly allows you to run programs at an incredible speed, in some cases close to native binaries. WASM runs resource-intensive processes without problems, even whole 3D game engines . In principle, you can compile the best C and C ++ numerical libraries in the world for WebAssembly, wrap them in ergonomic JS APIs, as the SciPy project for Python does. In the end, such projects already exist..


WebAssembly lets you run code in a browser with almost no overhead.

We also noticed that the Javascript community is ready to introduce new syntax if it helps people more efficiently solve their problems . Perhaps you should try to emulate some of the key syntax elements that make numerical programming more understandable and flexible in MATLAB, Julia, and Python - this is matrix multiplication, multidimensional slicing, broadcast array operations, and so on. And again we found that many agree with us .

All these premises lead to the question: how suitable is the web platform for scientific computing? At a minimum, it can help with communications in the processes that we encounter at Mozilla (and which many in the industry and academia encounter). With the ever-improving Javascript core and the ability to add syntax extensions for numerical programming, JS itself may become more attractive to scientists. WebAssembly seemed to allow the use of serious scientific libraries. The third leg of the chair is the web environment for creating scientific documents. On this last element, we focused our experiments, which led us to Iodide.

Anatomy Iodide

Iodide is a tool that provides scientists with a familiar workflow for creating great interactive documents using the full power of a web platform. The work is built in the form of “reports” - in fact, this is a web page that you fill out with your content. Plus some tools to iteratively examine the data and modify the report to create the final document. Once it is ready, you can send a link directly to it. If colleagues and employees want to view the code, then with one click switch to study mode. If you want to experiment with the code and use it as the basis for your work, one more click becomes a fork.

Next, we will talk about some experimental ideas on how to make this workflow more flexible.

Report Mode and Study Mode

Iodide seeks to link research, explanation and collaboration in one place. The central place here is the ability to move between a beautiful report and a useful environment for iterative research with scientific calculations.

When you create a new Iodide Notebook, you enter Explore View. This is a set of panels, including an editor for writing code, a console for viewing output data, a tool for viewing the workspace for examining variables created during a session, and a panel “Preview Report”.


Editing Markdown code in Iodide research mode

By clicking on the Report button in the upper right corner, you can expand the contents of the preview panel to the entire window. Readers who are not interested in technical details can focus on this presentation of the document without delving into the code. When the reader clicks on the report link, the code starts automatically. To view the code, click the Explore button in the upper right corner. From there, you can make a copy of the notebook for your own research.


Transition from research mode to report mode

Whenever you share a link to an Iodide notebook, your coworker always gets access to both of these views. A clean, readable document is never separated from the underlying code and editing environment.

Live, interactive documents with the power of a web platform

Iodide docs live in a browser. This means that the computing engine is always available. Each document is a live interactive report with running code. Moreover, since the calculation takes place in the browser simultaneously with the presentation, there is no need to call the language backend in another process. Thus, interactive documents are updated in real time, opening up the possibility of smooth 3D visualizations . Low latency and high frame rate even meet VR requirements .


Contributor Devin Bailey examines her brain MRI data

Sharing and reproducibility

Reliance on the web simplifies a number of workflow elements compared to other tools. Sharing is implemented natively: the document and code are accessible at the same URL and you do not need, say, to insert a link to the script in the Google Docs footnotes. The computational core is a browser, and libraries are loaded with an HTTP request, like any script - no additional languages, libraries or tools are required. And since browsers provide compatibility, Notepad looks the same on all computers and OSs.

To ensure collaboration, we created a fairly simple server where notepads are saved. There is a public instance of iodide.ioto experiment with Iodide and publish your work. But you can create a private instance behind the firewall (we at Mozilla do this for some internal documents). But it is important to note that the notebooks themselves are not tied to a single Iodide server. If the need arises, it is easy to transfer the work to another server or export the notebook as a package for sharing on other services, such as Netlify or GitHub Pages (for more information on exporting packages, see the “What's Next?” Section below). Passing computing to the client allows us to focus on creating a truly excellent environment for exchange and collaboration, without having to allocate computing resources in the cloud.

Pyodide: Python science stack in browser

When we started thinking about making the web better for scientists, we focused on ways that can make working with Javascript easier, such as compiling existing scientific libraries into WebAssembly and packing them into simple JS APIs. When we outlined the idea to the developers of WebAssembly in Mozilla , they proposed a more ambitious idea: if many scientists prefer Python, then go to their field - compile the Python scientific stack to run in WebAssembly.

We thought it sounded intimidating, that it would be a huge project, and that it would never provide satisfactory performance ... but two weeks later , Mike Droettboomwas a working Python implementation, working inside an Iodide notepad. Over the next few months, we added Numpy, Pandas, and Matplotlib, the most used modules in the Python science ecosystem. Thanks to the help of Kirill Smelkov and Roman Yurchak from Nexedi , support for Scipy and scikit-learn appeared. Since then, we continue to slowly add other libraries .

Running the Python interpreter inside the Javascript virtual machine adds overhead to performance, but they are surprisingly small. Compared to native code, in our tests the code runs 1-12 times slower in Firefox and 1-16 times slower in Chrome. Experience has shown that productivity is enough for comfortable interactive research.


Matplotlib in the browser supports interactive features not available in static environments.

Moving Python to the browser creates magical workflows. For example, you can import and process data in Python and then access the resulting objects from Javascript (in most cases, the conversion happens automatically) to display them using JS libraries such as d3 . Even more magically, you access the browser API from Python code, for example, to manipulate the DOM without using Javascript .

Of course, much more can be said about Pyodide and it deserves a separate article - we will consider it in more detail next month.

JSMD (JavaScript MarkDown)

Like Jupyter and R-Markdown in R, the Iodide editor allows you to freely alternate code and notes, breaking the code into pieces that change and run as separate units. Our implementation of this idea corresponds to the implementation of R Markdown and the “cell mode” in MATLAB: instead of the explicitly cell-based interface, the contents of the Iodide notepad are just a text document that uses special syntax to delimit certain types of cells. We call this text format JSMD.

Following MATLAB, code fragments begin with signs%%followed by a string indicating the language. We currently support snippets containing Javascript, CSS, Markdown (and HTML), Python, a special 'fetch' snippet that simplifies resource loading, and plugins that extend the functionality of Iodide by adding new cell types.

We found this format quite convenient. It simplifies the use of text tools such as diff viewer and your own favorite text editor. You can perform standard text operations (cut / copy / paste) without the need to learn cell management commands. See the JSMD documentation for more details .

What's next?

It’s worth repeating that this is only an alpha version: we continue to polish the interface and fix errors. But in addition to this, there are a number of ideas for the following experiments. If any of these ideas seem particularly useful to you, let us know! Better yet, help develop!

Advanced Collaboration Features

As already mentioned, we created a very simple backend that allows you to simply save work, view work done by others, quickly fork and expand other people's notebooks. But these are only the first steps in a collaborative workflow.

Now we are looking at three more great features:

1. Google Docs style comment threads.
   
2. The ability to propose changes in someone else's notebook through the fork / merge mechanism, as in GitHub.
   
3. Editing notebooks at the same time, as in Google Docs.

At the moment, we prioritize in approximately this order, but if you decide to arrange them differently or you have other suggestions, do not hesitate to report it!

More languages!

We discussed with the R and Julia communities the possibility of compiling these languages ​​in WebAssembly to use them in Iodide and other browser projects. At first glance, this is doable, but the implementation will be a little more complicated than for Python. As in Python, some interesting workflows open: for example, you can apply statistical models in R or solve differential equations in Julia, and then display the results using the browser API. If you study these languages, please let me know - in particular, we want to get help from experts on FORTRAN and LLVM.

Export Notebooks

Early versions of Iodide were standalone HTML executable files that included both the JSMD code used in the analysis and the JS code to run Iodide itself, but we moved away from this architecture. More recent experiments have convinced us that the benefits of working with the Iodide server outweigh the benefits of managing files on the local system. However, these experiments showed that you can make a standalone Iodide executable notepad by adding the Iodide code along with any data and libraries used in one large HTML file. It can be great, but it will be useful as a perfectly reproducible and archived picture.

Iodide Browser Extension for Text Editors

Although many scientists are used to working in browser-based programming environments, some people will never give up their favorite text editor. We really want Iodide to go where people find it convenient to work, including those who prefer to enter code in another editor but want to access the interactive and iterative features that Iodide provides. To meet this need, we began to think about creating an easy browser extension and some simple APIs to allow Iodide to communicate with client-side editors.

Feedback and help are welcome!

We are not trying to solve all the problems of scientific data and scientific computing, and Iodide is not a universal solution. If you need to process terabytes of data on GPU clusters, Iodide is unlikely to help you with anything. If you publish magazine articles and you just need to write a document in LaTeX, then there are better tools. If you don’t like the idea of ​​transferring everything to the browser, there is no problem - there are many really amazing tools that you can use for science, and we are grateful for that! We don’t want to change someone’s habits, and many scientists don’t need web-based communications. Super! Do as you feel comfortable!

But from scientists who make or want to make content for the Internet, I would like to hear what tools you need in your work!

Please go toiodide.io , try this tool and leave a review (but again: keep in mind that the project is in the alpha version - please do not use it for any critical work, and remember that everything can change in alpha). You can fill out a short form , tickets and bug reports on Github are also welcome . Requests for functions and general thoughts leave in our Google group or in Gitter .

If you want to take part in creating Iodide, the source code is published on Github . Iodide covers a wide range of software disciplines: from the development of modern interfaces and scientific computing to compilation and transpilation, so there are a lot of interesting things!