I welcome you, dear Khabrovites!

Today I wanted to start a story about one interesting product that I presented on the Tibbo Habr . This product may be useful to a wide range of IT professionals, including system administrators, information security specialists, and, finally, simple developers who have no, and have to program communication with devices and other low-level input / output.

We will talk about the programmable terminal / sniffer IO Ninja (hereinafter I will omit the word "emulator" and say simply "terminal"). I suspect that the very definition of “terminal / sniffer” may look rather unusual, if not strange. So let's start with the story of the emergence of IO Ninja.

Motivation and History

The very first product of our company was the EM100 embedded module for converting the serial interface of Serial RS232 to Ethernet 10BASE-T (now its analogue is the EM1206 module ).

On this device, the interaction protocols between the firmware (on the modules) and the virtual port drivers, as well as the configuration utilities (on the PC) were tested and run.

As you know, in our imperfect world, not a single program works immediately after writing, and we had many months to debug communication protocols. In our particular case of debugging a Serial-to-Ethernet converter, this resulted in the need to use the following utilities in parallel:

• Serial terminal;
• TCP terminal;
• UDP terminal;
• Serial Monitor
• Ethernet sniffer.

In addition to debugging our protocols, it was also necessary:

• Listen and accept incoming TCP connections;
• Send broadcast (UDP) UDP packets;
• Accept responses to them from multiple nodes;
• Send and be able to display binary data in hexadecimal form.

There were no problems with monitors for Ethernet and Serial - even free products like Wireshark (which was then called Ethereal) completely satisfied our requests. But with the terminals we ran into problems.

Like monitors, there are a great many terminals. But there is a nuance: most of them are very lame in terms of sending and displaying binary data. If this opportunity is provided, then in an extremely inconvenient form - a full-fledged hexadecimal viewer of incoming packages and an outgoing editor (with the ability to select, copy, paste, etc.) was not found in any of the terminals we tested. In addition, the situation is also bad in terms of receiving TCP connections and it is completely obscure - with support for working with broadcast UDP packets.

It was after an unsuccessful search that the idea was born to create a universal utility for debugging communication protocols, which would be able to do all of the above and would do it well. Yes - dream so dream! - It would be programmable and would allow us to easily add analyzers of our own protocols and test scripts (such as, for example, waiting for commands and sending answers dynamically generated according to certain rules).

Dreams of programmability, of course, for a long time remained just dreams. But the utility for sending and displaying binary data, while with the support of receiving TCP connections and broadcast UDP broadcasts, was slandered quite quickly. It was called ingenuously - SockTerm - and looked like this:



In addition to supporting socket modes that are absent in standard terminals, this utility has demonstrated the extreme convenience of the logging approach implemented in it, namely, a hexadecimal “sheet” with gluing new packets to the previous ones.

Work on the utility continued in his spare time from the main projects and tasks for a couple of years. The logging module was gradually refined, expanded and eventually reorganized as a full-fledged engine (its capabilities will be described in more detail below). This engine was quite suitable for use both in terminals and in sniffers - which was enthusiastically implemented. The result was the first version of IO Ninja.



This utility allowed us to work with all the vehicles we are interested in (Serial, TCP, UDP) both in sniffer mode for passive listening and in terminal mode for directional data transmission and reception. Multiple modes of operation were implemented in the form of plug-ins (written, like IO Ninja itself, in C ++), which in theory allowed us to finish support for new modes with sufficient ease, and in the long run publish SDKs and give customers the opportunity to do this themselves. However, in practice the latter was never realized - we were haunted by the very dreams of programmability, and plugins in C ++ are not the best architecture in this case.

What we needed instead was to make the ninja scripted. But the problem is that in languages ​​without pointers and address arithmetic, working with binary data (and this is an integral part of any protocol analyzer) turns into a nightmare like “pull out the byte at index N, shift it 8 bits to the left, add it to the byte at index N + 1 and etc. ". And scripting languages ​​with pointers in nature do not exist. More precisely, it did not exist before - we corrected this annoying misunderstanding by creating the Jancy language .

This article does not intend to tell about all the possibilities of the Jancy language (although, if dear Habrovites show interest, relevant articles will certainly appear). Now we are only talking about the fact that in our hands - after a long series of half-dead prototypes (and then a prototype that was quite viable and used to create the second, scripted version of IO Ninja - you can see its screenshot below), a language with safe pointers and address arithmetic.



Moreover, this language is designed so binary compatible with C / C ++ that a direct call between Jancy and C ++ is possible, and this leads, firstly, to a high efficiency of interaction between the script and the host C ++ application, and secondly, to ease creating C / C ++ libraries for use from Jancy scripts. In addition to ABI (binary) compatibility, Jancy also has a high degree of compatibility at the source code level, which allows you to copy and use (often without any modifications) C code taken from public sources.

In 2014, work was completed on the creation and release of the third version of IO Ninja, based on the Jancy language. It will be discussed further.

If you highlight the most significant features of the new IO Ninja, then these will be:

• Programmability;
• Logging engine;
• Package builder;
• Universal redirector;
• Combining all types of utilities for debugging communications in one product.

The fact that the ninja can be turned into with the help of programmability will be described in subsequent parts of the article; in the same, let's see what is available “out of the box”.

Logging engine

Since the time of the non-programmable first version of IO Ninja, the logging engine offered something that was not available in other terminals and sniffers. This engine stands out primarily by the approach to displaying log data.



Instead of a pair of “main list” and “recording details” windows (as in most monitors and log viewers), the IO Ninja engine offers a “endless sheet” log with alternating text messages and binary data - something like an heterogeneous HTML page growing unlimitedly downward. By the way, it was in connection with the unlimited growth of the log that the development of our own engine was required - otherwise, you could screw something like Gecko or WebKit.

At the same time, despite the fact that the size of the log is actually unlimited (it is really limited only by free space on the disk), the IO Ninja engine provides “smooth” scrolling, without any direct dependence on the amount of memory consumed on the size of the log.

One of the reasons for avoiding standard terminals was the lack of convenient tools for working with binary data, so it is not surprising that the IO Ninja logging engine is literally “tuned” for this task.

Binary data can be displayed in hexadecimal or text form, with customizable line width and tabs and in any of the UTF encodings (including variations with reverse byte order). Packets of the same type (for example, incoming) are optionally glued together. Bonding can be configured to "always", "never" and "if the interval does not exceed a certain threshold." Bonding places are highlighted (again, optionally) with color - checkers.

The IO Ninja log also supports collapse / expand records, which may be necessary to switch between a short and full description and is used, for example, in a network sniffer:

Package builder

Speaking about support for working with binary data, one cannot but mention the tools of IO Ninja for preparing outgoing packages.

For the simplest cases, IO Ninja has a text editor for packages. This editor understands all well-known escape sequences (\ n \ e \ x01 \ u2661, etc.), so introducing a couple of special characters is not a problem. And if you want to make a package entirely consisting of special characters? Welcome to IO Ninja Package Builder!

Package Builder is one of IO Ninja's strengths, available out of the box. In addition to the modern hex editor with support for all UTF encodings, editing nibbles (nibbles), restrictions on the packet size above and below, convenient highlight highlighting, as well as all other features that you can expect from a good editor (insert, overwrite, copy, paste, undo, redo) - in IO Ninja there is a tool for creating packages by template.

Remember, we talked about a high degree of compatibility between Jancy and C? Binary package preparation is one of the places where the benefits of this compatibility are reaped.

Phase 1 : find the sources for implementing the protocol we need in C in publicly available sources;

Phase 2: copy descriptions of package structures in IO Ninja and, possibly, retouch a little (after all, Jancy and C are different languages);

Phase 3 : ???

Phase 4 : PROFIT! You can fill out the package fields from the property editor!

Instead of a conclusion

This concludes the introductory review, and in the next part of the article we will demonstrate several real life scenarios of our company in which IO Ninja helped solve certain practical problems better than their analogues.

In the meantime, you can download and try the latest version of IO Ninja for Windows or Linux at tibbo.com/ioninja/downloads . A related IDE based on NetBeans is also available for download (however, more on this another time).

The following parts of the article

Part 2: http://habrahabr.ru/company/tibbo/blog/255221/
Part 3: http://habrahabr.ru/company/tibbo/blog/256803/