As promised, we are releasing our linux terminal emulator library for LGPL, which we use to display virtual machine consoles in the cloud. It is called, respectively, pyte (PYthon Terminal Emulator).

According to our own estimates, the coverage of the "text" functionality of console_codes is approaching 100% (from 80 to 90%, as suggested by pessimists among opportunists among developers).

Not implemented: codes for loading fonts into the character generator of the VGA adapter, VESA energy-saving control, sound signaling (codes for controlling the frequency and duration of sound), palette control, own charsets; in general, everything that is not related to the text.

But all the other complex functions are implemented, such as locking screen regions for recording, scrolling, managing line wrapping modes, the correct processing of attributes for various types of text deletion, etc. - all that is necessary for existing applications, such as nano, adom (in the picture a fragment of ESC codes and the resulting image is just from ADOM), vim, emacs, mc, aptitude, dialog, yast2, etc. for full rendering.

The library is written in python and sharpened for the convenience of manipulating the screen, abstracting from the graphical representation of the image, which allows it to be used in code that performs further transformations (for example, transferring a screen to JS or serializing to a database).

The second important feature of the library is the support of diffs (differences) between the previous and current state of the screen. This is necessary for economical transmission of images through channels of limited speed (read, the Internet) and rendering on devices with limited performance (read, javascript).

History of creation

In the beginning, the vt102 library was remarkable in its naivety . She somehow draws part of the vt102 esc codes, not thinking at all about unicode, subtle nuances of scrolling screen regions, performance and stability. As the author wrote about her, she was intended for cheating in the game nethack , and no more.

We started with her. For some time we tried to correct errors and refine, but then it turned out that firstly vt102 does not support some buttons (PgUp, for example), and secondly ... As I wrote earlier, the main advantage of the linux terminal is that it expects linux by default.

I had to 'from scratch' implement all the functionality of the linux console (with the above limitations on the features of the serial port compared to VGA adapters). Thus, the library smoothly renamed from vt102 to pyte (Python Terminal Emulator).

Despite the fact that the library is now written from scratch, we decided to publish it also under LGPL, mainly out of respect for the author vt102, the code of which we at least tactfully deleted, but some of the ideas still used, for example, the concept of the dispatcher.

Internal organization

The library consists of the screen and stream classes. Screen contains a copy of the screen (characters and attributes), Stream is a finite state machine responsible for processing ESC and (in particular) CSI sequences. The sequence of characters feeds to the stream, and we have a situation where the program using the library does not know anything about “characters” and operates with a stream of bytes with a customizable encoding (incremental decoder, so that there are no strange squares due to a gap in the middle of UTF- 8 characters). In principle, we work not only with UTF-8, but also with other encodings. We do not use this functionality in the cloud right now, but we have implemented it for the sake of a clear conscience.

After eating the entire sequence of characters that form the CSI sequence (this is a special class of ESC sequences starting with ESC codes [, followed by parameters and a command, for example, ESC [1; 31m generates a red bold tone color). Each output generates a particular event (symbol output, scrolling, etc.) for all subscribers, including the library itself that processes these events. Of course, among those who wish there may be external code, for example, in this way we learn about the appearance of new characters and inform them of the Java script in the control panel.

The objects inside the Screen are stored like regular python objects. We had a version using faster array arrays, but we ran into some difficulties with converting between a library-friendly format and an array, so the performance of arrays was greatly blurred by constant conversions. By the way, the use of snake objects simplifies access to their giblets and, most importantly, serialization to the database (which is the “killer feature” of our console, which survives the transfer of a virtual machine between servers without breaks and zeroing the screen contents).

By the way, this library works great under pythons from 2.6 to 3.2+ and even under PyPy .

Future plans

In the near future, it is planned to supplement the library with scrolling support (the same Shift-PgUp), which, by the way, we will add to our cloud.

In addition, there is an idea to do a little useless perfectionism and meet the standards better than the Linux console itself, that is, go through vttest without a single error.

The thing that we do not plan yet (although we will accept commits with great pleasure) is extended Unicode support - support for normalizing characters, characters of different widths, etc. In principle, there is everything in python for this - but we do not need such (we have not yet set our sights on the Asian market).

If this library will be useful to someone, we will be grateful for reports on use, bug reports and complaints.

References

easy_install pyte
github.com/selectel/pyte

Who say thank you?
bobry for the code,
o_l and akme for agreeing to publish under the LGPL.