We embed the Lua interpreter in the project for the microcontroller (stm32)

In fairly large applications, a significant part of the project is business logic. It is convenient to debug this part of the program on a computer, and then embed it in the project for the microcontroller, expecting that this part will be executed exactly as it was intended without any debugging (ideal case).

Since most programs for microcontrollers are written in C / C ++, for these purposes they usually use abstract classes that provide interfaces to low-level entities (if a project is written only using C, function pointer structures are often used). This approach provides the required level of abstraction over the iron, however, it is fraught with the need for constant re-compilation of the project with the subsequent programming of the non-volatile memory of the microcontroller with a large binary firmware file .

However, there is another way - using a scripting language that allows you to debug business logic in real time on the device itself or load work scripts directly from external memory, without including this code in the microcontroller firmware.

I chose Lua as the scripting language.

Why Lua?

There are several scripting languages ​​that you can embed in a project for a microcontroller. A few simple BASIC-like, PyMite, Pawn ... Each has its pros and cons, a discussion of which is not included in the list of issues discussed in this article.

Briefly about what is good specifically lua - can be found in the article "Lua in 60 minutes . " This article inspired me a lot and, for a more detailed study of the issue, I read the official guide-book from the author of the language Robert Jeruzalimsky " Programming in Lua " (available in the official Russian translation).

I would also like to mention the eLua project. In my case, I already have a ready-made software low-level layer for interaction with both the peripherals of the microcontroller and other required peripherals located on the device board. Therefore, I have not considered this project (since it is recognized to provide the very layers for connecting the Lua core with the peripherals of the microcontroller).

About the project in which Lua will be embedded

By tradition , my sandbox project will be used as the quality of the field for experiments (link to the commit with the already integrated lua with all the necessary improvements described below).

The project is based on the stm32f405rgt6 microcontroller with 1 MB non-volatile and 192 KB of RAM (the older 2 blocks with a total capacity of 128 KB are currently used).

The project has a FreeRTOS real-time operating system to support the hardware peripheral infrastructure. All memory for tasks, semaphores and other FreeRTOS objects is allocated statically at the linking stage (located in the .bss area of ​​RAM). All FreeRTOS entities (semaphores, queues, task stacks, etc.) are parts of global objects in the private areas of their classes. However, the FreeRTOS heap is still allocated to support the malloc , free , calloc functions (required for functions such as printf ) that are redefined to work with it . There is a raised API for working with MicroSD (FatFS) cards, as well as debugging UART (115200, 8N1).

About the logic of using Lua as part of a project

For the purpose of debugging business logic, it is assumed that commands will be sent by UART, packed (as a separate object) into finished lines (ending with the character "\ n" + 0-terminator) and sent to the lua machine. In case of unsuccessful execution, output by means of printf (since it was previously involved in the project ). When the logic is debugged, it will be possible to download the final business logic file from the file from the microSD card (not included in the material of this article). Also, for the purpose of debugging Lua, the machine will be executed inside a separate FreeRTOS thread (in the future, a separate thread will be allocated for each debugged business logic script in which it will be executed with its environment).

Inclusion of the lua submodule in the project

The official mirror of the project on github will be used as the source of the lua library (since my project is also posted there. You can use the sources directly from the official site ). Since the project has an established system for assembling submodules as part of the project, individual CMakeLists for each submodule, I created a separate submodule in which I included this fork and CMakeLists to maintain a single build style.

CMakeLists builds the sources of the lua repository as a static library with the following submodule compilation flags (taken from the submodule configuration file in the main project):

SET(C_COMPILER_FLAGS "-std=gnu99;-fshort-enums;-fno-exceptions;-Wno-type-limits;-ffunction-sections;-fdata-sections;")
SET(MODULE_LUA_COMP_FLAGS "-O0;-g3;${C_COMPILER_FLAGS}"

And flags of specification of the processor used (set in the root CMakeLists ):

SET(HARDWARE_FLAGS
        -mthumb;
        -mcpu=cortex-m4;
        -mfloat-abi=hard;
        -mfpu=fpv4-sp-d16;)

It is important to note the need for the root CMakeLists to specify a definition that allows not to use double values ​​(since the microcontroller does not have hardware support for double. Only float):

add_definitions(-DLUA_32BITS)

Well, it remains only to inform the linker about the need to assemble this library and include the result in the layout of the final project:

Defining functions for working with memory

Since Lua itself does not deal with memory, this responsibility is transferred to the user. However, when using the bundled lauxlib library and the luaL_newstate function from it, the l_alloc function is bound as a memory system. It is defined as follows:

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud; (void)osize;  /* not used */
  if (nsize == 0) {
    free(ptr);
    return NULL;
  }
  else
    return realloc(ptr, nsize);
}

As mentioned at the beginning of the article, the project already has overridden malloc and free functions, but there is no realloc function . We need to fix it.

In the standard mechanism for working with the FreeRTOS heap, the heap_4.c file used in the project does not have a function for resizing a previously allocated memory block. In this regard, it is necessary to make its implementation on the basis of malloc and free .

Since in the future it is possible to change the memory allocation scheme (using another heap_x.c file), it was decided not to use the interiors of the current scheme (heap_4.c), but to make a higher-level add-in. Though less effective.

It is important to consider that the methodrealloc not only removes the old block (if one existed) and creates a new one, but also moves data from the old block to the new one. Moreover, if the old block had more data than the new one, the new one is filled with the old ones to the limit, and the remaining data is discarded.

If this fact is not taken into account, then your machine will be able to execute such a script three times from the line " a = 3 \ n ", after which it will fall into a hard fault. The problem can be solved after studying the residual image of the registers in the hard fault handler, from which it will be possible to find out that the crash occurred after trying to expand the table in the bowels of the interpreter code and its libraries. If you call a script like ' print' test '", then the behavior will vary depending on how the firmware file is assembled (in other words, the behavior is not defined).

In order to copy data from the old block to the new one, we need to know the size of the old block. FreeRTOS heap_4.c (like others files that provide methods for working with the heap) does not provide an API for this, so I’ll have to add my own. I used the vPortFree function as a basis and cut its functionality to the following form:


Now it’s small, write realloc based on malloc , free , and vPortGetSizeBlock :

Add support for working with stdout

As it becomes known from the official description, the lua interpreter itself is not able to work with I / O. For these purposes, one of the standard libraries is connected. For output, it uses the stdout stream . The luaopen_io function from the standard library is responsible for connecting to the stream . To support working with stdout (unlike printf ), you will need to override the fwrite function . I redefined it based on the functions described in the previous article .


Without its definition, the print function in lua will execute successfully, but there will be no output. Moreover, there will be no errors on the Lua stack of the machine (since formally the function was executed successfully).

In addition to this function, we will need the fflush function (for the interactive mode to function, which will be discussed later). Since this function cannot be overridden, you will have to name it a little differently. The function is a stripped down version of the fwrite function and is intended to send what is now in the buffer with its subsequent cleaning (without additional carriage transfer).

Retrieving strings from a serial port

To get strings for a lua machine, I decided to write a simple uart-terminal class, which:

• receives data on a serial port byte-by-byte (in interrupt);
• adds the received byte to the queue, where the stream receives it from;
• in a stream of bytes, if this is not a line feed, sent back in the form in which it arrived;
• if a line feed has arrived (' \ r '), then 2 bytes of terminal carriage return are sent (" \ n \ r ");
• after sending the response, the handler of the byte that arrived (line layout object) is called;
• controls the pressing of the delete character key (to avoid deleting service characters from the terminal window);

Links to sources:

• UART class interface is here ;
• UART base class is here and here ;
• class uart_terminal here and here ;
• creating a class object as part of the project here .

Additionally, I note that for this object to work properly, you need to assign a priority to UART interruption in the allowable range for working with FreeRTOS functions from the interrupt. Otherwise, you can get interesting hard-to-debug errors. In the current example , the following options for interrupts are set in the FreeRTOSConfig.h file .


In the project itself, an nvic class object sets the priority of the 0x9 interrupt, which is in the valid range (the nvic class is described here and here ).

String formation for a Lua machine

Bytes received from the uart_terminal object are transferred to an instance of a simple class serial_cli, which provides a minimal interface for editing the string and transferring it directly to the thread in which the lua-machine is executed (by calling the callback function). Upon accepting the character '\ r', a callback function is called. This function should copy a line to itself and “release” the control (since the reception of new bytes is blocked during a call. This is not a problem with correctly prioritized streams and a sufficiently low UART speed).

Links to sources:

• serial_cli description files here and here ;
• creating a class object as part of the project here .

It is important to note that this class considers a string longer than 255 characters invalid and discards it. This is intentional, because the lua interpreter allows you to enter constructs line by line, waiting for the end of the block.

Passing a string to the Lua interpreter and its execution

The Lua interpreter itself does not know how to accept block code line by line, and then execute the whole block on its own. However, if you install Lua on a computer and run the interpreter in interactive mode, we can see that the execution is line-by-line with the corresponding notation as you type, that the block is not yet complete. Since the interactive mode is what is provided in the standard package, we can see its code. It is located in the lua.c file . We are interested in the doREPL function and everything that it uses. In order not to come up with a bicycle, to get the interactive mode functions in the project, I made the port of this code into a separate class, which I named lua_repl by the name of the original function, which uses printf to output information to the console and has a public add_lua_string method to add a string obtained from the object of the serial_cli class described above.

References:

• class description lua_repl here ;
• code here , here and here ;

The class is made according to the singleton Myers pattern, since there is no need to give several interactive modes within the same device. An object of class lua_repl receives data from an object of class serial_cli here .

Since the project already has a unified system for initializing and servicing global objects, the pointer to the object of class lua_repl is passed to the object of the global class player :: base here . In the start method of an object of the player :: base class (declared here . It is also called from main), the init method is calledan object of class lua_repl with a task priority of FreeRTOS 3 (in a project, you can assign a task priority from 1 to 4. Where 1 is the lowest priority and 4 is the highest). After successful initialization, the global class starts the FreeRTOS scheduler and interactive mode starts its work.

Porting Issues

Below is a list of the problems that I encountered during the Lua port of the machine.

2-3 single-line scripts of variable assignment are executed, then everything falls into hard fault

The problem was with the realloc method. It is required not only to re-select the block, but also to copy the contents of the old one (as described above).

When trying to print a value, the interpreter falls into hard fault

It was already more difficult to detect the problem, but in the end I managed to find out that snprintf was used for printing. Since lua stores values ​​in double (or float in our case), printf (and its derivatives) with floating point support is required (I wrote about the intricacies of printf here ).

Requirements for non-volatile (flash) memory

Here are some measurements that I made to judge how much non-volatile (flash) memory needs to be allocated to integrate the Lua machine into the project. Compilation was performed using gcc-arm-none-eabi-8-2018-q4-major. The version of Lua 5.4 was used. Below in the measurements, the phrase “without Lua” means the non-inclusion of the interpreter and methods of interaction with it and its libraries, as well as an object of the lua_repl class in the project. All low-level entities (including overrides for the printf and fwrite functions ) remain in the project. The FreeRTOS heap size is 1024 * 25 bytes. The rest is occupied by global project entities.

The summary table of results is as follows (all sizes in bytes):

Build options	Without lua	Core only	Lua with base library	Lua with libraries base, coroutine, table, string	luaL_openlibs
-O0 -g3	103028	220924	236124	262652	308372
-O1 -g3	74940	144732	156916	174452	213068
-Os -g0	71172	134228	145756	161428	198400

RAM requirements

Since the consumption of RAM depends entirely on the task, I will give a summary table of the consumed memory immediately after turning on the machine with a different set of libraries (it is displayed by the print (collectgarbage ("count") * 1024 command ).

Composition	RAM used
Lua with base library	4809
Lua with libraries base, coroutine, table, string	6407
luaL_openlibs	12769

In the case of using all libraries, the size of the required RAM is significantly increased in comparison with the previous sets. However, its use in a considerable part of applications is not necessary.

In addition, 4 kb is also allocated to the task stack, in which the Lua-machine is executed.

Further use

For full use of the machine in the project, you will need to further describe all the interfaces required by the business logic code for the hardware or service objects of the project. However, this is the topic of a separate article.

Summary

This article described how to connect a Lua machine to a project for a microcontroller, as well as launch a full-fledged interactive interpreter that allows you to experiment with business logic directly from the command line of the terminal. In addition, the requirements for the hardware of the microcontroller were considered for different configurations of the Lua machine.