Cmake packer assembly or kitty training
- You will not say how many degrees are below zero now?
- What?
- Well, I …. I'm training.
- Train better .. (looks at Morgunov) ... on cats.
The article describes the experience of using CMAKE and LZ4 with some bias on embedded systems. Those who are familiar with Makefiles or even CMAKE can safely skip the first few paragraphs.
Cats
As cats, it is better to take a small opensource project on the network consisting of a small number of files and several goals (executable files and libraries). It is difficult to find such a project, because many popular projects have already overgrown with a bunch of files. However, the LZ4 packer turned out to be a good candidate . LZ4 is still young, and most importantly, it is quite simple and aimed at SUPER FAST information compression, maybe that's why it is small, because a large piece of code will not work quickly (it will not be optimized in the cache).
LZ4 can quickly compress information, which for the embedded world is more likely to mean little performance and memory requirements. I managed to adapt it to compression for the LPC2478, while the amount of RAM that it took was 12k (4k input buffer + 4k output + 4k dictionary). I think this volume can be reduced to 256 +256 + 256, naturally due to the loss of compression ratio. Nevertheless, even this is a good option so that you do not make up your cunning algorithms for compressing the polling history of any systems if you have "almost" repeating records of 30 ~ 100 bytes in size. To those who are interested in the compression algorithm, I note that LZ4 is an optimized version of the LZ77 algorithm.
Separate praise is the unpacking function. It is very simple and fits in dozens of lines of code, this allows you to achieve the decompression speed of 1GB / s on modern PCs.
What is the build system for embedded systems for?
As a rule, any project in embedded systems starts with
Source code → assembly system → compiler → executable file (s)
Of course, your newfangled proprietary IDE will do its best to do all the build work for you, just click Build all. As your “code base” grows, you will think about collecting the existing code as often as possible and in various ways.
The reasons for having several options for building the code may be the presence of the following factors:
- several projects
- several boards
- even several types of controllers
- testing parts of code on a PC
- several development branches due to the large team
The first reaction to such requirements is always: “let's put in an ifdef”, however, very soon you realize that ifdefs alone can not do or get around, but the code does not look very nice.
Makefiles
The Make utility enters the arena, and brutal men take the compiler and the keys to it and begin to write assembly rules. Make quickly mastered, however, if you already have three projects and many folders, each of which has a library, you have to be more diligent in writing Makefiles. The assembly diagram with Make looks something like this:
Source code -> GNU MAKE -> compiler -> executable file (s)
CMAKE
CMAKE is an advanced make. Based on the assembly rules described in CmakeLists.txt, CMAKE generates assembly control files for MAKE and other utilities and development environments (a list of supported utilities can be obtained by running "cmake –help"). Now the diagram looks like this:
Source code → CMAKE → GNU MAKE or NMAKE or .... → compiler → executable file (s)
Among the advantages of CMAKE I note:
- The Lisp-o assembly rules language itself is similar and more developed than MAKE. A good chance to try something different from the traditional C paradigm.
- The utility is cross-platform, your build will not depend on the IDE and the working OS.
- Not only assembly rules are generated, but also projects for Code-blocks, Eclipse and Visual Studio.
- There are ample opportunities for generating the source code itself, as well as many modules, which allows you to replace Automake and Qmake.
- There are automatic assembly systems (CDASH), packaging (CPACK) and testing (CTEST).
- CMAKE is popular. Many development teams translate their projects under it (for example, KDE).
The main advantage in the field of embedded solutions is that, in the end, your code becomes more portable, you are one step closer to collecting a part of your application for PC for more comfortable debugging. You can run BUILD tests automatically, I don’t know if Code Red or CoCox allows it, but with CMAKE you are not dependent on the IDE.
minimal project
The following is a simple (but not the simplest) CmakeLists.txt file.
cmake_minimum_required (VERSION 2.8)
PROJECT(LZ4 C)
set(SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR}/)
ADD_DEFINITIONS("-Wall")
ADD_DEFINITIONS("-W")
ADD_DEFINITIONS("-Wundef")
ADD_DEFINITIONS("-Wcast-align")
ADD_DEFINITIONS("-Wno-implicit-function-declaration")
ADD_DEFINITIONS("-O3 -march=native -std=c99")
INCLUDE_DIRECTORIES (${SRC_DIR})
set(LZ4_SRCS_LIB lz4_decoder.h lz4_encoder.h ${SRC_DIR}lz4.c ${SRC_DIR}lz4hc.c ${SRC_DIR}lz4.h ${SRC_DIR}lz4_format_description.txt)
set(LZ4_SRCS ${LZ4_SRCS_LIB}
${SRC_DIR}xxhash.c
${SRC_DIR}bench.c
${SRC_DIR}lz4c.c
)
set(FUZZER_SRCS
#вставим сюда какой-нибудь комментарий
${LZ4_SRCS_LIB}
${SRC_DIR}fuzzer.c
# ${SRC_DIR}ooops_commented_source_file.c
)
# устанавливает цель lz4c
add_executable(lz4c ${LZ4_SRCS})
# устанавливает цель fuzzer
add_executable(fuzzer ${FUZZER_SRCS})
Сmake_minimum_required sets the minimum version of Cmake. The PROJECT command sets the name of the project, as well as the programming language (s).
Commands can be written in both lowercase and uppercase letters. The most common set command is some analogue of assigning and creating variables, like 'Let' in Basic, it starts to get a little annoying, but then you get used to it. Other variables are also predefined in CMAKE, one of them is $ {CMAKE_CURRENT_SOURCE_DIR} . Like make SRC_DIR is the variable name, and $ {SRC_DIR}- value. Variables in CMAKE are treated as lists, but can sometimes be interpreted as strings or numbers. Lists of files can be easily broken into several lines.
ADD_DEFINITIONS should actually be used to specify preprocessor macros, but here it is simply used to specify compiler keys.
add_executable sets the purpose of the assembly — the executable.
Configuration
Already such a small CmakeLists.txt allows you to assemble the LZ4 packer (currently revision 94). Open the command console and run cmake with the generator indicated (see screenshot of eran). Сmake produces some basic messages, which in this case will be ignored - nothing unusual happened - a Makefile and a project for CodeBlocks are generated.
The presence of a preliminary configuration phase in the form of starting Cmake before make allows you to change the compilation process based on the arguments for starting CMAKE and the software version (OS, compiler) on which the assembly takes place. You can also copy files, download files, apply patches, generate source files and more. Cmake has the ability to pass additional user-defined arguments. Cmake can also run various utilities during the configuration process. I will try to write about this in the next part.
With a big stretch, we can say that in terms of assembly, CMAKE is capable of performing Portage + automake functionality in Gentoo Linux, but compared to portage, Cmake is much more compact than the python distribution, it is more portable to many platforms (primarily Windows).
Build start
The assembly is launched through a) CodeBlocks b) make (I advise you to try also 'make VERBOSE = 1' and 'make help'). Additional goodies over make are the color output to the console, as well as the percentage of completion.
Meow!
Now let's have some fun with our lz4 packer.
C: \ Desktop \ lz4> lz4c.exe -H
*** LZ4 Compression CLI, by Yann Collet (May 1 2013) ***
Usage:
lz4c.exe [arg] input output
Arguments:
-c0 / -c: Fast compression (default)
-c1 / -hc: High compression
-d: decompression
-y: overwrite without prompting
-H: Help (this text + advanced options)
Advanced options:
-t: test compressed file
-B #: Block size [4-7] (default: 7)
-x: enable block checksum (default: disabled)
-nx: disable stream checksum (default: enabled)
-b #: benchmark files, using # [0-1] compression level
-i #: iteration loops [1-9] (default: 3), benchmark mode only
input: can be 'stdin' (pipe) or a filename
output: can be 'stdout' (pipe) or a filename or 'null'
example: lz4c -hc stdin compressedfile.lz4
Packing yourself.
C: \ Desktop \ lz4> lz4c.exe lz4c.exe lz4c.lz4 *** LZ4 Compression CLI, by Yann Collet (May 1 2013) *** Compressed 98828 bytes into 56586 bytes ==> 57.26% Done in 0.03 s ==> 3.04 MB / s
We launch benchmark.
C: \ Desktop \ lz4> lz4c.exe -b lz4c.exe lz4c.lz4 *** LZ4 Compression CLI, by Yann Collet (May 1 2013) *** lz4c.exe: 98828 -> 56567 (57.24%), 163.3 MB / s, 467.5 MB / s lz4c.lz4: 56586 -> 56278 (99.46%), 273.1 MB / s, 1442.1 MB / s TOTAL: 155414 -> 112845 (72.61%), 191.3 MB / s, 620.0 MB / s
My computer is not very fast, and I do not claim to be fully tested, but these figures are very impressive.
Total
So, I hope this article will help you in developing the Cmake utility. In the next part I will try to write about further research in the issue of assembling LZ4. Those who are curious, the final version of the file I wrote was accepted by the author of the program, however, in the current version of LZ4 there are two CmakeLists files, one written by me and used to build, and the other used to build libraries, although it is possible to combine these two files I have not yet reached my hands. Guru Сmake dare!