msvcore - c ++, cross-platform library, bicycle and 12 years of development

Hi Habr, I want to talk about the creation on which I spent my youth , it would be better to drink and smoke .

Msvcore is a cross-platform c ++ library written from scratch based on the principles of optimality, openness and simplicity. At least it was laid as a basic idea. What happened in the end ...

A bit of history


It all started back in 2004, when I started working with someone like a system administrator of all trades, and at the same time began to get involved in c ++. And, as I remember now, MFC with its templates and CString strings. Then the idea arose to write my own lines, simple and understandable. And away we go.

Unfortunately, I only kept the archive from October 2005, and I will restore events from it. You can look at it on a github . The earliest date in the archive is dated October 10, 2004, in the absence of another, this day can be considered the birthday of the library (Date: Sun, 10 Oct 2004 12:50:42 GMT).

Interestingly, the library, unlike others, was created through evolution. Frequently used code was added to it, and it was appended to current tasks. Tasks became more complicated, the library grew. But there is a minus in this, you may not find functions that, it would seem, are required to be in every library. Therefore, she and the bike, which is being added in the development process.

The first component in my library was the MString string class, the full name is MyString.

class MString{ 
	char * data; // Указатель на данные
	unsigned int sz; // Размер строки
	unsigned int rsz; // Размер выделенной памяти
}

Binary strings, with one buffer, ending with an extra null byte, which was required for most standard functions: open, strlen (), ...

After the first large program, with static arrays, illogical logic and other jambs of a novice programmer, dynamic arrays were required. The common name for the first dynamic arrays is MMatrix. Basic principle: Parent element with pointers to the first and last elements of the array and, a counter of elements. Elements of the array had pointers to the previous, next elements and data. Each version of the data was made its own copy of the class. Templates? No, not heard. Later, array classes were developed almost once a year.

One of the first to create the MSVCF class was to work with configuration files.
The analogs of the standard functions were written: itos () and stoi (), itos64 () and stoi64 (), ftos () and stof () for translating a number into a string and back for int, int64, float. itob () and btoi () too, but for binary strings. stos () converter short to char. explode () for breaking a string into parts. rts () (read to sumbol) and company, to search for a character / string in a string. The ILink class was created, which was required to parse paths and links to parts and is still in use. At some point, it took the IHeader class to work with http headers. The MSVThreads feature set for creating new threads is still relevant. MTime class created to work with time.

These were the beginnings of the library, which still has a lot to go through. The volume of text in the library is 115kb.

Basic principles


The library was sharpened on optimality, comprehensibility, logic and many other good words. In fact, it is still full of scary code written in one line. Only last year, code began to be written with indentation between characters.

To use the library, you need to include two MString.cpp and VString.cpp files in the project. Attempts to connect the library as a plug-in failed because of the difficulty of constantly reassembling two projects instead of one. Also, you cannot apply changes on the fly in the plugin library. Initially, the library was included in the project itself, but over time, the assembly began to take a long time, so it was divided into parts so that when changing it was required to reassemble only part of the project. Library-based projects usually consist of three main files: ProjectName.cpp, MString.cpp, VString.cpp. The program code is written in ProjectName.cpp, the next two are connected from the library. This often does not use the standard code separation into cpp and h files, to speed up code writing. The library and programs were sometimes written days and nights,

A word about cross-platform. After meeting Linux in 2006, the library was dubbed for assembly in gcc. Then under WinCE, Android (Jni) and even Flash (Crossbridge). It is worth noting that all programs are initially written in Windows and MSVS, and only then are transferred to the final platform for adding platform-dependent parts and testing. Which saves a lot of time and effort.

I basically did not use libraries, except for those that are pointless and difficult to rewrite: zlib, openssl, pcre, mysql.

Lines


The concept of lines was built for a long time. The options using the reference counter, when several variables point to the same memory block, did not work optimally enough, they needed to use locks and more complex logic, and in those days I put a lot of emphasis on economy and simplicity.

At some point, virtual strings of VString appeared, first as a separate class, and subsequently, solving the problem of converting strings between classes as a base.
As a result, today the picture is as follows: VString

Class(Virtual String) - virtual strings, contains a pointer to the data and its size. The class does not work with memory allocation / deallocation. It allows you to quickly perform operations with strings that do not require data changes: receiving part of a string, searching by string. The class replaces the commonly used pointer to char * strings ending in zero. I consider the last option for working with strings to be not optimal and simply dangerous; incorrect data in a string should not cause errors. The main problem with VString is keeping track of the relevance of the data pointed to by a variable of this class.

class VString{
public:
	unsigned char *data;
	unsigned int sz;
functions...
};

Class MString (My String) - standard strings with memory allocation for them. Memory is reallocated each time the line is resized, which makes the use of these lines extremely slow. These strings should be used where other string options are not suitable. Used as variables in classes. The main problem here is the need to use locks when accessing from multiple threads.

class MString: public VString{ functions... };

The SString class (Stack String) is a string on the stack. The same lines, but with the allocation of memory on the stack, allocate 1kb by default, MString is used for larger lines. Great speed, but a lot of extra volume. Used as temporary variables. Born in pursuit of a decrease in memory allocation / deallocation operations.

class SStringX : public VString{
	unsigned char sdata[stacksize];
	MString mdata;
functions...
};

Class HLString (Line String) - stores strings in a chain of memory blocks. By default, 4kb memory blocks or data size is allocated. Allocating memory in advance speeds up the operation of data addition operations. The class overloads the + operator and allows you to write code in the form: MString text (HLString () + “text” + a + 111); The class also allocates memory on the stack for the first memory block, by default 4kb. Use cases: addition of many lines, numbers to one line. It is also often used by feint ears to temporarily store strings. HLString ls; VString s = ls.addnfr (“Text”); - Adds an unfragmented row. There are advantages to allocating / freeing large blocks of memory, which is much faster than using MString for the same number of lines.

Class tstring(Temp String, Thread String) - Temporary or streaming strings. An idea that came only a year ago, and she had to hurry for five years. Basically, this idea is built on HLString, unfragmented strings and __thread variables. Each thread has its own instance of the HLString variable, from which interesting prospects arise. TString allocates memory in a thread-bound HLString, which is definitely faster than allocating memory through malloc () / free (). The problem with this class is that it does not free memory until all TString variables are destroyed. At certain points in the program, all variables must be destroyed, otherwise the program gradually uses all available memory with the corresponding consequences.

These are the five types of lines that I use when writing programs. VString - for working with strings and string fragments, MString - for storing data in classes, SString - for collecting strings from substrings on the stack, HLString - for collecting large strings on the fly, TString - for temporary strings.

Arrays


The first program built using static arrays showed that something needs to be done with this. Since then, the idea of ​​writing arrays has come up almost every year.

MMatrix (My Matrix) - the first attempts to work with pointers, constant crashes and an endless search for errors. They consisted of a parent element with pointers to the first and last element of the array, and, in fact, elements of the array, with pointers to the previous and next elements, as well as data. Propagated by simply copying the class and adding the necessary functions. Templates are not our method. We also optimized for the task: but let's throw out the pointer to the previous element and save as many as four bytes.

LMatrix(Live Matrix) ~ 2007 - forever alive, like grandfather Lenin. You can take a look at the code, but even I don’t want to delve into it and remember what principles it worked on.

UMatrix (Unlimited Matrix) ~ 2008 - a dynamic array from a chain of memory blocks with storage of several array elements in one block. Able to combine all the elements in one memory block. Here the idea was realized to allocate memory immediately for a block of elements, reducing the work with memory functions. A bitmask is used to determine free / occupied items. These ideas will be used in the following array options. Templates are still not our method, but copying with your hands is quite difficult, so the code generator of arrays was written.

IMatrix(Ideal Matrix) ~ 2009 - a vector, the entire array in one memory block, with a lack of space moves to a larger memory block. Subsequently, it was of little use and practically not used.

OMatrix (Object Matrix) ~ 2010 - UMatrix repeats the general idea, but if the first idea is a chain of objects, then here is the idea of ​​separation. Here, unlike UMatrix, a list of free objects and a pass through them are implemented. This class is used as a memory allocator, allowing you to quickly get / free memory for variables.
Matrices have ended, sheets have begun. And along with them code generators were abandoned and templates were accepted for use.

Mlist(My List) ~ 2013 - The class automatically wraps the user class in its own, adding pointers to the previous and next elements.

IList (Ideal List) ~ 2014 - IMatrix with the replacement of the code generator by templates, the same vector.

OList (Object List) ~ 2015 - Similarly, OMatrix rewritten for templates.

UList (Unlimited List) ~ 2015 - Replacement of UMatrix, templates, beautiful and logical code.

AList (Auto List) 2015 - A dynamic array with a set of memory allocators, from the standard new / free, UList, HLString, OList to the ability to write your own.

TrieList 2016 - Implementation of the Trie tree for quick search, based on AList.

Unfortunately, arrays are gradually overgrown with patterns; on the one hand, this simplifies development and, on the other, complicates understanding and change for your tasks.

Library usage


The simplest example is to create an empty project and add a cpp file with the contents:

#define USEMSV_GENERALCPP
#define PROJECTNAME "projectname"
#define PROJECTVER PROJECTNAME ## _versions
#include "../../opensource/msvcore/msvcore.cpp"
Versions PROJECTVER[]={
	// new version to up
	"0.0.0.1", "10.10.2013 21:24"
};
int main(int args, char* arg[]){
	ILink link; mainp(args, arg, link);
	print(PROJECTNAME, " v.", PROJECTVER[0].ver," (", PROJECTVER[0].date, ").\r\n");
return 0;
}

Add the files ".. \ .. \ opensource \ msvcore \ VString.cpp" and ".. \ .. \ opensource \ msvcore \ MString.cpp" to the project and write the code.

When adding several cpp files, you should use the same code up to #include ... msvcore.cpp inclusively by deleting the line #define USEMSV_GENERALCPP. You can also connect additional library extensions by specifying them before msvcore.cpp is connected, for example #define USEMSV_OLIST makes it possible to use OList arrays. The list of extensions can be seen in msvcore.cpp

By default in the library:

  • Platform-dependent files are connected, various defines are defined.
  • String classes are available: VString, MString, TString, SString (), HLString.
  • Function sets are available: rts () - search by line, PartLine () - analogue for working with VString.
  • The ILink class is for working with paths and links, the MTime class is for working with time, the ConfLine class is for working with configuration files, the Logs class is for logging, the MSVEF class is an analogue of regular expressions.
  • Class UList - dynamic arrays, class MMatrix - the first dynamic arrays, a set of MTE classes of universal classes for storing various data types.
  • MRect class for working with regions, MRGB class for working with colors.
  • The mbool class is for working with bit masks, functions for converting strings to numbers, encodings, various formats and vice versa.
  • Library sets for wince, flash.
  • A set of cross-platform functions for working with files: CreateFile (), ReadFile (), WriteFile (), GetFilePointer () ... GetFileInfo (), CloseHandle (), MkDir (), LoadFile (), SaveFile ().
  • The Readdir class uses MSVEF as a filter to list the files in a folder.
  • Console print function print (VString string, VString string1 = VString (), ...).
  • Functions for creating StartThread () threads.
  • Lock classes: TLock - critical sections, CLock - condition variables / condition variables, UGLock - automatic lock / unlock TLock.
  • Classes of working with buffers: SendDataL - a linear buffer, SendDataRing - a circular buffer, RecvData - a buffer for receiving data.
  • Classes of working with the network: ConIp - to establish a connection and open a port. Functions: GetIP () - getting ip by domain name, ifrecv () - checking the availability of data in the socket, ifsend () - checking write permissions on the socket, gettip () and getcip () - returns ip and port, server and client.

Add-ons are indicated before enabling msvcore.cpp:

#define USEMSV_ITOS ITos
class, previous version of SString. Deprecated

#define USEMSV_INTERCEPT
A set of functions for analyzing machine codes and intercepting functions.

#define USEMSV_CPPXCC
The XCC class is a C ++ code parser.

#define USEMSV_INTERCEPT_MALLOC
Code for intercepting system functions for working with memory, used to search for memory leaks.

#define USEMSV_XDATACONT
Classes for working with data formats. The XDataCont class parses JSON and XML, the rest of the classes are deprecated.

#define USEMSV_CJX CjxCont
class with binary data format implementation, binary json.

#define USEMSV_MLIST, USEMSV_ILIST, USEMSV_OLIST, USEMSV_ALIST, USEMSV_TRIELIST
Connects the corresponding classes of dynamic arrays: MList, IList, OList, AList, TrieList.

#define USEMSV_AMF
The amfe and amfd classes for converting / parsing AMF format.

#define USEMSV_PCRE Enables
libraries with pcre2 regular expression functions.

#define USEMSV_CONSOLE
The PipeLine and PipeLine2 classes to start other processes.

#define USEMSV_MWND
Almost a separate library for working with windows, graphics, images. Allows you to work with graphics on all platforms supporting the library. Contains platform-independent primitive drawing functions. It is worth talking about it separately. Uses the CxImage library for encoding / decoding image formats.

#define USEMSV_CONSOLELINE
A set of classes for working with the console.

#define USEMSV_OPENSSL
Functions and classes for working with openssl and encryption. The MySSL class for setting / receiving ssl connections and working with them. Functions: RsaCreateKeys () - creates two Rsa keys, the functions RsaPublicEncode (), RsaPublicDecode (), RSAPrivateEncode (), RsaPrivateDecode () - encrypt / decrypt with the public / private key, the functions AesEncode () and AesDecode () are encoded / decoded by the Aes algorithm. It also contains functions for working with certificates.

#define USEMSV_WEBSOCKETS
A set of functions for working with WebSockets and the WebSocketsCli class - an implementation of the WebSockets client.

#define USEMSV_MYSQL
The MySQLCon class is a wrapper for working with MySQL, uses mysql-connector-c.

#define USEMSV_MSL_FL
Connects the MSL interpreter - my version of the programming language, most resembling php. Msl Fast Line - msl version 4. Without generating pseudocode, executes text commands. It was written about a week in October 2013. It is also worth talking about it separately.

#define USEMSV_MSL_FV
Msl Five - the fifth version of the language. With code byte generation and other goodies. It was developed in September 2015.

#define USEMSV_HTTP
Classes and functions for working with http requests. The GetHttp and GetHttp2 classes. When connecting openssl functions supports https requests. IHeader class for working with http headers. MCookie class for working with cookies.

#define USEMSV_CONFLINE ConfLineOptions
class for working with configuration files.

#define USEMSV_NESTAPI, USEMSV_NESTAPI2
Classes and functions for the server side of my NestApi protocol.

#define USEMSV_STORMSERVER
Server platform, which I would like to write a separate post about. About ten years, I tried to write good servers and it finally succeeded. Almost perfect solution.

#define USEMSV_LIGHTSERVER
The LightServer class is a simple and easy server for which you can write your own handler. The LightServerHttp class is a simple https server that returns a test page.

#define USEMSV_TGBOTS
The TgBot and TgBots classes implement telegram bots.

#define USEMSV_ANDROID
Functions and classes simplifying work when compiling for android.

#define USEMSV_TRAFFIX
Classes for listening to traffic.

#define USEMSV_BUFECHO
Functions for changing console options.

Conclusion


I hope you are interested in this bike named after me and I will have something to tell you more. In my opinion, there are quite interesting things here, starting with strings, arrays, wrappers for openssl are written, msl or a class for creating bots in a telegram may be of interest, and finally, stormserver is a platform for creating various servers. About the latter, I will write a separate article on the development of servers, from a simple server echo to complex http and proxies.

As I said, the library is pretty much biased towards the developed programs. As practice shows, in itself it is difficult to develop without specific tasks.

I wrote this library in order to learn how to program well and understand what the code does from beginning to end. And I must say, there is progress, but still work and work.

In future plans, once again, rewrite from scratch, completely get rid of the old code, put in order. And maybe finally write the documentation. It is also planned to complete the implementation of new string options, as a replacement for the current one, and new dynamic arrays.

Library code and several projects

Also popular now: