Reduced overhead for golang utilities
The goal of the work is to reduce the overhead of storing a large number of utilities written in golang.
One of the side effects of golang static compilation is the relatively large overhead of storing the runtime and all the libraries used inside each executable. For example, a small utility that only does what it accesses through the network to the server and executes the simple commands received - it weighs 5.5 MB.
When there is only one such utility - in modern conditions it is still not scary. When utilities accumulate and there are already several tens or hundreds of them - purely humanly it becomes a pity for the hundreds of megabytes flowing “out of nowhere”.
To solve this problem I wrote a multiex library, which I share with the community. With its help, you can combine several programs into one executable file with minimal changes inside the program code and without changing the external behavior. The idea was taken from busybox - all programs are compiled into a single file, and the executable code is selected at startup based on the name of the file being launched.
To connect programs, you need to call the registrar function, for example like this:
The program name is transferred in the structure - if the executable file is called with the name test, then the function f1 will be called. Next is the normal operation of the function without changes, including it is possible to parse the passed command line arguments in the usual way.
./test asdf
./test can be the name of an executable file, a hard link or a symbolic link.
In addition, you can call the f1 function if the first parameter of the executable file is --multiex-command = test. In this case, the --multiex-command = test parameter will be deleted and the f1 function will also be able to parse its arguments as usual.
./any --multiex-command = test asdf
The library has already implemented the program of the same name with the name multiex, at the moment it can:
1. Display a brief help about what kind of file it is and what programs are included in it - if the file is called with an unexpected name
2. Create symbolic links to all programs included in the executable file. Links are created in the same folder where the executable file itself lies.
./any --multiex-command = multiex install
This does not create a link to multiex itself.
In github.com/rekby/multiex-example shows an example of programs to connect:
1. Registration of functions directly in the main compiled module. With this method, the main assembly module knows about the specific functions to be included, and when adding a new function, you need to add its enumeration to the registration list. In this case, the included programs themselves do not need to add a dependency on multiex - just have the exported Main function.
2. Plug-ins are imported into the assembly module, and each module registers its functions during the initialization process. Here the assembly module knows only the list of included modules, and not every function. Each module itself determines what functions it will export and to export a new function, it is enough to correct only the module with this function.
3. Naturally, both methods can be combined (in the example, the combined version is just shown).
Moreover, each program can maintain its independence and, if necessary, be compiled separately from the others into its own file and run as usual. This is done by adding the main subfolder to the source code of the program. Moreover, this file can be the same for all projects and simply be copied without changes wherever needed.
github.com/rekby/multiex
One of the side effects of golang static compilation is the relatively large overhead of storing the runtime and all the libraries used inside each executable. For example, a small utility that only does what it accesses through the network to the server and executes the simple commands received - it weighs 5.5 MB.
When there is only one such utility - in modern conditions it is still not scary. When utilities accumulate and there are already several tens or hundreds of them - purely humanly it becomes a pity for the hundreds of megabytes flowing “out of nowhere”.
To solve this problem I wrote a multiex library, which I share with the community. With its help, you can combine several programs into one executable file with minimal changes inside the program code and without changing the external behavior. The idea was taken from busybox - all programs are compiled into a single file, and the executable code is selected at startup based on the name of the file being launched.
To connect programs, you need to call the registrar function, for example like this:
func f1(){
if os.Args[1] == "asdf" {
println("ok")
}
}
multiex.Register(multiex.ExecutorDescribe{Name: "test", Function: f1})
The program name is transferred in the structure - if the executable file is called with the name test, then the function f1 will be called. Next is the normal operation of the function without changes, including it is possible to parse the passed command line arguments in the usual way.
./test asdf
./test can be the name of an executable file, a hard link or a symbolic link.
In addition, you can call the f1 function if the first parameter of the executable file is --multiex-command = test. In this case, the --multiex-command = test parameter will be deleted and the f1 function will also be able to parse its arguments as usual.
./any --multiex-command = test asdf
The library has already implemented the program of the same name with the name multiex, at the moment it can:
1. Display a brief help about what kind of file it is and what programs are included in it - if the file is called with an unexpected name
2. Create symbolic links to all programs included in the executable file. Links are created in the same folder where the executable file itself lies.
./any --multiex-command = multiex install
This does not create a link to multiex itself.
In github.com/rekby/multiex-example shows an example of programs to connect:
1. Registration of functions directly in the main compiled module. With this method, the main assembly module knows about the specific functions to be included, and when adding a new function, you need to add its enumeration to the registration list. In this case, the included programs themselves do not need to add a dependency on multiex - just have the exported Main function.
2. Plug-ins are imported into the assembly module, and each module registers its functions during the initialization process. Here the assembly module knows only the list of included modules, and not every function. Each module itself determines what functions it will export and to export a new function, it is enough to correct only the module with this function.
3. Naturally, both methods can be combined (in the example, the combined version is just shown).
Moreover, each program can maintain its independence and, if necessary, be compiled separately from the others into its own file and run as usual. This is done by adding the main subfolder to the source code of the program. Moreover, this file can be the same for all projects and simply be copied without changes wherever needed.
github.com/rekby/multiex