Using Rebar and GProc

Zert January 27, 2011 at 22:32

Using Rebar and GProc

Using Rebar

This tutorial may contain outdated information, since Rebar is very actively developing without maintaining compatibility with previous versions.

When developing on Erlang, you often have to collect dependencies from different sources, keep track of their necessary versions, create OTP releases for distributing projects. Things are pretty routine and unpleasant. In order for the development to deliver less unpleasant moments, Basho created a very convenient tool - Rebar. In this article, I will try to uncover the benefits of using it in a real-life example using third-party dependencies and creating configurable OTP releases.

You can download Rebar at hg.basho.com/rebar/downloads/rebar. This is one single file containing several beam-modules. It should be placed in any convenient place that enters the search path for PATH executables, for example, ~/bin/or /usr/local/bin/.
Let's get down to the project.
First, create a directory in which our project will be located ( gpt) and go to it:

$ mkdir gpt && cd gpt

In it, we will create a subdirectory where we will save the source files directly of our application:

$ mkdir -p apps / gpt && cd apps / gpt

Making the application skeleton:

$ rebar create-app appid = gpt

The appid parameter defines the name of our application and, accordingly, the prefix of the source files.

$ ls -1 src
gpt_app.erl
gpt.app.src
gpt_sup.erl

Add a src/gpt.app.srcdescription of the application and a dependency on gproc to the blank for the .app file ( ):

  {description, "GProc tutorial"},
  ...
  {applications,
   [
    kernel,
    stdlib
    gproc% <--- Application depends on gproc
   ]},
  ...

We go back to the top-level directory where our project is stored, create a subdirectory in it reland go there:

$ cd ../../
$ mkdir rel && cd rel

Rel will contain the files needed to create the release - all that is required to run the project, all its runtime dependencies.
Using it, rebarcreate a blank for the node by passing its name in the parameter nodeid:

$ rebar create-node nodeid = gptnode

Editing the file reltool.config:

  ...
  {lib_dirs, ["../deps", "../apps"]},% <--- In these directories, reltool will also look for dependencies in our application
  {rel, "gptnode", "1",
   [
    kernel,
    stdlib
    sasl
    gproc,% <--- gproc application
    gpt% <--- Our application
   ]},
  ...

Next, you can edit the file files/vm.argsby changing, say, the name of the node:

-name [email protected]

on the

-sname gptnode @ localhost

Let's go back to the top-level directory:

$ cd ../

and create a file rebar.configwith the following contents:

%% Dependencies will lie here
{deps_dir, ["deps"]}.
%% Subdirectories into which rebar should look
{sub_dirs, ["rel", "apps / gpt"]}.
%% Compiler Options
{erl_opts, [debug_info, fail_on_warning]}.
%% Dependency List
%% The master branch of the corresponding git repository will be cloned into the gproc directory.
{deps,
 [
  {gproc, ". *", {git, "http://github.com/esl/gproc.git", "master"}}
 ]}.

Now we are ready to create a release. Let's execute several commands rebar(the output of commands is omitted):

$ rebar get-deps
$ rebar compile
$ rebar generate

The command get-depsdownloads the dependencies. In our case, this is the gproc application. The command compile, obviously, causes compilation of all source files, and generatecreates a release.
The directory rel/gptnodecan be safely moved to other hosts (of course, subject to binary compatibility, since the release includes the Erlang virtual machine). After creating the release, run what happened:

(cd rel / gptnode && sh bin / gptnode console)

Make sure that all the necessary applications are running:

(gptnode @ localhost) 1> application: which_applications ().
[{sasl, "SASL CXC 138 11", "2.1.9.2"},
 {gpt, "GProc tutorial", "1"},
 {gproc, "GPROC", "0.01"},
 {stdlib, "ERTS CXC 138 10", "1.17.2"},
 {kernel, "ERTS CXC 138 10", "2.14.2"}]

We are interested in gpt and gproc. As you can see, they are on this list.

Using gproc

So, we rebarfigured it out, learned how to create a simple project and work with it. Let's get down to gproc.
As you know, applications in Erlang, as a rule, consist of many processes that exchange messages.
In order for processes to know whom to send a message to, it is necessary to have a registrar that converts some coordinates into a process identifier. By default, Erlang / OTP provides process registration under the atom name. This is wasteful, since atoms are not collected by the garbage collector, and, having been created once, they live until the completion of the operation of the entire node, which will necessarily lead to the exhaustion of all memory, if necessary, register processes under unique names. Moreover, such an approach is inconvenient, since one would have to convert different terms into an atom, compose some rules for this, in addition, a process can be registered under only one name. Registering processes as atom using functionerlang:register/2valid only for a small number of long-lived processes, whose name should not change, the analogue is global variables in imperative programming languages.
To circumvent these restrictions, the following scheme is often used:

the registrar process is launched, creating the ets-table and being its owner;
when starting processes that need to be registered, they send a message to the registrar containing the coordinates for registration (any erlang term) and their identifier;
the registrar writes this mapping to the ets table and enables monitoring of the process with erlang:monitor/2;
upon completion, the registered process either explicitly sends a message about its deregistration, or the registrar receives a message 'DOWN'when this process crashes, after which it deletes its record from the ets table;

This scheme is very often used, almost every application has its own implementation, with its own features and bugs. Of course, that there is a natural desire to replace this registrar with something unique. And the solution to the problem came in the form of the developer Ulf Wiger and his gproc application (https://github.com/esl/gproc).
The application API can be found at github.com/esl/gproc/blob/master/doc/gproc.md .

Local registration

Consider the simplest case - local (on the current node) registration of a process by an arbitrary term.
The source code for examples can be taken here: github.com/Zert/gproc-tutorial.git
The code of the processes that we will register through gproc is in the file gpt_proc.erl. In gpt_sup.erlis the supervisor code for this process group. When the function gpt_sup:start_worker/1is called, our process will be launched and registered under the name that is passed to the function as the only argument. In this case, it is a number.
We launched the node using the above command and performed a series of process launches with different identifiers in it:

(gptnode @ localhost) 1> [gpt_sup: start_worker (Id) || Id <- lists: seq (1,3)].
(gpt_proc: 29) Start process: 1
(gpt_proc: 29) Start process: 2
(gpt_proc: 29) Start process: 3
[{ok, <0.61.0>}, {ok, <0.62.0>}, {ok, <0.63.0>}]

A function call gproc:add_local_name(Name)registers the process that calls it under the name Name(this function is just a wrapper over gproc:reg({n,l,Name}), where n- name, l- local). After that, the function gproc:lookup_local_name(Name)will return the process identifier.
Now let’s tell one of the processes to start waiting for the process to start and register with the name 4. The code responsible for this:

handle_info ({await, Id},
            #state {id = MyId} = State) ->
    gproc: await ({n, l, Id}),
    ? DBG ("MyId: ~ p. ~ NNewId: ~ p.", [MyId, Id]),
    {noreply, State};

Here, the function gproc:await/1is called with an argument of the following form: {n, l, Id}. For some reason, it does not have a wrapper, but oh well.

(gptnode @ localhost) 2> gproc: lookup_local_name (1)! {await, 4}.
{await, 4}

Starting the process with identifier 4, we will first see a message from it, and then from the first waiting process:

(gptnode @ localhost) 3> gpt_sup: start_worker (4).
(gpt_proc: 29) Start process: 4
(gpt_proc: 45) MyId: 1.
NewId: 4.
{ok, <0.66.0>}

Let's stop the process of receiving a message stop:

handle_info (stop, State) ->
    {stop, normal, State};

and stop it:

(gptnode @ localhost) 4> gproc: lookup_local_name (1)! stop.
stop

After that, the process is automatically deleted from the registrar database:

(gptnode @ localhost) 5> gproc: lookup_local_name (1).
undefined

Global registration

It is well known that Erlang is wildly distributed. This implies a transparent exchange of messages between nodes, in other words, having a process identifier, you can send a message to it without knowing which node it is on. Local registration of the process with gproc allows you to map an arbitrary term to the process identifier within one erlang node, while on the other node you cannot get the identifier value using this term.
In order for any node in the cluster to be able to register its processes so that they are accessible from other nodes, there is a global registration. GProc implements a call gproc:add_global_name/1to implement this action. Let's look at an example.
First, we will build two nodes combined in a cluster, andrebarit will help us, as it has the ability to create configuration files according to a given template. When creating a cluster, the following details must be considered:

Set the same cookie values for nodes
Give them different names.
Pass the appropriate parameters to the required application kernelon each node
When using GProc, pass the desired node roles to it.

The first two points are set in the file files/vm.args:

## Name of the node
-sname {{node}}
## Cookie for distributed erlang
-setcookie gptnode

Here {{node}}is the placeholder that will be filled in when the release is created. The virtual machine parameter -setcookiesets the cookie value for this node; in the cluster, all nodes must have the same values.
The second two points are set in the file files/app.config. Placeholders will also be used here:

 %% GProc
 {gproc, {{gproc_params}}},
%% Kernel
 {kernel, {{kernel_params}}},

To fill in the placeholders, we indicate in the file reltool.configthat the previous two files should be processed as templates:

  {template, "files / app.config", "etc / app.config"},
  {template, "files / vm.args", "etc / vm.args"}

We create two configuration files, one for each node: vars/dev1_vars.configand vars/dev2_vars.config. The file dev1_vars.configwill contain the following placeholder values:

%% etc / app.config
{gproc_params,
"[
  {gproc_dist, {['gpt1 @ localhost'],
                [{workers, ['gpt2 @ localhost']}]}}
 ] "}.
{kernel_params,
"[
  {sync_nodes_mandatory, ['gpt2 @ localhost']},
  {sync_nodes_timeout, 15000}
 ] "}.
%% etc / vm.args
{node, "gpt1 @ localhost"}.

For the file, the dev2_vars.configparameters sync_nodes_mandatoryand nodewill change places. Let's analyze them in more detail.
The parameter gproc_distrefers to the gproc application; it is a tuple of two lists. The first list is nodes that are able to become a leader (master), the second list contains key-value tuples, so far we only need one key - workerswhich sets up a list of nodes that are simple members of the cluster (slave).
There are two parameters to the kernel application. The first sync_nodes_mandatoryis a list of nodes that must be present in the cluster. Second,sync_nodes_timeout- time in milliseconds that each node will wait for the nodes from the previous list to appear. If during this time the nodes did not appear, then the node will stop. Let's make it value 15 seconds in order to have time to start them both hands.
The value nodewill be written to the startup parameters of the virtual machine, this is its name.
Now create two releases using the following rule from the Makefile:

dev1 dev2:
    mkdir -p dev
    (cd rel && rebar generate target_dir = .. / dev / $ @ overlay_vars=vars/$@_vars.config)

Go to the directory dev/dev1, launch the second terminal window (or create a new window in screen ), переходим в директориюdev / dev2 . Запускаем в каждом шелле./bin/gptnode console`. Let's see the list of available nodes in the first Erlang shell:

(gpt1 @ localhost) 1> nodes ().
[gpt2 @ localhost]

We see that the second node started normally and connected to the cluster. In order not to be wise for a long time, we will globally register the process of the current shell under some term:

(gpt1 @ localhost) 2> gproc: add_global_name ({shell, 1}).
true

In another window, try to request the process identifier for this term:

(gpt2 @ localhost) 2> gproc: lookup_global_name ({shell, 1}).
<3358.70.0>

As you can see, successfully. By sending a message to this process, we can receive it on the first node:

(gpt2 @ localhost) 3> gproc: lookup_global_name ({shell, 1})! {the, message}.
message

We read it on the first node with the command flush():

(gpt1 @ localhost) 3> flush ().
Shell got {the, message}
ok

Conclusion

That's all. I began to write an article for myself, since the documentation on rebar is very scarce and is constantly forgotten after the next use. Along the way, I started using gproc, and in order not to get up twice, I placed everything in one article.

Tags:

Using Rebar and GProc