March 21, 2012 at 16:01Using parse_transform
Disclaimer: The tool described has a controversial reputation. I do not urge to use it wherever I get, I only familiarize myself with the concepts used in order to reduce some of the thrill of technology.
The written sources as well as a text copy of the article can be found on the github .
parse_transform - AST change mechanism before compilation. Designed to change the meaning of constructions (semantics) without going beyond Erlang syntax.
Unfortunately, there is little information about this on the Web, which makes the entry threshold very high for non-guru erlang.
In this article, I’ll talk a bit about AST erlang, give an example of simple transformations, and also show the process of writing parse_transform to create a stateless gen_server (the task doesn’t make much sense, but it will work as an example), and in the end I’ll give a link on a set of beginner transformer.
Just in case: the definition of AST
It is better to see the AST once than to read its description a hundred times . Therefore, we will write a small module to see how each line is transformed.
So, the source code for astdemo.erl :
To see the AST, you need to set the parse_file function from the epp module on this file:
It can be seen that each expression is converted to a tube of length at least 3, while the first two elements are always a type and a string, followed by a description specific to it. If it is not clear what is in a particular place, the documentation is at your service.
Let's do a dummy-parse_transform now to see what we have to deal with next. To do this, create a module that will deal with the transformation, and instead of manipulating the AST, just print it.
So demo_pt.erl :
Insert the corresponding directive in astdemo.erl :
We compile:
As you can see, the AST is the same (accurate to line breaks), but this time it was printed at compile time.
It should be noted that the compiler directives have already been deleted in the AST that arrived for transformation.
What is conveyed in the options, the curious reader will probably find out on his own. This article is about AST.
Let's do a thing that is useless in practice for training - rename the function "hello / 0" to "hi / 0". This will be easy to do, since hello / 0 is not called from within the module, but only has the ability to be called from the outside. Therefore, it is enough to change the list of exports and the title of the function.
Since AST (Forms binding) is a list, each element of which is a form of a very short list of types, it is logical to pass all Forms through a mutator function. Since the task is simple, and the transformation of each expression does not depend on the rest of the content, lists: map is suitable for us .
A function that will change the export and function headers will look something like this:
We enable this function by changing the code of the parse_transform function :
We compile demo_pt , make sure that we don’t mess it up.
We try to compile astdemo with the new transformer :
Perfectly! It worked as they wanted. Time to do something a little more useful.
Sometimes when writing a module with the behavior of gen_server, there is no need to drag State along with it, since there is nothing to store in it, and dragging State from handle_anything into the final expression clogs the code. Let's do a parse_transform, which allows us to define handle_call / 2, handle_cast / 1, handle_info / 1 . Or not. To make the article a little shorter, I will show only the transformation handle_call / 2 -> handle_call / 3 , and those who are interested will redefine everything else.
The behavior of gen_server requires the definition of handle_call (for simplicity) this way ( documentation ):
Since we get rid of the need to consider State, let our syntax be like this:
We will train on it. Defined handle_call in our syntax and its analogue in canonical form for comparing and writing a transformer.
Everything was written as it was last time - looking at the output of epp: parse_file and customizing what is there to what is needed.
Success! It remains to finish the owl and put it on the github.
An interested reader, I hope, met AST in Erlang, and also received an approximate idea of the methods of its transformation. Perhaps someone first learned about parse_transform.
The article contains information that should be enough to start writing your own transformation. A little lower will be criticism and a link to a library useful for transformations.
First, using parse_transform (if it is in a separate project) adds dependency to your project. In the case of rebar, this is not fatal.
Secondly, people reading (and especially editing) such code may not immediately understand the concept. Therefore, we need not only good documentation, but also a noticeable link to it at the beginning of the source.
Thirdly, the ability to write your own dialects is very limited. Before the AST gets under your scalpel, it works out a regular parser. Therefore, introducing tricky keywords and your own operators can break the parser, greatly complicating the task.
parse_trans is a useful thing for writing parse_transforms. It allows you to make a recursive map onto a tree, which is extremely useful when modifying expressions at a variable depth. The examples have a very concise way of rewriting the “!” Operator on a call to gproc: send .
The written sources as well as a text copy of the article can be found on the github .
What is parse_transform
parse_transform - AST change mechanism before compilation. Designed to change the meaning of constructions (semantics) without going beyond Erlang syntax.
Unfortunately, there is little information about this on the Web, which makes the entry threshold very high for non-guru erlang.
What will we do
In this article, I’ll talk a bit about AST erlang, give an example of simple transformations, and also show the process of writing parse_transform to create a stateless gen_server (the task doesn’t make much sense, but it will work as an example), and in the end I’ll give a link on a set of beginner transformer.
AST in Erlang
Just in case: the definition of AST
It is better to see the AST once than to read its description a hundred times . Therefore, we will write a small module to see how each line is transformed.
So, the source code for astdemo.erl :
-module(astdemo).
-export([hello/0, hello/2]).
hello() ->
hello("world", 1).
hello(_What, 0) ->
ok;
hello(What, Count) ->
io:format("Hello, ~s~n", [What]),
hello(What, Count - 1).
To see the AST, you need to set the parse_file function from the epp module on this file:
Eshell V5.8.5 (abort with ^ G)
1> {ok, Forms} = epp: parse_file ("astdemo.erl", [], []), io: format ("~ p ~ n", [Forms]).
[{attribute, 1, file, {"astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 2, export, [{hello, 0}, {hello, 2}]},
{function, 4, hello, 0,
[{clause, 4, [], [],
[{call, 5,
{atom, 5, hello},
[{string, 5, "world"}, {integer, 5.1}]}]}]},
{function, 7, hello, 2,
[{clause, 7, [{var, 7, '_ What'}, {integer, 7.0}], [], [{atom, 8, ok}]},
{clause, 9,
[{var, 9, 'What'}, {var, 9, 'Count'}],
[],
[{call, 10,
{remote, 10, {atom, 10, io}, {atom, 10, format}},
[{string, 10, "Hello, ~ s ~ n"},
{cons, 10, {var, 10, 'What'}, {nil, 10}}]},
{call, 11,
{atom, 11, hello},
[{var, 11, 'What'},
{op, 11, '-', {var, 11, 'Count'}, {integer, 11,1}}]}]}]}},
{eof, 12}]
ok
It can be seen that each expression is converted to a tube of length at least 3, while the first two elements are always a type and a string, followed by a description specific to it. If it is not clear what is in a particular place, the documentation is at your service.
Parse_transform / 2 function
Let's do a dummy-parse_transform now to see what we have to deal with next. To do this, create a module that will deal with the transformation, and instead of manipulating the AST, just print it.
So demo_pt.erl :
-module(demo_pt).
-export([parse_transform/2]).
parse_transform(Forms, _Options) ->
io:format("~p~n", [Forms]),
Forms.
Insert the corresponding directive in astdemo.erl :
-module(astdemo).
-compile({parse_transform, demo_pt}).
-export([hello/0, hello/2]).
...........
We compile:
Eshell V5.8.5 (abort with ^ G)
1> c (astdemo).
[{attribute, 1, file, {"./ astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 3, export, [{hello, 0}, {hello, 2}]},
{function, 5, hello, 0,
[{clause, 5, [], [],
[{call, 6,
{atom, 6, hello},
[{string, 6, "world"}, {integer, 6.1}]}]}]},
{function, 8, hello, 2,
[{clause, 8, [{var, 8, '_ What'}, {integer, 8,0}], [], [{atom, 9, ok}]},
{clause, 10,
[{var, 10, 'What'}, {var, 10, 'Count'}],
[],
[{call, 11,
{remote, 11, {atom, 11, io}, {atom, 11, format}},
[{string, 11, "Hello, ~ s ~ n"},
{cons, 11, {var, 11, 'What'}, {nil, 11}}]},
{call, 12,
{atom, 12, hello},
[{var, 12, 'What'},
{op, 12, '-', {var, 12, 'Count'}, {integer, 12,1}}]}]}]},
{eof, 13}]
{ok, astdemo}As you can see, the AST is the same (accurate to line breaks), but this time it was printed at compile time.
It should be noted that the compiler directives have already been deleted in the AST that arrived for transformation.
What is conveyed in the options, the curious reader will probably find out on his own. This article is about AST.
First transformations
Let's do a thing that is useless in practice for training - rename the function "hello / 0" to "hi / 0". This will be easy to do, since hello / 0 is not called from within the module, but only has the ability to be called from the outside. Therefore, it is enough to change the list of exports and the title of the function.
Single form transformer
Since AST (Forms binding) is a list, each element of which is a form of a very short list of types, it is logical to pass all Forms through a mutator function. Since the task is simple, and the transformation of each expression does not depend on the rest of the content, lists: map is suitable for us .
A function that will change the export and function headers will look something like this:
% hello_to_hi replaces occurences of hello/0 with hi/0
hello_to_hi({attribute, Line, export, Exports}) ->
% export attribute. Replace {hello, 0} with {hi, 0}
HiExports = lists:map(
fun ({hello, 0}) -> {hi, 0};
(E) -> E
end, Exports),
{attribute, Line, export, HiExports};
hello_to_hi({function, Line, hello, 0, Clauses}) ->
% Header of hello/0. Just replace hello with hi
{function, Line, hi, 0, Clauses};
hello_to_hi(Form) ->
% Default: do not modify form
Form.
Now all together
We enable this function by changing the code of the parse_transform function :
parse_transform(Forms, _Options) ->
HiForms = lists:map(fun hello_to_hi/1, Forms),
io:format("~p~n", [HiForms]),
HiForms.
We compile demo_pt , make sure that we don’t mess it up.
Check
We try to compile astdemo with the new transformer :
Eshell V5.8.5 (abort with ^ G)
1> c (astdemo).
[{attribute, 1, file, {"./ astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 3, export, [{hi, 0}, {hello, 2}]},
{function, 5, hi, 0,
[{clause, 5, [], [],
[{call, 6,
{atom, 6, hello},
[{string, 6, "world"}, {integer, 6.1}]}]}]},
{function, 8, hello, 2,
[{clause, 8, [{var, 8, '_ What'}, {integer, 8,0}], [], [{atom, 9, ok}]},
{clause, 10,
[{var, 10, 'What'}, {var, 10, 'Count'}],
[],
[{call, 11,
{remote, 11, {atom, 11, io}, {atom, 11, format}},
[{string, 11, "Hello, ~ s ~ n"},
{cons, 11, {var, 11, 'What'}, {nil, 11}}]},
{call, 12,
{atom, 12, hello},
[{var, 12, 'What'},
{op, 12, '-', {var, 12, 'Count'}, {integer, 12,1}}]}]}]},
{eof, 13}]
{ok, astdemo}
2> astdemo: hi ().
Hello world
okPerfectly! It worked as they wanted. Time to do something a little more useful.
Stateless gen_server parse_transform
Sometimes when writing a module with the behavior of gen_server, there is no need to drag State along with it, since there is nothing to store in it, and dragging State from handle_anything into the final expression clogs the code. Let's do a parse_transform, which allows us to define handle_call / 2, handle_cast / 1, handle_info / 1 . Or not. To make the article a little shorter, I will show only the transformation handle_call / 2 -> handle_call / 3 , and those who are interested will redefine everything else.
Concept
The behavior of gen_server requires the definition of handle_call (for simplicity) this way ( documentation ):
handle_call(Request, From, State) ->
.....
{reply,Reply,NewState}.
Since we get rid of the need to consider State, let our syntax be like this:
handle_call(Request, From) ->
.....
Reply.
Transformation plan
- Find and change in export handle_call / 2 to handle_call / 3
- Among the function definitions for handle_call / 2, add the State parameter and frame the final expression in each clause in {reply, ..., State}
Cat
We will train on it. Defined handle_call in our syntax and its analogue in canonical form for comparing and writing a transformer.
-module(sl_gs_demo).
-behavior(gen_server).
-compile({parse_transform, sl_gs}).
-export([handle_call/2, ref_handle_call/3]).
-export([handle_cast/2, handle_info/2]).
-export([init/1, terminate/2, code_change/3]).
% This will be transformed
handle_call(Req, From) ->
{Req, From}.
% That's what handle_call should finally look like
ref_handle_call(Req, From, State) ->
{reply, {Req, From}, State}.
% Dummy functions to make gen_server happy
% Exercise: Try to insert them automatically during transformations :)
handle_cast(_, State) -> {noreply, State}.
handle_info(_, State) -> {noreply, State}.
init(_) -> {ok, none}.
terminate(_, _) -> ok.
code_change(_, State, _) -> {ok, State}.
The code
Everything was written as it was last time - looking at the output of epp: parse_file and customizing what is there to what is needed.
-module(sl_gs).
-export([parse_transform/2]).
parse_transform(Forms, _Options) ->
lists:map(fun add_missing_state/1, Forms).
add_missing_state({attribute, Line, export, Exports}) ->
% export attribute. Replace {handle_call, 2} with {handle_call, 3}
NewExports = lists:map(
fun ({handle_call, 2}) -> {handle_call, 3};
% You can add more clauses here for other function mutations
(E) -> E
end, Exports),
{attribute, Line, export, NewExports};
add_missing_state({function, Line, handle_call, 2, Clauses}) ->
% Mutate clauses
NewClauses = lists:map(fun change_call_clause/1, Clauses),
% Finally, change arity in header
{function, Line, handle_call, 3, NewClauses};
add_missing_state(Form) ->
% Default
Form.
change_call_clause({clause, Line, Arguments, Guards, Body}) ->
% Change arity in clauses.
NewArgs = Arguments ++ [{var, Line, 'State'}], % Add State argument
% Then replace last statement of each clause with corresponding tuple
NewBody = change_call_body(Body),
{clause, Line, NewArgs, Guards, NewBody}.
change_call_body([Statement | Rest=[_|_] ]) -> % Rest has to be non-empty list for this
% Recurse to change only last statement
[Statement|change_call_body(Rest)];
change_call_body([LastStatement]) ->
% Put it into tuple. Lines are zero to omit parsing LastStatement
[{tuple,0, [{atom,0,reply},
LastStatement,
{var,0,'State'}]
}].
Health Check
Eshell V5.8.5 (abort with ^ G)
1> c (sl_gs_demo).
{ok, sl_gs_demo}
2> {ok, D} = gen_server: start_link (sl_gs_demo, [], []).
{ok, <0.39.0>}
3> gen_server: call (D, hello).
{hello, {<0.32.0>, # Ref <0.0.0.83>}}
Success! It remains to finish the owl and put it on the github.
Summary
An interested reader, I hope, met AST in Erlang, and also received an approximate idea of the methods of its transformation. Perhaps someone first learned about parse_transform.
The article contains information that should be enough to start writing your own transformation. A little lower will be criticism and a link to a library useful for transformations.
Criticism of the method
First, using parse_transform (if it is in a separate project) adds dependency to your project. In the case of rebar, this is not fatal.
Secondly, people reading (and especially editing) such code may not immediately understand the concept. Therefore, we need not only good documentation, but also a noticeable link to it at the beginning of the source.
Thirdly, the ability to write your own dialects is very limited. Before the AST gets under your scalpel, it works out a regular parser. Therefore, introducing tricky keywords and your own operators can break the parser, greatly complicating the task.
Parse_trans library
parse_trans is a useful thing for writing parse_transforms. It allows you to make a recursive map onto a tree, which is extremely useful when modifying expressions at a variable depth. The examples have a very concise way of rewriting the “!” Operator on a call to gproc: send .