Metaprogramming with JavaScript Examples

    This article is another attempt to rethink metaprogramming that I periodically make. The idea is refined each time, but this time it was possible to pick up quite simple and understandable examples that are both very compact and illustrative, have real useful applications and do not drag libraries and dependencies. At the time of publication, I will report this topic on OdessaJS , therefore, the article can be used as a place for questions and comments on the report. The format of the article makes it possible to more fully present the material than in the report, the listeners of which are not exempt from reading .

    UPD:Updated video version of the article on Youtube (the lecture was recorded at the Kiev Polytechnic Institute on April 18, 2019 as part of the course “100 video lectures on programming” ):

    A popular understanding of metaprogramming is usually very vague, and more often than not, it ends with the following options:
    • Compilation templates and macros
    • A program that changes itself
    • Program generating another program

    I propose the following definition:
    Metaprogramming is a programming paradigm built on a programmatic change in the structure and behavior of programs.
    And then we will analyze how it works, why it is needed and what advantages and disadvantages we get in the end.

    What is modeling?


    The concept of metaprogramming is closely related to modeling , because the method itself involves increasing the level of abstraction of models by removing metadata from the model. As a result, we get a metamodel and metadata . During early or late binding (when compiling, translating or running a program), we again get the model from the metamodel and metadata in an automatic programmatic way. The created model can change many times, without changing the program code of the metamodel, and often, even without stopping the program.

    Surprisingly, a person is able to successfully solve problems, the complexity of which exceeds the capabilities of his memory and thinking, using the construction of models and abstractions. The accuracy of these models determines their usefulness for decision making and development of control actions. The model is always inaccurate and displays only a small part of reality, one or more of its sides or aspects. However, in limited conditions of use, the model may be indistinguishable from the real object of the subject area. There are physical, mathematical, simulation and other types of models, but we will be interested in, first of all, information data models and program logic models. Programming paradigms, these are the models of program logic, for example, imperative, declarative, functional, eventful. Our task, as programmers, we can designate is not writing code, but, first of all, building models and abstractions. Thus, metaprogramming allows us to raise the level of abstraction in models,


    What is metaprogramming?


    Metaprogramming is not something new; you have always used it if you have practical programming experience in any language and in any applied field of application. All programming paradigms, at least for computers of Fonneimann architecture , somehow inherit the basic principles of modeling from this architecture. The most important principle of von Neumann architecture is mixing data and instructions that define the logic of data processing in one universal memory. That is, the absence of a fundamental difference between the program and the data. This has many consequences, firstly, the machine needs to distinguish where the command is, and where is the number and what is its bit depth and type, where is the address, and where is the array, where is the string, and where is the length of this string, in what encoding is it presented, etc. ., up to complex structures, like objects and visibility areas. All this is determined by metadata; without metadata, nothing happens at all in programming languages ​​for Fonneimann architecture. Secondly, the program gets access to the memory in which it is stored, other programs, their source code, and can process the code as data, which allows translation and interpretation, automated testing and optimization, introspection and debugging, dynamic linking and much more another.

    Definitions:
    MetadataIs data about data. For example, the type of a variable is the metadata of a variable, and the names and types of parameters of a function are the metadata of this function.
    Introspection is a mechanism that allows a program to get metadata about memory structures during operation, including metadata about variables and functions, types and objects, classes and prototypes.
    Dynamic binding (or late binding) is a call to a function through an identifier that turns into a call address only at the execution stage.
    A metamodel is a model of a high level of abstractness from which metadata is derived, and which dynamically generates a specific model when receiving metadata.
    Metaprogramming - This is a programming paradigm built on a programmatic change in the structure and behavior of programs.

    So, you cannot start using metaprogramming from today, but you can consciously analyze, analyze and apply the tool. This is paradoxical, but many tend to separate data and logic using Fonneimann architecture. Meanwhile, they should not be separated, but combined in the right way. There are other architectures, for example, analog solvers, digital signal processors (DSP), programmable logic integrated circuits (FPGAs), and others. In these architectures, the calculations are not carried out imperatively, that is, not by the sequence of processing operations specified by the algorithm, but by parallel working digital or analog elements, realizing a lot of mathematical and logical operations in real time and having a ready-made answer at any time. These are analogues of reactive and functional programming. In FPGAs, circuit switching occurs during reprogramming, while in DSP imperative logic controls small-scale circuit switching in real time. Metaprogramming is also possible for systems with non-imperative or hybrid logic, for example, I see no reason why one FPGA cannot reprogram another.

    Now consider a generalized modelshown in the diagram of the software module. Each module necessarily has an external interface and software logic. And such components as configuration, state, and read-only memory may either be absent or play a major role. The module receives requests from other modules through the interface and respond to them by exchanging data in certain protocols. The module sends requests to the interfaces of other modules from anywhere in its program logic, so incoming communications are connected by an interface, and outgoing communications are scattered throughout the module. Modules are part of larger modules and are themselves built from several or many submodules. The generalized model is suitable for modules of any scale, ranging from functions and objects, to processes, servers, clusters and large information systems. When modules interact, requests and responses are data, but theynecessarily contain metadata that affect how the module will process the data or how it tells another module to process the data. Usually, the set of metadata is limited by what the protocol necessarily requires to read the structure of the transmitted data. In binary formats, metadata is less than in syntactic formats used to serialize data (such as JSON and MIME). Information about the structure of binary formats, for the most part, is located at the receiving module in the form of struct (structures for C, C ++, C #, and other languages) or is “wired” into the logic of the interpreting moduleanother way. It is quite difficult to separate where data processing using metadata ends and metaprogramming begins. Conventionally, you can define such a criterion: when metadata does not just describe structures, but increases the abstraction of program code in a module that interprets data and metadata, metaprogramming begins here. In other words, when there is a transition from a model to a metamodel . The main sign of such a transition is the expansion of the universality of the module, and not the expansion of the universality of the protocol or data format. The diagram on the right shows how metadata is extracted from the data and enter the module, changing its behavior during data processing. Thus, the abstract metamodel contained in the module at the execution stage turns intospecific model .

    Before starting to consider the techniques and techniques of metaprogramming, I would like to give one quote, which I always give when it comes to metaprogramming. It suggests the idea that metaprogramming is a reflection of such a fundamental law on which all cybernetic systems are based. That is, “living” systems in which there is control, correction of behavior and activity parameters through regulation with feedback. This allows systems to reproduce their state and structure under different conditions and with different modifications, while maintaining significant and varying behavior, including generating derivative systems for this.

    “That's what I mean by a producing work or, as I called it last time,“ opera operans. ” In philosophy, there is a distinction between "natura naturata" and "natura naturans" - the generated nature and the generating nature. By analogy, one could form - "cultura culturata" and "cultura culturans". Say, the novel “In Search of Lost Time” is built not as a work, but as “cultura culturans” or “opera operans”. This is what the Greeks called the Logos. ”
    // Merab Mamardashvili "Lectures on ancient philosophy"

    How does metaprogramming work?


    Based on the definition, you need to parse the following three questions:
    1. When do changes occur?
    2. What exactly is changing?
    3. What changes are happening with?

    When changes occur : metaprogramming development time , for example, when the IDE analyzes your code as data, helping to modify it, suggesting the names of objects and functions, their types and even generates templates or automatically builds code blocks from diagrams or visual modeling tools, for example in visual editors of user interfaces, databases, and other CAD / CAM CAD tools. Examples of changes in compilation time : translators, including for creating typed algorithms from untyped ones and for generating code from a language with a higher level of abstraction into a language executed in a specific environment, up to the OS and hardware platform. But we are more interestedchanging the behavior of programs during their work , we will consider this in more detail below.

    So, I propose the following classification of metaprogramming according to the time of changes in behavior and structure:
    • during development (Design time)
    • at compile time
    • while the application is running (Run time)
      • during task execution (Just-in-Time)
      • between tasks (Lazy)
        • by time (Timer)
        • by external call (Pull)
        • by event (Push)
    Interpreting and linking Just-in-Time is not the best way, but sometimes it is the only possible way if metadata comes along with the data. But metadata, nevertheless, changes less frequently than requests occur, so the model can be built in advance and cached in anticipation of requests and data. Update the model for specific calls, to minimize calls, or you can periodically interrogate the source of metadata storage for changes. It’s best, of course, to have a notification channel from the source so that it triggers an update on a push basis.

    What exactly is changing?
    • data types and data structures ;
    • identifiers (names of classes, types, variables, both inside the module, and names by which the module refers to other modules);
    • calls (names of functions and methods, dynamic linking, including using the stub / skeleton pattern in which, in a single module, a stub is constructed that represents an object, class or function located in the address space of the remote module, so that all calls to stub were identical to remote calls);
    • parameters of data processing algorithms or parameters of models that may vary;
    • substitution of expressions, formulas and logical expressions, regular expressions, etc .;
    • the code itself is dynamically interpreted (metadata is usually declarative, but this is not necessary, it may be imperative or may contain fragments of an imperative code);
    • serialization / deserialization of data, objects and classes, as well as marshaling with code correction when transferring from one module to another (usually address correction, but there may be other corrections).

    What changes are happening with?
    • Parsing and translation of syntactic structures (with parsing by string operations or regular expressions).
    • Access to identifiers by name or index (including object parameters, associative arrays, etc.).
    • Full introspection (see definition above if the concept is unusual).
    • Individuation of objects of the first class (main method):
      • functions through closures;
      • objects through dynamic creation and impurities;
      • all the conceivable and inconceivable means that a programming language provides.

    Why do we need metaprogramming?


    Now we can highlight the main tasks and cases when metaprogramming significantly simplifies the implementation or even makes the solution possible:
    • Extending functionality, increasing software versatility.
    • Dynamic subject areas when changes are in regular mode.
    • Simplification of intersystem integration is a separate issue, but it helps a lot.

    Example 1


    Consider the simplest example of extracting metadata from a model and building a metamodel (see the example on github ). First we define the problem of the example: there is an array of lines, you need to filter them according to certain rules: the length of the matching lines should be from 10 to 200 characters inclusive, but excluding lines from 50 to 65 characters in length; the line should begin on “Mich” and not begin on “Abu”; the line must contain “V” and not contain “Lev”; the line must end with "ov" and must not end with "iov". Define the data for an example:

    let names = [
      'Marcus Aurelius Antoninus Augustus',
      'Darth Vader',
      'Victor Michailovich Glushkov',
      'Gottfried Wilhelm von Leibniz',
      'Mao Zedong',
      'Vladimir Sergeevich Soloviov',
      'Ibn Arabi',
      'Lev Nikolayevich Tolstoy',
      'Muammar Muhammad Abu Minyar al-Gaddafi',
      'Rene Descartes',
      'Fyodor Mikhailovich Dostoyevsky',
      'Benedito de Espinosa'
    ];
    

    We implement logic without metaprogramming:
    function filter(names) {
      let result = [], name;
      for (let i=0; i= 10 && name.length <= 200 &&
          name.indexOf('Mich') > -1 &&
          name.indexOf('V') === 0 &&
          name.slice(-2) === 'ov' &&
          !(
            name.length >= 50 && name.length <= 65 &&
            name.indexOf('Abu') > -1 &&
            name.indexOf('Lev') === 0 &&
            name.slice(-3) === 'iov'
          )
        ) result.push(name);
      }
      return result;
    }
    

    Select metadata from the model for solving the problem and form them in a separate structure:
    {
      length: [10, 200],
      contains: 'Mich',
      starts: 'V',
      ends: 'ov',
      not: {
        length: [50, 65],
        contains: 'Abu',
        starts: 'Lev',
        ends: 'iov'
      }
    }
    

    We are building a metamodel:
    function filter(names, conditions) {
      let operations = {
        length: (s, v) => s.length >= v[0] && s.length <= v[1],
        contains: (s, v) => s.indexOf(v) > -1,
        starts: (s, v) => s.indexOf(v) === 0,
        ends: (s, v) => s.slice(-v.length) === v,
        not: (s, v) => !check(s,v)
      };
      function check(s, conditions) {
        let valid = true;
        for (let key in conditions) valid &= operations[key](s, conditions[key]);
        return valid;
      }
      return names.filter(s => check(s, conditions));
    }
    

    The advantage of solving the problem using metaprogramming is obvious, we got a universal string filter, with configurable logic. If filtering needs to be performed not once, but several, on the same metadata configuration, then the metamodel can be wrapped in a closure and get cached individual functions to speed up work.

    Example 2


    The second example we will write immediately with the help of metaprogramming (see the example on github ), because if I imagine its dimensions in size in govnokode, then I get scared. Description of the task: you need to make HTTP GET / POST requests from specific URLs or download data from files and transfer received or read data via HTTP PUT / POST to other URLs and / or save them to files. There will be several such operations and they need to be performed at various time intervals. The task can be described as metadata as follows:

    [
      { interval: 5000, get: 'http://127.0.0.1/api/method1.json', save: 'file1.json' },
      { interval: '8s', get: 'http://127.0.0.1/api/method2.json', put: 'http://127.0.0.1/api/method4.json', save: 'file2.json' },
      { interval: '7s', get: 'http://127.0.0.1/api/method3.json', post: 'http://127.0.0.1/api/method5.json' },
      { interval: '4s', load: 'file1.json', put: 'http://127.0.0.1/api/method6.json' },
      { interval: '9s', load: 'file2.json', post: 'http://127.0.0.1/api/method7.json', save: 'file1.json' },
      { interval: '3s', load: 'file1.json', save: 'file3.json' },
    ]
    

    We solve the problem using metaprogramming:
    function iterate(tasks) {
      function closureTask(task) {
        return () => {
          console.dir(task);
          let source;
          if (task.get) source = request.get(task.get);
          if (task.load) source = fs.createReadStream(task.load);
          if (task.save) source.pipe(fs.createWriteStream(task.save));
          if (task.post) source.pipe(request.post(task.post));
          if (task.put) source.pipe(request.put(task.put));
        }
      };
      for (let i = 0; i < tasks.length; i++) {
        setInterval(closureTask(tasks[i]), duration(tasks[i].interval));
      }
    }
    

    We see that we wrote “beautiful columns” and we can make another convolution by taking out the metadata already inside the metamodel. What a metamodel configured by metadata will look like:
    function iterate(tasks) {
      // Metamodel configuration metadata
      //
      let sources = {
        get: request.get,
        load: fs.createReadStream
      };
      let destinations = {
        save: fs.createWriteStream,
        post: request.post,
        put: request.put
      };
      // Metamodel logic
      //
      function closureTask(task) {
        return () => {
          console.dir(task);
          let verb, source, destination;
          for (key in sources) if (task[key]) source = sources[key](task[key]);
          for (key in destinations) if (task[key]) source.pipe(destinations[key](task[key]));
        }
      }
      for (let i = 0; i < tasks.length; i++) {
        setInterval(closureTask(tasks[i]), duration(tasks[i].interval));
      }
    }
    

    I note that in the example closures are used to individuate tasks.

    Example 3


    The second example uses the duration function, which returns a value in milliseconds that we did not consider. This function interprets the interval value specified as a string in the format: “Dd Hh Mm Ss”, for example “1d 10h 7m 13s”, each component of which is optional, for example “1d 25s”, if the function receives a number, then it gives it back, this it’s necessary for the convenience of specifying metadata if we set the interval directly in milliseconds.

    // Parse duration to seconds, example: duration('1d 10h 7m 13s')
    // Parse duration to seconds
    // Example: duration('1d 10h 7m 13s')
    //
    function duration(s) {
      let result = 0;
      if (typeof(s) === 'string') {
        let days = s.match(/(\d+)\s*d/),
          hours = s.match(/(\d+)\s*h/),
          minutes = s.match(/(\d+)\s*m/),
          seconds = s.match(/(\d+)\s*s/);
        if (days) result += parseInt(days[1]) * 86400;
        if (hours) result += parseInt(hours[1]) * 3600;
        if (minutes) result += parseInt(minutes[1]) * 60;
        if (seconds) result += parseInt(seconds[1]);
        result = result * 1000;
      } if (typeof(s) === 'number') result = s;
      return result;
    }
    

    Now we implement the interpretation configured by metadata:
    function duration(s) {
      if (typeof(s) === 'number') return s;
      let units = {
        days: { rx: /(\d+)\s*d/, mul: 86400 },
        hours: { rx: /(\d+)\s*h/, mul: 3600 },
        minutes: { rx: /(\d+)\s*m/, mul: 60 },
        seconds: { rx: /(\d+)\s*s/, mul: 1 }
      };
      let result = 0, unit, match;
      if (typeof(s) == ='string') for (let key in units) {
        unit = units[key];
        match = s.match(unit.rx);
        if (match) result += parseInt(match[1]) * unit.mul;
      }
      return result * 1000;
    }
    

    Example 4


    Now let's look at the introspection metaprogramming used to integrate the modules. First, we define the remote methods on the client using this structure and show how to use these calls when writing application logic:

    let ds = wcl.AjaxDataSource({
      read: { get:  'examples/person/read.json' },
      insert: { post: 'examples/person/insert.json' },
      update: { post: 'examples/person/update.json' },
      delete: { post: 'examples/person/delete.json' },
      find: { post: 'examples/person/find.json' },
      metadata: { post: 'examples/person/metadata.json' }
    });
    ds.read({ id: 5 }, (err, data) => {
      data.phone = '+0123456789';
      ds.update(data, () => console.log('Data saved'));
    });
    

    Now we initialize from the metadata received from another module and show that the application logic has not changed:
    let ds = wcl.AjaxDataSource({
      introspect: { post: "examples/person/introspect.json" }
    });
    ds.read({ id:3 }, (err, data) => {
      data.phone ="+0123456789";
      ds.update(data, () => console.log('Data saved'));
    });
    

    In the following example, we will create a local data source with the same interface as the remote one and show that the application logic has not changed either:
    let ds = wcl.MemoryDataSource({ data: [
      {
        id: 1, name: 'Person 1', phone: '+380501002011',
        emails: [ 'person1@domain.com' ], age: 25
      },
      {
        id: 2, name: 'Person 2', phone: '+380501002022',
        emails: [ 'person2@domain.com', 'person2@domain2.com' ],
        address: { city: 'Kiev', street: 'Khreschatit', building: '26' }
      },
      {
        id: 3, name: 'Person 3', phone: '+380501002033',
        emails: [ 'person3@domain.com' ],
        tags: [ { tag: 'tag1', color: 'red' }, { tag: 'tag2', color: 'green' } ]
      },
    ]});
    ds.read({ id: 3 }, (err, data) => {
      data.phone ="+0123456789";
      ds.update(data, () => console.log('Data saved'));
    });
    

    conclusions


    Metaprogramming Techniques
    • Task description style: declarative (metadata), using imperative and functional inserts
    • Hashes (associative arrays) do not know the key in advance: let a = {}; a [key] = value;
    • Interpretation of lines, we invent our own syntaxes or take the generally accepted ones (json, js, regexp ...)
    • Impurities (mixins): we don’t know in advance where to add function mixin (a) {a.fn = () => {...}}
    • Closures: we personalize the functions fn = (a => () => a * 2) (value)
    The consequences of metaprogramming
    • Code size: often decreases sharply, but sometimes it can increase slightly
    • Performance: slightly decrease, but with proper implementation remains approximately the same
    • Flexibility: the program code is becoming more universal, the scope of software is expanding
    • Integration: Usually greatly simplified and requires fewer code changes.
    • Pleasure from work: metaprogramming is more interesting, therefore more fun and motivation
    • Development speed: development takes longer and maintenance is much easier, a lot of time is saved

    Related Links


    1. Full source code examples on Github: https://github.com/tshemsedinov/metaprogramming
    2. Report Slides: http://www.slideshare.net/tshemsedinov/javascript-36636872

    Old articles to track idea development

    1. Metaprogramming http://habrahabr.ru/post/137446/
    2. Dynamic interpretation of metamodels http://habrahabr.ru/post/154891/
    3. Extended dynamic interpretation scheme http://blog.meta-systems.com.ua/2011/01/blog-post_28.html
    4. The use of metamodel in designing databases with several abstract layers: Part 1 http://habrahabr.ru/post/119317/
    5. The use of metamodel in designing databases with several abstract layers: Part 1 http://habrahabr.ru/post/119885/
    6. Integration of information systems http://habrahabr.ru/post/117468/
    7. Introduction of the meta-level http://blog.meta-systems.com.ua/2011/01/blog-post.html
    8. Metamodel in the problems of integration of information systems http://blog.meta-systems.com.ua/2010/07/blog-post.html
    9. Call or metadata integration? http://blog.meta-systems.com.ua/2009/10/blog-post_18.html
    10. Model and metamodel http://blog.meta-systems.com.ua/2009/10/blog-post_05.html

    Only registered users can participate in the survey. Please come in.

    Do you use metaprogramming?

    • 42.6% Always wrote like that, not realizing that this is metaprogramming 101
    • 12.6% Always wrote like that, realizing that this is metaprogramming 30
    • 12.6% First time I learned from the article, I'll try to apply 30
    • 20.6% I used to listen by the edge of my ear, I will pay more attention 49
    • 11.3% Metaprogramming is crazy crazy 27

    Also popular now: