DRuby aka DRb - the basis of distributed systems in Ruby. Principle of work and pitfall

    Recently, The dRuby book - distributed and parallel computing with Ruby (a translation of a Japanese book written by the author of the library itself) was released. In this article, I will try to give an overview of the book chapters regarding the DRb library. If you want to familiarize yourself with the topic in more detail, you can buy or download the book . I must say right away that I will not talk about thread synchronization in this post, nor about the Rinda library.

    Suppose you are writing a system that works with more than one process. For example, you have a web server that runs tasks that run for a long time in the background. Or you just need to ensure that data is transferred from one process to another and coordinate. For such situations, you need a DRb library. It is written entirely in Ruby and is included in the standard library, so you can start working with it instantly. To connect it, just write the require 'drb'

    advantages of the DRb library for the most part stem from the dynamism of the Ruby language itself.
    Firstly, when spending minimal effort on the preparatory stage, then you work with objects without hesitation where they are located: in one process or in another. The library completely disguises all the technical details from you.
    Secondly, you are not required to hard-code the interface. Any ruby ​​object can expose its interface to the outside - this way you can either use the functionality of one of the standard classes of type Hashor Queue, or you can make your own class with any interface. In addition, nothing prevents you from changing the interface right in the process of execution, and even using it method_missingto process any requests. And of course, updating the server interface does not affect the client at all, unless the client calls methods that have changed the signature or behavior. Thus, the server and client are as independent as possible.
    And finally, the client is not even obliged to know the classes of objects that the server returns to him, he can use them without it. Thus, the server is free to hide as many details as it pleases.
    But, of course, there are pitfalls, and there are plenty of them. Fortunately, dRuby is uncomplicated in understanding, but understanding its structure allows most of the problems simply to be avoided. The documentation for this library, unfortunately, does not clarify many points, so the article will be interesting for both beginners and people who have already worked with the library.

    To verify that everything works, open two irb terminals. I admit that I don’t know how big the differences are in ruby ​​1.8, so let's agree that we are discussing version 1.9 (especially since 1.8 - will soon cease to be supported, hooray!)

    Conditionally, these two terminals are a server and a client. The server must provide a front object that will receive requests. This object can be any object: even an object of a built-in type, even a module with a specially created interface. In turn, the client connects to the server and interacts with this object.
    For example, let's start the server in the first terminal and expose an ordinary array.

    require 'drb'
    front = []
    DRb.start_service('druby://localhost:1234', front)
    front << 'first'
    # если вы запускаете сервер не в консоли, а отдельным скриптом - обязательно добавьте строку DRb.thread.join
    

    Now connect the client. We recognize the first element of the array and write another element to the array

    require 'drb'
    DRb.start_service
    remote_obj = DRbObject.new_with_uri('druby://localhost:1234')
    p remote_obj
    p remote_obj[0]
    remote_obj << 'second'
    


    Now you can call from the first terminal front[1]and see that there is a line 'second'. And you can connect another client, and from it also operate on the same front-end object.

    As you already noticed, the server is started by the command DRb.start_service(keep an eye on the register!). The method takes as an argument a string with the address of the view 'druby://hostname:port'and a front object. The front object is the object that will receive requests.
    When the server starts in a separate script, you must writeDRb.thread.joinat the end of the script. The fact is that the DRb server is launched in a separate thread, and Ruby shuts down the program as soon as the main thread is completed. Therefore, if the main thread does not wait for the DRb server stream to close, then both streams will be completed ahead of schedule, and the server will immediately become unavailable. Be prepared for the fact that the execution of the method DRb.Thread.joinblocks the current thread until the server is turned off.

    In order to connect to the server, you must call the method DRbObject.new_with_uriand pass it the address at which the server starts. This method will return a proxy objectremote_obj. Requests (method calls) to the proxy object are automatically transferred to the object on the remote server, the called method is executed there, and then the result is returned to the client calling the method. (However, not all methods are called on the server. For example, judging by the behavior, the method #classis executed locally)
    The DRb.start_serviceclient will discuss the meaning of the command a little later.

    Let’s finally figure out how the method of the remote object is executed. To do this, calling the method of the proxy object serializes (marshals) the name of the method and the list of arguments, transfers the resulting string using the TCP protocol to the server, which deserializes the call arguments, executes the method on the front-end object, serializes the result and passes it back to the client. Everything looks simple. In fact, you work with a remote object in the same way as with a regular one, and a lot of actions for remote execution of the method, the proxy object and the server are hidden from you.

    But not so simple. Remote method invocation is expensive. Imagine that a method on a server "pulls" a lot of argument methods. This would mean that the server and the client, instead of doing the calculations, the lion's share of the time would access each other using a relatively slow protocol (and it’s good if the two processes are located on the same machine). To prevent this, arguments and results between processes are passed by value, not by reference (the marshalization of the object stores only the internal state of the object, and does not know itobject_id- the object will be serialized at first, and then deserialized will only be a copy of the original object, but not the same object, so the transfer is automatically done by copy). In Ruby, usually everything is passed by reference, and in dRuby, usually by value. Thus, if you execute front[0].upcase!on the server, the value front[0]will change, and if you execute remote_obj[0].upcase!, you will receive the first element in upper case, but the value on the server will not change, since it remote_obj.[](0)is a copy of the first element. This call can be considered similar to the method. front[0].dup.upcase!
    However, you can always define the behavior of dRuby in such a way as to pass arguments and the result by reference, but more on that later.

    Now is the time to talk about the first problem. Not all objects are marshalized. For example, Proc and IO objects, as well as threads (Thread objects) cannot be marshaled and transmitted over a copy. dRuby in this case proceeds as follows: if the marshalization did not work, then the object is passed by reference.
    So, how is the object passed by reference? Recall C. There, pointers are used for this purpose. In Ruby, the role of a pointer is fulfilled object_id. To transfer an object by reference, an object of the class is used DRbObject.
    DRbObject- This is, in fact, a proxy object for transmission by reference. An instance of this class DRbObject.new(my_obj)contains an object_idobjectmy_objand the URI of the server where the object came from. This allows you to intercept a method call and pass it to the very object on the remote machine (or in another terminal) to which the method was intended.

    Let's make a method for our server

    def front.[](ind)
      DRbObject.new(super)
    end
    

    And run the code from the client.

    remote_obj.[0].upcase!
    


    The new method #[]returned not a copy of the first element, but a link, so that after the method has been executed, the upcase!front-end object has changed, it is easy to check by executing, for example, a command puts remote_objor puts frontfrom the client and server, respectively.

    But to write every time DRbObject.newis laziness. Fortunately, there is another way to pass an object by reference, rather than by value. To do this, it is enough to make the object non-marshalizable. This is easy to do, just include a module in the object DRbUndumped.
    my_obj.extend DRbUndumped
    class Foo; include DRbUndumped; end
    

    Now the object my_objand all objects of the class Foowill be automatically passed by reference (and Marshal.dump(my_obj)will produce TypeError 'can\'t dump').

    I will give an example that I have encountered in practice. The server sets the hash as the front object, in which the values ​​are tickets (from the inside the ticket is a state machine). Then remote_obj[ticket_id]gives a copy of the ticket. But this does not allow us to change the ticket status on the server, only locally. Let's get DRbUndumpedin the classroom Ticket. Now we get from the hash not a copy of the ticket, but a link to it - and any actions with it now happen not on the client, but directly on the server.

    And now it's time to remember the promise and tell us why you need to callDRb.start_serviceat the client. Imagine that an array is indicated by the front object on your server, as in the first example.
    Now let the client call the method. remote_obj.map{|x| x.upcase}
    In fact, the map method with a block argument is called on the front object. And we, as we recall, cannot be marshaled. So this block argument is passed by reference. The method mapon the server will access it with instructions yield, which means the client is the server! But since the client has to be a server from time to time, it means that it also has to start the DRb server using the method start_service. It is not necessary to specify the URI of this server. How it works from the inside, I do not know, but it works. And as you already noticed, the differences between the client and server are less than it might seem.

    There is a risk of stumbling into a new nuisance. Suppose the method returned a link (not a copy) to the object generated directly in the method. If the server did not save this object anywhere separately (for example, did not put it in a special hash), then the server does not have a link to it. The client on the remote machine has, but the server does not! Therefore, sooner or later the troll will come to you for mail for this object will come GC - the garbage collector. This means that after some time DRbObjectthe client’s type link will “go bad” and will point to nowhere. Attempting to access the methods of this object will cause an error.
    Therefore, care must be taken that the server stores links to returned objects, at least until they are used by the server. There are several solutions for this:
    1) save all returned objects passed by reference to an array - then the garbage collector will not collect them, because the link is used;
    2) send the client a link to the block. For example:
    Instead of this code:
    Ticket.send :include, DRbUndumped
    def front.get_ticket
      Ticket.new
    end
    foo = remote_obj.get_ticket
    foo.start
    foo.closed? # Здесь ссылка foo может уже не иметь оригинала на сервере. Особенно, если метод start длится достаточно долго.
    


    It should be written like this:
    Ticket.send :include, DRbUndumped
    def front.get_ticket
      object_to_reference = Ticket.new
      yield object_to_reference
    end
    remote_obj.get_ticket do |foo|
      foo.start
      foo.closed?
    end
    

    A valid local variable on the server cannot be collected by the garbage collector. So, inside the block, the link is guaranteed to work.
    3) The book describes another way - you need to wedge into the process of creating a link at the stage of receiving the object_idobject and try at this moment to somehow delay the garbage collection process. You can automatically add an element to the hash and store the object forever (as you can guess, the memory will run out sooner or later), you can store a link to the object and clear it manually, you can clear this hash once every few minutes.
    The last method can be implemented by doing
    require 'drb/timeridconv'
    DRb.install_id_conv(DRb::TimerIdConv.new)
    

    before starting the server. For more information, see Chapter 11 of the book - Handling Garbage Collection. It seems interesting to me, and perhaps after reading it you will have new ways to use the manipulation of the garbage collection process. But still, I think that in practice it is better to use the second method - and issue links in a block. More reliable and understandable.

    It remains to illuminate, probably, the last moment. Suppose you pass an object Fooas a link. The client does not know about any class, Fooand yet this does not prevent him from working with the object. In essence, the client operates on the class object DRbObject. Everything is as usual.
    Now imagine that you are transmitting not a link, but a copy. Serialization on the server saved the state of the object and the name of its class. The client received the string and is trying to deserialize it. Of course, this does not work for him, because the client cannot create an object of a nonexistent class Foo. Then deserialization will return an object of the type DRb::DRbUnknownthat will store the buffer with the marshaled object. This object can be passed on (for example, to the task queue). You can also find out the name of the class, load the appropriate library with the class and call the method reload- then another attempt will be made to deserialize. This is

    No, yet this is not the last moment. I promised not to write about synchronization, but still I’ll say a few words.
    For distributed programming, synchronization of actions and atomicity of operations are critical concepts. The server starts in a separate thread. And for each request to the server, a separate thread is automatically created in which this request is processed. So it is simply necessary to prohibit different threads from accessing the same information at the same time. So, when programming distributed and parallel systems, use:
    1) the construction lock = Mutex.new; lock.synchronize{ do_smth }
    2) the standard library module MonitorMixin
    3) the standard library classes Queue,SizedQueue

    Good luck using DRb! I hope I prevented someone from long hours trying to understand why the object does not change, although you use the destructive method, how to make the method that accepts the block work on the client, and why the received link worked and worked - and suddenly stopped.
    However, in the book you will find much more, especially about the Rinda library and its counterparts.

    Also popular now: