ZeroRPC - a lightweight, reliable library for distributed communication between servers

    Earlier, I needed to implement some kind of my own statsd-like metric collection server, but with a few narrow-specific features, under which no ready-made or semi-finished solution went without a good file. In this regard, it was decided to implement a simple client-server protocol in python using tcp / udp sokets. I will make a reservation that I was familiar with network programming, and I remain so far as, although there was a general understanding of the tcp / ip stack. The decision on the forehead on synthetics proved to be wonderful, but as soon as I loaded it with more or less real data (only about 20k messages per second from several streams), and it began to show its underwater pebbles. I guess I just couldn’t prepare raw sockets correctly, but the problem needed to be solved quickly, There was no time for a deep understanding and study of network programming, so I started looking for solutions where at least half of them would have been invented for me. Search led me to the libraryZeroRPC , which was not so long ago, as I understand it, released into the world from the bowels of dotCloud.

    I was surprised that I found only one mention of this development on the Habré, and even then it was slippery, so I decided to write this note.

    As the developers of the library themselves write on the official website, ZeroRPC is a lightweight, reliable, language-independent library for distributed communication between servers.
    ZeroRPC is built on top of the ZeroMQ and MessagePack stack , which have already been mentioned more than once on the Haber. And out of the box it supports stream responses, heartbeats, timeout detection, and even automatic reconnection after some file.

    It sounded good and I decided to give it a try. By and large, I have not yet studied all the features of the library and will only tell you how I applied it in my case. But, perhaps, it will already interest you to read and, perhaps, try ZeroMQ in your projects.

    Installation is simple and straightforward, like most python libraries: it will attract a couple more libs, among which:

    pip install zerorpc

    • gevent
    • msgpack-python,
    • pyzmq

    It is not set in a couple of seconds, so if the installation suddenly stopped in one place, do not panic - soon everything will continue.

    Actually, according to the documentation, you can already start using it without writing any code. I willingly believe, but I was not very interested in this. The examples on the main page of the site looked more interesting. Server

    zerorpc --server --bind tcp://*:1234 time

    import zerorpc
    class HelloRPC(object):
        def hello(self, name):
            return "Hello, %s" % name
    s = zerorpc.Server(HelloRPC())

    import zerorpc
    c = zerorpc.Client()
    print c.hello("RPC")

    Copy paste, start up, work.
    Cool, but hello worlds always work, so let's complicate the primary tests a little

    import zerorpc, time
    class StreamingRPC(object):
    	count = 0
        def commit(self, cid):
        	self.count += 1
        	return str(cid)+'_done'
    s = zerorpc.Server(StreamingRPC())

    import zerorpc
    c = zerorpc.Client()
    for i in range(0,290000):
    	print c.commit(i)

    And we’ll run several clients in order to see what happens with the packages.
    I was most concerned about this question, because in my implementation I came across the fact that with a heavy load I lost more than 40% of messages on udp sockets, which is actually not very surprising ...
    The test certainly showed that everything was not so fast as it was on sockets, but not a single packet loss. A little later, I conducted a couple more tests with threads and writing to the database and was completely satisfied. After several days of testing, I rewrote the server and client socket code for my metric, and now two weeks later the metric collects a bunch of data and “not a single gap”. Moreover, increasing the speed of data collection, I did not see a drawdown in performance. At the moment, the metric collects in the peak up to 100k messages per second and feels great.

    Of course, in my case, ZeroRPC is most likely redundant and I’m sure that if I were a more knowledgeable programmer in my place, he would be able to defeat packet loss, hang up sessions and break socket connections and implement a more productive solution.
    But then I would not meet ZeroRPC and would not talk about it here.
    And maybe other features of this library can be very useful in working days of developers reading this article.

    I recommend to see and read the report of Jérôme Petazzoni from dotCloud about why and why they did it
    Project website -
    GitHub - etsy / statsd

    Also popular now: