October 22, 2013 at 16:20ZeroMQ. Chapter 1: Getting Started
- Transfer
- Tutorial
Hello!
I want to start a free translation of the book "ZeroMQ. Use ZeroMQ and learn how to apply different message patterns." I am sure that many will want to deal with this interesting library.
Content
Welcome to ZeroMQ! This chapter is an introduction to ZeroMQ and gives the reader a general idea of what a message queuing system is and, most importantly, what ZeroMQ is. In this chapter we will talk about the following topics:
People throughout their lives socially interact with each other. Programs behave this way. The program must communicate with another program, since we live in the world of the Internet. We have UDP, TCP, HTTP, IPX, WebSocket and other protocols for connecting applications.
Nevertheless, approaches of such a low level complicate our lives; we need something simpler and faster. High-level abstractions sacrifice speed and flexibility, while dealing directly with low-level details is not easy. ZeroMQ shows what the solution is, giving us the convenience of using high-level methods with the speed of a low-level approach.
Before starting to understand ZeroMQ, let's first look at the general concept of message queuing.
Message Queuing, or, technically, FIFO (First In First Out), is one of the basic and well-studied data structures. There are various queue implementations, such as a priority queue or a two-way queue, that have different properties, but the general idea is that the data is added to the queue when it appears or the caller is ready.
However, the message queue provides guarantees that the message will be delivered no matter what happens. The message queue allows asynchronous interaction between loosely coupled components, and also provides a strict queue sequence. In case of insufficient resources, which prevents you from immediately processing the data sent, you can put them in a message queue on the server, which will store data until the client is ready.
Message queuing plays an important role in scaling distributed systems because it supports asynchronous communications. We give a brief information about the difference between synchronous and asynchronous systems.
In ordinary synchronous systems, tasks are processed one at a time. A task is considered not processed until its processing is completed. This is the easiest way to organize work.
We can also implement threads in this system. In this case, the processing of each task would be carried out in parallel.
In a multi-threaded model, threads are controlled by the operating system itself on a single processor or multiple processors / cores.
Asynchronous I / O (AIO) allows the program to continue execution while processing I / O requests. AIO is a must in real-time applications. With AIO, we can handle multiple tasks in one thread.
This is the traditional way of programming, after the start of the process, we wait for its completion. The disadvantage of this approach is that it blocks the execution of the program while the task is in the process of processing. But there is another AIO approach. In AIO, a task that is process independent is still ongoing. In the second chapter, we will take a closer look at AIO and use it in ZeroMQ.
You may wonder why you should use the message queue instead of processing all threads for a single-threaded or multi-threaded approach. Let's look at a situation where you have a web application similar to Google Images, in which you allow users to enter some URLs. Once they submit the form, your application displays all the images for the given URL. Nonetheless:
In this case, you know that you need to add the URL data to the queue and process it. So, you need a message queuing system.
Up to this point, we examined what a message queue is, which led us to the goal of this book, that is, to ZeroMQ. The ZeroMQ Society has defined it as "steroids sockets." Formally, ZeroMQ is defined as a message library that helps developers create distributed and parallel applications.
The first thing we should learn about ZeroMQ is that it is not a traditional message queuing system such as ActiveMQ, WebSphereMQ, or RabbitMQ. ZeroMQ is different. She gives us the tools to create our own message queuing system. This is a library.
It runs on a variety of architectures from ARM to Itanium and is supported in more than 20 programming languages.
ZeroMQ is simple. We can do some asynchronous I / O, and ZeroMQ can also queue messages from the I / O stream. ZeroMQ I / O streams work asynchronously when communicating with network traffic, so they can do the rest for you. If you've worked with sockets before, then you should know that this is not easy. However, ZeroMQ makes it much easier to work with them.
ZeroMQ is fast. The Second Life website managed to get 13.4 microseconds of continuous latency and up to 4,100,000 messages per second. ZeroMQ can use the multicast transport protocol, which is an effective method for transmitting data in various directions.
So, it's time to start writing code after we've figured out what the message queue is and what ZeroMQ is like. And of course, we will start with the famous Hello World program.
Let's look at a situation where we have a server and a client. The server responds
The following is the server code that sends a message to the
The following is the code for the client that sends the message
We have the first main architecture: request-response, as shown in the following diagram:
Let's take a closer look at the code to understand how it works.
First we create a context and a socket. The method
The server binds the socket
This simple hello world program shows us an example of using the first request-response pattern.
We use a request-response template (pattern) to send a message from a client to one or more servers and get a response to each sent message. This is most likely the easiest way to use ZeroMQ. Requests for answers should be in order.
The following is the response part of the request-response pattern:
The server uses the socket
This book is fully dedicated to queues. You may be surprised to find out what we mean when we talk about strategy
To understand how it works, let's say that the streams in the previous figure send 16, 2, 6, and 8 packets per second, respectively, but only 12 packets per second can be processed at the output. In this case, we could transmit 4 packets per second, but Stream 2 transmits only 2 packets per second. Fair-queue rules are that there can be no free exits, unless all exits are free at the same time. Thus, you can allow Stream 2 to transfer its 2 packets per second and split the remaining 10 between the remaining threads.
This inbound message routing strategy uses
The following is the query part of the request-response pattern:
The client uses
This part combines the request and response sections and shows how the response to someone’s request occurs and how to respond to it.
The client sends a message to the server using
After sending the message, the client waits to receive a response. This is done using
Each programming language has its own approach for processing strings. Erlang generally does not have strings per se (they are represented there as lists of characters). In the C programming language, strings with a zero at the end. Strings in C are basically character arrays, where '\ 0' means the end of the string. String errors are common and result from many vulnerabilities.
According to Miller et al. (1995), 65 percent of Unix vulnerabilities are due to string errors such as zero byte or buffer overflow, so string processing in C should be done carefully.
When using ZeroMQ, this falls on your shoulders so that the message is reliably formatted so that other applications can read it. ZeroMQ only knows the size of the message.
Using different programming languages in one application is usually the case. If applications written in a programming language that does not add a null byte at the end of a line must somehow communicate with applications written in C, then you will get strange results.
You can send a message such as world, as in our example with a zero byte, namely:
But in Erlang you would send the same message as follows:
Suppose our client, written in C, connects to ZeroMQ, a service written in Erlang, and we send a message to
Lines in ZeroMQ are fixed in length and sent without a zero byte. Thus, the ZeroMQ string is transmitted as a few bytes (the string itself in this example), as well as the length.
It is very useful to know which version of ZeroMQ you are using. In some cases, to avoid unwanted surprises, you need to know the exact version. For example, there are some differences between ZeroMQ 2.x and ZeroMQ 3.x, such as using legacy methods; therefore, if you know the exact version of ZeroMQ that is installed on your computer, then you can avoid using obsolete methods.
In this chapter, we looked at what the message queue is and also made a small introduction to ZeroMQ. We examined how ZeroMQ processes strings and the first of the patterns (request-response). And they also wrote the first hello world app.
You can download the resources for this article from the link.
Thank you all very much for your attention!
I want to start a free translation of the book "ZeroMQ. Use ZeroMQ and learn how to apply different message patterns." I am sure that many will want to deal with this interesting library.
Content
- Chapter 1: Getting Started
- Chapter 2: Introducing Sockets
- Chapter 3: Using Socket Topology
- Chapter 4: Additional Patterns
Welcome to ZeroMQ! This chapter is an introduction to ZeroMQ and gives the reader a general idea of what a message queuing system is and, most importantly, what ZeroMQ is. In this chapter we will talk about the following topics:
- Overview of what constitutes a message queue
- Why use ZeroMQ and what sets it apart from other message queuing technologies
- Client / Server Architecture Basics
- Consider the first pattern: request-response
- How can we handle strings in C
- Checking Installed ZeroMQ Versions
Start
People throughout their lives socially interact with each other. Programs behave this way. The program must communicate with another program, since we live in the world of the Internet. We have UDP, TCP, HTTP, IPX, WebSocket and other protocols for connecting applications.
Nevertheless, approaches of such a low level complicate our lives; we need something simpler and faster. High-level abstractions sacrifice speed and flexibility, while dealing directly with low-level details is not easy. ZeroMQ shows what the solution is, giving us the convenience of using high-level methods with the speed of a low-level approach.
Before starting to understand ZeroMQ, let's first look at the general concept of message queuing.
Message queue
Message Queuing, or, technically, FIFO (First In First Out), is one of the basic and well-studied data structures. There are various queue implementations, such as a priority queue or a two-way queue, that have different properties, but the general idea is that the data is added to the queue when it appears or the caller is ready.
However, the message queue provides guarantees that the message will be delivered no matter what happens. The message queue allows asynchronous interaction between loosely coupled components, and also provides a strict queue sequence. In case of insufficient resources, which prevents you from immediately processing the data sent, you can put them in a message queue on the server, which will store data until the client is ready.
Message queuing plays an important role in scaling distributed systems because it supports asynchronous communications. We give a brief information about the difference between synchronous and asynchronous systems.
In ordinary synchronous systems, tasks are processed one at a time. A task is considered not processed until its processing is completed. This is the easiest way to organize work.
We can also implement threads in this system. In this case, the processing of each task would be carried out in parallel.
In a multi-threaded model, threads are controlled by the operating system itself on a single processor or multiple processors / cores.
Asynchronous I / O (AIO) allows the program to continue execution while processing I / O requests. AIO is a must in real-time applications. With AIO, we can handle multiple tasks in one thread.
This is the traditional way of programming, after the start of the process, we wait for its completion. The disadvantage of this approach is that it blocks the execution of the program while the task is in the process of processing. But there is another AIO approach. In AIO, a task that is process independent is still ongoing. In the second chapter, we will take a closer look at AIO and use it in ZeroMQ.
You may wonder why you should use the message queue instead of processing all threads for a single-threaded or multi-threaded approach. Let's look at a situation where you have a web application similar to Google Images, in which you allow users to enter some URLs. Once they submit the form, your application displays all the images for the given URL. Nonetheless:
- If you use a single-threaded queue, then in the case of a large number of users, the program will not be able to process data for all URLs
- If you use a multi-threaded approach for the queue, then your application may be subject to a DDoS attack
- You will lose data for all URLs in the event of a hardware failure
In this case, you know that you need to add the URL data to the queue and process it. So, you need a message queuing system.
Introduction to ZeroMQ
Up to this point, we examined what a message queue is, which led us to the goal of this book, that is, to ZeroMQ. The ZeroMQ Society has defined it as "steroids sockets." Formally, ZeroMQ is defined as a message library that helps developers create distributed and parallel applications.
The first thing we should learn about ZeroMQ is that it is not a traditional message queuing system such as ActiveMQ, WebSphereMQ, or RabbitMQ. ZeroMQ is different. She gives us the tools to create our own message queuing system. This is a library.
It runs on a variety of architectures from ARM to Itanium and is supported in more than 20 programming languages.
Simplicity
ZeroMQ is simple. We can do some asynchronous I / O, and ZeroMQ can also queue messages from the I / O stream. ZeroMQ I / O streams work asynchronously when communicating with network traffic, so they can do the rest for you. If you've worked with sockets before, then you should know that this is not easy. However, ZeroMQ makes it much easier to work with them.
Performance
ZeroMQ is fast. The Second Life website managed to get 13.4 microseconds of continuous latency and up to 4,100,000 messages per second. ZeroMQ can use the multicast transport protocol, which is an effective method for transmitting data in various directions.
Hello world
So, it's time to start writing code after we've figured out what the message queue is and what ZeroMQ is like. And of course, we will start with the famous Hello World program.
Let's look at a situation where we have a server and a client. The server responds
world whenever it receives a message hello from the client. The server runs on port 4040, but the client, accordingly, sends messages to the same port. The following is the server code that sends a message to the
world client:#include
#include
#include
#include "zmq.h"
int main (int argc, char const *argv[])
{
void* context = zmq_ctx_new();
void* respond = zmq_socket(context, ZMQ_REP);
zmq_bind(respond, "tcp://*:4040");
printf("Starting…\n");
for(;;)
{
zmq_msg_t request;
zmq_msg_init(&request);
zmq_msg_recv(&request, respond, 0);
printf("Received: hello\n");
zmq_msg_close(&request);
sleep(1); // sleep one second
zmq_msg_t reply;
zmq_msg_init_size(&reply, strlen("world"));
memcpy(zmq_msg_data(&reply), "world", 5);
zmq_msg_send(&reply, respond, 0);
zmq_msg_close(&reply);
}
zmq_close(respond);
zmq_ctx_destroy(context);
return 0;
}
The following is the code for the client that sends the message
hello to the server:#include
#include
#include
#include "zmq.h"
int main (int argc, char const *argv[])
{
void* context = zmq_ctx_new();
printf("Client Starting….\n");
void* request = zmq_socket(context, ZMQ_REQ);
zmq_connect(request, "tcp://localhost:4040");
int count = 0;
for(;;)
{
zmq_msg_t req;
zmq_msg_init_size(&req, strlen("hello"));
memcpy(zmq_msg_data(&req), "hello", 5);
printf("Sending: hello - %d\n", count);
zmq_msg_send(&req, request, 0);
zmq_msg_close(&req);
zmq_msg_t reply;
zmq_msg_init(&reply);
zmq_msg_recv(&reply, request, 0);
printf("Received: hello - %d\n", count);
zmq_msg_close(&reply);
count++;
}
// We never get here though.
zmq_close(request);
zmq_ctx_destroy(context);
return 0;
}
We have the first main architecture: request-response, as shown in the following diagram:
Let's take a closer look at the code to understand how it works.
First we create a context and a socket. The method
zmq_ctx_new()creates a new context. It is thread safe, so context can be used in multiple threads.zmq_socket(2)creates a new socket in a specific context. ZeroMQ sockets are not thread safe, so they should only be used in the thread in which they were created. Traditional sockets are synchronous, while ZeroMQ sockets have the ability to create one queue on the client side and the other on the server side, asynchronously controlling the request-response pattern. ZeroMQ automatically organizes connection setup, reconnection, disconnection, and content delivery. In Chapter 3 we will examine in more detail the difference between traditional sockets and ZeroMQ sockets. The server binds the socket
ZMQ_REPand the 4040th port and begins to wait for requests and responds each time it receives a message. This simple hello world program shows us an example of using the first request-response pattern.
Request-response pattern
We use a request-response template (pattern) to send a message from a client to one or more servers and get a response to each sent message. This is most likely the easiest way to use ZeroMQ. Requests for answers should be in order.
Answer
The following is the response part of the request-response pattern:
void* context = zmq_ctx_new();
void* respond = zmq_socket(context, ZMQ_REP);
zmq_bind(respond, "tcp://*:4040");
The server uses the socket
ZMQ_REP to receive messages from the Internet and send responses to clients. For routing incoming messages from the ZMQ_REP fair strategy fair-queue, and for outgoing - last-peer.Fair-queue strategy
This book is fully dedicated to queues. You may be surprised to find out what we mean when we talk about strategy
fair-queue. This is an algorithm for planning and allocating resources that is fair by definition. To understand how it works, let's say that the streams in the previous figure send 16, 2, 6, and 8 packets per second, respectively, but only 12 packets per second can be processed at the output. In this case, we could transmit 4 packets per second, but Stream 2 transmits only 2 packets per second. Fair-queue rules are that there can be no free exits, unless all exits are free at the same time. Thus, you can allow Stream 2 to transfer its 2 packets per second and split the remaining 10 between the remaining threads.
This inbound message routing strategy uses
ZMQ_REP. Cyclic planning is the easiest way to implement a fair queue strategy, which is also used in ZeroMQ.Inquiry
The following is the query part of the request-response pattern:
void* context = zmq_ctx_new();
printf("Client Starting….\n");
void* request = zmq_socket(context, ZMQ_REQ);
zmq_connect(request, "tcp://localhost:4040");
The client uses
ZMQ_REQ to send messages and receive responses from the server. All messages are sent using a round-robin ( round-robin) routing strategy . Inbound routing strategy is this last-peer. ZMQ_REQ does not throw any messages. If there are no services available to send the message, or all services are busy, then an operation is sent zmq_send(3)that will be blocked until one of the servers becomes available for sending the message. ZMQ_REQ compatible with ZMQ_REP and types ZMQ_ROUTER. In chapter 4 we will consider ZMQ_ROUTER.Message sending
This part combines the request and response sections and shows how the response to someone’s request occurs and how to respond to it.
printf("Sending: hello - %d\n", count);
zmq_msg_send(&req, request, 0);
zmq_msg_close(&req);
The client sends a message to the server using
zmq_msg_send(3). This is another message and sends it to the socket.int zmq_send_msg(zmq_msg_t *msg, void *socket, int flags)
zmq_msg_send accepts three parameters, namely the message, socket and flag:- The message parameter is reset during the request, so if you want to send a message to several sockets, you need to copy it.
- A successful
zmq_msg_send()request does not indicate if the message was sent over the network. - Flag can be either a parameter
ZMQ_DONTWAITorZMQ_SNDMORE. ZMQ_DONTWAITindicates that the message should be sent asynchronously.ZMQ_SNDMOREindicates that the message is composite and the rest of the message is in transit.
After sending the message, the client waits to receive a response. This is done using
zmq_msg_recv(3).zmq_msg_recv(&reply, request, 0);
printf("Received: hello - %d\n", count);
zmq_msg_close(&reply);
zmq_msg_recv(3)receives part of the message from the socket, as specified in the parameter, socket and saves the response to the parameter message.int zmq_msg_recv (zmq_msg_t *msg, void *socket, int flags)
zmq_msg_recv accepts three parameters, namely message, socket, and flags.- Previously received messages (if any) are invalid
- A flag parameter may be
ZMQ_DONTWAITthat indicates that the operation should be performed asynchronously
Working with strings in C
Each programming language has its own approach for processing strings. Erlang generally does not have strings per se (they are represented there as lists of characters). In the C programming language, strings with a zero at the end. Strings in C are basically character arrays, where '\ 0' means the end of the string. String errors are common and result from many vulnerabilities.
According to Miller et al. (1995), 65 percent of Unix vulnerabilities are due to string errors such as zero byte or buffer overflow, so string processing in C should be done carefully.
When using ZeroMQ, this falls on your shoulders so that the message is reliably formatted so that other applications can read it. ZeroMQ only knows the size of the message.
Using different programming languages in one application is usually the case. If applications written in a programming language that does not add a null byte at the end of a line must somehow communicate with applications written in C, then you will get strange results.
You can send a message such as world, as in our example with a zero byte, namely:
zmq_msg_init_data_(&request, "world", 6, NULL, NULL);
But in Erlang you would send the same message as follows:
erlzmq:send(Request, <<"world">>)
Suppose our client, written in C, connects to ZeroMQ, a service written in Erlang, and we send a message to
world this service. In this case, Erlang will output world. If we send a message with a null byte, Erlang will output the following [119,111,114,108,100,0]. Instead of the desired line, we got a list containing some numbers, these are ASCII codes. However, this is no longer interpreted as a string. Lines in ZeroMQ are fixed in length and sent without a zero byte. Thus, the ZeroMQ string is transmitted as a few bytes (the string itself in this example), as well as the length.
Checking ZeroMQ Version
It is very useful to know which version of ZeroMQ you are using. In some cases, to avoid unwanted surprises, you need to know the exact version. For example, there are some differences between ZeroMQ 2.x and ZeroMQ 3.x, such as using legacy methods; therefore, if you know the exact version of ZeroMQ that is installed on your computer, then you can avoid using obsolete methods.
#include
#include "zmq.h"
int main (int argc, char const *argv[])
{
int major, minor, patch;
zmq_version(&major, &minor, &patch);
printf("Installed ZeroMQ version: %d.%d.%d\n", major, minor, patch);
return 0;
}
Summary
In this chapter, we looked at what the message queue is and also made a small introduction to ZeroMQ. We examined how ZeroMQ processes strings and the first of the patterns (request-response). And they also wrote the first hello world app.
You can download the resources for this article from the link.
Thank you all very much for your attention!