Introducing interprocess communication on Linux

Interprocess communication ( Inter-process communication (IPC) ) - is a set of methods for exchanging data flows between processes. Processes can be launched both on the same computer and on different ones connected by a network. IPC can be of several types: “signal”, “socket”, “semaphore”, “file”, “message” ...

In this article I want to consider only 3 types of IPC:

1. named pipe
2. shared memory
3. semaphore

Digression: this article is an educational one and is aimed at people who are still embarking on the path of system programming. Its main idea is to get acquainted with various ways of interaction between processes on a POSIX-compatible OS.

Named pipe

To send messages, you can use the mechanisms of sockets, channels, D-bus and other technologies. You can read about sockets on every corner, and write a separate article about D-bus. Therefore, I decided to dwell on low-sounding technologies that meet POSIX standards and give working examples.

Consider passing messages over named pipes. Schematically, the transfer looks like this:

To create named pipes, we will use the function, mkfifo () :

#include 
int mkfifo(const char *pathname, mode_t mode);

The function creates a special FIFO file with the name pathname , and the mode parameter sets the access rights to the file.

Note: mode is used in conjunction with the current umask value as follows: ( mode & ~ umask) . The result of this operation will be the new umask value for the file we create. For this reason, we use 0777 ( S_IRWXO | S_IRWXG | S_IRWXU ) to not overwrite a single bit of the current mask.

Once a file is created, any process can open this file for reading or writing, just as it would a regular file. However, for the correct use of the file, it is necessary to open it simultaneously by two processes / threads, one for receiving data (reading a file) and the other for transferring (writing to a file).

If the FIFO file is successfully created, mkfifo () returns 0 (zero). In case of any errors, the function returns -1 and sets the error code to errno variable .

Typical errors that may occur during channel creation:

• EACCES - there is no permission to run (execute) in one of the directories in the pathname path
• EEXIST - the pathname file already exists, even if the file is a symlink
• ENOENT - does not exist any directory mentioned in pathname , or is a broken link
• ENOSPC - No space to create a new file
• ENOTDIR - one of the directories mentioned in pathname is not really one
• EROFS - an attempt to create a FIFO file on a read-only file system

Reading and writing to the created file is done using the read () and write () functions .

Example

mkfifo.c

#include 
#include 
#include 
#include 
#define NAMEDPIPE_NAME "/tmp/my_named_pipe"
#define BUFSIZE        50
int main (int argc, char ** argv) {
    int fd, len;
    char buf[BUFSIZE];
    if ( mkfifo(NAMEDPIPE_NAME, 0777) ) {
        perror("mkfifo");
        return 1;
    }
    printf("%s is created\n", NAMEDPIPE_NAME);
    if ( (fd = open(NAMEDPIPE_NAME, O_RDONLY)) <= 0 ) {
        perror("open");
        return 1;
    }
    printf("%s is opened\n", NAMEDPIPE_NAME);
    do {
        memset(buf, '\0', BUFSIZE);
        if ( (len = read(fd, buf, BUFSIZE-1)) <= 0 ) {
            perror("read");
            close(fd);
            remove(NAMEDPIPE_NAME);
            return 0;
        }
        printf("Incomming message (%d): %s\n", len, buf);
    } while ( 1 );
}

[ download ]

We open the file for reading only ( O_RDONLY ). And they could use the O_NONBLOCK modifier, designed specifically for FIFO files, so as not to wait for the file to be opened for writing on the other hand. But in the above code, this method is inconvenient.

We compile the program, then run it:

$ gcc -o mkfifo mkfifo.c
$ ./mkfifo

In the adjacent terminal window, do:

$ echo 'Hello, my named pipe!' > /tmp/my_named_pipe

As a result, we will see the following output from the program:

$ ./mkfifo 
/tmp/my_named_pipe is created
/tmp/my_named_pipe is opened
Incomming message (22): Hello, my named pipe!
read: Success

Shared memory

The next type of interprocess communication is shared memory . Schematically, we depict it as a certain named area in memory that is accessed simultaneously by two processes:

To allocate shared memory, we will use the POSIX function shm_open () :

#include 
int shm_open(const char *name, int oflag, mode_t mode);

The function returns the file descriptor that is associated with the memory object. This descriptor can later be used by other functions (for example, mmap () or mprotect () ).

The integrity of the memory object is preserved, including all data associated with it, until the object is disconnected / deleted ( shm_unlink () ). This means that any process can access our memory object (if it knows its name) until we explicitly call shm_unlink () in one of the processes .

The oflag variable is a bitwise OR of the following flags:

• O_RDONLY - open read-only
• O_RDWR - open with read and write permissions
• O_CREAT - if the object already exists, then there is no effect from the flag. Otherwise, the object is created and access rights are set for it in accordance with mode.
• O_EXCL - setting this flag in combination with O_CREATE will cause the shm_open function to return an error if a shared memory segment already exists.

How the value of the mode parameter is set is described in detail in the previous paragraph “message passing”.

After creating a shared memory object, we set the size of the shared memory by calling ftruncate () . At the input of the function is the file descriptor of our object and the size we need.

Example

The following code demonstrates creating, modifying, and deleting shared memory. It also shows how, after creating shared memory, the program exits, but the next time we start it, we can access it until shm_unlink () is executed .

shm_open.c

#include 
#include 
#include 
#include 
#include 
#include 
#define SHARED_MEMORY_OBJECT_NAME "my_shared_memory"
#define SHARED_MEMORY_OBJECT_SIZE 50
#define SHM_CREATE 1
#define SHM_PRINT  3
#define SHM_CLOSE  4
void usage(const char * s) {
    printf("Usage: %s  ['text']\n", s);
}
int main (int argc, char ** argv) {
    int shm, len, cmd, mode = 0;
    char *addr;
    if ( argc < 2 ) {
        usage(argv[0]);
        return 1;
    }
    if ( (!strcmp(argv[1], "create") || !strcmp(argv[1], "write")) && (argc == 3) ) {
        len = strlen(argv[2]);
        len = (len<=SHARED_MEMORY_OBJECT_SIZE)?len:SHARED_MEMORY_OBJECT_SIZE;
        mode = O_CREAT;
        cmd = SHM_CREATE;
    } else if ( ! strcmp(argv[1], "print" ) ) {
        cmd = SHM_PRINT;
    } else if ( ! strcmp(argv[1], "unlink" ) ) {
        cmd = SHM_CLOSE;
    } else {
        usage(argv[0]);
        return 1;
    }
    if ( (shm = shm_open(SHARED_MEMORY_OBJECT_NAME, mode|O_RDWR, 0777)) == -1 ) {
        perror("shm_open");
        return 1;
    }
    if ( cmd == SHM_CREATE ) {
        if ( ftruncate(shm, SHARED_MEMORY_OBJECT_SIZE+1) == -1 ) {
            perror("ftruncate");
            return 1;
        }
    }
    addr = mmap(0, SHARED_MEMORY_OBJECT_SIZE+1, PROT_WRITE|PROT_READ, MAP_SHARED, shm, 0);
    if ( addr == (char*)-1 ) {
        perror("mmap");
        return 1;
    }
    switch ( cmd ) {
    case SHM_CREATE:
        memcpy(addr, argv[2], len);
        addr[len] = '\0';
        printf("Shared memory filled in. You may run '%s print' to see value.\n", argv[0]);
        break;
    case SHM_PRINT:
        printf("Got from shared memory: %s\n", addr);
        break;
    }
    munmap(addr, SHARED_MEMORY_OBJECT_SIZE);
    close(shm);
    if ( cmd == SHM_CLOSE ) {
        shm_unlink(SHARED_MEMORY_OBJECT_NAME);
    }
    return 0;
}

[ download ]

After creating the memory object, we set the size of shared memory we need by calling ftruncate () . Then we accessed shared memory using mmap () . (Generally speaking, even using the mmap () call itself, you can create shared memory. But the difference between the shm_open () call is that the memory will remain allocated until the computer is deleted or rebooted.)

You need to compile the code this time with the -lrt option :

$ gcc -o shm_open -lrt shm_open.c

We look at what happened:

$ ./shm_open create 'Hello, my shared memory!'
Shared memory filled in. You may run './shm_open print' to see value.
$ ./shm_open print
Got from shared memory: Hello, my shared memory!
$ ./shm_open create 'Hello!'
Shared memory filled in. You may run './shm_open print' to see value.
$ ./shm_open print
Got from shared memory: Hello!
$ ./shm_open close
$ ./shm_open print
shm_open: No such file or directory

We use the create argument in our program both to create shared memory and to modify its contents.

Knowing the name of the memory object, we can change the contents of shared memory. But once we call shm_unlink () , the memory ceases to be available to us and shm_open () without the O_CREATE parameter returns the error “No such file or directory”.

Semaphore

A semaphore is the most commonly used method for synchronizing threads and for controlling the simultaneous access of multiple threads / processes to shared memory (for example, a global variable). The interaction between processes in the case of semaphores is that processes work with the same data set and adjust their behavior depending on this data.

There are two types of semaphores:

1. a semaphore with a counter (counting semaphore), which determines the limit of resources for processes accessing them
2. binary semaphore, which has two states “0” or “1” (more often: “busy” or “not busy”)

Consider both types of semaphores.

Semaphore with a counter

The meaning of a semaphore with a counter is to give access to a certain resource to only a certain number of processes. The rest will wait in line when the resource is freed.

So, to implement the semaphores, we will use the POSIX function sem_open () :

#include 
sem_t *sem_open(const char *name, int oflag, mode_t mode, unsigned int value);

In the function for creating a semaphore, we pass the name of the semaphore, built according to certain rules and control flags. This way we get a named semaphore.
The name of the semaphore is constructed as follows: at the beginning is the symbol "/" (slash), followed by Latin characters. The slash symbol should no longer apply. The semaphore name can be up to 251 characters long.

If we need to create a semaphore, then the O_CREATE control flag is passed . To start using an existing semaphore, then oflag is zero. If used together with the flag O_CREATE pass flag O_EXCL , the function sem_open () will return an error if the semaphore with the specified name already exists.

The mode parameter sets permissions in the same way as explained in previous chapters. And the value variable initializes the initial value of the semaphore. Both mode and value parameters are ignored when a semaphore with the specified name already exists, and sem_open () is called along with the O_CREATE flag .

To quickly open an existing semaphore, we use the construction:

#include 
sem_t *sem_open(const char *name, int oflag);

where only the semaphore name and control flag are indicated.

Example semaphore with counter

Consider an example of using a semaphore to synchronize processes. In our example, one process increases the value of the semaphore and waits for the second to reset it in order to continue further execution.

sem_open.c

#include 
#include 
#include 
#include 
#define SEMAPHORE_NAME "/my_named_semaphore"
int main(int argc, char ** argv) {
    sem_t *sem;
    if ( argc == 2 ) {
        printf("Dropping semaphore...\n");
        if ( (sem = sem_open(SEMAPHORE_NAME, 0)) == SEM_FAILED ) {
            perror("sem_open");
            return 1;
        }
        sem_post(sem);
        perror("sem_post");
        printf("Semaphore dropped.\n");
        return 0;
    }
    if ( (sem = sem_open(SEMAPHORE_NAME, O_CREAT, 0777, 0)) == SEM_FAILED ) {
        perror("sem_open");
        return 1;
    }
    printf("Semaphore is taken.\nWaiting for it to be dropped.\n");
    if (sem_wait(sem) < 0 )
        perror("sem_wait");
    if ( sem_close(sem) < 0 )
        perror("sem_close");
    return 0;
}

[ download ]

In one console, run:

$ ./sem_open 
Semaphore is taken.
Waiting for it to be dropped.       <-- здесь процесс в ожидании другого процесса
sem_wait: Success
sem_close: Success

In the neighboring console, run:

$ ./sem_open 1
Dropping semaphore...
sem_post: Success
Semaphore dropped.

Binary semaphore

Instead of a binary semaphore, for which the sem_open function is also used, I will consider a much more commonly used semaphore called a mutex.

A mutex is essentially the same as a binary semaphore (that is, a semaphore with two states: busy and not busy). But the term “mutex” is more often used to describe a scheme that protects two processes from using shared data / variables at the same time. While the term “binary semaphore” is more often used to describe a construct that restricts access to a single resource. That is, the binary semaphore is used where one process “occupies” the semaphore and the other “releases” it. While the mutex is freed by the same process / thread that occupied it.

It is impossible to do without a mutex in writing, for example, a database to which many clients can access.

To use the mutex, you must call the pthread_mutex_init () function:

#include 
int pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *mutexattr);

The function initializes the mutex (variable mutex ) with the mutexattr attribute . If mutexattr is NULL , then the mutex is initialized to the default value. If the function succeeds (return code 0), the mutex is considered initialized and “free”.

Typical errors that may occur:

• EAGAIN - insufficient resources (except memory) to initialize the mutex
• ENOMEM - not enough memory
• EPERM - no rights to perform the operation
• EBUSY - an attempt to initialize a mutex that has already been initialized, but not destroyed
• EINVAL - mutexattr value is not valid

To occupy or free the mutex, we use the functions:

int pthread_mutex_lock(pthread_mutex_t *mutex);
int pthread_mutex_trylock(pthread_mutex_t *mutex);
int pthread_mutex_unlock(pthread_mutex_t *mutex);

The pthread_mutex_lock () function , if mutex is not already busy, then it takes it, becomes its owner and immediately exits. If the mutex is busy, it blocks the further execution of the process and waits for the mutex to free.
The pthread_mutex_trylock () function is identical in behavior to the pthread_mutex_lock () function , with one exception - it does not block the process if mutex is busy, but returns an EBUSY code.
The pthread_mutex_unlock () function frees a busy mutex.

Return codes for pthread_mutex_lock () :

• EINVAL - mutex is incorrectly initialized
• EDEADLK - The mutex is already taken by the current process.

Return codes for pthread_mutex_trylock () :

• EBUSY - Mutex is already taken
• EINVAL - mutex is incorrectly initialized

Return codes for pthread_mutex_unlock () :

• EINVAL - mutex is incorrectly initialized
• EPERM - the calling process is not the owner of the mutex

Mutex example

mutex.c

#include 
#include 
#include 
#include 
static int counter; // shared resource
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
void incr_counter(void *p) {
    do {
        usleep(10); // Let's have a time slice between mutex locks
        pthread_mutex_lock(&mutex);
        counter++;
        printf("%d\n", counter);
        sleep(1);
        pthread_mutex_unlock(&mutex);
    } while ( 1 );
}
void reset_counter(void *p) {
    char buf[10];
    int  num = 0;
    int  rc;
    pthread_mutex_lock(&mutex); // block mutex just to show message
    printf("Enter the number and press 'Enter' to initialize the counter with new value anytime.\n");
    sleep(3);
    pthread_mutex_unlock(&mutex); // unblock blocked mutex so another thread may work
    do {
        if ( gets(buf) != buf ) return; // NO fool-protection ! Risk of overflow !
        num = atoi(buf);
        if ( (rc = pthread_mutex_trylock(&mutex)) == EBUSY ) {
            printf("Mutex is already locked by another process.\nLet's lock mutex using pthread_mutex_lock().\n");
            pthread_mutex_lock(&mutex);
        } else if ( rc == 0 ) {
            printf("WOW! You are on time! Congratulation!\n");
        } else { 
            printf("Error: %d\n", rc);
            return;
        }
        counter = num;
        printf("New value for counter is %d\n", counter);
        pthread_mutex_unlock(&mutex);
    } while ( 1 );
}
int main(int argc, char ** argv) {
    pthread_t thread_1;
    pthread_t thread_2;
    counter = 0;
    pthread_create(&thread_1, NULL, (void *)&incr_counter, NULL);
    pthread_create(&thread_2, NULL, (void *)&reset_counter, NULL);
    pthread_join(thread_2, NULL);
    return 0;
}

[ download ]

This example demonstrates the joint access of two threads to a shared variable. One thread (the first thread) in automatic mode constantly increases the counter variable by one, while occupying this variable for a whole second. This first thread gives the second access to the count variable for only 10 milliseconds, then again takes it for a second. In the second thread, it is proposed to enter a new value for the variable from the terminal.

If we had not used the mutex technology, then what value would be in the global variable, while two streams were simultaneously accessed, we do not know. Also during startup, the difference between pthread_mutex_lock () and pthread_mutex_trylock () becomes apparent.

You need to compile the code with the additional parameter -lpthread :

$ gcc -o mutex -lpthread mutex.c

We start and change the value of the variable simply by entering a new value in the terminal window:

$ ./mutex 
Enter the number and press 'Enter' to initialize the counter with new value anytime.
1
2
3
30      <--- новое значение переменной
Mutex is already locked by another process.
Let's lock mutex using pthread_mutex_lock().
New value for counter is 30
31
32
33
1      <--- новое значение переменной
Mutex is already locked by another process.
Let's lock mutex using pthread_mutex_lock().
New value for counter is 1
2
3

Instead of a conclusion

In the following articles, I want to look at d-bus and RPC technologies. If you are interested, let me know.
Thanks.

UPD: Updated the 3rd chapter about semaphores. Added a sub-chapter about the mutex.