Our approach to coloring threads
We at the company always strive to increase the maintainability of our code, using generally accepted practices, including in matters of multithreading. This does not solve all the difficulties that an ever-increasing load brings, but simplifies support - it also wins code readability and the speed of developing new features.
We now have 47,000 users daily, about 30 servers in production, 2,000 API requests per second, and daily releases. Miro service has been developing since 2011, and in the current implementation, user requests are processed in parallel by a cluster of heterogeneous servers.
The main value of our product is collaborative user boards, so the main burden falls on them. The main subsystem that controls most of the competitive access is the stateful system of user sessions on the board.
For each openable board on one of the servers, the state rises. It stores both application runtime data necessary to ensure collaboration and display of content, as well as system data, such as binding to processing threads. Information about which server the state is stored in is written to a distributed structure and is available to the cluster as long as the server is running, and at least one user is on the board. We use Hazelcast to provide this part of the subsystem. All new connections to the board are sent to the server with this state.
When connecting to the server, the user enters the receiving stream, whose sole task is to bind the connection to the state of the corresponding board, in the flows of which all further work will occur.
Two streams are associated with the board: network, processing connections, and “business”, responsible for business logic. This allows you to transform the execution of heterogeneous tasks of processing network packets and executing business commands from serial to parallel. Processed network commands from users form applied business tasks and direct them to the business stream, where they are processed sequentially. This avoids unnecessary synchronization when developing application code.
The division of code into business / application and system is our internal convention. It allows you to distinguish between the code responsible for the features and capabilities for users, from the low-level details of communication, sheduling and storage, which are the service tool.
If the receiving stream detects that there is no state for the board, the corresponding initialization task is set. State initialization is handled by a separate type of thread.
The types of tasks and their direction can be represented as follows:
Such an implementation allows us to solve the following problems:
The subsystem described above in the implementation is quite nontrivial. The developer has to keep in mind the scheme of the system and take into account the reverse process of closing boards. When closing, you must remove all subscriptions, delete entries from the registries and do this in the same streams in which they were initialized.
We noticed that bugs and complexities of code modification that arose in this subsystem were often associated with a lack of understanding of the execution context. Juggling threads and tasks made it difficult to answer the question in which particular thread a particular piece of code is executing.
To solve this problem, we used the method of coloring threads - this is a policy aimed at regulating the use of threads in the system. Colors are assigned to threads, and methods define the scope for executing within threads. Color here is an abstraction, it can be any entity, for example, an enumeration. In Java, annotations can serve as the color marking language:
Annotations are added to the method, with the help of which you can set the validity of the method. For example, if the annotation of a method allows yellow and red, then the first thread can call the method, and for the second, such a call will be erroneous.
Invalid colors can be set:
You can add and remove thread privileges in the dynamics:
Lack of annotation or annotation as in the example below says that the method can be executed in any thread:
Android developers may be familiar with this approach for annotations MainThread, UiThread, WorkerThread, etc. P.
The coloring of threads uses the principle of self-documenting code, and the method itself lends itself well to static analysis. Using static analysis, you can say before the code is executed that it is written correctly or not. If we exclude Grant and Revoke annotations and assume that the stream upon initialization already has an unchangeable set of privileges, then this will be a flow-insensitive analysis - a simple version of static analysis that does not take into account the order of calls.
At the time of the implementation of the flow coloring method, there were no ready-made solutions for static analysis in our devops infrastructure, so we went the simpler and cheaper way - we introduced our annotations, which are uniquely associated with each type of flows. We began to check their correctness with the help of aspects in runtime.
For aspects, we use the aspectj library and the maven plugin, which provides weaving when compiling the project. Weaving was initially configured to load-time when loading classes with ClassLoader. However, we were faced with the fact that weaver sometimes behaved incorrectly when loading the same class competitively, as a result of which the original byte of the class code remained unchanged. As a result, this resulted in very unpredictable and difficult to reproduce production behavior. Perhaps in current versions of the library there is no such problem.
The solution on aspects allowed us to quickly find most of the problems in the code.
It is important not to forget to always keep annotations up to date: they can be deleted, add laziness, weaving aspects can be turned off altogether - in this case the coloring will quickly lose its relevance and value.
One of the varieties of coloring is the GuardedBy annotation from java.util.concurrent. It delimits access to fields and methods, indicating which locks are necessary for correct access.
Modern IDEs even support the analysis of this annotation. For example, IDEA displays this message if something is wrong with the code:
The method of coloring threads is not new, but it seems that in languages such as Java, where multi-threaded access often goes to mutable objects, its use not only as part of the documentation, but also at the compilation stage, assembly could greatly simplify the development of multi-threaded code.
We still use implementation on aspects. If you are familiar with a more elegant solution or analysis tool that allows you to increase the stability of this approach to system changes, please share it in the comments.
We now have 47,000 users daily, about 30 servers in production, 2,000 API requests per second, and daily releases. Miro service has been developing since 2011, and in the current implementation, user requests are processed in parallel by a cluster of heterogeneous servers.
Competitive Access Control Subsystem
The main value of our product is collaborative user boards, so the main burden falls on them. The main subsystem that controls most of the competitive access is the stateful system of user sessions on the board.
For each openable board on one of the servers, the state rises. It stores both application runtime data necessary to ensure collaboration and display of content, as well as system data, such as binding to processing threads. Information about which server the state is stored in is written to a distributed structure and is available to the cluster as long as the server is running, and at least one user is on the board. We use Hazelcast to provide this part of the subsystem. All new connections to the board are sent to the server with this state.
When connecting to the server, the user enters the receiving stream, whose sole task is to bind the connection to the state of the corresponding board, in the flows of which all further work will occur.
Two streams are associated with the board: network, processing connections, and “business”, responsible for business logic. This allows you to transform the execution of heterogeneous tasks of processing network packets and executing business commands from serial to parallel. Processed network commands from users form applied business tasks and direct them to the business stream, where they are processed sequentially. This avoids unnecessary synchronization when developing application code.
The division of code into business / application and system is our internal convention. It allows you to distinguish between the code responsible for the features and capabilities for users, from the low-level details of communication, sheduling and storage, which are the service tool.
If the receiving stream detects that there is no state for the board, the corresponding initialization task is set. State initialization is handled by a separate type of thread.
The types of tasks and their direction can be represented as follows:
Such an implementation allows us to solve the following problems:
- There is no business logic in the receiving stream that could slow down the new connection. This type of stream on the server exists in a single copy, so delays in it will immediately affect the opening time of the boards, and if there is an error in the business code, it can be easily hung.
- State initialization is not performed in the business flow of boards and does not affect the processing time of business commands from users. It may take some time, and business flows process several boards at once, so the opening of new boards does not directly affect existing ones.
- Parsing network commands is often faster than executing them directly, so the configuration of the network thread pool may be different from the configuration of the business thread pool in order to efficiently use system resources.
Flow coloring
The subsystem described above in the implementation is quite nontrivial. The developer has to keep in mind the scheme of the system and take into account the reverse process of closing boards. When closing, you must remove all subscriptions, delete entries from the registries and do this in the same streams in which they were initialized.
We noticed that bugs and complexities of code modification that arose in this subsystem were often associated with a lack of understanding of the execution context. Juggling threads and tasks made it difficult to answer the question in which particular thread a particular piece of code is executing.
To solve this problem, we used the method of coloring threads - this is a policy aimed at regulating the use of threads in the system. Colors are assigned to threads, and methods define the scope for executing within threads. Color here is an abstraction, it can be any entity, for example, an enumeration. In Java, annotations can serve as the color marking language:
@Color
@IncompatibleColors
@AnyColor
@Grant
@RevokeAnnotations are added to the method, with the help of which you can set the validity of the method. For example, if the annotation of a method allows yellow and red, then the first thread can call the method, and for the second, such a call will be erroneous.
Invalid colors can be set:
You can add and remove thread privileges in the dynamics:
Lack of annotation or annotation as in the example below says that the method can be executed in any thread:
Android developers may be familiar with this approach for annotations MainThread, UiThread, WorkerThread, etc. P.
The coloring of threads uses the principle of self-documenting code, and the method itself lends itself well to static analysis. Using static analysis, you can say before the code is executed that it is written correctly or not. If we exclude Grant and Revoke annotations and assume that the stream upon initialization already has an unchangeable set of privileges, then this will be a flow-insensitive analysis - a simple version of static analysis that does not take into account the order of calls.
At the time of the implementation of the flow coloring method, there were no ready-made solutions for static analysis in our devops infrastructure, so we went the simpler and cheaper way - we introduced our annotations, which are uniquely associated with each type of flows. We began to check their correctness with the help of aspects in runtime.
@Aspect
public class ThreadAnnotationAspect {
@Pointcut("if()")
public static boolean isActive() {
… // здесь учитываются флаги, определяющие включены ли аспекты. Используется, например, в ряде тестов
}
@Pointcut("execution(@ThreadAnnotation * *.*(..))")
public static void annotatedMethod() {
}
@Around("isActive() && annotatedMethod()")
public Object around(ProceedingJoinPoint joinPoint) throws Throwable {
Thread thread = Thread.currentThread();
Method method = ((MethodSignature) jp.getSignature()).getMethod();
ThreadAnnotation annotation = getThreadAnnotation(method);
if (!annotationMatches(annotation, thread)) {
throw new ThreadAnnotationMismatchException(method, thread);
}
return jp.proceed();
}
}
For aspects, we use the aspectj library and the maven plugin, which provides weaving when compiling the project. Weaving was initially configured to load-time when loading classes with ClassLoader. However, we were faced with the fact that weaver sometimes behaved incorrectly when loading the same class competitively, as a result of which the original byte of the class code remained unchanged. As a result, this resulted in very unpredictable and difficult to reproduce production behavior. Perhaps in current versions of the library there is no such problem.
The solution on aspects allowed us to quickly find most of the problems in the code.
It is important not to forget to always keep annotations up to date: they can be deleted, add laziness, weaving aspects can be turned off altogether - in this case the coloring will quickly lose its relevance and value.
Guardedby
One of the varieties of coloring is the GuardedBy annotation from java.util.concurrent. It delimits access to fields and methods, indicating which locks are necessary for correct access.
public class PrivateLock {
private final Object lock = Object();
@GuardedBy (“lock”)
Widget widget;
void method() {
synchronized (lock) {
//Access or modify the state of widget
}
}
}
Modern IDEs even support the analysis of this annotation. For example, IDEA displays this message if something is wrong with the code:
The method of coloring threads is not new, but it seems that in languages such as Java, where multi-threaded access often goes to mutable objects, its use not only as part of the documentation, but also at the compilation stage, assembly could greatly simplify the development of multi-threaded code.
We still use implementation on aspects. If you are familiar with a more elegant solution or analysis tool that allows you to increase the stability of this approach to system changes, please share it in the comments.