The intricacies of the "encrypted pipeline" in the process of developing an IMAP client on Scala + Akka + Spray
More recently, I switched from my beloved object-oriented C ++ to a new for me and still not quite clear functional Scala. The reasons for the transition are a completely different story. But one of them was the presence of good enough, judging by the reviews, support for the actor model - using the Akka library. I have long dreamed of trying out on my own experience all the described advantages of this technology, and the existing implementations in C ++ (CAF_C ++ and Theron), which I turned a little in small tests, turned out to be raw enough for my needs. The most canonical (in my opinion) decision of the actor model - Erlang - I dismissed, because I thought that it would take me too much time to master it, and not the fact that I could find the third-party libraries I needed for this far from universal language. Therefore, as a result, my choice fell on Scala in conjunction with Akka, especially since I once began to study Scala for a long time, but abandoned it for inexpediency. However, as it turned out, this time I didn’t choose the best time for my experiment, which I became convinced of only after a fairly solid part of the project was already completed.
Development from the very beginning went at an accelerated pace: almost all the necessary functionality that was needed for my application was abundantly available on the Internet as third-party libraries. And the lack of something that has not yet been written specifically for Scala has more than been compensated by the rich variety of various components for Java. However, the problem, as always, arose in an unexpected place.
The fact is that at some point my application needs to connect to the IMAP server to read and process received mail messages. And since the actor model implies asynchronous work with the network, I needed a library that could connect to the mail server and receive mail asynchronously in order to fit beautifully and organically into the structure of my new application. After a short search, I came across akka-camel module, which allows you to use the apache-camel library as a message channel for actors. And camel, as it turned out, can connect, among other things, to mail servers. In addition, when specifying the necessary connection parameters, camel can only read fresh (with the \ Recent) flag, delete read messages, or copy / move them to a specially created folder. I could not even dream of more. To get started in SBT, you just had to mention the dependencies akka-camel, camel-core and camel-mail.
Creating an actor and connecting it to an IMAP server took just a few lines of code. And in the application log the text of the message fell out, which I myself sent to the mail for the test. I already started to rub my hands in satisfaction and think about the next task, but I decided just in case to try connecting to a work box, which will receive mail for processing as a result. And here my actor threw an exception and “fell”. As it turned out, he could not correctly parse the server response. On the Internet, I did not find any information about this error and possible solutions. And a little depressed. Somehow I did not really want to spend time studying protocol specifications and writing my client. And I reluctantly, in the name of saving time, decided to step back from the intended course of complete asynchrony and use the JavaMail synchronous blocking library. However, in the same place, with the same exception, this library fell. After that, I already firmly decided that abandoning ideals is the way for weaklings and lazy people, but I still write my own IMAP client, with asynchrony and actors. Moreover, I did not need to implement the entire IMAP, the functionality was very limited: authorization, selecting the INBOX folder, receiving a list of messages, reading a specific message, copying the message to another folder, and deleting.
I especially did not have to choose where to start. As you know, Akka developers at some point abandoned Netty for network I / O in favor of Spray. In the future, the development of Akka and Spray was so closely intertwined that even their documentation mutually referred to each other, and pieces of code from spray.io smoothly migrated to akka.io. And here here the main catch was waiting for me: once, during the development of versions 2.x, Akka adopted the idea of channels used in Spray (they are also “pipelines”, they are pipelines in English), which allow ( according to the authors, it is easy to create network protocols that support back pressure - that is, the ability to “tighten the valve” so that the pipes do not clog in case the recipient does not have time to process the data stream from the sender, filter, separate, to multiply data and what else not to do with them. But something in these “pipelines” went wrong, and they, without ever leaving the experimental stage, were declared deprecated. The last announced innovation from Akka, the purpose of which is to completely replace the channels, is “reactive streams”, which have already been written about on the hub But since this innovation is still in the announcement stage, in the latest version of akka 2.3.6 it is not there yet, but there are no channels anymore. The channels remained in the spray, but all the documentation on them leads to the outdated Akka Version 2.2.0-RC1 documentation, which no longer reflects all current reality. And the new Akka documentation says the channels stayed in Spray. Generally, The first version of my mail client turned out to be approximately similar to the long-suffering child of Frankenstein - collected from different pieces of disparate and sometimes conflicting documentation. I immediately decided to abandon the "pipeline" in view of the transcendental complexity of this concept that seemed to me sky-high, and therefore my client worked directly with the stream of characters from the server in the form of ByteString. More precisely, with fragments of this stream, since no one guarantees that the answer of interest will come in one whole piece, or two answers will not go together. In some miraculous way, through a bunch of matyots poured into the monitor and rewritten pieces of code, I managed to screw SSL / TLS encryption to my actor using several pieces of code found in different places. The code,
With the implementation of each subsequent stage of their modest functionality, my client became more monstrous. In the end, after another iteration, at 3 in the morning, I realized that you can’t live like that anymore, I wrote to myself in TODO to try the same blocking JavaMail, but through POP3, without the ability to move messages through folders, I went to bed.
However, being a stubborn scumbag, the next morning (or rather, at lunch), instead of trying to tame JavaMail, I first got into the Spray sources on the github. I spent a couple of days studying them and adapting the received information to my needs, but the time spent paid off handsomely. First of all, in the source code, I came across a ConnectionHandler class that was not described elsewhere in the documentation, which made life much easier for me and my creation, putting a lot of things in their places. It was while studying the application of this class in spray-can that I understood how and where these “pipelines” can be used, about which I found out from the documentation only what tasks they are called to solve, but not howthey do it. In the same source code, I discovered how you can “connect pipes” —that is, combine several “pipes” (stages) of a pipeline (PipelineStage) into one common “pipeline”, what this leads to and how it is supposed to be used. And I also found out why and how exactly the SSL encryption that I screwed up on the day before works, which until now has remained a black box for me, which "just works and doesn’t have to get into it."
For those who are interested in the details: the “pipeline” consists of parts, in the original they are called “stages” or “stages” (English stages), but I will call them “pipes” to maintain imagery. These "pipes" in the code are combined using the >> operator, the order matters. The first are the "pipes" closest to the client, and the last to the server. That is, everything that goes from the client passes through the “pipeline” from left to right, from the server - on the contrary, from right to left. For example, the “pipe” performing the encryption is indicated last, therefore everything that the client sends to the “pipe” undergoes all the necessary transformations first, and only then it is encrypted, and eventually the encrypted data is sent to the server. And vice versa, everything that the server sends is first decrypted, then transformed by the rest of the “pipeline”. Why do you need all this plumbing? For a wide variety of things. For example, to filter data sent or received. Or to transform some entities into others, which is useful when implementing protocols. Let's say there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected. there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected. there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected.
The main point that at first completely baffled me was the presence of the conceptual concepts of “Event” and “Command”, and the corresponding breakdown of the pipeline into two: the event pipeline and the command pipeline within one class: PipelineStage. It was these very not understood by me at the very beginning of the concept (well, the manuals, especially the scattered and incomprehensible ones, that only losers read to the end, normal guys go right ahead and fill up cones) made me think of pipelines as bad and decided for myself that it was too much complicated and not worth the time spent. It seemed to me that this was somehow connected with the very “back pressure”, which would have to be taken into account and implemented, although I did not need it at all. And this, in addition to the fact that at first I didn’t understand, where to stick one “pipe”, and how to put something into it so that it reaches the server. And then how to get the answer out of her. And then there were two more of these pipes. On the other hand, if it were not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from. if not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from. if not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from.
In general, as a result of my research, I got a new class responsible for connecting to the IMAP server, which took such a short and concise look:
I got this class by simplifying the spray.can.client.HttpClientConnection class. It is inherited from spray.io.ConnectionHandler, and that, in turn, is from akka.Actor. That is, he is an ordinary actor. In fact, this is a plumber, let's call him Stanislav. Stanislav is responsible for the pipeline from the client to the server and the delivery of data through this pipeline, back and forth. It paves this pipeline when initializing the actor (i.e., himself) with standard context.actorOf (...). The pipeline property is the very pipeline assembled in this case from three pipes - a frontend, a response parser from the server, and an SSL / TLS encryptor. And now, all the data sent to the plumber Stanislav (by sending him, as an actor, messages to the operator!, By the tell method, or by any other available means), he carefully puts it into the pipe and sends it to the server. And everything that comes in response from the server will also be carefully taken out of the pipe and sent to the client. Such is he, our hardworking guy Stasik.
As for the "pipes" I used.
SslTlsSupport is Spray’s standard SSL / TLS encryption capability. It requires a special context (returned by the pipelineContext method), and also requires that the client side of the connection be kept open, even after the server closes the connection on its side (the so-called semi-open connection).
ResponseParsing- this is an object already written by me with the apply () function, which returns an instance of the "pipe" responsible for parsing - parsing the stream of characters from the server (in the form of "raw" Tcp.Received messages) to case classes of specific answers that are understood and processed already target actor (which is my IMAP client). The parser is also responsible for monitoring the integrity of the returned data: wait for additional data if the response from the server is not complete, and also separate several responses from each other if they came in one piece. This greatly unloaded the code of my client, which now has turned from a terrible patchwork monster into a simple, understandable, direct and uncomplicated guy Vasily, a crony friend of our neat man Stasik (if he hadn’t been friends yet, Stas just came and took on a lot of dirty work). Mass of tests,
Finally, eventFrontend is a function that returns an instance of a “pipe” - PipelineStage, the essence of which is one: transmit all “events” (that is, data that has passed through the entire pipeline from the server and has already undergone all the necessary changes) to the client, that is, Vasya, whose address Stanislav knows thanks to the variable passed in the class constructor.
I didn’t do any special rendering of commands, for lack of such a need. All commands to the server are sent using simple Tcp.Write.
Here, in fact, all the plumbing. As an epilogue, I can say that the client itself is a Finite-state machine based on Akka.FSM. I just fell in love with the implementation of this concept in Akka, since writing an automaton and unit tests for him is such an exciting mini-game.
Start
Development from the very beginning went at an accelerated pace: almost all the necessary functionality that was needed for my application was abundantly available on the Internet as third-party libraries. And the lack of something that has not yet been written specifically for Scala has more than been compensated by the rich variety of various components for Java. However, the problem, as always, arose in an unexpected place.
The fact is that at some point my application needs to connect to the IMAP server to read and process received mail messages. And since the actor model implies asynchronous work with the network, I needed a library that could connect to the mail server and receive mail asynchronously in order to fit beautifully and organically into the structure of my new application. After a short search, I came across akka-camel module, which allows you to use the apache-camel library as a message channel for actors. And camel, as it turned out, can connect, among other things, to mail servers. In addition, when specifying the necessary connection parameters, camel can only read fresh (with the \ Recent) flag, delete read messages, or copy / move them to a specially created folder. I could not even dream of more. To get started in SBT, you just had to mention the dependencies akka-camel, camel-core and camel-mail.
First try
Creating an actor and connecting it to an IMAP server took just a few lines of code. And in the application log the text of the message fell out, which I myself sent to the mail for the test. I already started to rub my hands in satisfaction and think about the next task, but I decided just in case to try connecting to a work box, which will receive mail for processing as a result. And here my actor threw an exception and “fell”. As it turned out, he could not correctly parse the server response. On the Internet, I did not find any information about this error and possible solutions. And a little depressed. Somehow I did not really want to spend time studying protocol specifications and writing my client. And I reluctantly, in the name of saving time, decided to step back from the intended course of complete asynchrony and use the JavaMail synchronous blocking library. However, in the same place, with the same exception, this library fell. After that, I already firmly decided that abandoning ideals is the way for weaklings and lazy people, but I still write my own IMAP client, with asynchrony and actors. Moreover, I did not need to implement the entire IMAP, the functionality was very limited: authorization, selecting the INBOX folder, receiving a list of messages, reading a specific message, copying the message to another folder, and deleting.
Second attempt
I especially did not have to choose where to start. As you know, Akka developers at some point abandoned Netty for network I / O in favor of Spray. In the future, the development of Akka and Spray was so closely intertwined that even their documentation mutually referred to each other, and pieces of code from spray.io smoothly migrated to akka.io. And here here the main catch was waiting for me: once, during the development of versions 2.x, Akka adopted the idea of channels used in Spray (they are also “pipelines”, they are pipelines in English), which allow ( according to the authors, it is easy to create network protocols that support back pressure - that is, the ability to “tighten the valve” so that the pipes do not clog in case the recipient does not have time to process the data stream from the sender, filter, separate, to multiply data and what else not to do with them. But something in these “pipelines” went wrong, and they, without ever leaving the experimental stage, were declared deprecated. The last announced innovation from Akka, the purpose of which is to completely replace the channels, is “reactive streams”, which have already been written about on the hub But since this innovation is still in the announcement stage, in the latest version of akka 2.3.6 it is not there yet, but there are no channels anymore. The channels remained in the spray, but all the documentation on them leads to the outdated Akka Version 2.2.0-RC1 documentation, which no longer reflects all current reality. And the new Akka documentation says the channels stayed in Spray. Generally, The first version of my mail client turned out to be approximately similar to the long-suffering child of Frankenstein - collected from different pieces of disparate and sometimes conflicting documentation. I immediately decided to abandon the "pipeline" in view of the transcendental complexity of this concept that seemed to me sky-high, and therefore my client worked directly with the stream of characters from the server in the form of ByteString. More precisely, with fragments of this stream, since no one guarantees that the answer of interest will come in one whole piece, or two answers will not go together. In some miraculous way, through a bunch of matyots poured into the monitor and rewritten pieces of code, I managed to screw SSL / TLS encryption to my actor using several pieces of code found in different places. The code,
With the implementation of each subsequent stage of their modest functionality, my client became more monstrous. In the end, after another iteration, at 3 in the morning, I realized that you can’t live like that anymore, I wrote to myself in TODO to try the same blocking JavaMail, but through POP3, without the ability to move messages through folders, I went to bed.
Third attempt
However, being a stubborn scumbag, the next morning (or rather, at lunch), instead of trying to tame JavaMail, I first got into the Spray sources on the github. I spent a couple of days studying them and adapting the received information to my needs, but the time spent paid off handsomely. First of all, in the source code, I came across a ConnectionHandler class that was not described elsewhere in the documentation, which made life much easier for me and my creation, putting a lot of things in their places. It was while studying the application of this class in spray-can that I understood how and where these “pipelines” can be used, about which I found out from the documentation only what tasks they are called to solve, but not howthey do it. In the same source code, I discovered how you can “connect pipes” —that is, combine several “pipes” (stages) of a pipeline (PipelineStage) into one common “pipeline”, what this leads to and how it is supposed to be used. And I also found out why and how exactly the SSL encryption that I screwed up on the day before works, which until now has remained a black box for me, which "just works and doesn’t have to get into it."
Enlightenment
For those who are interested in the details: the “pipeline” consists of parts, in the original they are called “stages” or “stages” (English stages), but I will call them “pipes” to maintain imagery. These "pipes" in the code are combined using the >> operator, the order matters. The first are the "pipes" closest to the client, and the last to the server. That is, everything that goes from the client passes through the “pipeline” from left to right, from the server - on the contrary, from right to left. For example, the “pipe” performing the encryption is indicated last, therefore everything that the client sends to the “pipe” undergoes all the necessary transformations first, and only then it is encrypted, and eventually the encrypted data is sent to the server. And vice versa, everything that the server sends is first decrypted, then transformed by the rest of the “pipeline”. Why do you need all this plumbing? For a wide variety of things. For example, to filter data sent or received. Or to transform some entities into others, which is useful when implementing protocols. Let's say there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected. there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected. there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one stage (in one of the “pipes”) this class is converted to the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” that the server understands. Still “pipes” can delay the delivery of data if, for example, the response from the server is incomplete, and an addition is expected.
The main point that at first completely baffled me was the presence of the conceptual concepts of “Event” and “Command”, and the corresponding breakdown of the pipeline into two: the event pipeline and the command pipeline within one class: PipelineStage. It was these very not understood by me at the very beginning of the concept (well, the manuals, especially the scattered and incomprehensible ones, that only losers read to the end, normal guys go right ahead and fill up cones) made me think of pipelines as bad and decided for myself that it was too much complicated and not worth the time spent. It seemed to me that this was somehow connected with the very “back pressure”, which would have to be taken into account and implemented, although I did not need it at all. And this, in addition to the fact that at first I didn’t understand, where to stick one “pipe”, and how to put something into it so that it reaches the server. And then how to get the answer out of her. And then there were two more of these pipes. On the other hand, if it were not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from. if not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from. if not for this misunderstanding, I would not have fully felt the full power of the “plumbing” approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it for itself separates two oncoming flows, so as not to get confused about where it goes from.
Result
In general, as a result of my research, I got a new class responsible for connecting to the IMAP server, which took such a short and concise look:
class Connection(client: ActorRef, remoteAddress: InetSocketAddress, sslEncryption: Boolean, connectTimeout: Duration)(implicit sslEngineProvider: ClientSSLEngineProvider) extends ConnectionHandler { actor =>
override def supervisorStrategy = SupervisorStrategy.stoppingStrategy
def tcp = IO(Tcp)(context.system)
log.debug("Attempting connection to {}", remoteAddress)
tcp ! Tcp.Connect(remoteAddress)//, timeout = Some(Duration(connectTimeout, TimeUnit.SECONDS)))
context.setReceiveTimeout(connectTimeout)
val pipeline = eventFrontend >> ResponseParsing() >> SslTlsSupport(512, publishSslSessionInfo = false)
override def receive: Receive = {
case connected: Tcp.Connected =>
val connection = sender()
connection ! Tcp.Register(self, keepOpenOnPeerClosed = sslEncryption)
client ! connected
context.watch(connection)
context.become(running(connection, pipeline, pipelineContext(connected)))
case Tcp.CommandFailed(_: Tcp.Connect) =>
throw new ConnectionFailure(1, "Failed to connect to IMAP server")
case ReceiveTimeout =>
log.warning("Connect timed out after {}", connectTimeout)
throw new ConnectionFailure(2, "Connect timed out")
}
def eventFrontend = new PipelineStage {
def apply(context: PipelineContext, commandPL: CPL, eventPL: EPL): Pipelines = new Pipelines {
val commandPipeline: CPL = commandPL
val eventPipeline: EPL = {
case event => client ! event
}
}
}
def pipelineContext(connected: Tcp.Connected) = new SslTlsContext {
def actorContext = context
def remoteAddress = connected.remoteAddress
def localAddress = connected.localAddress
def log = actor.log
def sslEngine = if (sslEncryption) sslEngineProvider(this) else None
}
}
I got this class by simplifying the spray.can.client.HttpClientConnection class. It is inherited from spray.io.ConnectionHandler, and that, in turn, is from akka.Actor. That is, he is an ordinary actor. In fact, this is a plumber, let's call him Stanislav. Stanislav is responsible for the pipeline from the client to the server and the delivery of data through this pipeline, back and forth. It paves this pipeline when initializing the actor (i.e., himself) with standard context.actorOf (...). The pipeline property is the very pipeline assembled in this case from three pipes - a frontend, a response parser from the server, and an SSL / TLS encryptor. And now, all the data sent to the plumber Stanislav (by sending him, as an actor, messages to the operator!, By the tell method, or by any other available means), he carefully puts it into the pipe and sends it to the server. And everything that comes in response from the server will also be carefully taken out of the pipe and sent to the client. Such is he, our hardworking guy Stasik.
As for the "pipes" I used.
SslTlsSupport is Spray’s standard SSL / TLS encryption capability. It requires a special context (returned by the pipelineContext method), and also requires that the client side of the connection be kept open, even after the server closes the connection on its side (the so-called semi-open connection).
ResponseParsing- this is an object already written by me with the apply () function, which returns an instance of the "pipe" responsible for parsing - parsing the stream of characters from the server (in the form of "raw" Tcp.Received messages) to case classes of specific answers that are understood and processed already target actor (which is my IMAP client). The parser is also responsible for monitoring the integrity of the returned data: wait for additional data if the response from the server is not complete, and also separate several responses from each other if they came in one piece. This greatly unloaded the code of my client, which now has turned from a terrible patchwork monster into a simple, understandable, direct and uncomplicated guy Vasily, a crony friend of our neat man Stasik (if he hadn’t been friends yet, Stas just came and took on a lot of dirty work). Mass of tests,
Finally, eventFrontend is a function that returns an instance of a “pipe” - PipelineStage, the essence of which is one: transmit all “events” (that is, data that has passed through the entire pipeline from the server and has already undergone all the necessary changes) to the client, that is, Vasya, whose address Stanislav knows thanks to the variable passed in the class constructor.
I didn’t do any special rendering of commands, for lack of such a need. All commands to the server are sent using simple Tcp.Write.
Epilogue
Here, in fact, all the plumbing. As an epilogue, I can say that the client itself is a Finite-state machine based on Akka.FSM. I just fell in love with the implementation of this concept in Akka, since writing an automaton and unit tests for him is such an exciting mini-game.