Writing an open source messenger

Why write?

Once upon a time, America Online was in one faraway country . And she had an amazing private Internet behind the fence, where instead of URLs there were “keywords”: something in between the web page address and the purchased keyword in the advertisement. Companies fought for interesting keywords, as they are now fighting for domains, and the advertisement looked like this: "visit us on the world wide web at www.example.com, or type AOL Keyword: 'banking'".

History has the property of repeating itself. Now the role of America Online is played by the main messengers: they are all behind fences, incompatible with each other, everyone invents their keywords, they want to grab the user and never let go. Companies are not interested in openness: larger players do not want to share users with smaller ones, much less become open. As a result, it is impossible to send a message even from WhatsApp to Facebook Messenger, despite the fact that both belong to the same company. And users value reliability and convenience above abstract openness, although it is annoying to many that some friends, for example, on Telegram, some on WhatsApp, and parents on Skype.

But unfortunately no one plays the role of the open Internet today. I want to change the situation. If XMPP failed, maybe someone else can? And here is the story about Tinode .

What is Tinode

Tinode is a fully open source messenger on Github. All client applications ( ReactJS and Android ) are licensed under Apache 2.0, in order to simplify the creation of commercial applications based on Tinode, a server under GPL 3.0. The goal of the project is to create a federated messenger that is simple and convenient for both users and operators. Delivered - and everything works like MySQL or Nginx. In the long term, the goal of the project is to create an open alternative to existing proprietary messengers, to repeat with respect to instant messengers what Android did with respect to operating systems for mobile phones.

What can he do

Support for multiple devices.

Everyone has a smartphone, sometimes not one, plus it is often convenient to use a web application from the main computer. Therefore, support for multiple devices was one of the main requirements for the project, which determined the basic architectural solutions. If a user logs in with a new device, then he doesn’t want him to start from scratch like in WeChat. And this means that you need both the address book and messages to be stored on the server, which was implemented.

Obviously, storing user information on a server is not suitable for everyone, as it creates risks of unwanted access: the more copies of data stored in different places, the higher the likelihood that something will go wrong. For this, the possibility of ephemeral messages and messages that are deleted from the server after delivery to the client is provided. Technically, there is the possibility of not storing contacts on the server permanently - the client sends them to the server at the time of connection (login), then they are deleted after logout. However, the authors considered this impractical difficult and did not.

Online status

Broadcasting online / offline user status in messengers is taken for granted, however, it is very difficult to implement a feature. She needs to “just work,” predictably and reliably. Reliability ruled out the generation of status on the client, as is implemented in some XMPP applications. In the case of Tinode, the server generates online status and sends it to the address book, which, again, requires storing contacts on the server and synchronizing them with client applications.

Protocol simplicity

I wanted to make the protocol so that the learning curve was gentle - you don’t need to know everything to get started. The specification turned out to be very compact : 10 client requests, 5 server responses. For example, compared to 200+ pages, only core XMPP , not counting extensions, is almost a note on a napkin.

The data presentation is separated from the network protocol. The protocol only requires a certain data structure, but does not require that they be transmitted over the network in any specific way. Now the server supports JSON over websocket and long polling, with and without TLS, plus gRPC over TCP. GRPC support was implemented by one developer in two weeks, including writing a text client in Python. Adding support for other data formats and protocols, such as MessagePack or Noise , is unlikely to take much longer.

Extensibility

On the one hand, I want everything to work right away, for example, so that the main functionality is comparable to WhatsApp and Telegram right out of the box. On the other hand, people have different needs and you need to be able to expand functionality. The search for balance is similar to the choice between monolithic architecture and microservices: it is undesirable to have an immutable monolith, and, similarly, it is bad to get a zoo of microservices, which management turns into a separate task.

It was decided to divide the functional into three parts - the main, network and auxiliary. The core is what allows Tinode to perform the core function of forwarding messages. Network - functionality of interaction in servers, as a format of transmitted data and network protocol. The auxiliary is something that solves someone's local problem, for example, supporting a specific database as a backend or some kind of authorization method, but does not affect other servers or user applications in any way. The main functionality is implemented in the main code. Network functionality is allocated, but it is also stored in the main repository in order to avoid creating incompatible servers, if possible. The auxiliary is implemented in the form of plugins - compiled Go interfaces (support for different databases, different authorizers, push notifications,

Other

• An opportunity, but not a requirement to link an account to a phone or email or anything else.
• User IDs that are difficult to guess and therefore difficult to send spam.
• Tags, allowing you to implement people search like in WeChat (and, like WeChat, embed a dating service in the messenger) or divide the organization into departments like in Slack.
• The ability to connect users without registration, necessary, for example, to organize a support service via chat.
• Interface and example of connecting chatbots.
• Plans for creating channels like in Telegram.

Why go?

The server for the messenger is essentially a router: it receives a message from one channel, somehow processes it, then transfers it to another channel or channels. Go (like Erlang, but that's another story) is ideal for creating such functionality because contains goroutine and chan primitives that make organizing threads and exchanging data between them efficient and easy.

Of course, the router can be written in C / C ++, and in Java. However, ceteris paribus, the code is likely to turn out to be more complex and require a lot of effort to avoid deadlocks.

And then what?

Federation

One of the main tasks for Tinode for the coming year is to create a platform for federation. So that anyone can start their Tinode server, which could exchange messages with any other server, just as it is possible with an email. Server clustering is already possible. Network exchange between the server and clients is via TLS websocket, which for an external observer is little different from simple HTTPS traffic.

Public DNS is likely to be used, at least initially. However, in the future, the search for chat servers will be carried out in the same way as it is done in Bittorrent - using DHT , a distributed hash of the table.

I would also like to avoid the problems for which XMPP is often criticized. For example, XMPP servers are very talkative. Up to half of the messages are duplicate when the XMPP client sends online notifications individually to each contact from the address book.

Reputation and Distributed Decision Making

Messaging systems are most similar to email. As you know, a significant part of email is spam. I don’t want to repeat the mistakes of others, therefore, mechanisms that limit spam should be immediately built into the system. The problem has not yet been completely solved, but there is a general direction:

• Cryptographic identification of the sending server.
  Initially, SMTP did not imply any identification of the sender at all. Do not step on this rake again. Each server that wants to establish contact will be presented with a cryptographic certificate. "Well, yes," you say, "I will now renew 100,500 certificates and each time I will appear as a new clean server." And you will be right. Therefore, the next paragraph.
• Distributed reputation accounting.
  When a new, unknown sender server knocks on us, we will make a request to known servers with a request to report the rating of the new server. And depending on the answer, we’ll set, for example, the speed with which the new server can send us messages.

If you look at the distributed decision-making system and accounting for reputation from a bird's eye view, you will notice a similarity with the blockchain. Perhaps blockchain (but not cryptocurrency) can be used as the basis for building a distributed reputation system, although it is not yet clear how.

Encryption

Well, what about these days without message encryption? Chats between two people are likely to be encrypted by OTR . With group chats is still unclear. All known encryption schemes for group chats either have significant shortcomings, or are heavy and difficult to implement. Also, it is not obvious how important the encryption of group chats is: "If two know the secret, this is no longer a secret, but if three, this is the market."

What do you think about this?