Titanium: messenger backend on Erlang

Titanium: Scalable Backend Core for Custom Messengers — From Prototype to Production

Developers building custom messengers often start with XMPP/ejabberd or Matrix/Synapse. These platforms provide ready-made transport, federation, and E2E encryption. But customizing for modern UX hits roadblocks.

XMPP with ejabberd on Erlang is compact and load-resistant, but:

• XML stanzas add parsing overhead.
• MUC rooms don't scale linearly: CPU spikes with participant growth.
• Partial XEP extensions lead to race conditions in business logic.

Matrix is powerful, but its layered architecture makes changes tricky: Synapse on Python struggles with GIL in real-time tasks.

Hybrid setups—like ejabberd for transport + FastAPI for API/OTP/reactions—cause split-brain state, XML<->JSON conversions, and extra queues. The single source of truth gets blurred across components.

Takeaway: It's simpler to build a backend from scratch for full control over state and performance.

Titanium v1: FastAPI Under Load

The first Titanium version on FastAPI with WebSocket handled core features: chats, UINs, media, pts-sync. The prototype sustained 500 RPS in a 1000-user scenario with 120 msg/s, 40 media/min, and 7% disconnects.

RPS was measured on full paths: send -> delivery -> ack -> sync. This covered:

• Access rights validation.
• pts updates (message points for delta-sync).
• Real-time WebSocket pushes.
• Retries on flaky networks.
• Update logs for offline clients.

Issues emerged at scale:

• GIL throttled WebSocket concurrency.
• PostgreSQL locked up on high contention in pts tables.
• Media storage needed temporary links, not permanent ones.

Titanium Architecture: Where the Truth Lives

Titanium defines strict state management rules:

• Server as source of truth: Never trust clients; all changes via API.
• Pts mechanism: Delta-sync via message points, Telegram-style.
• Update log: Immutable log for state replay.
• Distributed state: Sharding by UIN, no sticky sessions.
• Media: Temporary S3-style signatures, 24h TTL.

Key components:

• Gateway: WebSocket + HTTP/2, rate limiting, JWT.
• Core: Business logic, pts, ACLs (by UIN/groups).
• Storage: PostgreSQL (states) + Redis (sessions/queues) + S3 (media).
• Workers: Celery/RQ for async tasks (media processing, bots).

Pts-sync example (pseudocode):

% Client requests delta
handle_get_updates(ReqPts) ->
  Updates = db:fetch_updates_after(ReqPts, UserId),
  NewPts = lists:max([U#update.pts || U <- Updates]),
  {reply, #{updates => Updates, pts => NewPts}}.

Switching to Erlang/OTP: Scalability and Resilience

Python couldn't handle 10k concurrent WebSockets. Erlang/OTP fixed it:

• Actor model: Each UIN gets a supervisor tree with gen_server.
• VM hot-reload with zero downtime.
• Built-in Mnesia/ETS for low-latency storage.
• Cowboy for WebSocket scaling to 1M+ connections per node.

Benchmarks:

| Scenario | FastAPI (Python) | Titanium (Erlang) |

|----------|-------------------|--------------------|

| 1k users, 120 msg/s | 500 RPS | 5k RPS |

| 10k WS | 20% CPU | 5% CPU |

| Media 100/min | 2s latency | 200ms |

Gen_server session example:

-module(session).
-export([start_link/1, handle_message/2]).

handle_call({send_message, Msg}, _From, State) ->
  case validate_rights(Msg) of
    ok ->
      NewPts = State#state.pts + 1,
      save_update(Msg#update{pts=NewPts}),
      broadcast_to_peers(Msg),
      {reply, ok, State#state{pts=NewPts}};
    {error, Reason} -> {reply, {error, Reason}, State}
  end.

Group Chats and Edge Cases

Groups are a beast. Ditching MUC for:

• Sharded rooms: Leader by hash(UIN_list), followers replicate deltas.
• Presence: ETS for online/offline, pubsub on reconnect.
• Bad networks: Optimistic send + server ACK, gap-filling from logs.

Handled cases:

• Duplicate sends (idempotency by msg_id).
• Packet reordering (timestamp + seq).
• Sleep/wakeup tabs (long-poll fallback).
• Bot webhooks with retries (dead letter queue).

Key Takeaways

• Titanium isn't an MTProto clone—it's a lean real-time stack: pts, logs, sharding.
• Erlang/OTP delivers 10x RPS at 1/4 CPU vs Python.
• Server truth prevents client-side inconsistencies.
• Media/bots offloaded async; core focuses on delivery.
• Scale: 3 nodes handle 50k users, 1k msg/s sans federation.

— Editorial Team