OLTP Database Architecture: From Principles to Implementation in Rust
Key OLTP engine subsystems have been implemented: unified storage in a single file per database, disk-backed storage with UNDO-log MVCC, WAL and ARIES-based recovery, a shared BufferPool, an in-memory engine, and a pgwire protocol compatible with PostgreSQL. The code is functional but requires load testing and stabilization of edge cases. This is not a final product but an early stage with established contracts.
Core Design Principles
The project relies on strict rules for decision-making. Each new decision must adhere to them or justify any deviation.
Restrictive by Default
Components have explicit limits with fail-closed behavior instead of throttling. This ensures predictability under load.
| Component | Limit | Action on Violation | SQLSTATE |
|-----------|---------|------------------------|----------|
| BufferPool | buffer_pool_size_mb | Eviction (CLOCK), WAL-first flush | — |
| TxnWriteSet | txn_max_write_set_mb | Reject DML | 54023 |
| UndoStore | undo_max_size_mb | Reject writes | 53100 |
| Connection pool | max_connections | Reject new connections | 53300 |
| Statement timeout | statement_timeout_ms | Cancel query | 57014 |
| Snapshot age | max_snapshot_age | Force-close stale snapshots | 40001 |
Fail-closed provides clear SQLSTATE codes for application retry or circuit breaker logic.
Contract-First Approach
Architectural contracts are defined in Rust traits: TableEngine, PageProvider, TransactionLogSink, StorageIo. This prevents implementation drift by enforcing invariants at compile time.
Why Rust?
Rust was chosen for its predictable latency, memory allocation control, and type safety. Send/Sync at async/sync boundaries, explicit errors instead of runtime issues. C++ requires additional discipline, while Go is unsuitable due to GC for OLTP tail latency.
Separation of Responsibilities
The database functions as a data engine without business logic: no triggers, PL/pgSQL, or stored procedures initially. User-defined functions may be added later with sandboxing and no side effects.
Hybrid Asynchronicity
- Network/pgwire layer: async (accept, TLS, send).
- Core (query execution, storage, WAL, MVCC): sync for correctness and debugging.
- Boundary is one-way via a bridge.
StorageIo is abstracted to allow future async I/O without rewriting transaction logic.
| Layer | Runtime | Examples |
|------|---------|---------|
| Network/Protocol | async | pgwire, TLS |
| Query Execution | sync | plan, execute |
| Storage/WAL | sync (async I/O later) | HeapStore, BufferPool |
| MVCC/Transactions | sync | snapshot, locks |
PostgreSQL Compatibility
Pgwire enables integration with drivers and ORMs without client migrations. Compatibility is at the wire level via a boundary layer, without copying PostgreSQL internals (MVCC heap, VACUUM). The core retains architectural freedom.
Platform Constraints and Reliability
Linux-only to focus on io_uring and eBPF without cross-platform compromises. Priorities: no data corruption, safe rollback of optimizations, resilience to user input, built-in security.
Key Takeaways
- Restrictive by Default with fail-closed for predictability under load.
- Contract-First via Rust traits for invariant control.
- Hybrid async/sync: core stability before I/O acceleration.
- Pgwire compatibility without copying PostgreSQL internals.
- Data engine without business logic for purity and testability.
— Editorial Team
No comments yet.