Optimizing Fastify Backends for High-Load Applications: Practical Solutions
Building scalable and high-performance backends demands a deep understanding of architectural trade-offs. In this article, we'll explore our experience developing a high-load application using Fastify, focusing on strategies to minimize database load, effectively identify users without traditional authentication, and protect against malicious requests, leveraging tools like Bloom filters and asynchronous operations.
ID Generation Strategies and Efficient Data Management
When designing systems that handle a large number of entities, such as users or images, the choice of identifier (ID) format becomes critically important. Standard GUIDs or UUIDs, while ensuring uniqueness, can be excessively resource-intensive for database indexing and backend processing. To reduce this overhead, we implemented a hybrid ID format: 19bc19e00ab-eh5kkeh99q7. The first part represents a Unix timestamp converted to hexadecimal, and the second is a random sequence of characters. The hyphen serves purely for improved readability.
Here's an example function for generating such an ID:
function generateID(): string {
const timestamp = Date.now().toString(16);
const randomPart = generateRandomSequence( 11 );
return timestamp + '-' + randomPart;
}This approach provides a "conditionally incremental" ID, as its initial part changes with every millisecond, which contributes to more efficient database indexing. An additional benefit is the ability to decode the entity's creation date directly from the ID, eliminating the need for a separate created_at field in tables. This reduces the volume of stored data and the number of write operations, though it does impose minor limitations on the convenience of building date-based queries.
Scaling with Bloom Filters and Passwordless Authentication
Developing an application designed for thousands of concurrent sessions and tens of thousands of images necessitates approaches that minimize CPU and memory consumption. Traditional SQL queries to check for viewed images or existing users quickly become a bottleneck. In this context, Bloom filters play an invaluable role.
A Bloom filter is a probabilistic data structure that can efficiently (e.g., with 99.9999% probability) determine if an element is not in a set. Its key advantage is extremely low memory and CPU overhead compared to classical data structures when checking for membership. The primary applications of Bloom filters in our project include:
- Tracking Viewed Images: For each user, a Bloom filter is maintained, storing the IDs of images already shown. This allows for quick checks to see if an image has already been presented to the user, eliminating duplicates from new batches.
- Identifying Existing Users: Upon backend startup, all registered user IDs are loaded into a global Bloom filter. This provides instant verification of whether an incoming
device_idis already known to the system.
To enhance user convenience and simplify app store moderation, a system for identification without explicit authentication was implemented. When the application is installed, a unique device_id is generated, which is then used to identify the user on the backend. To protect against the generation of fake device_ids and abuse, a multi-stage verification process is employed:
- Fastify JSON Schema Validation: The initial validation of the
device_idformat occurs at the Fastify level using a JSON schema that definesminLength,maxLength, and apattern.
```javascript
fastify.post( '/backend', {
schema: {
body: {
type: 'object',
required: [ 'device_id' ],
properties: {
device_id: {
type: 'string',
minLength: 23,
maxLength: 23,
pattern: '^[a-z0-9]{11}-[a-z0-9]{11}$'
}
}
}
}
} );
```
- User Bloom Filter: After passing the Fastify schema, the
device_idis checked against the Bloom filter of existing users. - ID Correctness Check:
* Creation Date: The first part of the ID, encoding the timestamp, must fall within the period between the project's launch and the current date.
* Random Part Entropy: The second part of the ID, representing a random sequence, is checked for entropy. This prevents the use of simple, predictable IDs (e.g., aaaaaaaaaaaa). The entropy must be above a certain threshold (e.g., 2).
```javascript
function getEntropyString( str: string ): number {
const frequencies = new Map();
for (const char of str) {
frequencies.set(char, (frequencies.get(char) || 0) + 1); }
return [...frequencies.values()].reduce((acc, count) => {
const p = count / str.length;
return acc - p * Math.log2(p);
},
0);
}
```
- Asynchronous User Insertion: If the ID is deemed new and valid, it's added to the Bloom filter of existing users, and an
INSERTquery is then asynchronously sent to the database. Omittingawaitprevents blocking the request processing, significantly reducing backend load. This approach accepts an extremely low probability of losing new user data during an unexpected backend restart, which is an acceptable trade-off for this type of application, as "loss" merely means resetting the swipe history for that user.
Optimizing Client Interaction and API Protection
Efficient interaction between the mobile application and the backend is critically important for performance. For serving static content, such as images, Nginx is used, operating on a separate domain. This maximally offloads the Fastify server, allowing it to focus on business logic, and enables effective content caching.
The mobile application receives images in batches of 10. The IDs of these images are also added to a "Viewed Images" Bloom filter specific to each user. This approach minimizes the number of API requests: the application loads two batches at startup and then requests a new batch after every 10 likes. This ensures a continuous content stream while the user interacts with the current batch.
To protect the API from excessive load and abuse, rate limiting is implemented using the rateLimit plugin for Fastify. Depending on the architecture, it can use Redis for distributed data storage or operate in the Node.js application's memory for single instances.
An important aspect is also the validation of user reactions (likes/dislikes), which are sent in batches of 10. This process involves several levels of validation:
- Fastify Schema Validation: The user ID, image ID, and the reaction itself (as an
enum'like' or 'dislike') are checked against a JSON schema. - User Bloom Filter: Confirmation of the
user_id's existence via the "Existing Users" Bloom filter. - Viewed Images Bloom Filter: Verification that the
image_idwas indeed shown to the current user, using their personal "Viewed Images" Bloom filter. - Reaction Batch Authenticity Check: A function,
trustLikeInBatch10, is implemented to analyze patterns within a batch of 10 reactions. If all reactions are identical (all likes or all dislikes) or if there's an excessively frequent alternation (e.g., 8 or more switches), the batch is considered unreliable and ignored. This helps filter out automated or malicious actions.
```javascript
function trustLikeInBatch10( batch: string[] ): boolean {
if( batch.length != feedLimit ) return false;
const likes = batch.filter( v => v === 'like' ).length;
if( likes === 0 || likes === feedLimit ) return false;
const switches = batch.slice(1).reduce( ( acc, val, i ) => acc + ( val !== batch[ i ] ? 1 : 0 ), 0 );
if( switches >= 8 ) return false;
return true;
}
```
- Asynchronous Reaction Writes: Similar to new users, reactions are asynchronously written to the database without awaiting the operation's completion and without using
FOREIGN KEYconstraints. This approach, while unconventional, significantly reduces database load during high-intensity data streams, sacrificing strict consistency for throughput. In this context, the potential loss of a few reactions is not critical for overall statistics and the recommendation system.
To enhance security and prevent exploits related to API validation details, errors generated by Fastify due to JSON schema mismatches are masked. Instead of a detailed problem description, the backend returns a generic response. This makes it harder for attackers to reverse-engineer data schemas.
fastify.setErrorHandler((
error: FastifyError,
request: FastifyRequest,
reply: FastifyReply ) =>
{
console.error( 'setErrorHandler', error );
countErrorStatistics();
reply.code(200).send( 'Hello World!' );
});The client application must be prepared to handle such generic responses, which is part of the overall security strategy.
Leveraging TypeScript for Code Reliability
Using TypeScript significantly enhances code reliability and maintainability. Specifically, when working with fixed sets of string states (e.g., image statuses), you can elegantly create a type from a string array. This ensures strict typing for variables, eliminating errors related to typos or the use of invalid values.
// Usually it's like this:
const imageStateArray1 = [ 'new', 'broken', 'gpt', 'ready', 'download', 'done', 'bad' ];
type tpImageState1 = 'new' | 'broken' | 'gpt' | 'ready' | 'download' | 'done' | 'bad';
// or better yet:
const imageStateArray2 = [ 'new', 'broken', 'gpt', 'ready', 'download', 'done', 'bad' ] as const;
type tpImageState2 = ( typeof imageStateArray2 )[ number ];The second approach, using as const and (typeof array)[number], is more preferable as it automatically infers the literal type from the array, ensuring synchronization between the array values and its type without duplication.
Key Takeaways
- The use of Bloom filters significantly reduces database load when checking for element existence, which is critically important for high-load systems.
- Custom ID generation, incorporating a timestamp, allows for embedding metadata and optimizing indexing, as well as eliminating redundant
created_atfields. - Asynchronous database write operations and the omission of
FOREIGN KEYconstraints boost backend throughput at the cost of potential, but acceptable, data loss during failures in high-intensity applications. - Detailed validation of incoming requests at the Fastify level and data entropy checks protect the API from abuse and malformed data.
- Masking Fastify validation errors enhances API security by obscuring the internal structure of schemas and making reverse-engineering more difficult for potential attackers.
— Editorial Team
No comments yet.