Backup failure due to GIF duplicates in Discourse

# How One GIF File Caused a Backup Failure in Discourse

During a backup of one of the forums on the Discourse platform, an anomalous incident occurred: the same 1.6 MB GIF file was duplicated 246,173 times. This ballooned the backup size by 377 GB, corrupted the Linux filesystem, and completely crashed the backup process. The issue wasn't due to a bug in the code but rather the platform's security architecture and limitations of the ext4 filesystem.

Why One File Became Hundreds of Thousands of Copies

Discourse uses a "safe upload" mechanism where every time a file is moved between different contexts (e.g., from a private message to a public post), the system creates a new copy with a unique SHA1 hash. Even though the file content remains identical, the platform treats it as a separate object. This is necessary to enforce security policies and prevent unauthorized access to media files.

In the case of a popular GIF—the dancing Rachel from the TV show Friends—users repeatedly inserted it into various topics, comments, and private messages. Each use in a new context generated a new database entry and a new file on disk, despite the binary content being exactly the same.

First Optimization Attempt: Hard Links

The Discourse team developed a solution based on grouping files by content hash during backup. For each group of duplicates, only one physical copy was kept, with the rest replaced by hard links to it. This approach drastically reduced backup sizes and worked fine on most instances.

However, testing on a large client forum revealed that the ext4 filesystem imposes a limit: a maximum of about 65,000 hard links per inode. With 246,173 duplicates, once the limit was hit, the system started creating new physical copies—not as duplicates, but as independent files. In the end, instead of one copy, there were ~181,000 additional files, which only partially solved the problem.

Final Solution: Dynamic Switching Between Links and Copies

The new solution accounts for filesystem limitations and works universally:

• The system starts by creating hard links, as before.
• When it receives an EMLINK error ("Too many links") from the Linux kernel, the process switches to local file copying.
• The new copy becomes the "primary" one for subsequent links, and the cycle repeats upon hitting the next limit.

This approach requires no filesystem tweaks, is compatible with any POSIX-compliant OS, and effectively reduces backup sizes even with extreme duplicate counts.

Example processing logic (simplified):

if hard_link_fails_with_emlink?
  create_new_physical_copy
  reset_link_counter
end

Key Takeaways

• Security vs. Efficiency: Architectural choices for context isolation can lead to unexpected side effects at scale.
• Filesystem Limits Are Real: Even low-level limits like 65K hard links per inode in ext4 can disrupt high-level apps.
• Universal Solutions Are Better: Adaptive algorithms that respond to OS errors are more reliable than static optimizations for ideal conditions.
• Popular Content Drives Load: Frequently used media files (emojis, reactions, memes) become hidden resource hogs.
• Production Data Testing Is Critical: Only real-world usage scenarios reveal these edge cases.

Lessons for Distributed Systems Developers

The Discourse incident highlights a classic dilemma: balancing data security with storage efficiency. When designing systems that handle user-generated content, consider:

• Application-level deduplication before writing to disk.
• Monitoring metrics related to file counts and duplicates.
• Graceful degradation when hitting system limits.

Although the problem stemmed from mass use of one GIF, its roots lie in architectural trade-offs. The solution found by the Discourse team can serve as an example for other projects facing similar challenges.

— Editorial Team