7 lessons learned when creating Reddit

Original author: highscalability.com
  • Transfer
UPD The original article is quite old - 2010. Now the situation is different.

In December 2010, Reddit had 829M views and 119 servers.
At the end of 2011 - 2.07B views and 240 servers.

Thanks to potomushto for updating.

UPD 2. Corrected the scheme for people with color perception problems. Thanks second_pilot and spiritedflow



Steve Huffman, one of the creators of Reddit, told the presentation what they learned while they built and developed Reddit to 7.5 million users per month, 270 million page views per month and more than 20 database servers.


Steve clarified that most of the lessons learned were obvious, so there would be no radically new ideas in the presentation. But Steve has a lot of experience, and if he could not see this rake, then perhaps you should pay attention to these “obvious things”.


Each of the 7 lessons will be considered in the corresponding section.
  • Fall often
  • Service Sharing
  • Open data schema
  • Avoid storing states
  • Memcache
  • Save redundant data
  • Get the most out of your work in the background


The most interesting point of Reddit architecture is "Lesson 6 - Save Redundant Data." The basic idea is very simple - we achieve speed by pre-calculating everything and caching it. Reddit uses maximum calculation. One gets the impression that everything you see on Reddit has been calculated in advance and cached, regardless of how many versions of the data they have to store. For example, they cache all 15 ways of sorting messages (hot, new, best old, this week, etc.) for a list of messages as soon as someone adds a link. Ordinary developers are afraid to make such an extreme cache, considering it a waste of resources. But the Reddit team thinks it's better to spend some resources than to slow down. Spending disk space and memory is better than keeping users waiting. So that,


Lesson 1: Fall often


The essence of the lesson is simple - automatically restart dropped or failed services.

The problem with using dedicated servers is that you are constantly responsible for their support. When your service crashes, you should repair it right now, even at 2 a.m. This creates a constant background stress in your life. You must constantly carry a computer with you, and you know that at any moment someone can call to report another disaster. Which will have to be raked. This approach makes life unbearable.

One of the methods to solve this problem is to restart the service that died or began to behave incorrectly. Reddit uses supervisors to restart applications. Special monitoring programs kill processes that eat up too much memory, processor time, or just hang. Instead of worries, just restart the system. Of course, you will have to read the logs to understand the reason for the fall. But such an approach will allow you to keep calm and reason.


Lesson 2: Sharing Services


The essence of the lesson is to group similar processes and data on different machines.

Performing too many actions on one machine will require a lot of context switching for tasks. Try to make each of your servers serve a particular data type in a certain way. This means that all your indexes will fall into the cache, and will not fall out of it. Keep similar data as close as possible to each other. Do not use Python threads. They slow down. Reddit developers divided everything into several services. Services include spam checking, image processing and query caching. This makes it easy to smash them into multiple machines. Thus, they solved the problem of process interactions. And when this problem is solved, the architecture becomes cleaner and it becomes easier to grow.


Lesson 3: Open Database Schema


The essence of the lesson - do not worry about the scheme.

Reddit developers spent a lot of time worrying about the structure of the data and its normalization. Schema changes became more and more expensive as they grew. Adding one column to 10 million rows required a bunch of locks, and it worked very poorly. Reddit used replication for scaling and backup. Updating the schema and supporting replication is hard work. Sometimes you had to restart replication, and the backup might not work all day. Releases also caused problems, because it was necessary to ensure that the software update and the update of the database schema passed simultaneously.


Instead, they use the Entity Table and the Data Table. In Reddit-e, everything is an entity - users, links, comments, sub-forums, awards, etc. Entities share several common attributes, for example votes for / against, type, date of creation. The data table contains 3 columns - thing id, key, value. A line for each attribute. There is a line for the title, for the link, the author, votes for spam, etc. Now, when adding a new feature, they don’t have to worry about the structure of the database. No need to add new tables for new entities or worry about updates. Easier to develop and maintain. Price - the inability to use cool relational features. You cannot use join s and you must manually ensure data integrity. You don’t have to worry about foreign keys when joining, or how to share data. In general, it turned out pretty well. Concerns about relational databases are a thing of the past.



Lesson 4: Avoid Storing Statees


The goal of each server is simple - to process a request of any type. And while Reddit was growing, they had to use more and more machines, and it was no longer possible to rely on the application server cache. Initially, they duplicated the state of each application server, which was a waste of RAM. They could not use memcached, because they kept in memory a huge amount of small data, which did not give an increase in speed. They rewrote a service to support memcache and no longer store state in application servers. Even if application servers crash, life does not stop. And scaling is now just a matter of adding new servers.


Lesson 5: Memcache


The bottom line is to use memcache for everything.

Reddit uses memcache for:
  1. Database data
  2. Session Data
  3. Pages generated
  4. Memoization (caching previously calculated results) of internal functions
  5. Limits on the number of requests
  6. Storing pre-calculated lists and pages
  7. Global locks




Now they store all the data in Memcachedb, not Postgres. This is an analogue of memcache, but saves data to disk. Very quick fix. All requests are generated by the same piece of code and are cached in memcache. Changed passwords, links are cached for 20 minutes or so. The same can be said for captcha. They use the same approach for links that do not want to be stored forever.

They built memoization into their framework. The calculated results are also cached: normalized pages, lists, and everything else.

They limit the number of requests using memcache + the limited lifetime of the request. A good way to protect your system from attacks. Without such a subsystem, any unscrupulous user could put systems down. Do not feng shui. Therefore, for users and indexers, they store data in memcache. If the user makes a repeated call in less than a second, he is kicked off. Ordinary users don’t click so fast, so they don’t notice anything. The Google indexer will process you as often as you allow, so when everything starts to slow down, just limit the frequency of requests. And the system will begin to work more calmly without creating inconvenience for users.

Everything in Reddit is a list. Home page, private messages, comment page. All of them are calculated in advance and placed in the cache. When you get a list, it is taken from the cache. Each link and each comment is likely stored in 100 different variations. For example, a link with 2 votes created 30 seconds ago is rendered and saved separately. After 30 seconds, it renders again. Etc. Every piece of HTML comes from the cache. Therefore, CPU time is not wasted on rendering pages. If everything starts to slow down, just add more cache.

When they come to work with their non-greasy database, they use memcache as a global lock. It works, although this is not the best option.


Lesson 6: Save Redundant Data


The essence of the lesson: to achieve maximum speed - calculate everything in advance and cache the result.

A guaranteed way to make a slow website is to have a normalized database, access it on demand, and then generate HTML. It takes forever even for one request. Therefore, if you have data that can be displayed in several formats, links on the main page, in incoming messages, profiles, store all these views separately. But when someone comes in and accesses the data, they are ready.

Each list can have 15 different sort orders (hot, new, best, old, current week). When someone adds a new link, they recount all the lists that this may concern. At first glance, this looks too wasteful, but it is better to spend money first than to slow down. Wasting disk space and memory is better than keeping the user waiting.


Lesson 7: Get the most out of your work in the background

The essence of the lesson: do a minimum on the server side, and tell the user that everything is ready.

If you need to do something, do it when the user is not waiting for you. Queue the job. When a user votes on Reddit, lists, karma of the user and many other things are updated. Therefore, when voting, the database is updated to mark the fact of voting. After that, the task is queued, and the task knows all 20 things that need to be updated. When the user returns, all necessary data will be cached.


What Reddit does in the background:

  1. Cache Lists
  2. Process pictures
  3. Discover cheaters
  4. Remove spam
  5. Rewards are calculated
  6. Update Search Index


There is no need to do these things while the user is waiting for you. For example, when Reddit became popular, attempts to increase the rating became more frequent. Therefore, they spend a lot of server time to detect such situations. The work of this service in the background does not prevent users from voting further.


The following is an architecture diagram:


Blue arrows indicate what happens when a request is received. Let's say someone leaves a link or votes. This data goes to the cache, the main database, and the job queue. The result is returned to the user. Other things that happen offline are represented by yellow arrows. Services like Spam, Precomputer, and Thumnailer receive jobs from the queue, execute them, and update the database. The main highlight of the technological solution is RabbitMQ.

PS It is very interesting to compare the lessons of Reddit with the practice of designing applications on Erlang. Let it fail, Supervision tree, mnesia. Russian translation of documentation for OTP can be found here .

PPS There is an idea to translate a couple of chapters from RabbitMQ in Action. How interesting would that be?

Also popular now: