Data protection in Evernote: how we get rid of broken disks

In our entry, “Three Data Protection Rules at Evernote,” we already talked about some of the measures we take to protect data so that our users can trust our service. In fact, there are many more such measures, and today I would like to talk about one important point: what do we do when hard drives fail.

You probably had to read stories about people who bought computers from their hands and found various information from the previous owner, with sometimes very confidential data. Therefore, Evernote takes the decommissioning of their old drives very seriously.

In our user data storage infrastructure, we use both hard drives and solid state drives (SSDs). Hard drives are mechanical in nature, and therefore, like all things with moving components, sooner or later break down. SSDs have a different nature of failure: such drives have a limited number of rewrite cycles, after which they become read-only.

We provide data redundancy with hardware RAID controllers. This means that a single drive failure does not affect the safety of your data. In addition, we take preventive measures to identify drives that may fail by tracking data transfer errors and forecasts provided by the drive itself.. If these values ​​reach a critical threshold, we replace the disk without waiting for a breakdown, usually on the same or next business day. Sometimes, disks simply break without warning, and then our task is to replace them as quickly as possible.

As a result, we still have all these broken disks that may contain user data. The ATA instruction set contains the Secure Erase feature , which overwrites each track on a disc, making data recovery almost impossible. All this is great, but for this you need the drive to be operational. And in our case, most of the failed drives no longer function, so this feature does not suit us.

Drives are expensive and usually come with a warranty (usually three years, sometimes even more). Manufacturers usually require customers to return broken drives for replacement as part of the warranty program. But since our disks may contain user data, and we do not have the opportunity to use the same Secure Erase function, we cannot afford to send disks for repair or replacement, and thereby risk user data. Fortunately for such cases, most manufacturers offer specific replacement programs, known as the Black Hole. The specific conditions may vary from company to company, but usually it’s enough for the client to send the front panel of the disk and some form or written statement about the physical destruction of the disk.

In general, our approach to working with broken disks is to destroy them, where we also adhere to the principle of redundancy.

There is such a National Institute of Standards and Technology ( NIST ), a US government agency whose tasks include the development and publication of technology manuals for other US agencies. These guides are available online for free and generally meet industry standards. NIST Publication Nos. 800-88 ( “Media Cleanup Guidelines” ) covers both physical and electronic recording forms. The approach in Evernote is based on this instruction.

Our work with broken disks consists of the following steps:

• Disks sent for destruction are stored in a safe place.
• The front panel of the disks is removed (see photo # 1, # 2 and # 3), which requires several different types of screwdrivers (what would we do without the iFixit sets !).
  
• The disk is placed in the Garner Products HD-2 demagnetizer (photo # 4), where the data is erased securely.
  
• Then the media is physically destroyed using a device that physically crushes the disk with a powerful wedge ( Garner Products PD-4 in photos # 5 and # 6). This makes the disk completely unusable (photo # 7 and # 8).
  
  
  
• The broken parts of the disk are then sent for recycling.
• The front panels (photo # 9) are sent to the respective manufacturers, and the disks that have arrived to replace them come into operation again.
  

The goal of all these operations is to ensure that the security of user data is never compromised when working with disks. This principle, combined with strict adherence to NIST guidelines and other industry standards, ensures that we use proven and reliable methods.