How not to restore data, or so you are so lucky

We all periodically encounter storage device failures. Hundreds of instructions have been written on the Internet how to read everything that is possible without special equipment from devices still responding to normal OS requests. But for many years I was unlucky, the disks either died completely, completely, or the file system was still available and I just read everything that was read in normal mode. And he was waiting. It had to happen that a dying disk came to me precisely in a state requiring more than the most basic actions?

And now this day has come. On a section on top of the software raid0 , which, frankly, kept all the nonsense, the files were unavailable, and the reason was the long-awaited I / O Error. It was unexpected that the carrier device of the second half of the reid did not appear at all in the fdisk outputalthough a list of files was available. Reboot - and the array is assembled, and the files are read perfectly. The SMART report shows a huge number of reassigned sectors. Just what you need. I turn off the drives and begin to think through a recovery strategy.

To read the surviving sectors, it was decided to use GNU ddrescue , a specialized version of dd for reading data from dying disks, which allows you to choose a reading strategy and can build a disk map to continue working after failures, and they will definitely work, because I work with a faulty device.

So, I connect the first disk to my desktop, and POST solemnly reports that one of the connected disks is really bad. The OS booted normally, except for a couple of messages about some ata device errors. After successfully loading fdisk -l, the patient sees, well, finally I will restore the integrity of the raid0 data .

ddrescue -ndAvv /dev/sdd1 bad mapbad

The process went quite briskly, 80 MB / s. Due to the noise of the fan cooling the disks, which was specially removed from the decommissioned server for such an occasion, the chilling sounds of reading the dying disk would not be heard, even if they were. Perhaps not hearing them is even better. The less you know the better you sleep.

By morning, the work of ddrescue was completed, and on the new drive lay the image and map of a dying comrade. But restarting ddrescue for some reason ended suspiciously quickly, and the card


“Something obviously went wrong,” suggested a sleepy inner voice. "And try fdisk -l " - and the mind is already connected. / dev / sdd was not in the output. “How is this even possible?” “Oh yes, the same thing was on the server,” it means, not everything was still lost: approximately such thoughts were in my head all day.

In the evening, the experimental disk again showed signs of life, and I had a difficult task: what to do with the image, because ddrescue with a finalized map refused to read anything, leaving me with an image file smaller than the size of the partition.
On the one hand, ddrescue can fill in -blocks with arbitrary data, then to determine the broken files. On the other hand, I didn’t ask him to create a spars file ( -S parameter ), and intensified googling of the question of how exactly the smaller-than-section of the image file will be filled out, remained unanswered.


I really did not want to rely on the case, and I decided to study the disk map with a calculator in my hands. It turned out that 96% of the contents of the disk were successfully read and fell into the first + block. But unfortunately, the sum of all + blocks did not give the size of the read image, which completely confused me with all the cards. That’s why I went the other way: since I have a good start and the devil knows what at the end, why not get the good old dd and create exactly what you can work with from an incomprehensible image?

dd if=bad of=bad_new bs=16M count=42751
dd if=/dev/zero of=bad_new bs=1024 seek=700432384 count=32139617

The result is a brand new image, which is now exactly the same size as the dying partition, namely 750153729024 bytes. Now let's get to the zeroes at the end. I do it manually


and run:

ddrescue -ndAvv /dev/sdd1 bad_new mapbad

Hurrah! Now all is well. The read blocks from the disk moved to an empty place of the new image at the desired addresses, and the map was finalized again, and my adventures were coming to an end. But maybe make the last leap and re - read - blocks in one sector? Now give ddrescue


and run like this:

ddrescue -ndAvv /dev/sdd1 -с1 bad_new mapbad

In general, he did not behave at all as I expected. Having stumbled upon a bad sector in the - block, he went on to the next - block until he caught hold of a confident reading from the last - block. An hour later, I stopped this mess using CTRL-C and started reading at normal speed. A few gigabytes from the end of the section were read until ddrescue again ran into bad sectors, which for some reason again led to the disappearance of / dev / sdd from the system and the finalization of the card, which was very swollen and was a mess of + , - and ?blocks. It didn’t make any sense to sort it all out anymore, and I decided to stop torturing the dying man and finally start assembling the array. But before that I wanted to find out what would happen if I used the fill mode. I prepare the placeholder and turn on the filling mode, not forgetting to replace the card with the original one:

printf "BADSECTOR" > tmpfile
ddrescue --fill-mode=- tmpfile bad mapbad

And I get another image of the right size! How he did this was a mystery to me, because I several times counted the sum of sizes + blocks from the card, and it did not match the size of the finished file.

And now, in the place of the one who gives the magnetic soul to God, his healthy brother is already, and I carry out the commands:

losetup /dev/loop1 bad_new 
mdadm --assemble -o /dev/md12 /dev/loop1 /dev/sdd1

But the result, to put it mildly, is not what I expected after 24 hours of hope of success:

mdadm: no RAID superblock on /dev/loop1
mdadm: /dev/loop1 has no superblock - assembly aborted

What to do, now let's try to talk with Google and sections:

#mdadm --examine /dev/loop1
mdadm: No md superblock detected on /dev/loop1.

- That is, how is it no md superblock detected ?
- And this one, who is still healthy?

#mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 64d38efd:b8e92c8c:f5292846:21864477
  Creation Time : Fri Oct 10 20:22:23 2008
     Raid Level : raid0
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Update Time : Fri Oct 10 20:22:23 2008
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 6f768a67 - correct
         Events : 1
     Chunk Size : 4K

- Version 0.90.00? And where should the mdadm 0.90.00 superblock be ?
- At the end of the section.
- And how to read it using mdadm ?
- No way.
- That is, how can it be? Well, so where is it more precise there?
- No further than 128K but no closer than 64K from the end of the section, aligned at the border of 64K, 4K size.

N-yes, I take dd and check:

dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128
dd if=last128 of=bad_new bs=1024 seek=732571873 count=128

Yes, the superblock recorded at the end of the section at the correct address, but it still does not want to assemble an array with two identical superblocks, so you need to give it that superblock from the dying one.

Replacing the drive again, if only I read:

#dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128
dd: reading `/dev/sdd1': Input/output error
0+0 records in
0+0 records out

“Yes, the Great Combinator has never experienced such a blow,” for some reason it was this immortal quote that surfaced in my memory.
Well, we don’t give up so easily. And here we are:

dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 conv=noerror

Oh, last128 size > 0 but <128K. And where is that? How to work with it? And how to make tac toe instead of the killed sectors? And like this:

dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 conv=noerror,sync

Here they are, the last 128K. I register them at the end of the image, reboot with a healthy disk, and the array is assembled!
There is a simple command left and:

#mount -o ro /dev/md11 /srv/old/
mount: wrong fs type, bad option, bad superblock on /dev/md11,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Well, this is somehow completely offensive. The last time the array was built on the server and ext3 was mounted. The output of dumpe2fs is not suspicious, the status of the clean section , the addresses of alternative superblocks are visible, but for some reason it is not mounted with them either. It would be necessary to treat it, but in this form and with half the data without backup it is impossible to treat. Eh, you have to throw another terabyte. Of course, in a good way it would be necessary to make an image from the now conditionally working disk using ddrescue , but all this is already very tired and therefore I do this:

dd if=/dev/md11 of=ext3 bs=16M conv=noerror,sync

And sleep, and in the morning I try:

losetup /dev/loop2 ext3 
mount -o ro /dev/loop2 /srv/old/

And it is mounted! The directory structure at first glance has not changed, and the files from which I have hashes were read without errors. Only 20 GB is free on the restored partition, so its loop-file received a well-deserved attribute:

chattr +i ext3

and write to fstab:

/srv/ext3	/srv/old	ext3		ro,loop	0	0

As a conclusion, I have read to here I recommend:

• set up automatic backup of important and not so data,
• periodically view SMART reports,
• use a USB to SATA device or similar to your drive for recovery,
• treat any disk from which you want to take an image, like a dying person and use GNU ddrescue ,
• remember that if it comes to recovering really important data, then there will definitely not be such luck.