June 21, 2011 at 20:00shared hotspare for mdadm
(I doubted writing here or in system administration)
I found that the Internet has very little (and not very intelligible) explained how mdadm works with shared (global) hot-swap disks. In a note, I will describe what it is and explain why shared hotspare are not marked as general in / proc / mdstat, but instead look like they are quite local.
(I’m writing not for beginners, so at a gallop across Europe)
If an array has redundancy and one of its disks fails, then it is possible to restore redundant information to a backup disk. If the disk is added to the array by hand (the administrator received a letter about the failure, he read the letter, woke up / got dressed, arrived at work, took out the failed disk, inserted the spare one, added it to the array, gave the command to restore redundancy), then such a disk is called cold- spare. Just a "spare disk".
If the server has an idle disk on which redundancy is restored immediately after the failure of any of the array disks, then this disk is called hot-spare. The main advantage is that it will rebuild (restore redundancy) even if the admin email missed or did not manage to arrive on time.
This is actually logical - if we have a choice - use hot-spare to restore the redundancy of the system partition or the data partition, then we need to restore the redundancy of the data partition. And if the system partition is already “occupied” by hot-spare, then there will be a trouble. Moreover, some manufacturers offer 1EE hotspare, in which the backup disk is also used for data storage (the empty space is “spread” between the disks of the array, providing the ability to quickly rebuild and increasing performance in normal mode).
It was an introduction. Now let's get to the bottom line.
In other words, for this to work, mdadm must be started in -F (follow) mode. He usually sends messages to the mail about the problems of the raid. Most modern distributions run it (if there are arrays), but it’s important to understand that it only services arrays that were assembled from mdadm.conf and not assembled by handles. (Yes, yes, here we are waiting for the setup).
As it is easy to understand from the above about mdadm / linux md, in / proc / mdstat there is no and there can be nothing about spare-group. Because these are mdadm’s personal thoughts and thoughts, and the kernel about it is neither a dream nor a spirit (files in / proc are created by kernel modules ...).
Thus, it is possible to provide shared hot-spare only with mdadm. There are two options: if the group is specified for the array going to load (/etc/mdadm/mdadm.conf), then you can specify hot-spare, something like this:
(I immediately answer the question of where so many smart words to take -
written in comparison with the output of mdadm here only spare-group. Please note - in the second array there is NO hot-spare, however, because If a group is specified, then in case of failure a disk from another array with the same group will be used. In our case, it will be / dev / md1.
Of course, all this will happen only if we have mdadm running in -F mode. In debian, it looks like this in ps output:
There may be several groups on the same system.
By the way, there is a minor muck: when you call mdadm with --detail, there will be no mention of spare-groups, you will need to add them yourself.
And here, alas, yok. As far as I know, mdadm does not support both local (which will belong to only one array) and general hotspare at the same time. If there are two arrays with one spare-group, then all hot-spare from one array can be used for the benefit of another.
The scenario is not as rare as it seems. Here is a simple topology:
SYS_ARRAY
DATA_ARRAY
2 hot-spare
It would be logical to make one hot-spare belong only to DATA_ARRAY, and make the second common so that it is used both as a reserve for SYS_ARRAY and as a "second reserve level" for DATA_ARRAY.
Alas, alas, alas, this is not (if they dissuade me in the comments, I will be very happy).
I found that the Internet has very little (and not very intelligible) explained how mdadm works with shared (global) hot-swap disks. In a note, I will describe what it is and explain why shared hotspare are not marked as general in / proc / mdstat, but instead look like they are quite local.
What is hot-spare?
(I’m writing not for beginners, so at a gallop across Europe)
If an array has redundancy and one of its disks fails, then it is possible to restore redundant information to a backup disk. If the disk is added to the array by hand (the administrator received a letter about the failure, he read the letter, woke up / got dressed, arrived at work, took out the failed disk, inserted the spare one, added it to the array, gave the command to restore redundancy), then such a disk is called cold- spare. Just a "spare disk".
If the server has an idle disk on which redundancy is restored immediately after the failure of any of the array disks, then this disk is called hot-spare. The main advantage is that it will rebuild (restore redundancy) even if the admin email missed or did not manage to arrive on time.
Local hot-spare
Usually a spare disk is added for the array, that is, if the array fails, then its spare disk is used. If a failure occurs in a neighboring array, then hot-spare from a “foreign” array is not used.This is actually logical - if we have a choice - use hot-spare to restore the redundancy of the system partition or the data partition, then we need to restore the redundancy of the data partition. And if the system partition is already “occupied” by hot-spare, then there will be a trouble. Moreover, some manufacturers offer 1EE hotspare, in which the backup disk is also used for data storage (the empty space is “spread” between the disks of the array, providing the ability to quickly rebuild and increasing performance in normal mode).
Global (General) hot-spare
However, it happens that there are many arrays of data. And they all need hot-spare drives. But I feel sorry for the drives. And then there is a desire to have a “common” disk, which can be used for any of the arrays (or even better, 2-3 such disks).It was an introduction. Now let's get to the bottom line.
linux md
mdadm (kernel module in the DM stack) does not support shared hot-spare. A disk can only be added as hot-spare to a specific array.But mdadm does support!
Exactly. mdadm supports, the kernel module does not. Mdadm implements a common hot-spare method of "throw hotspare from one array to another, damaged".In other words, for this to work, mdadm must be started in -F (follow) mode. He usually sends messages to the mail about the problems of the raid. Most modern distributions run it (if there are arrays), but it’s important to understand that it only services arrays that were assembled from mdadm.conf and not assembled by handles. (Yes, yes, here we are waiting for the setup).
spare-group
For the ability to distribute disks between different arrays, there is a concept of spare-group, that is, a group within which a drive can be thrown. There can be many such groups - and hot-spare can only be transferred between them.As it is easy to understand from the above about mdadm / linux md, in / proc / mdstat there is no and there can be nothing about spare-group. Because these are mdadm’s personal thoughts and thoughts, and the kernel about it is neither a dream nor a spirit (files in / proc are created by kernel modules ...).
Thus, it is possible to provide shared hot-spare only with mdadm. There are two options: if the group is specified for the array going to load (/etc/mdadm/mdadm.conf), then you can specify hot-spare, something like this:
ARRAY / dev / md1 level = raid1 num-devices = 2 metadata = 1.2 spares = 1 spare-group = myhostparegroupname name = server: 1 UUID = 18219495: 03fda335: 3f1ad1ee: a5f5cd44
devices = / dev / sda, / dev / sdb, / dev / sdc
ARRAY / dev / md2 level = raid1 num-devices = 2 metadata = 1.2 spare-group = myhostparegroupname name = server: 2 UUID = 18219495: 03fda335: 3f1ad1ee: a5f5cd45
devices = / dev / sdd, / dev / sde
(I immediately answer the question of where so many smart words to take -
mdadm --detail --scan --verbose) It’s written in comparison with the output of mdadm here only spare-group. Please note - in the second array there is NO hot-spare, however, because If a group is specified, then in case of failure a disk from another array with the same group will be used. In our case, it will be / dev / md1.
Of course, all this will happen only if we have mdadm running in -F mode. In debian, it looks like this in ps output:
/ sbin / mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
There may be several groups on the same system.
By the way, there is a minor muck: when you call mdadm with --detail, there will be no mention of spare-groups, you will need to add them yourself.
Local && global hotspare
And here, alas, yok. As far as I know, mdadm does not support both local (which will belong to only one array) and general hotspare at the same time. If there are two arrays with one spare-group, then all hot-spare from one array can be used for the benefit of another.
The scenario is not as rare as it seems. Here is a simple topology:
SYS_ARRAY
DATA_ARRAY
2 hot-spare
It would be logical to make one hot-spare belong only to DATA_ARRAY, and make the second common so that it is used both as a reserve for SYS_ARRAY and as a "second reserve level" for DATA_ARRAY.
Alas, alas, alas, this is not (if they dissuade me in the comments, I will be very happy).
