Using Linux in a SAN network. Little trick

Good day to all.
I do not think that the situation described by me is typical of most, but nevertheless I think it will be useful to know. My story about the method of protection "from a fool" in the case when you have a SAN network, there are several storage systems (storage system) that do not support the technology of restricting access to certain LUNs (Logical Unit Number) and Linux-based servers (in our In case all actions will be carried out using OS SLES 10SP1).


Background


In 2008, I had to put into operation the computing infrastructure of one customer. A small SAN network was built with a certain number of storeys (about 10 pieces of a different class). Most of these stories were intended to build the file system of the computing cluster, so initially the problem was not obvious and did not even think about.

And the problem is that for some infrastructure subsystems a hundredth was allocated that does not know how to restrict access to certain LUNs from certain hosts (for each manufacturer it is called differently Port Masking, LUN MAPPING, etc.), it’s more accurate to say do it, but additional purchase is required. license, but this was not done. And here the problem arose - all the servers that connect to this story see all its sections. It seems that this is not a big deal, but you need to protect the customer from the possibility of formatting someone else’s partition, because The data of each subsystem is very important.

Total we have:
  • LSI storage system (more precisely one of its partners)
  • Servers based on SLES10 SP1 with two-headed FC adapter
  • A huge desire to hedge ourselves and the customer from mistakes.


Step One, Introductory


I’ll immediately make a reservation for those who do not have a strong knowledge of the topic:
Each device in the SAN network has a unique WWN (World Wide Name) address, it is some analogue to the MAC address of the network adapter, but, like in the case of Ethernet, the switch does not know what is inside the packet coming to the switch, therefore it is impossible to make a restriction on the switch (zoning), you can only limit the scope of the Data Storage Systems for certain nodes, which was done.

The range of solutions was limited to the following options:
  1. The presence in the driver of the ability to use only certain LUNs
  2. Use OS tools to block unnecessary LUNs


First option

He fell away after looking at the information on this module. It contains only the option to scan the device and determines the maximum number of LUNs. So this option disappears

Second option

This option immediately directs us to "carry out land work" in the direction of the udev subsystem. It is she who is responsible in linux for initializing devices, assigning them specialized names and other actions. Simply put, we need to correctly describe the rule for udev, which will create only the devices we need.
Looking ahead, I’ll say that this decision will limit only the creation of specials. devices (like / dev / sdb, / dev / sdc, etc.), but physically the host will see all the LUNs available to it.

Step Two, Research



We proceed to a detailed study of the topic.

After a quick study of the documentation, as well as those rules that are in the system, you can isolate the following blocks that are of interest to us:

  1. Block "SUBSYSTEM", which defines the subsystem. Subsystems are of three types: “block” (block devices), “char” (character devices) and “pipe” (FIFO devices).
    man mknod
    We are interested in the "block" subsystem, because wheels belong to this class.

  2. udev can view device attributes. They usually store various data, such as:
    • type of manufacturer
    • model name
    • various hardware data.


    To find out the parameters we need, we need:
    The lssci utility - displays all the SCSI devices that are available on the system, as well as the name of the spec. the device that is attached to this SCSI device
    Utility udevinfo - shows all the necessary data for the requested device.

    Let's get started:

    First, we need to display a list of all devices that are available on this server: The output of this command will be large, so I will give only a part. So now we know:

    # lssci
    SCSI ID TYPE VENDOR MODEL REV. device
    [5:0:0:10] disk XXX YYYYY 0619 /dev/sdr
    [5:0:0:20] disk XXX YYYYY 0619 /dev/sds
    [5:0:0:21] disk XXX YYYYY 0619 /dev/sdu
    ......
    [5:0:1:10] disk XXX YYYYY 0619 /dev/sdan
    [5:0:1:20] disk XXX YYYYY 0619 /dev/sdao
    [5:0:1:21] disk XXX YYYYY 0619 /dev/sdar




    SCSI ID is an identifier in the format H: B: T: L (Host: Bus: Target: Lun) Host is the serial number of the adapter that is responsible for this device, Bus (Channel) is the SCSI channel number on the adapter, Target is the number FC port on the store, LUN - identifier of the section allocated on the store.
    VENDOR - the name of the manufacturer of the brand;
    MODEL - the name of the model
    REV. - FirmWare version on the
    device device - special. device accessible to the user

    In this operation, we were interested in the field “device” and “SCSI ID”. The remaining fields are ignored.

  3. The next step is to get a list of storage attributes that we need:
    • driver name
    • manufacturer name
    • model name


    To do this, we will need the udevinfo utility, as well as the sysfs file system (for kernel versions 2.6), because it contains all the information we need. It is not necessary to request information for each device, as lsscsi output contains duplicate entries. These are paths to a specific LUN through different devices (for example, 1 FC port on the host and 3 Port on the storage), and since this is the same storage, the parameters we need will be the same for everyone.

    # udevinfo -a -p /sys/block/sds

    looking at device '/block/sds':
    KERNEL=="sds"
    SUBSYSTEM=="block"
    SYSFS{stat}==" 151707 2355 3422965 430068 436954 12273 18585533 1098588 0 651944 1528756"
    SYSFS{size}=="2129838080"
    SYSFS{removable}=="0"
    SYSFS{range}=="16"
    SYSFS{dev}=="65:32"

    looking at device '/devices/pci0000:00/0000:00:04.0/0000:08:00.0/host5/rport-5:0-0/target5:0:0/5:0:0:20':
    ID=="5:0:0:20"
    BUS=="scsi"
    DRIVER=="sd"
    SYSFS{ioerr_cnt}=="0x7"
    SYSFS{iodone_cnt}=="0xec62d"
    SYSFS{iorequest_cnt}=="0xec62d"
    SYSFS{iocounterbits}=="32"
    SYSFS{retries}=="5"
    SYSFS{timeout}=="60"
    SYSFS{state}=="running"
    SYSFS{rev}=="0619"
    SYSFS{model}=="YYYYY "
    SYSFS{vendor}=="XXX "
    SYSFS{scsi_level}=="6"
    SYSFS{type}=="0"
    SYSFS{queue_type}=="simple"
    SYSFS{queue_depth}=="64"
    SYSFS{device_blocked}=="0"
    ....
    looking at device '/devices/pci0000:00/0000:00:04.0/0000:08:00.0':
    ID=="0000:08:00.0"
    BUS=="pci"
    DRIVER=="mptfc"
    SYSFS{modalias}=="pci:v00001000d00000646sv00001000sd00001020bc0Csc04i00"
    SYSFS{local_cpus}=="00000000,00000000,00000000,0000000f"
    SYSFS{irq}=="169"
    SYSFS{class}=="0x0c0400"
    SYSFS{subsystem_device}=="0x1020"
    SYSFS{subsystem_vendor}=="0x1000"
    SYSFS{device}=="0x0646"
    SYSFS{vendor}=="0x1000"





Short summary:

  1. DRIVER == “sd” is the name of the driver that identifies this device. Why exactly “sd” and not the FC “mptfc” driver - because exactly how the “sd” driver is responsible for determining the disk device received from SCSI / FC adapters.
  2. SYSFS {vendor} == “XXX” is the repository manufacturer. If you pay attention, then his name is not just “XXX”, but also a number of spaces. For this reason, the output of lsscsi is not so informative, and if you use only “XXX”, the rule simply won’t work.
  3. SYSFS {model} == "YYYYY" - the name of the model. The same reason as in the paragraph above.
  4. ID == “5: 0: 0: 20” - an identifier in the format H: B: T: L. It also needs to be remembered, because we need it further.
  5. BUS == "scsi" - type of bus. We will need to more accurately identify the device


So, now we have all the necessary information in order to write a rule.

Step Three, the office writes!



Now we proceed directly to writing the rule. We will need udev documentation.
I give the final rule and the description of the options: And now the explanation is what's what: This option tells udev that if the subsystem works with any non-block device, then this rule should not be applied. This is the most important part, in fact, for the sake of this part, this article was written. The rule sets the behavior: if a device gets through the BUS == “scsi” bus and it uses the DRIVER == “sd” driver, we also have the manufacturer SYSFS {vendor} == “XXX” and the model name is SYSFS {model} == “ YYYYY ”and the LUN of this device corresponds to ID ==" *: *: *: 20 ", then this device must follow all other rules and create a special. device in the / dev directory.

# If not a block device (ex. tape) ignore this rule
SUBSYSTEM!="block", GOTO="go_ignore_end"
# Phisical device section
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:20", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:21", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", OPTIONS+="ignore_device,last_rule"

LABEL="go_ignore_end"




SUBSYSTEM!="block", GOTO="go_ignore_end"


BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:20", GOTO="go_ignore_end"
BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", ID=="*:*:*:21", GOTO="go_ignore_end"




The record ID == "*: *: *: 20" is a record in the format H: B: T: L, which describes only the LUN identifier, the rest of the fields can be any. This allows us to skip all devices from storage that have a given LUN. If you need to expand the list of LUNs, then records are simply added and the identifier we need is written in the last field. The main thing is that they are placed before the next rule. This rule applies the same conditions as the previous one, only OPTIONS + = “ignore_device, last_rule” tells udev to ignore all further actions with this device. Those. all devices except the specified ones will be ignored. This rule is a simple end-of-file mark. It should be the last on the list.

BUS=="scsi", DRIVER=="sd", SYSFS{vendor}=="XXX ", SYSFS{model}=="YYYYY ", OPTIONS+="ignore_device,last_rule"




LABEL="go_ignore_end"


Step Four, the final battle


Now we just have to save this rule and apply it.
We save this rule in the /etc/udev/rules.d folder in the correct format. I used the name "52-ignore-unused-devices.rules". After this, something else needs to be done. If the adapter driver is in the ramdisk, then you need to remove it from there and load it only after the udev subsystem starts, otherwise this rule will not work, because devices will be created before udev starts.

To do this, you need to make changes to the / etc / sysconfig / kernel file:

  1. Delete the driver FC device in the variable INITRD_MODULES = "...."
  2. Add it to the variable MODULES_LOADED_ON_BOOT = "..."
  3. Regenerate the ramdisk with the mkinitrd command.


Now you can restart the server and make sure that only those devices that we need are created. The rest, although visible in the system, are not specified. files, and therefore the user can not do anything with them. It will look like this: As a result, our task was solved elegantly and beautifully. It will also speed up loading, especially if you use the built-in multipath mechanism. I hope this will be useful for at least one person. Materials that helped a lot

# lssci

SCSI ID TYPE VENDOR MODEL REV. device
[5:0:0:10] disk XXX YYYYY 0619 -
[5:0:0:20] disk XXX YYYYY 0619 /dev/sds
[5:0:0:21] disk XXX YYYYY 0619 /dev/sdu
......
[5:0:1:10] disk XXX YYYYY 0619 -
[5:0:1:20] disk XXX YYYYY 0619 /dev/sdao
[5:0:1:21] disk XXX YYYYY 0619 /dev/sdar








Also popular now: