Configuring SCST Target on CentOS 7 Using FC QLogic Adapter
Background
In my infrastructure, among other things, there are three ESXi 5.5 servers connected to the HP MSA1000 SAN (a piece of iron already around 10 years old) via optics through the HP StorageWorks 2 / 8q switch, using QLogic QLA200 adapters. Of course, the storage is very productive, but also very modest in volume. Since virtualization is now a trend, there was an irresistible desire to get a really large storage, also connected via optics with block access. I won’t explain why it’s blocky - holivars on this topic began on the Noah's Ark, and this does not apply to the subject of this article. Well, as it always happens, the budget doesn’t even allow you to buy a brand-name solution, so you can only become the hero of the program “Crazy hands”.
As a result of a not very long study, the SCST target was chosen for the following reasons:
- High performance
- Stability
- Ability to install on Centos (our standard for Linux servers)
- Support for QLogic FC adapters
- VAAI Support
Next, you had to pick up the hardware. In the bins, a 2U server with 6 baskets was found, 4 3TB drives (inferior, two of them WD Green), QLogic 2460 FC adapter ( SCST supports QLogic 22xx / 23xx / 24xx / 25xx / 26xx adapters ).
The search for ready-made manuals for the Centos 7 + FC SCST bundle did not lead to anything, so I had to collect information piece by piece for quite some time. As a result, everything turned out and I am satisfied, ready to purchase new discs. While
Action
We carry out the installation and basic configuration of Centos 7 in the Minimal assembly (this is my preference, you can install any assembly). Everyone has their own “basic setting,” so I won’t go into details. Do not forget to update:
[root@localhost ~]# yum update -y
[root@localhost ~]# reboot
Find out the name of our FC adapter, as well as its RISC ID:
[root@localhost ~]# dmesg | grep -E 'qla2xxx .* QLogic | Found an ISP'
We get something like this:
[ 1.881084] qla2xxx [0000:05:04.0]-001d: Found an ISP2422 irq 24 iobase 0xffffc90001e06000.
[ 2.901460] qla2xxx [0000:05:04.0]-00fb:1: QLogic QLA2460 - PCI-X 2.0 Single Channel 4Gb Fibre Channel HBA.
Where the RISC ID is ISP2422 and the model is QLogic QLA2460.
Next, you need to download and install the QConvergeConsoleCLI utility and the latest firmware for our adapter from here . We extract the .BIN firmware file, rename it in accordance with the RISC ID (see table below), and put it in / lib / firmware on the server. Most likely there will already be such a file, so just replace the existing one.
RISC ID - Firmware
ISP 21XX - ql2100_fw.bin
ISP 22XX - ql2200_fw.bin
ISP 2300 -
ql2300_fw.bin
ISP 24XX - ql2400_fw.bin
ISP 25XX - ql2500_fw.bin
-ql2500_fw.bin
26 ql2500_fw.bin .bin
In my case - ql2400_fw.bin We update the
firmware of our adapter:
[root@localhost ~]# /opt/QLogic_Corporation/QConvergeConsoleCLI/qaucli -fc -b all /lib/firmware/ql2400_fw.bin
Download the current QLogic SCST driver:
[root@localhost ~]# yum install git -y
[root@localhost ~]# git clone git://git.QLogic.com/scst-qla2xxx.git
Install the necessary packages for the subsequent assembly of the kernel, as well as svn:
[root@localhost ~]#yum install gcc ncurses-devel kernel-devel lsscsi patch subversion bc bison net-tools -y
Download SCST sources:
[root@localhost ~]# cd /
[root@localhost /]# svn co https://svn.code.sf.net/p/scst/svn/trunk scst
In the SCST directory, we make a soft link to the QLogic SCST driver:
[root@localhost /]# cd scst
[root@localhost scst]# ln -s ../scst-qla2xxx/drivers/scsi/qla2xxx qla2x00t_git
Further, in accordance with the recommendations of the SCST developers, you should patch the kernel.
We will assemble the new kernel in the new user environment:
[root@localhost /]# useradd builder
We give rights to use yum and yum-builddep without a password:
[root@localhost /]# echo 'builder ALL=(ALL) NOPASSWD: /usr/bin/yum, /usr/bin/yum-builddep' >>/etc/sudoers
[root@localhost /]# visudo –sc
Go to the new user environment:
[root@centos-test /]# su builder
We launch the script, kindly prepared for us by the developers:
[builder@localhost /]# ./scst/scripts/rebuild-rhel-kernel-rpm
Install the new kernel:
[builder@localhost /]$ su -c 'rpm -ivh --force /home/builder/rpmbuild/RPMS/x86_64/kernel-*.rpm'
Reboot:
[builder@localhost /]$ su –c reboot
Check the current kernel:
[root@localhost ~]# uname -r
3.10.0-229.el7.centos.scst.x86_64
If everything is ok, move on. Unload and add the default qla2xxx kernel module to the blacklist:
[root@localhost /]# echo blacklist qla2xxx >/etc/modprobe.d/blacklist-qla2xxx.conf
[root@localhost /]# rmmod qla2xxx
Set the SCST build mode to release:
[root@localhost /]# cd /scst
[root@localhost scst]# make 2release
Let's collect the necessary kernel modules:
[root@localhost scst]# BUILD_2X_MODULE=y CONFIG_SCSI_QLA_FC=y CONFIG_SCSI_QLA2XXX_TARGET=y make all install
We verify that everything went well:
[root@localhost ~]# ls -l /lib/modules/`uname -r`/extra/qla2*
-rw-r--r-- 1 root root 1511833 Sep 10 11:10 /lib/modules/3.10.0-229.11.1.el7.centos.scst.x86_64/extra/qla2x00tgt.ko
-rw-r--r-- 1 root root 10699423 Sep 10 11:10 /lib/modules/3.10.0-229.11.1.el7.centos.scst.x86_64/extra/qla2xxx_scst.ko
You can check the complete list of installed modules:
[root@localhost ~]# ls -l /lib/modules/`uname -r`/extra
total 25128
drwxr-xr-x 2 root root 4096 Sep 10 11:10 dev_handlers
-rw-r--r-- 1 root root 4309043 Sep 10 11:10 iscsi-scst.ko
-rw-r--r-- 1 root root 3100544 Sep 10 11:10 isert-scst.ko
-rw-r--r-- 1 root root 1511833 Sep 10 11:10 qla2x00tgt.ko
-rw-r--r-- 1 root root 10699423 Sep 10 11:10 qla2xxx_scst.ko
-rw-r--r-- 1 root root 5619115 Sep 10 11:10 scst.ko
-rw-r--r-- 1 root root 474173 Sep 10 11:10 scst_local.ko
[root@localhost ~]# ls -l /lib/modules/`uname -r`/extra/dev_handlers
total 3960
-rw-r--r-- 1 root root 305582 Sep 10 11:10 scst_cdrom.ko
-rw-r--r-- 1 root root 292948 Sep 10 11:10 scst_changer.ko
-rw-r--r-- 1 root root 328092 Sep 10 11:10 scst_disk.ko
-rw-r--r-- 1 root root 308934 Sep 10 11:10 scst_modisk.ko
-rw-r--r-- 1 root root 292972 Sep 10 11:10 scst_processor.ko
-rw-r--r-- 1 root root 292876 Sep 10 11:10 scst_raid.ko
-rw-r--r-- 1 root root 312024 Sep 10 11:10 scst_tape.ko
-rw-r--r-- 1 root root 704076 Sep 10 11:10 scst_user.ko
-rw-r--r-- 1 root root 1199482 Sep 10 11:10 scst_vdisk.ko
Add them all to the kernel:
[root@localhost ~]# for m in scst qla2xxx_scst qla2x00tgt scst_vdisk scst_user scst_disk ...; do modprobe $m; done
Something like this should appear in dmesg (I give an example from my version):
[10753.981044] [21513]: scst: SCST version 3.1.0-pre1 loaded successfully (max mem for commands 709MB, per device 283MB)
[10753.981051] [21513]: scst: Enabled features: TRACING
[10753.982152] [21519]: scst: Management thread started
[10754.024379] qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.04.00.05.2.2-SCST.03-k-p.
[10754.030864] [21527]: qla2x00t: Initializing QLogic Fibre Channel HBA Driver target mode addon version 3.1.0-pre1
[10754.032397] [21527]: qla2x00t: Target mode driver for QLogic 2x00 controller registered successfully
[10754.032409] [21527]: scst: Target template qla2x00t registered successfully
[10754.038638] [21528]: scst: Virtual device handler vdisk_fileio for type 0 registered successfully
[10754.038647] [21528]: scst: Virtual device handler vdisk_blockio for type 0 registered successfully
[10754.038653] [21528]: scst: Virtual device handler vdisk_nullio for type 0 registered successfully
[10754.038658] [21528]: scst: Virtual device handler vcdrom for type 5 registered successfully
[10754.044786] [21529]: scst: Virtual device handler "scst_user" registered successfully
[10754.048905] [21530]: scst_user: Cleanup thread started
[10754.052682] [21531]: scst: Device handler "dev_disk" for type 0 registered successfully
[10754.052691] [21531]: scst: Device handler "dev_disk_perf" for type 0 registered successfully
We rebuild the ram-disk for initial initialization (initrd) so that the qla2xxx kernel module is replaced by qla2xxx_scst:
[root@localhost ~]# mkinitrd -f /boot/initramfs-`uname -r`.img `uname -r`
Install scstadmin:
[root@localhost /]# cd scst
[root@localhost scst]# make -C scstadmin -s install
Next, you need to create the file /etc/scst.conf, I will give an example of mine:
HANDLER vdisk_blockio {
DEVICE disk1 {
filename /dev/sdb
}
}
TARGET_DRIVER qla2x00t {
TARGET ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ {
enabled 1
LUN 0 disk1
}
}
Explanation:
HANDLER vdisk_blockio is the choice of handler, in my case I wanted block access, if you want file access, then vdisk_fileio will be.
Filename / dev / sdb - a disk for your target
TARGET ХХ: ХХ: ХХ: ХХ: ХХ: ХХ: ХХ: ХХ - instead of Xs, enter WWN of your FC adapter
Example command with which you can find out WWN:
cat /sys/class/scsi_host/host*/device/fc_host/host*/node_name
We use our configuration file:
[root@localhost scst]# scstadmin -config /etc/scst.conf
Collecting current configuration: done.
-> Checking configuration file '/etc/scst.conf ' for errors.
-> Done, 0 warnings found.
-> Opening device 'disk1' using handler 'vdisk_blockio': done.
-> Adding device 'disk1' at LUN 0 to driver/target 'qla2x00t/ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ': done.
-> Enabling driver/target 'qla2x00t/ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ:ХХ': done.
All done.
Well, and probably we want this whole thing to start automatically if the target reloads.
Turn on autostart of SCST itself:
[root@localhost /]# systemctl enable scst.service
Add kernel modules to startup:
[root@localhost /]# vi /etc/modules-load.d/scst_modules.conf
scst_disk
scst_user
scst_vdisk
scst
libcrc32c
crc_t10dif
qla2x00tgt
And finally, add the application of the config to rc.local:
[root@localhost /]# chmod +x /etc/rc.d/rc.local
[root@localhost /]# vi /etc/rc.d/rc.local
scstadmin –config /etc/scst.conf
If everything is done correctly, then if, after rebooting the server, run:
[root@localhost ~]# scstadmin -config /etc/scst.conf
The following will come out:
Collecting current configuration: done.
-> Checking configuration file '/etc/scst.conf' for errors.
-> Done, 0 warnings found.
-> Applying configuration.
-> Done, 0 change(s) made.
All done.
This completes the target setting. Below are a couple of useful notes.
Important! Do not forget to configure Zoning on your FC switch, if one is used, otherwise the initiators will not see the new target.
Also, in the case of ESXi, after setting the target, as well as in the case of its reboot, you need to reboot all the initiating hosts, otherwise the drive will not pick up.
Still such a small performance note. On Centos 7, using fio, I measured 84 iops and avg latency 12 msec per record with iodepth = 1. Which is actually not bad for WD Green.
Update 10/05/2016! I was going to raise the target according to my own article, but an error came up with a kernel patch: No matching put_page_callback patch found for kernel version.
Developed by SCST saved us from having to patch the kernel in the latest version 3.3. So you can skip the section on rebuilding the kernel, instead, just follow these steps:
After [root @ localhost scst] # ln -s ../scst-qla2xxx/drivers/scsi/qla2xxx qla2x00t_git:
[root @ localhost scst] yum isntall - y bzip2 rpm-build
[root @ localhost scst] make rpm
In the process, an error most likely comes out that is related to the lack of certain Perl modules, you will need to google it and immediately there will be a solution in the first pages, I just made it and forgot to write down - I repent.
After that, install the RPMs and continue from this point:
[root @ localhost /] # echo blacklist qla2xxx> /etc/modprobe.d/blacklist-qla2xxx.conf