September 9, 2011 at 17:56Configuring iSCSI initiator on linux
Abstract: how open-iscsi (iSCSI initiator on linux) works, how to configure it and a little bit about the iSCSI protocol itself.
Lyrics: There are many articles on the Internet that explain quite well how to configure ISCSI target, however, for some reason, there are practically no articles about working with the initiator. Despite the fact that target is technically more complicated, there is more administrative fuss with initiator - there are more complicated concepts and not very obvious principles of work.
Before telling about ISCSI - a few words about different types of remote access to information in modern networks.
Explorer. At its core, it is 'scsi over tcp'. The SCSI protocol itself (without the letter 'i') is a very complicated design, because it can work through different physical environments (for example, UWSCSI is a parallel bus, SAS is a serial bus - but they have the same protocol). This protocol allows you to do much more than just “stick drives to the computer” (as invented in SATA), for example, it supports device names, the presence of several links between the block device and the consumer, and support for switching (yeah, SAS switch, such even is in nature), connecting several consumers to one block device, etc. In other words, this protocol was simply requested as the basis for a network block device.
target - the one who provides the block device. The closest analogue from the ordinary computer world is a server.
initiator - a client who uses a block device. Customer equivalent.
WWID - a unique identifier for the device, its name. An analogue of a DNS name.
LUN - the number of the "piece" of the disk that is being accessed. The closest analogue is the hard disk partition.
ISCSI brings the following changes: WWID disappears, in its place comes the notion of IQN (iSCSI Qualified Name) - that is, pure name, similar to the degree of confusion with DNS (with slight differences). Here is an IQN example: iqn.2011-09.test: name.
IETD and open-iscsi (server and client for Linux) bring another very important concept, which is most often not written about in the manuals for iscsi - portal. Portal is, roughly speaking, several targets that are announced by a single server. There is no analogy with www, but if the web server could be asked to list all of its virtualhosts, then that would be it. portal indicates a list of targets and available IPs that can be accessed (yes, iscsi supports several routes from initiator to target).
This article is not about target, so I give a very brief description of what target does. He takes a block device, slaps the name and LUN to it, and publishes it on his portal, after which he allows everyone (authorization to taste) to contact him.
Here is an example of a simple configuration file, I think it will be clear from it what the target does (the configuration file using the IET example):
(complex from simple differs only in export options). Thus, if we have target, then we want to connect it. And here the difficult part begins, because the initiator has its own logic, it does not at all look like the trivial mount for nfs.
Open-iscsi is used as the initiator. So, the most important thing is that it has operating modes and status . If we give a command in the wrong mode or do not take into account the state, the result will be extremely discouraging.
So, the operating modes:
From this list, the life cycle is quite understandable - first find, then connect, then disconnect, then reconnect. Open-iscsi keeps the session open, even if the block device is not in use. Moreover, it keeps the session open (up to certain limits, of course), even if the server goes into reboot. The iscsi session is not the same as an open TCP connection, iscsi can transparently reconnect to the target. Disconnection / connection - operations that are controlled "from the outside" (either from other software or by hand).
A bit about the condition. After discovery, open-iscsi remembersall found targets (they are stored in / etc / iscsi /), in other words, discovery is a constant operation, NOT at all appropriate, for example, dns resolving). Found targets can be deleted by hand (by the way, a common mistake is when open-iscsi, as a result of experiments and settings, has a bunch of found targets, which when trying to login into them crawl a lot of errors due to the fact that half of the targets are old lines config, which for a long time do not exist on the server, but remember open-iscsi). Moreover, open-iscsi allows you to change the settings of a remembered target — and this “memory” affects the further work with targets even after rebooting / restarting the daemon.
The second question that torments many in the beginning is where does it get after connecting? open-iscsi creates, even if it is a network, but a BLOCK device of the SCSI class (it is not for nothing that it says “I say”), that is, it receives a letter in the / dev / sd family, for example, / dev / sdc. The first free letter is used, because for the rest of the system, this is a block device - a typical hard drive, no different from being connected via usb-sata or simply directly to sata.
This often causes a panic "how can I find out the name of the block device?" It is output in the verbose output of iscsiadm (# iscsiadm -m session -P 3).
Unlike SAS / UWSCSI, iSCSI is accessible to anyone. To protect against such, there is a username and password (chap), and their transfer to iscsiadm is another headache for novice users. It can be implemented in two ways - by changing the properties of the target already found earlier and setting the username / password in the open-iscsi configuration file.
The reason for such difficulties is that the password and login process are not attributes of the user, but of the system. ISCSI is a cheap version of the FC infrastructure, and the concept of “user” in the context of the person behind the keyboard is not applicable here. If your sql database is on the iscsi block device, then of course you will want the sql server to start itself, and not after a minute of personal attention of the operator.
This is a very important file, because in addition to the login / password, it also describes the behavior of open-iscsi when it finds errors. It may not give a “backward” error immediately, but with a pause (for example, about five minutes, which is enough to reboot the data server). It also controls the login process (how many times to try, how long to wait between attempts) and any subtle tuning of the process itself. Note that these parameters are quite important for work and you need to understand how your iscsi will behave if you remove the power cord for 10-20 s, for example.
I don’t really like quoting easily-found mana and lines, so I’ll give a typical scenario for using iscsi:
first we find the target we need, for this we need to know the initiator IP / dns name:
Another issue that is important in serious decisions is the support of several routes to the source. The beauty of iscsi is the use of a regular ip, which can be processed in the usual way, like any other traffic (although in practice it is usually not routed, but only switched - the load is too great there). So, iscsi supports multipath in the “do not resist” mode. Open-iscsi alone cannot connect to multiple IPs of the same target. If it is connected to several IPs of the same target, this will lead to the appearance of several block devices.
However, there is a solution - this is multipathd, which finds disks with the same identifier and processes them as expected in multipath, with custom policies. This article is not about multipath, so I won’t explain the mystery of the process in detail, however, here are some important points:
Lyrics: There are many articles on the Internet that explain quite well how to configure ISCSI target, however, for some reason, there are practically no articles about working with the initiator. Despite the fact that target is technically more complicated, there is more administrative fuss with initiator - there are more complicated concepts and not very obvious principles of work.
ISCSI
Before telling about ISCSI - a few words about different types of remote access to information in modern networks.
NAS vs SAN
There are two methods of accessing data located on another computer: file (when a file is requested from a remote computer, and which file systems do it, nobody cares), typical representatives of NFS, CIFS (SMB); and block - when blocks are requested from a remote computer from a disk medium (similar to how they are read from a hard disk). In this case, the requesting party creates a file system on the block device, and the server that sends the block device does not know about the file systems on it. The first method is called NAS (network attached storage), and the second is SAN (storage area network). The names generally indicate other signs (SAN means a dedicated network to storage), but it so happened that NAS are files, and SANs are block devices over the network. And although everyone (?) Understandsscsi over tcp
One of the access protocols for block devices is iscsi. The letter 'i' in the name does not refer to Apple products, but to InternetTerminology
The following terms are accepted in the SCSI world:target - the one who provides the block device. The closest analogue from the ordinary computer world is a server.
initiator - a client who uses a block device. Customer equivalent.
WWID - a unique identifier for the device, its name. An analogue of a DNS name.
LUN - the number of the "piece" of the disk that is being accessed. The closest analogue is the hard disk partition.
ISCSI brings the following changes: WWID disappears, in its place comes the notion of IQN (iSCSI Qualified Name) - that is, pure name, similar to the degree of confusion with DNS (with slight differences). Here is an IQN example: iqn.2011-09.test: name.
IETD and open-iscsi (server and client for Linux) bring another very important concept, which is most often not written about in the manuals for iscsi - portal. Portal is, roughly speaking, several targets that are announced by a single server. There is no analogy with www, but if the web server could be asked to list all of its virtualhosts, then that would be it. portal indicates a list of targets and available IPs that can be accessed (yes, iscsi supports several routes from initiator to target).
target
This article is not about target, so I give a very brief description of what target does. He takes a block device, slaps the name and LUN to it, and publishes it on his portal, after which he allows everyone (authorization to taste) to contact him.
Here is an example of a simple configuration file, I think it will be clear from it what the target does (the configuration file using the IET example):
Target iqn.2011-09.example: data
IncomingUser username Pa $$ w0rd
Lun 0 Path = / dev / md1
(complex from simple differs only in export options). Thus, if we have target, then we want to connect it. And here the difficult part begins, because the initiator has its own logic, it does not at all look like the trivial mount for nfs.
Initiator
Open-iscsi is used as the initiator. So, the most important thing is that it has operating modes and status . If we give a command in the wrong mode or do not take into account the state, the result will be extremely discouraging.
So, the operating modes:
- Search for targets (discovery)
- Connect to target
- Work with a connected target
From this list, the life cycle is quite understandable - first find, then connect, then disconnect, then reconnect. Open-iscsi keeps the session open, even if the block device is not in use. Moreover, it keeps the session open (up to certain limits, of course), even if the server goes into reboot. The iscsi session is not the same as an open TCP connection, iscsi can transparently reconnect to the target. Disconnection / connection - operations that are controlled "from the outside" (either from other software or by hand).
A bit about the condition. After discovery, open-iscsi remembersall found targets (they are stored in / etc / iscsi /), in other words, discovery is a constant operation, NOT at all appropriate, for example, dns resolving). Found targets can be deleted by hand (by the way, a common mistake is when open-iscsi, as a result of experiments and settings, has a bunch of found targets, which when trying to login into them crawl a lot of errors due to the fact that half of the targets are old lines config, which for a long time do not exist on the server, but remember open-iscsi). Moreover, open-iscsi allows you to change the settings of a remembered target — and this “memory” affects the further work with targets even after rebooting / restarting the daemon.
Block device
The second question that torments many in the beginning is where does it get after connecting? open-iscsi creates, even if it is a network, but a BLOCK device of the SCSI class (it is not for nothing that it says “I say”), that is, it receives a letter in the / dev / sd family, for example, / dev / sdc. The first free letter is used, because for the rest of the system, this is a block device - a typical hard drive, no different from being connected via usb-sata or simply directly to sata.
This often causes a panic "how can I find out the name of the block device?" It is output in the verbose output of iscsiadm (# iscsiadm -m session -P 3).
Login
Unlike SAS / UWSCSI, iSCSI is accessible to anyone. To protect against such, there is a username and password (chap), and their transfer to iscsiadm is another headache for novice users. It can be implemented in two ways - by changing the properties of the target already found earlier and setting the username / password in the open-iscsi configuration file.
The reason for such difficulties is that the password and login process are not attributes of the user, but of the system. ISCSI is a cheap version of the FC infrastructure, and the concept of “user” in the context of the person behind the keyboard is not applicable here. If your sql database is on the iscsi block device, then of course you will want the sql server to start itself, and not after a minute of personal attention of the operator.
configuration file
This is a very important file, because in addition to the login / password, it also describes the behavior of open-iscsi when it finds errors. It may not give a “backward” error immediately, but with a pause (for example, about five minutes, which is enough to reboot the data server). It also controls the login process (how many times to try, how long to wait between attempts) and any subtle tuning of the process itself. Note that these parameters are quite important for work and you need to understand how your iscsi will behave if you remove the power cord for 10-20 s, for example.
Quick reference
I don’t really like quoting easily-found mana and lines, so I’ll give a typical scenario for using iscsi:
first we find the target we need, for this we need to know the initiator IP / dns name:
iscsiadm -m discovery -t st -p 192.168.0.1-t st is the send targets command. iscsiadm -m node(list found for login) iscsiadm -m node -l -T iqn.2011-09.example:data(log in, that is, connect and create a block device). iscsiadm -m session(display a list of what they are connected to) iscsiadm -m session -P3(display it, but in more detail - at the very end of the output there will be an indication of which block device belongs to which target). iscsiadm - m session -u -T iqn.2011-09.example:data(log out from a specific one) iscsiadm -m node -l(log in to all detected targets) iscsiadm -m node -u(log out from all targets) iscsiadm -m node --op delete -T iqn.2011-09.example:data(remove target from detected ones).mulitpath
Another issue that is important in serious decisions is the support of several routes to the source. The beauty of iscsi is the use of a regular ip, which can be processed in the usual way, like any other traffic (although in practice it is usually not routed, but only switched - the load is too great there). So, iscsi supports multipath in the “do not resist” mode. Open-iscsi alone cannot connect to multiple IPs of the same target. If it is connected to several IPs of the same target, this will lead to the appearance of several block devices.
However, there is a solution - this is multipathd, which finds disks with the same identifier and processes them as expected in multipath, with custom policies. This article is not about multipath, so I won’t explain the mystery of the process in detail, however, here are some important points:
- When using multipath, you should set small timeouts - switching between bad paths should be fast enough
- In conditions of a more or less fast channel (10G and higher, in many cases gigabits), load parallelism should be avoided, since the ability to use bio coalesing is lost, which in some types of load can unpleasantly hit the target.