SAS Interface: History, Storage Organization Examples

    Last time we looked at everything related to SCSI technology in a historical context : by whom it was invented, how it was developed, what varieties it has, and so on. We concluded that the most modern and relevant standard is Serial Attached SCSI, it appeared relatively recently, but got a rapid development. The first implementation “in silicon” was shown by LSI in January 2004, and in November of that year, SAS entered the top most popular queries on storagesearch.com .

    Let's start with the basics. How do devices on SCSI technology work? In the SCSI standard, everything is built on a client / server concept.

    The client, called the initiator (English initiator), sends different commands and waits for their results. Most often, of course, the SAS controller acts as the client. Today, SAS controllers are HBAs and RAID controllers, as well as storage controllers inside external storage systems.

    The server is called the target device (English target), its task is to accept the initiator's request, process it and return data or confirmation of the command execution back. The role of the target device can be a separate disk or a whole disk array. In this case, the SAS HBA inside the disk array (the so-called external storage system), designed to connect servers to it, works in Target mode. Each target device (“target”) is assigned a separate SCSI Target ID.

    To connect clients to the server, the Data Delivery Subsystem is used, in most cases, this tricky name simply hides cables. Cables are available both for external connections and for connections inside servers. Cables vary from generation to generation of SAS. Today there are three generations of SAS:

    - SAS-1 or 3Gbit SAS
    - SAS-2 or 6Gbit SAS
    - SAS-3 or 12 Gbit SAS - is preparing to be released in mid-2013.




    Internal and external SAS cables

    Sometimes this subsystem may include SAS expanders or expanders. By expanders (Eng. Expanders, expanders, but the word “expander” has taken hold in Russian) is understood as devices that help deliver information from initiators to targets and vice versa, but transparent to target devices. One of the most typical examples is an expander, which allows you to connect several target devices to one initiator port, for example, an expander chip in a disk shelf or on a server backplane. Thanks to such an organization, servers can have more than 8 disks (the controllers that are currently used by leading server manufacturers are usually 8-port), and disk shelves can be any number needed.

    An initiator connected to a target device by a data delivery system is called a domain. Any SCSI device contains at least one port, which can be a port of the initiator, target device, or combine both functions. Ports can be assigned identifiers (PIDs).

    Target devices consist of at least one logical unit number (LUN). It is the LUN that identifies which of the disks or partitions of this target device the initiator will work with. It is sometimes said that target provides the LUN to the initiator. Thus, to fully address the desired storage, a pair of SCSI Target ID + LUNs is used.

    As in the well-known joke (“I do not lend, and the First National Bank does not sell seeds”) - the target device usually does not act as a “sending command”, and the initiator does not provide a LUN. Although it is worth noting that the standard allows the fact that one device can be both the initiator and the target, but in practice it is used little.

    There is a protocol for “communication” of devices in SAS, according to “good tradition” and on the recommendation of OSI, divided into several layers (from top to bottom): Application, Transport, Link, PHY, Architecture and Physical.

    SAS includes three transport protocols. Serial SCSI Protocol (SSP) - used to work with SCSI devices. Serial ATA Tunneling Protocol (STP) - for interfacing with SATA drives. Serial Management Protocol (SMP) - for managing a SAS factory. Thanks to STP, we can connect SATA drives to SAS controllers. Thanks to SMP, we can build large (up to 1000 disk / SSD-devices in one domain) systems, as well as use SAS zoning (more about this in the article about the SAS switch).

    The link layer is used to manage connections and transfer frames. PHY level - used for such things as setting the connection speed and encoding. At the architectural level are issues of expanders and topology. The physical layer determines the voltage, waveform of the connection, etc.

    All interaction in SCSI is based on the commands that the initiator sends to the target device and expects their result. These commands are sent in the form of command description blocks (Command Description Block or CDB). The block consists of one byte of the command code and its parameters. The first parameter is almost always LUN. A CDB can have a length of 6 to 32 bytes, although recent SCSI versions allow variable length CDBs.

    After receiving the command, the target device returns a confirmation code. 00h means that the command was accepted successfully, 02h means an error, 08h means a busy device.

    Teams fall into 4 broad categories. N, from the English "non-data", are intended for operations not directly related to the exchange of data. W, from “write” - write data received by the target device from the initiator. R, as you might guess from the word “read”, is used for reading. Finally B - for two-way data exchange.

    SCSI commands exist quite a lot, so we list only the most frequently used ones.

    Test unit ready (00h) - check whether the device is ready, whether there is a disk in it (if it is a tape drive), if the disk has untwisted and so on. It is worth noting that in this case the device does not perform full self-diagnostics, for this there are other commands.
    Inquiry (12h) - get the main characteristics of the device and its parameters
    Send diagnostic (1Dh) - perform a self-diagnosis of the device - the results of this command are returned after diagnosis with the command Receive Diagnostic Results (1Ch)
    Request sense (03h) - the command allows you to get the status of the previous command - the result of this command can be a message like “no error”, and various failures, starting with the lack of a disk in the drive and ending with serious problems.
    Read capacity (25h) - allows you to find out the size of the target device
    Format Unit (04h) - serves to destructively format the target device and prepare it for data storage.
    Read (4 options) - reading data; exists in the form of 4 different commands differing in the length of the CDB
    Write (4 options) - write. As well as for reading in 4 options
    Write and verify (3 options) - recording data and checking
    Mode select (2 options) - setting various device parameters
    Mode sense (2 options) - returns the current device parameters

    And now let's look at some typical examples of organizing data storage on SAS.

    Example one, a storage server.

    What is it and what does it eat with? Large companies such as Amazon, Youtube, Facebook, Mail.ru and Yandex use this type of server in order to store content. Content is understood as video, audio information, pictures, results of indexing and processing of information (for example, Hadoop, so popular recently in the USA), mail, etc. To understand the task and the proper choice of equipment for it, you need to additionally know a few introductory, without which it is impossible. First and foremost, the more disks, the better.



    Data center of one of the Russian Web 2.0 companies

    Processors and memory in such servers are not heavily used. The second - in the world of Web 2.0, information is stored geographically distributed, several copies on various servers. 2-3 copies of information are stored. Sometimes, if requested frequently, they store more copies for load balancing. Well, the third, based on the first and second, the cheaper - the better. In most cases, all of the above leads to the use of Nearline SAS or SATA high-capacity drives. Typically Enterprise-level. This means that such disks are designed to work 24x7 and cost significantly more than their counterparts used in desktop PCs. The case usually choose one where you can insert more disks. If it's 3.5 '', then 12 drives in 2U.



    Typical 2U Storage Server

    Or 24 x 2.5 '' in 2U. Or other options in 3U, 4U, etc. Now, having the case, the number of disks and their type, we must choose the type of connection. In fact, the choice is not very large. And it comes down to using an expander or non-expander backplane. If we use an expander backplane, then the SAS controller can be 8-port. If expanderless, then the number of ports of the SAS controller should equal or exceed the number of disks. And finally, the choice of controller. We know the number of ports, 8, 16, 24, for example, and select a controller based on these conditions. Controllers come in 2 types, RAID and HBA. They differ in that the RAID controllers support RAID levels of 5.60, 0.60 and have a sufficiently large amount of memory (512MB-2GB today) for caching. HBA has either no memory at all, or very little memory. Besides, HBA either do not know how to do RAID at all, or they can only simple, low-computational levels. RAID 0/1 / 1E / 10 is a typical set for HBA. Here we need HBA, they are much cheaper, so we do not need data protection at all and we strive to minimize the cost of the server.



    16-port SAS HBA

    An example of the second is the Exchange mail server. As well as MDaemon, Notes and other similar servers.

    Everything here is not so obvious as in the first example. Depending on how many users the server should serve, the recommendations will be different. In any case, we know that the Exchange database (the so-called Jet database) is best stored on RAID 5/6 and it is well cached using SSDs. Depending on the number of users, we determine the necessary storage volumes “today” and “for growth”. Remember that the server lives 3-5 years. Therefore, the "growth" can be limited to a 5-year perspective. Then it will be cheaper to completely change the server. Depending on the size of the disks, choose a case. With backplane easier, it is recommended to use expanders, since the requirements for the price are not as stringent as in the previous case, and in general, the server will rise in price by $ 50- $ 100, and sometimes more, we will outlast it for the sake of reliability and functionality. We choose SAS or NL-SAS / Enterprise SATA drives depending on the volume. Further, data protection and caching. Let's choose a modern 4/8 port controller that supports RAID 5/6/50/60 and caching on SSD. For LSI, this is any MegaRAID other than 9240 with CacheCade 2.0 caching, or Nytro MegaRAID with SSD “on board”. For Adaptec, these are controllers that support MAX IQ. For caching in both cases (except Nytro MegaRAID), you will need to take a couple of SSDs using Enterprise-class e-MLC technology. Intel, Seagate, Toshiba, etc. have such. Prices and companies to choose from. If you do not pay extra damage for the brand, then in the line of servers IBM, Dell, HP, find similar products and go! supporting RAID 5/6/50/60 and caching on SSD. For LSI, this is any MegaRAID other than 9240 with CacheCade 2.0 caching, or Nytro MegaRAID with SSD “on board”. For Adaptec, these are controllers that support MAX IQ. For caching in both cases (except Nytro MegaRAID), you will need to take a couple of SSDs using Enterprise-class e-MLC technology. Intel, Seagate, Toshiba, etc. have such. Prices and companies to choose from. If you do not pay extra damage for the brand, then in the line of servers IBM, Dell, HP, find similar products and go! supporting RAID 5/6/50/60 and caching on SSD. For LSI, this is any MegaRAID other than 9240 with CacheCade 2.0 caching, or Nytro MegaRAID with SSD “on board”. For Adaptec, these are controllers that support MAX IQ. For caching in both cases (except Nytro MegaRAID), you will need to take a couple of SSDs using Enterprise-class e-MLC technology. Intel, Seagate, Toshiba, etc. have such. Prices and companies to choose from. If you do not pay extra damage for the brand, then in the line of servers IBM, Dell, HP, find similar products and go! Toshiba, etc. Prices and companies to choose from. If you do not pay extra damage for the brand, then in the line of servers IBM, Dell, HP, find similar products and go! Toshiba, etc. Prices and companies to choose from. If you do not pay extra damage for the brand, then in the line of servers IBM, Dell, HP, find similar products and go!



    Nytro MegaRAID SSD Caching RAID Controller

    Third example, do -it-yourself external storage system.

    So, the most serious knowledge of SAS, of course, is required for those who manufacture data storage systems or want to make them with their own hands. We will focus on a fairly simple storage system, the software for which is produced by Open-E. Of course, you can do storage on Windows Storage Server, Nexenta, AVRORAID, Open NAS, and any other software suitable for these purposes. I just outlined the main directions, and then manufacturers' websites will help you. So, if this is an external system, then we almost never know how many drives an end user will need. We must be flexible. To do this, there is the so-called JBOD - external shelves for disks. They include one or two expanders, each of which has an input (4-port SAS connector), an output to the next expander, the remaining ports are wired to connectors, designed to connect drives. Moreover, in two-expander systems, the first disk port is routed to the first expander, the second port to the second expander. This allows you to build fault-tolerant chains of JBODs. The head server may have internal disks in its composition, or may not have them at all. In this case, “external” SAS controllers are used. That is, controllers with ports "out". The choice between a SAS RAID controller or SAS HBA depends on the control software that you choose. In the case of Open-E, this is a RAID controller. You can take care of the caching option on the SSD. If your storage system will have a lot of disks, then the Daisy Chain solution (when each subsequent JBOD connects to the previous one or to the host server) is not suitable for many reasons. In this case, the head server is either equipped with several controllers, or a device called a SAS switch is used. It allows you to connect one or more servers to one or more JBODs. We will analyze SAS switches in more detail in the following articles. For external storage systems, it is highly recommended that you use only SAS drives (including NearLine) due to increased fault tolerance requirements. The fact is that the SAS protocol incorporates much more functions than SATA. For example, monitoring write-read data all the way with checksums (T.10 End-to-End protection). And the path, as we already know, can be very long. For external storage systems, it is highly recommended that you use only SAS drives (including NearLine) due to increased fault tolerance requirements. The fact is that the SAS protocol incorporates much more functions than SATA. For example, monitoring write-read data all the way with checksums (T.10 End-to-End protection). And the path, as we already know, can be very long. For external storage systems, it is highly recommended that you use only SAS drives (including NearLine) due to increased fault tolerance requirements. The fact is that the SAS protocol incorporates much more functions than SATA. For example, monitoring write-read data all the way with checksums (T.10 End-to-End protection). And the path, as we already know, can be very long.



    Multi-disk JBOD

    Finally, I want to share some information about the current adaptation of SAS by global equipment manufacturers. SAS today is the de facto standard for server systems and professional workstations. The server systems of the vast majority of both A- and B-brands include SAS controllers, both HBA and RAID. In the field of external storage systems, the main equipment manufacturers (HP, EMC, NetApp, IBM) for several years already transferred the internal architecture of their systems to SAS. Thus, Fiber Channel discs have become a real exotic in the last couple of years. Fiber Channel continues to live and develop, mainly as a way to connect servers to storage systems, although in the field of Low-End, Mid-Range and professional systems, SAS is gaining an increasing share.

    With this, our excursion into the world of history and theory of SCSI in general and SAS in particular came to an end, and next time I will tell you in more detail about the application of SAS in real life.

    Also popular now: