May 31, 2019 at 12:15The history of the assembly of the "village supercomputer" of spare parts from eBay, Aliexpress and a computer store. Part 1
Good day, dear Khabrovchians!
I want to tell you a long and, as I hope, fascinating, and maybe useful story for someone who assembled a “village supercomputer” from a Dell server node board, Nvidia Tesla K20 GPU, and what was bought here and there from all kinds of online stores or in computer stores in your city.
The story began with the fact that my friend, a programmer, who at the same time was also an astronomer, began studying neural networks. Their “full-time specialist” quit and the topic was hung on the “closest specialist”. I myself am not a programmer, just a “radio mechanic for repairing computers (c) my diploma,” because assembling all kinds of interesting computer glands is an interesting and enjoyable task for me. Unfortunately, I work in a different field.
To more clearly formulate the problem, I created a topic on the forum “Iron Ghosts of the Past” where it was discussed for a long time. At first there was a rather naive idea of “building a 4-way SLI on a GTX 580 3Gb” which gradually transformed into an understanding - you need to build a server! The prices for server motherboards were shamelessly biting until I came across on Youtube an interesting video about launching a Chinese server board on 2 non-standard format processors.
Here is the video:
In this video I was especially pleased with the budget price of the system.
However, a consultation with more knowledgeable comrades who were dealing with Chinese servers convinced me - “We do not need Chinese happiness!” According to their reviews, Chinese servers were simply monstrously unreliable. And I began to look at Avito options with Dell server boards. I have two laptops from this company and they have only positive impressions. Very reliable equipment.
On Avito, a Dell PowerEdge C6220 server node board was found in the process of communicating with the seller of which - he prompted me an excellent site where there was a publication about how one craftsman launched such a board, here is the link . And there was a link to the American forum, where powerful workstations were assembled on such boards. This topic is here .
I read the whole topic from beginning to end, I decided on the goals, objectives and ways to achieve them. The task was formulated as follows: "Assemble a dual-processor server on the Dell PowerEdge C8220 node board with a Tesla K10 or K20 GPU." The choice fell on specialized GPUs after a discussion with a person for whom the system was actually going to - having “cards” capable of long-term calculations with double accuracy and error control of ECC memory, he could use them for his scientific work, and not just for training neural networks. What he actually was very happy.
To discuss and capture the history of the assembly process on the Iron Ghosts of the Past forum, I created a corresponding topic where I actually wrote about the process and posted photos. Those interested can familiarize themselves .
The task was set and I started searching for components. At the time when it all started, I did not yet have an eBay registration and in the beginning my friends bought the necessary spare parts, to whom I paid the costs of purchase and shipping. Later, I registered there myself and began to buy directly, although sometimes I have to ask for help from those who have accounts in Shopotama and similar services. Not all necessary spare parts are sent directly from the USA to Russia.
The first with eBau was bought motherboard Dell PowerEdge C8220 0083N0. According to Dell's documentation, it applied to board version 1.2 with 3 PCI-E 16x slots. The two usual ones near the power button and the third on the other side of the board are non-standard, under the so-called GPGPU-riser, which was included in the so-called Edge Slot.
Photo of the board, the same 0083N0, photo from eBay.
And this is my photo, a ruler is attached to the board to understand the scale.
By that time, a GPGPU riser came to me in the same Edge Slot.
Here is a photo of where it is connected to the sample to its regular place.
At the same time, a power adapter was purchased on eBay, from ATX to this C6100 power connector. They are sold on eVau two types, 12 and 18 pin. We need the latter, and also DC-DC boosting to turn + 5VSB from ATX PSU to + 12VSB of Dell server. And of course, the “mother” response block in the connector in order to install the jumper necessary to start the board and output the PS_ON signal from it. By the way, it is with a non-standard contact pitch of 2.0 mm. Of course, desperate guys can poke a screwdriver or nail directly into the connector on the board, but I preferred to do everything culturally.
In addition, for the trial launch of the board, the cheapest Xeon E5-2604 V1 were purchased from Aliexpress and with eBau, a pair of DDR3 ECC REG memory slots that were sold as compatible with the Dell PowerEdge C8220. First of all, I used coolers Alpine 20 Plus C0 for LGA 2011 which I had to modify - the edges were sawn up by the grinder which rested on the memory slots, the spring washers were removed from the fastening screws, and a couple of nuts were screwed onto the threads - so as not to screw the screws too deep and Do not break the board. The LGA 2011 server sockets are arranged slightly different than regular sockets and for radiator screws there the thread should be short. By the way, the coolers worked well, despite the fact that they were purely aluminum.
And so, the moment came when the processors arrived, I captured their installation in a photo for memory.
And here are the very aluminum Alpine coolers installed.
The assembled and running system.
My old faithful Chiftek 550 W power supply was connected to the system, a USB hub for 4 devices that included a keyboard, a mouse and a USB flash drive with Ubuntu, a card reader was connected to the USB card reader on the board, into which I inserted a Chinese USB sound card, also connected a VGA monitor and patch cord to the 100 Mbps IPMI port, which is called Delicated-NIC. Next to it are two 10Gbe ports that work on a normal twisted pair of “copper” and fully support the usual 100/1000 network.
In this form, the system was started and it turned out that the board at startup checks the memory for a very long time. And in the BIOS splash screen, she called herself the Dell DCS 6220.
On this, I will finish the first part of my story so as not to tire the grateful readers.
Link to part 2: habr.com/en/post/454448
I want to tell you a long and, as I hope, fascinating, and maybe useful story for someone who assembled a “village supercomputer” from a Dell server node board, Nvidia Tesla K20 GPU, and what was bought here and there from all kinds of online stores or in computer stores in your city.
The story began with the fact that my friend, a programmer, who at the same time was also an astronomer, began studying neural networks. Their “full-time specialist” quit and the topic was hung on the “closest specialist”. I myself am not a programmer, just a “radio mechanic for repairing computers (c) my diploma,” because assembling all kinds of interesting computer glands is an interesting and enjoyable task for me. Unfortunately, I work in a different field.
To more clearly formulate the problem, I created a topic on the forum “Iron Ghosts of the Past” where it was discussed for a long time. At first there was a rather naive idea of “building a 4-way SLI on a GTX 580 3Gb” which gradually transformed into an understanding - you need to build a server! The prices for server motherboards were shamelessly biting until I came across on Youtube an interesting video about launching a Chinese server board on 2 non-standard format processors.
Here is the video:
In this video I was especially pleased with the budget price of the system.
However, a consultation with more knowledgeable comrades who were dealing with Chinese servers convinced me - “We do not need Chinese happiness!” According to their reviews, Chinese servers were simply monstrously unreliable. And I began to look at Avito options with Dell server boards. I have two laptops from this company and they have only positive impressions. Very reliable equipment.
On Avito, a Dell PowerEdge C6220 server node board was found in the process of communicating with the seller of which - he prompted me an excellent site where there was a publication about how one craftsman launched such a board, here is the link . And there was a link to the American forum, where powerful workstations were assembled on such boards. This topic is here .
I read the whole topic from beginning to end, I decided on the goals, objectives and ways to achieve them. The task was formulated as follows: "Assemble a dual-processor server on the Dell PowerEdge C8220 node board with a Tesla K10 or K20 GPU." The choice fell on specialized GPUs after a discussion with a person for whom the system was actually going to - having “cards” capable of long-term calculations with double accuracy and error control of ECC memory, he could use them for his scientific work, and not just for training neural networks. What he actually was very happy.
To discuss and capture the history of the assembly process on the Iron Ghosts of the Past forum, I created a corresponding topic where I actually wrote about the process and posted photos. Those interested can familiarize themselves .
The task was set and I started searching for components. At the time when it all started, I did not yet have an eBay registration and in the beginning my friends bought the necessary spare parts, to whom I paid the costs of purchase and shipping. Later, I registered there myself and began to buy directly, although sometimes I have to ask for help from those who have accounts in Shopotama and similar services. Not all necessary spare parts are sent directly from the USA to Russia.
The first with eBau was bought motherboard Dell PowerEdge C8220 0083N0. According to Dell's documentation, it applied to board version 1.2 with 3 PCI-E 16x slots. The two usual ones near the power button and the third on the other side of the board are non-standard, under the so-called GPGPU-riser, which was included in the so-called Edge Slot.
Photo of the board, the same 0083N0, photo from eBay.
And this is my photo, a ruler is attached to the board to understand the scale.
By that time, a GPGPU riser came to me in the same Edge Slot.
Here is a photo of where it is connected to the sample to its regular place.
At the same time, a power adapter was purchased on eBay, from ATX to this C6100 power connector. They are sold on eVau two types, 12 and 18 pin. We need the latter, and also DC-DC boosting to turn + 5VSB from ATX PSU to + 12VSB of Dell server. And of course, the “mother” response block in the connector in order to install the jumper necessary to start the board and output the PS_ON signal from it. By the way, it is with a non-standard contact pitch of 2.0 mm. Of course, desperate guys can poke a screwdriver or nail directly into the connector on the board, but I preferred to do everything culturally.
In addition, for the trial launch of the board, the cheapest Xeon E5-2604 V1 were purchased from Aliexpress and with eBau, a pair of DDR3 ECC REG memory slots that were sold as compatible with the Dell PowerEdge C8220. First of all, I used coolers Alpine 20 Plus C0 for LGA 2011 which I had to modify - the edges were sawn up by the grinder which rested on the memory slots, the spring washers were removed from the fastening screws, and a couple of nuts were screwed onto the threads - so as not to screw the screws too deep and Do not break the board. The LGA 2011 server sockets are arranged slightly different than regular sockets and for radiator screws there the thread should be short. By the way, the coolers worked well, despite the fact that they were purely aluminum.
And so, the moment came when the processors arrived, I captured their installation in a photo for memory.
And here are the very aluminum Alpine coolers installed.
The assembled and running system.
My old faithful Chiftek 550 W power supply was connected to the system, a USB hub for 4 devices that included a keyboard, a mouse and a USB flash drive with Ubuntu, a card reader was connected to the USB card reader on the board, into which I inserted a Chinese USB sound card, also connected a VGA monitor and patch cord to the 100 Mbps IPMI port, which is called Delicated-NIC. Next to it are two 10Gbe ports that work on a normal twisted pair of “copper” and fully support the usual 100/1000 network.
In this form, the system was started and it turned out that the board at startup checks the memory for a very long time. And in the BIOS splash screen, she called herself the Dell DCS 6220.
On this, I will finish the first part of my story so as not to tire the grateful readers.
Link to part 2: habr.com/en/post/454448