May 24, 2007 at 07:59how we tested dual-core opterons
Well. To tell you the truth, we still test them, but there is already one feature that has become obvious: memory does not pull two cores. Well, that’s not all. We have a new cluster, in which there are some blades, into which dual-processor boards are inserted, on which 285 dual-core opterons are installed, each of which is stuck with 4 gigabytes of memory and it’s connected to the outside world through Hyper Transport like this here: the
outside world (including 1Gb copper ethernet) --HT-- cpu0 --2xHT-- cpu1
We make the simplest test (well, actually the program is not so simple - this is MG from NASA Parallel Benchmark) in the following configuration: either we add 4 processes of this task to one node - for each core by a process, or 4 processes to different processors in two different blades. As a result, the first configuration produces 2700 parrots, and the second 4500. Think about it, interaction via ethernet, which is much less efficient than zerocopy on NUMA, and even in such a configuration when all traffic extends through one link, and even when instead of an optimized protocol everything is swinging computationally powerful TCP is much more efficient.
Of course, the reason is obvious: two cores compete for one memory controller. When we use 4 controllers and asynchronous data delivery between them, everything turns out to be much faster. But then the question arises: why the hell are we forced to buy multi-core processors, convincing them that they have some unsurpassed performance? Indeed, it would be better to add some kind of logic instead of the second core, like DMA. Or energy consumption would be reduced, just throwing the core.
Eh. But what is most depressing is where developers have focused their activities: to put even more cores into the processor, without at all reducing competition for access to memory. Well, yes, yes, Intel has a common second-level cache, AMD will have a third-level cache, but this is a common resource, yes, probably with several banks. Yes, AMD will bring this all to two memory controllers, but why is it so complicated and energy inefficient? If this does not solve the problem. Because, as before, for one controller (even if by some miraculous miracle it is possible to convince Linux to expand address spaces so that they fall under different memory controllers and in different banks of the Third cache), two cores will compete.
And now, attention, drum roll and all that jazz, processors with integrated graphics cores roll out onto the stage. My question arises: how in their favorite SMP configurations with one subsystem of memory access will they feed their favorite kernels with data? Why do we need idle processors (how many, if you count on parrots?) 30% of the time? Why not make more simple processors, but with DMA and its own memory. By the way, do you know that BlueGene / L runs on processors for embedded electronics ?:)
Yes, even if you want Intel and AMD to sell solutions on the same chip, why not just separate each core into its own memory? Well, much more effective. Eh. I do not understand the logic of the manufacturers. Moreover, I do not understand the logic of those who spend money on all this “splendor”, because the shortcomings are obvious.
In short, all the bad ones, except for IBM, which did something similar in Cells. But again, it’s not clear, this is a manic desire to put everything in one chip. Not, well, yes, they were slow external tires for a long time. But now there are consecutive ones - insanely fast and they consume little energy, and even they can be safely let through the optical fiber.
Here. No, of course, you can find application for the second core: some kind of virtualization, or, as in our case, run all system processes on, for example, even cores, and settlement on odd ones, you can win, 10 percent, which is not enough , in principle, if the calculation time is 10 days. But still, intuition protests against the complication of such an already difficult thing as a processor. Complication is, of course, only quantitative. But ... Huh. Anyway, the transistors then switch, the air heats up, the ice melts, America drowns, nations emigrate, we become crowded in the Urals :)
PS Now, they’ll probably come and say that we ourselves are fools, that in x86-64 16 registers were invented for this purpose, to unload the memory subsystem during calculations, and that the compiler should be optimized and used correctly. To which I will answer: used the recommended AMD compiler - PGI. Intel’s also tried, carefully watching to optimize the same memory accesses. The result is the same.
outside world (including 1Gb copper ethernet) --HT-- cpu0 --2xHT-- cpu1
We make the simplest test (well, actually the program is not so simple - this is MG from NASA Parallel Benchmark) in the following configuration: either we add 4 processes of this task to one node - for each core by a process, or 4 processes to different processors in two different blades. As a result, the first configuration produces 2700 parrots, and the second 4500. Think about it, interaction via ethernet, which is much less efficient than zerocopy on NUMA, and even in such a configuration when all traffic extends through one link, and even when instead of an optimized protocol everything is swinging computationally powerful TCP is much more efficient.
Of course, the reason is obvious: two cores compete for one memory controller. When we use 4 controllers and asynchronous data delivery between them, everything turns out to be much faster. But then the question arises: why the hell are we forced to buy multi-core processors, convincing them that they have some unsurpassed performance? Indeed, it would be better to add some kind of logic instead of the second core, like DMA. Or energy consumption would be reduced, just throwing the core.
Eh. But what is most depressing is where developers have focused their activities: to put even more cores into the processor, without at all reducing competition for access to memory. Well, yes, yes, Intel has a common second-level cache, AMD will have a third-level cache, but this is a common resource, yes, probably with several banks. Yes, AMD will bring this all to two memory controllers, but why is it so complicated and energy inefficient? If this does not solve the problem. Because, as before, for one controller (even if by some miraculous miracle it is possible to convince Linux to expand address spaces so that they fall under different memory controllers and in different banks of the Third cache), two cores will compete.
And now, attention, drum roll and all that jazz, processors with integrated graphics cores roll out onto the stage. My question arises: how in their favorite SMP configurations with one subsystem of memory access will they feed their favorite kernels with data? Why do we need idle processors (how many, if you count on parrots?) 30% of the time? Why not make more simple processors, but with DMA and its own memory. By the way, do you know that BlueGene / L runs on processors for embedded electronics ?:)
Yes, even if you want Intel and AMD to sell solutions on the same chip, why not just separate each core into its own memory? Well, much more effective. Eh. I do not understand the logic of the manufacturers. Moreover, I do not understand the logic of those who spend money on all this “splendor”, because the shortcomings are obvious.
In short, all the bad ones, except for IBM, which did something similar in Cells. But again, it’s not clear, this is a manic desire to put everything in one chip. Not, well, yes, they were slow external tires for a long time. But now there are consecutive ones - insanely fast and they consume little energy, and even they can be safely let through the optical fiber.
Here. No, of course, you can find application for the second core: some kind of virtualization, or, as in our case, run all system processes on, for example, even cores, and settlement on odd ones, you can win, 10 percent, which is not enough , in principle, if the calculation time is 10 days. But still, intuition protests against the complication of such an already difficult thing as a processor. Complication is, of course, only quantitative. But ... Huh. Anyway, the transistors then switch, the air heats up, the ice melts, America drowns, nations emigrate, we become crowded in the Urals :)
PS Now, they’ll probably come and say that we ourselves are fools, that in x86-64 16 registers were invented for this purpose, to unload the memory subsystem during calculations, and that the compiler should be optimized and used correctly. To which I will answer: used the recommended AMD compiler - PGI. Intel’s also tried, carefully watching to optimize the same memory accesses. The result is the same.
