How SpaceX uses GPUs to calculate rocket engines

Elon Musk breaks into the automotive and space industries with Tesla and SpaceX. It is interesting that the first completely abandons internal combustion engines, while the second, on the contrary, is trying to invent new technologies for burning fuel and to carry out a manned flight to Mars.

At the last GPU Technology Conference organized by Nvidia, we learned that delivering a group of people to Mars and vice versa is not an easy task. One of the challenges of such a mission is the need for a large and efficient rocket engine that can deliver a lot of material into orbit, ”explained Adam Lichtl, director of the SpaceX research team, to us. With a team of several dozen programmers, he is trying to cope with the difficult task of improving the simulation of combustion inside a rocket engine. To shorten flights to Mars, you also need a large engine.

“One of the main problems is cosmonaut exposure to radiation,” Lichtl explains. - To fly to Mars, you must wait for the desired window in accordance with the amount of energy that a rocket can produce. Those who played in the Kerbal Space Program know that in order to change the orbit, for example, around the Earth, you need to give yourself a change in speed and change the orbit to the one that intersects the orbit of Mars. We need vehicles that give the maximum change in speed to make the trip shorter. The second obstacle to the work of the expedition to Mars, consisting of people, not just robots, is the lack of infrastructure. Pioneers, crossing new lands, build their own houses for themselves, and they need to get food. It's pretty hard to do on Mars. ”



You not only need to take a bunch of everything with you to Mars in order to maintain a colony there, but also somehow create fuel on Mars to return to Earth. All this affects the development of the engine.

The ISS weighs 450 tons and for its construction 36 shuttle flights and five Proton launches were needed. NASA Mars Design Reference Architecture wants to launch 300 tons of materials into orbit and assemble a ship there to fly to Mars. It is advisable to do this with three flights of 100 tons each. Building and refueling a ship in Earth’s orbit is a relatively simple matter, but the rocket must be able to work on methane, since it can be synthesized from the water of Mars and carbon dioxide from the atmosphere of Mars. The current SpaceX Merlin 1D engines run on kerosene, which has more density and traction. It is produced by distillation of oil.

And if these problems are few, one more is added. Computational fluid dynamics (CFD), a program for simulating the movement of liquids and gases and their combustion inside engines, is poorly suited for simulating the operation of rockets.

“Methane is a fairly simple hydrocarbon, well suited as a fuel,” says Lichtl. - Difficulty in developing an engine that works efficiently on it. But VG for rocket engines is extremely complex. ”

SpaceX collaborates with various institutes and Sandia national laboratories in the development of its own VG software, which will allow in the future to create improved versions of Merlin engines suitable for flying to Mars and capable of working on methane.

Lichtl is a good choice for the head of rocket engine development (in my opinion, this is generally a dream job). As a graduate work at Carnegie Melon University, he developed simulators for high energy physics, and then worked in nat. Brookhaven laboratories on the same subject. Then he worked as an analyst and worked on strategic planning at Morgan Stanley, and then at another financial company. Lichtl joined the SpaceX team as chief engine engineer two years ago, and rose to the head of the engine development department for flying to Mars.

Stephen Jones, chief software developer at SpaceX, who previously worked as an engineer at Nvidia, is leading a project to develop his own software for VG. The program has no name yet, and the techniques used in SpaceX can be used to improve all kinds of combustion engines - even automobile ones.

Existing VG software is not suited for missile counting

At a Texas factory, SpaceX tries to change various injectors and other parameters to squeeze the most power out of the engines, and tests them every day. The tests are expensive, and even if you install sensors over the entire surface of the engine, you still won’t be able to find out what is going on inside. It is much better to simulate all engine components and fuel behavior, and reduce the set of options for installing injectors to a minimum, after which it is already possible to build a real engine and test some of the most optimal options.

“Another property of the engines that we have mastered is the instability of combustion as a result of propagating pressure waves and the release of chemical energy. This has always made it difficult to develop rocket engines. This is the curse of the industry. The engine starts to shake, and either falls apart, or just shakes so hard that you can’t install any payload on it. ”

They tried to increase the efficiency of automobile engines and turbines using VG calculations. Due to their small size, this is much easier to do than for large rocket engines. Therefore, you can’t just take the software and scale it.

At the molecular level, reactions in rocket fuel occur in about 10 ^-9 - 10 ^-11 seconds, and heat transfer through advection occurs at time intervals of the order of 10^-6 - 10 ^-7 sec. Acoustic vibrations in the combustion chamber occur at time intervals of the order of 10 ^-3 - 10 ^-4 sec. This is too large a scale for calculation in the SH simulation.

Another problem with the calculations is the physical size of the engine. The length of the combustion chamber can be a meter, and at the other end of the Kolmogorov scale are sizes of the order of 1 μm, which control the dispersion rate in the fuel flow and determine the rate of combustion in the chamber. It turns out a million times difference, which must be taken into account in the calculations.

“If you start dividing the grid for calculating the SH by two degrees, you will need to divide it about 20 times to accommodate such a length scale,” says Licht. If you try to uniformly distribute the grate throughout the combustion chamber, you will need to process the yottabytes (10 ²⁴ ) of the data. This is unrealistic. ”

In this regard, rocket developers simulate what is happening either with a rough approximation and many characteristics, or in more detail, but on a smaller scale.

“Why not do it at the same time? What is interesting in turbulence is that, despite the large scale spread, the flux density is low. You do not need to count everything down to microns at every point. By its nature, it is fractal. And fractality allows us to concentrate computing power where they are needed. It's kind of a fancy compression algorithm. ”

It is only necessary to carry out all calculations on compressed fractal data without the need to decompress them. To do this, engineers came up with a technique called local fractal compression of microwaves (wavelets local fractal compression). For simulations, an adaptive grid with different resolutions in different places is used. Typically, such techniques involve predetermined lattice parameters, but SpaceX has succeeded in making a simulation with dynamically adapting parameters.

To demonstrate the SH, Jones showed a simulation of the fuel and oxidizer that explode inside the box, and the calculation of the propagation of shock waves that bounce and move inside the fluid, resulting in a huge amount of fractal turbulence. ( link - video, the simulation starts at 36 minutes). In the simulation, the change in gas density is calculated in increments of several milliseconds.



“If you were to do these calculations the usual way around the grid, you would have to calculate 300 million nodes,” says Jones. “And we did it with the help of one GPU, because we calculate the parameters only in those places where it is necessary, and the algorithm itself knows where exactly it is necessary to conduct the calculation.”

Although the specific GPU was not named, it was probably the Tesla GPU coprocessor of the Kepler class, and apparently it used parallel computing capabilities to calculate the lattice.

Here is another simulation on one GPU, showing gas ignition and temperature distribution (in the video from 37:45), where the gas explodes and bounces off the walls:



Yellow areas are the hottest. Jones says that you can make an image magnification of 60,000 times and all the same, internal structures will be visible there.

Lichtl says they tried to use similar techniques before, and these simulations are based on the work of Jonathan Rigel, a professor from Iowa University.

“However, without a GPU and without our architecture, even the simplest simulations would have to be calculated on thousands of processors for months. GPU dramatically speeds up the process. Well, that in SpaceX we were allowed to start from scratch in the manufacture of VG. What for? Then, if you are a little mistaken in the classic version of the VG simulator, it usually gets away with you. And when simulating combustion, if you made a mistake somewhere, all processes begin to go differently. ”

For example, a small error in the gas temperature leads to a large error in the estimation of its kinetic energy, because they are connected by an exponential dependence.

Another nice feature of their software, coupled with the GPU - they are well calculated chemical kinetic models that describe the bi-directional chemical reactions that occur when the fuel meets the oxidizing agent. The burning of hydrogen with oxygen is not a simple reaction, leading to the appearance of two water molecules from two hydrogen molecules and one - oxygen. You have to calculate 23 possible reactions and 11 intermediate to calculate all possible outcomes. When methane is burned with oxygen, 53 intermediate reactions and 325 final variants appear.

“It’s not for you to show figurines to sparrows,” says Lichtl. - It is impossible to overestimate the benefits of parallel architecture. By reducing the number of necessary grid points for calculation, you can transfer valuable computing resources to something more important, for example, to miscalculation of chemical reactions. "

This is exactly what SpaceX software does, calculating chemical reactions and turbulence of fuel in the combustion chamber and at the exit of the nozzle (video from 43 minutes):



In this case, the acoustic properties of combustion, including the Mach disk and the Mach ring, all obtained from the mathematical model, are shown , and pretty well imitating the real combustion shown in the photo at the top left. At the exit of the nozzle, the gases are transparent, and their speed exceeds the speed of sound. Faced with a shock wave, they slow down and contract, becoming white.

Further software development involves scaling to many GPUs working in clusters.

“Now the code runs on one machine. In the near future we will deal with parallelization, ”says Lichtl. There is no need to

wait until Nvidia completes its NVLink . Now they can make clusters based on the PCI-Express bus. The possibilities of using the MPI protocol are also considered, when different servers will work in parallel, calculating data for the VG.

An intermediate solution in anticipation of NVLink could be porting software on clusters from IBM Power8 processors, and using the InfiniBand network interface, which works with processors through the IBM's Coherent Accelerator Processor Interface, or CAPI. This architecture is used by the U.S. Department of Energy in its largest Summit and Sierra supercomputers, scheduled for 2017. In addition, there is a development from Mellanox Technologies , a ConnectX-4 network interface that allows you to work with Power8 through CAPI at speeds up to 100 Gb / s. There are several companies making the OpenPower system with which SpaceX will also be able to collaborate if it decides to go this route.

The main thing is that Lichtl wants to purchase a server with many GPUs. What can not but rejoice the company Nvidia.