Lessons from space accidents: the defeat and triumph of Apollo 13

In the same April days of 1970, probably the most dramatic story of everything that happened in space was played out. Three astronauts who went to the moon were in mortal danger and were forced to return home for three days, overcoming various difficulties that arose. This is a very beautiful story about how small changes in the specification can lead to big problems, the well-coordinated work of hundreds of people at the MCC in the emergency mode, about courage and professionalism.

Cause

As is often the case in complex technical systems and large projects, the cause of the accident was laid years before the flight of Apollo 13, and the accident itself consisted of a complex chain of events, moreover, the absence of any link would lead to the absence of an accident.

Design

In order to understand what happened, it is necessary to talk about the design of the Apollo service module: The

energy subsystem of the Apollo service module consisted of two hydrogen tanks, two oxygen tanks and three fuel cells. Fuel cells, consuming hydrogen and oxygen, produced electricity and water, which was consumed by the crew for drinking and equipment for cooling. It was a very efficient system, better than solar panels, provided that the flight would be no longer than 2-3 weeks.

This is the oxygen tank of the Apollo service module. It is so well insulated that it is able to store liquid oxygen for years. Liquid oxygen is stored in it in a supercritical fluid state., and, therefore, exhibits the properties of both liquid and gas. As you know, with expansion, the gas temperature decreases. The thermal insulation is so good that liquid oxygen would cool and lose its supercritical properties simply from expansion at normal consumption for fuel cells. Therefore, I had to install a special heater to maintain the required temperature and pressure. In zero gravity, liquid oxygen in a supercritical state had a bad habit of exfoliating into liquid and gaseous layers, which led to incorrect readings of the level sensor. Therefore, it was necessary to install a special turbine for mixing oxygen, and for the crew, the procedure for mixing oxygen in tanks was added to the “housework” set so that after it the Houston MCC could get the correct data on the amount of oxygen on board.

Minor specification change

1965 year. Five more years before the Apollo 13 flight, another year before the first unmanned AS-201 flight, even the Gemini program made its first manned flight only this year. Active work is underway on the Apollo ship. Due to the sheer scale of the work, NASA contractors hire subcontractors to produce the necessary elements. The Apollo service module was made by North American Aviation, and the tanks for it were made by a Beech Aircraft subcontractor. Since the fuel cells produced 28 volts of voltage, the operating voltage of 28 volts was specified in the tank specification. However, already in the process of developing the service module, it turned out that in preparation for the launch, Apollo will receive electricity from ground-based generators of the launch complex, and they have an operating voltage of 65 volts (a completely normal situation, when many qualified people make a big project, no jokes). Therefore, the specification had to be redone. Beech Aircraft engineers changed the equipment of the oxygen tank, but forgot to change only one thing under the new voltage - thermostat contacts. They are designed to open the heater circuit if necessary. Quality control at all levels - Beech Aircraft, North American Aviation and NASA did not notice this error.

Moving tanks

1968 year. The tanks, which eventually ended up on Apollo 13, are installed in the service module, which will become part of Apollo 10. Since changes were made to the tanks, after a while it was decided to install newer tanks on the Apollo 10, and to remove, upgrade and install the already installed tanks on another service module. In the process of removing the tanks, the workers forgot to unscrew one bolt, and the winch, which had already begun to raise the shelf with tanks, stalled and dropped the tanks back into the rack. The height of the fall was ridiculous, only 5 cm, but for space technology this is a serious emergency. The incident was documented, the tank was tested, it was considered serviceable and sent for modernization. Obviously, the modernization was not associated with a serious disassembly of the tank (this is important for understanding the next stage).

The same tank number two in the foreground, photographed after installation.

Cocking

March 27, 1970, two weeks before the start of Apollo 13. The so-called pre-launch training countdown - a complete simulation of the launch with refueling the vehicle with working fluids, transition to the flight atmosphere, in short, everything except the real “Ignition” command. The simulation was successful with one exception - tank number two refused to empty after testing. In the process of solving the problem, the engineers suggested that in the fall of 1968, the lower drain fitting was damaged. Theoretically, this is an emergency, it is necessary to postpone the start and change the tanks. But, on the other hand, the lower drain fitting is used only once - during the training pre-launch reading. In flight, it is not needed, and you can fly with an idle fitting. Therefore, it was proposed to use a heater for oxygen bleeding, to gasify oxygen, which will evaporate by gravity through the top fitting. The decision was agreed with the spacecraft commander astronaut Jim Lovell. Jim signed the documents based on the available data: the proposed solution is the best one invented, no other malfunctions were found, the lower connection is not needed during the flight, and changing the tanks will take forty-five hours, apart from checking the new tanks, which will disrupt the pre-launch preparation schedule and delay the launch for a month.
When the heater was turned on with a ground voltage of 65 volts, the contacts of the thermostat, designed for 28 volts, were welded in the on position, the heater lost the ability to turn off: a

closed contact, a photograph of a natural experiment to reproduce the accident.

The temperature sensor inside the tank, made to measure the operating temperature in the region of -207 degrees, had an upper measurement limit of +27 degrees. The engineer controlling the work could get only two parameters - “the heater is on” and “the temperature is not higher than +27 degrees.” In reality, the constantly turned-on heater quickly evaporated oxygen and, while continuing to work in an empty tank, heated up to +540 degrees. Somewhere in the huge complex of buildings on the launch pad, there was a recorder fixing the heater's direct current instead of on-off cycles, but no one looked at his tape before the accident. The heater, heated to +540 degrees, melted the Teflon insulation, and the wires turned into a detonator.
Abnormal heating could not be fixed directly - the tank was well insulated, so a fire in the service module could not occur, and the abnormal temperature lasted until the pre-start filling, when new liquid oxygen cooled the inside of the tank.

Characters

The crew of Apollo 13

From left to right: Lovell, Swayert, Hayes. They took pictures in a hurry because of a replacement in the crew.

Jim Lovell - the commander, a veteran of the space program, made two flights on the "Gemini" and fly over the moon on the "Apollo 8".
John Swaigert is a command module pilot. The first space flight, in a backup crew, was transferred to the main crew a few days before the flight due to the fact that the astronaut of the main crew, Ken Mattingly, was in contact with astronaut Charles Duke, who had rubella, and was not immune to rubella. The first bachelor among astronauts.
Fred Hayes is the pilot of the lunar module. First flight to space.

Houston MCC

Gene Kranz is the leader of the “white team”, the main flight shift (there were four shifts in total), and the leading flight director.


"White team." The mission is unknown.

Crash

55 hours 54 minutes flight. The next inclusion of the tank mixing system (it was switched on regularly, more often than once a day) caused a short circuit in tank number two. Teflon insulation caught fire:

Teflon burning in oxygen, a photograph of a natural experiment to reproduce an accident.

The combustion of Teflon in oxygen caused a sharp heating of the tank and an increase in pressure exceeding the strength limits of the tank. The top lid of the tank tore off:

Lid failure, photograph of a full-scale experiment to reproduce the accident.

A sharp increase in pressure tore off the sheathing sheet of the service module section:

Failure of the panel, photograph of a full-scale experiment to reproduce the accident.

In addition, the shake from the failure of the tank cap caused an abnormal closure of the valves of the fuel cells 1 and 3, leading to their shutdown after three minutes, and led to a violation of the tightness of the pipelines of the oxygen tank number one. After 130 minutes, the pressure in the oxygen tank number one dropped to zero - the command module lost water and energy. It was at a distance of 320,000 kilometers from Earth.

Photo from a ground-based telescope.

“Houston, we had a problem here.”

Flight Scheme.

Yes, the phrase-meme sounded in the original just like that: "Houston, we've had a problem here." Astronauts and the MCC, of ​​course, did not have a full knowledge of the situation described in the previous paragraph, so the first minutes people tried to figure out what happened. In any space incident, first of all, it is necessary to establish whether this is a real problem or a failure of sensors / telemetry. First, the crew rebooted the computer, reported the indicators, connected the fuel cells to different power buses to figure out what was happening. But fifteen minutes later, Jim Lovell said that he was observing a leak of some gas from the service module - the problem was obviously very serious, and it was far from over.
At that time, another catchphrase sounded at the MCC - “Guys, let's solve the problem.” Let's not make things worse by guesswork. ”
The initial task was to try to save the remaining oxygen in tank number one. Despite all attempts, the leak site could not be isolated, the pressure continued to fall. The only possibility was the inclusion of the lunar module, which became a lifeboat. The work had to be carried out very quickly, simultaneously turning off the command module and including the lunar. The inclusion of the lunar module according to the instructions took about three hours. The leak rate increased, and when it became clear that the fuel cell would last less than fifteen minutes, I had to change the start-up procedure on the go. A separate problem was navigation. It was necessary to rewrite the data of the gyrostabilized platform of the command module, recalculate (the angles of docking of the command and the lunar module were not strictly 180 degrees) and enter the data into the gyro-stabilized platform of the lunar module. The complex procedure was successfully completed. The command module disconnected, the lunar module took control.

Difficult choice and first path correction

The next task was the choice of return mode. All the Apollo flied along such a trajectory (the so-called free return trajectory), which allowed the Moon to fly around and a normal landing on Earth. Because of this, all the Apollo landings were near the lunar equator. At the same time, there were emergency return modes, when a sufficiently long impulse with the engines returned the ship to Earth without flying around the moon:
After a rather intense meeting (another catch phrase was said - “Failure is an unacceptable option” ), it was decided to stay on the trajectory free return. Arguments:

1. The option of shooting the lunar module to reduce the mass, which would have been necessary to slow down, became impossible.
2. The option of using the engines of the lunar module, while there was enough fuel, did not give a noticeable gain in time. A reset of the service module could disrupt the thermal regime of the heat shield at the bottom of the command module. Damage to the heat shield made landing impossible.
3. Consumables (water, electricity) were estimated to be enough for free return.
4. The main engine of the service module may have been damaged and its use was a big risk.
5. The ship was already close to the moon and emergency return maneuvers became less and less profitable.

However, Apollo 13 has already stepped off the path of free return. A slight deviation was necessary for landing in the selected area. Therefore, I had to make a correction with the lunar module landing engine, turning it on for 30 seconds. A separate problem was navigation. Pieces of thermal insulation, torn out during the destruction of the oxygen tank, scattered in abundance around the ship, becoming false stars. Therefore, I had to use the Sun to check the accuracy of orientation. Fortunately, the data was transferred correctly, and the gyrostabilized platform worked correctly, navigation was kept accurate, and the correction was successful.

PC + 2 maneuver

The free return trajectory also required a little maneuver. It was held two hours after the perisulation (the pericenter of the orbit around the moon), therefore it was simply called “PeriSeleniy +2” (PC + 2). Thanks to him, the landing point shifted from the Indian to the Pacific Ocean, where the main landing support ships were regularly located, and the landing time moved forward by 10 hours. The lunar module landing engine, designed for one inclusion before landing, was turned on for the second time, and it worked for 4 minutes 24 seconds.

Strict savings

The lunar module was powered by batteries, not fuel cells. Therefore, on the one hand, oxygen was abundant, because it was used to fill the lunar module after reaching the surface of the moon. On the other hand, electricity and water were in severe shortage. The lunar module was designed to work two people for a day and a half, but now it had to provide three people for four days. Therefore, after the ship left due to the disk of the moon, all possible measures were taken to save electricity and water. Water was consumed by people and spent on cooling equipment. Therefore, everything that was possible was turned off in the lunar module, and people had to endure thirst. In preparation for bedtime, the command module closed the curtains on the windows out of habit, it quickly cooled down and did not warm up until landing.

Insert the square pin into the round hole

In English, there is an idiom - "square pin in a round hole" - "square peg in a round hole". It denotes a person out of place. And in the flight of Apollo 13, the idiom became a reality. Despite the abundance of oxygen, the crew had a breathing problem. The fact is that the exhaled carbon dioxide must be absorbed by something. More than 15% of carbon dioxide in the inhaled air leads to visual impairment, then consciousness, and, as a result, death. In the lunar module there were round cans of lithium hydroxide that absorbed carbon dioxide. But they were not enough. There were enough cans of lithium hydroxide in the command module, but they were square:

Photo from the movie Apollo 13, but the meaning is conveyed very correctly.

Therefore, the task arose of quickly creating a way to insert a square pin into a round hole. The special group, taking the same materials as those that were on the Apollo 13, quickly assembled the adapter and wrote the assembly instructions. The idea was quite simple - the canister was placed in a bag into which air was supplied from the pump of the air system. They took the bag from the flight suit packaging, the hose from the spacesuits, fastened it with electrical tape, put the bent cover of the flight plan as a spacer for even distribution of air, and closed the regular hole in the canister with a toe and covered with the same electrical tape.

Adapter assembly and working device. It blows with something elusively dear ... The

instructions were transmitted to Apollo 13, and the same adapter was assembled in space. The carbon dioxide problem has been resolved.

Critical procedure

Parallel to all these matters, extremely hard work was underway to create a procedure for launching a command module. Without the inclusion of the command module, landing was impossible. And the inclusion was complicated by the fact that its batteries were already partially discharged, and the procedure for turning on the completely turned off command module was not only not developed in advance, it was not thought about and was not checked in the simulator. John Aaron, the same "steel-eyed rocket"who saved Apollo 12, led the group creating this procedure. In conditions of limited time (the command module was flying inexorably toward Earth), they came up with the idea of ​​deploying a power bus, which was designed for emergency power supply to the lunar module. Literally a few amperes, which were not enough to run the command module systems, were received from the lunar module, and the procedure was ready on time.

Battery explosion

At 97 hours of flight time, an explosion occurred in one of the batteries of the lunar module. Hydrogen and oxygen normally released during battery operation accumulated in the compartment of one of the batteries and an accidental spark led to an explosion. Fortunately, this explosion did not bring any special problems, three batteries worked as well, and the fourth one slightly decreased its charge.

Problem out of nowhere

All the while returning from the moon, a problem accumulated near the ship, the cause of which no one could establish. For a normal landing, the ship should be in a fairly narrow range of angles of entry into the atmosphere. The angle is too small - and the ship will bounce off the atmosphere like a flat stone from the water, the angle is too large, and the ship will burn out due to too much heat during braking. And an unknown force during the flight led to the fact that the angle of entry slowly but steadily decreased. It seems that all the forces acting on the ship were taken into account. Even the discharge of urine overboard was forbidden so that the reactive force did not spoil the trajectory, but in vain - the angle went beyond permissible limits. Another correction was needed. To save electricity, it was carried out manually, not including a computer. The lunar module landing engine turned on for 14 seconds at 10% of the thrust, for the third time. Time was also measured manually by wristwatch. The correction was satisfactory, the angle of entry into the atmosphere was within acceptable limits, but later another correction will be required.
After the flight, it was found that the reactive force was created by the water evaporating from the cooling system of the lunar module. Prior to this, the lunar module was never in free flight so long that this small force became noticeable.

Another correction

At 108 hours of flight, the safety membrane of the boost tank burst. Helium, designed for a single operation before landing on the moon, was warmed up even before the first correction after the accident. The pressure in the tank was constantly growing, and a breakthrough of the safety membrane was inevitable. The loss of boost gas meant that the lunar module's landing engine could no longer be started. But one more correction was needed. I had to use the lunar module orientation engines. Fortunately, a small impulse was needed, only 22 seconds of much less powerful orientation engines, and the correction for 137 hours of flight was successful.

Landing

Before landing, it was necessary to perform many operations. Firstly, it was necessary to transfer ballast to the command module - unnecessary things that would replace 45 kilograms of moonstones. Cameras, other small things, and a tablet that was planned to be left on the moon became such ballast - Lovell decided to take it as a souvenir.
Secondly, it was necessary to include the command module, and, given several days of downtime and the abundance of moisture on the walls, this was an exciting moment. Fortunately, the procedure was developed correctly, and the module returned to service for a short battery life. Then the service module was undocked. He was moving away, spinning slowly, and astronauts were surprised to see how much damage was caused by the accident - the whole panel was torn down:


Thirdly, it was necessary to undock the lunar module. The astronauts sadly saw off the module, which did not deliver them to the moon, but saved their lives.
And fourthly, it was necessary to verify the correct orientation of the command module. Since the final part of the flight took place in the shadow of the Earth, the time of the Earth’s coverage of the Moon was measured. The time coincided with the calculations of the MCC, the orientation was correct, and then the computers worked.

The cooling of the lunar module again managed to reduce the angle of entry into the atmosphere, so the ship passed dense layers of the atmosphere longer than usual without communication, which probably caused people to worry. And after the connection was restored, the last intrigue remained - whether the parachutes would work. Fortunately, they worked, and on the air the whole planet could rejoice for the successful return of the astronauts home.

The lessons

• «Аполлон-13» очень наглядно показал, к каким большим проблемам могут привести небольшие изменения спецификации, небольшие проблемы документации и небольшие недоработки в тестировании.
• Второй урок, который мы можем извлечь из этой истории — это то, что квалифицированная, сработавшаяся, мотивированная команда, с квалифицированным лидером сделает всё возможное, и даже чуть-чуть больше для успеха проекта, преодолевая самые разные проблемы.
• А вот фраза «Провал — это недопустимый вариант» сейчас иногда опасно неверно используется как своя противоположность. «Аполлон-13» успешно вернулся только потому, что огромные ресурсы были потрачены на отработку сценариев различных аварий, резервирование важных систем, и прочие меры по обеспечению безопасности. Сказать «Провал недопустим» можно только тогда, когда проблема уже есть, для воодушевления команды. Пытаться успокоить себя словами «провала не будет», игнорируя управление рисками и прочие меры обеспечения безопасности — это преступная небрежность.
• И последний урок, не из сферы управления проектами, а общефилософский. «Аполлон-13» не высадился на Луну, и, вроде бы, это поражение. Но то, какое количество трудностей было преодолено, какой профессионализм был показан людьми, и какое качество показала техника — это настоящий триумф миссии.

Epilogue and notes

This is a post from the “Space Accident Lessons” series. The first post of this series for those interested.
I understand very well that in this post there is a lot about technology, very little about people, and much remains behind the scenes. Compensate for this imbalance by watching the movie Apollo 13, it is worth a look, even if you are not particularly keen on astronautics.

Sources of information:

1. Wikipedia and the sources indicated in its articles.
2. Lovell Jim and Kluger Jeffrey, "Lost Moon: The Perilous Voyage of Apollo 13." There is a translation of Khartikov, but it is very uneven, often difficult places are transferred correctly, and then there is some kind of translation blunder. If there are no other translations, reading is recommended. There is an audiobook in English.
3. “Moon Machines”, Science Channel, TV series 2008, episode “Lunar Module”.
4. “When We Left Earth”, Discovery Channel, TV series 2008, fourth series.
5. "From the Earth to the Moon", HBO, TV series 1998.
6. JSC Digital Image collection - photo materials.
7. Apollo Expeditions To The Moon .
8. "" Failure to EXCLUDE from the list of features! " "- NK, 2006.

For navigation: posts by the tag "Space lessons"