Software for the Boeing-737 Max was written by outsourcers earning $ 9 per hour

In the midst of the crisis around the Boeing-737 Max, it still remains a mystery: how the company, famous for its careful design approach, made, apparently, children's mistakes in the development of software that led to two disasters with human casualties. Engineers working for the company for many years say the development was complicated due to the delegation of part of the work to low-paid contractors.

Software flaws may leave the planes chained to the ground for another month - this week, US regulators discovered additional problems. The software for the 737-Max series was written at a time when Boeing laid off experienced engineers and put pressure on suppliers.

Moreover, the icon of American aircraft construction and its subcontractors trusted temporary workers who earned only $ 9 per hour to develop and test their software. Often, these were workers from countries with undeveloped aircraft construction, namely from India.

“Yesterday's graduates, hired by the Indian software company HCL Technologies Ltd, occupy several rows of tables at the Boeing Field offices in Seattle (officially King County International Airport, at this airport Boeing has its own hangar and conducts aircraft tests - approx. Transl.)” says Mark Rabin, a former Boeing engineer who worked in the 737-Max series aircraft testing team.

HCL encoders are usually designed according to specifications sent from Boeing. But, according to Rabin, “this is a controversial decision, since it is much less effective than just letting Boeing engineers write code.” He recalls that “often it was necessary to redo everything several times, since the code was written incorrectly.”

Support from Indian companies may have brought other benefits. Over the past few years, Boeing has won several tenders for the supply of military and commercial aircraft to India, for example, a $ 22 billion contract for SpiceJet Ltd. This contract includes 100 737-Max 8 aircraft and is the largest order in the history of Indian Airlines, traditionally collaborating with Airbus.

According to the findings published on social networks, HCL engineers participated in the development and testing of software for PFD (Primary flight display, the primary flight display - approx. Transl.), And employees of another Indian company, Cyient Ltd., were engaged in instrumentation software intended for flight tests.

Costly delay

In one post, an HCL employee described his job responsibilities as follows: “I quickly made a crutch to solve the production problem and not delay the flight tests of the 737-Max (delaying each flight costs a huge amount to Boeing).”

Boeing said it did not trust HCL and Cyient engineers to develop the MCAS (Maneuvering Characteristics Augmentation System), which is associated with the crashes of the JT-610 Lion Air flight near Jakarta in October 2018 and the ET302 Ethiopian Airlines flight in Addis Ababa in March 2019. Also, according to Boeing, none of these companies is associated with a problem discovered after the disaster - the signal lamp in the cabin is not working for most buyers.

“Boeing has many years of experience working with suppliers and partners around the world,” says the company’s official. “Our main goal is to always be sure that our products are safe, of the highest quality and that they are complied with all the rules.”

In turn, HCL in an official statement states that “it has a strong and long-standing business relationship with Boeing and is proud of the work that the company has done for its customers. However, HCL does not comment on what kind of work it was. HCL is in no way associated with current problems with the 737 Max. ”

Recent emulator tests conducted by the US Federal Aviation Administration revealed that software problems are at a deeper level. Shares of the company this week fell in price after regulators discovered a problem with the microcircuit starting to give critical answers with a delay, if it is overloaded with requests.

The development of the 737 Max began 8 years ago, and the engineers who worked on it complained about pressure from managers. There were demands to limit the changes, potentially creating additional costs.

“Boeing did everything possible, everything you can imagine, to reduce costs, including moving development from Puget Sound (the region in Washington state where Boeing’s production facilities are located - approx. Transl.), Because it costs too expensive, ”says Rick Ludtke, a former flight test engineer who was laid off in 2017. “This can be understood from a business perspective. Gradually, over time, it became clear that this impaired the design ability of Puget Sound engineers. ”

Rabin, a former programmer who was laid off in 2015, recalls how one of the managers at the general meeting said that Boeing does not need seniors, as their products are already quite mature. “I was shocked that in a hall filled with a couple of hundreds of mostly senior engineers, we were seriously told that we were not needed ...”

A typical jet liner consists of millions of parts, and millions of lines of code, and Boeing has long passed a large part of the work for suppliers who simply follow the detailed drawings.

Starting with the 787 Dreamliner launched in 2004, Boeing has sought to increase profits by providing higher-level specifications instead of drawings, and then inviting suppliers to work out the parts themselves. The idea was “they are experts, you know, and they will take care of these things for us,” says Frank McCormick, a former flight test engineer who later worked as a consultant for regulators and manufacturers. “It was just stupid.”

An additional reason for transferring work abroad is sales. In exchange for the 11 billionth contract with Air India, signed in 2005, Boeing pledged to invest $ 1.7 billion in Indian companies. This, of course, was a boon to HCL, Cyient, and other companies whose programmers were widely used in the computer industry but were not yet involved in aircraft manufacturing.

Rockwell Collins, an electronics manufacturer for aircraft cockpits, was one of the first aircraft manufacturing companies to transfer much of its work to India, where, since 2000, HCL began testing their software. By 2010, HCL had over 400 people working on software development and testing for Rockwell Collins, based in offices in Chennai and Bangalore.

In the same year, Boeing, together with HCL, opened the so-called “center of excellence” in Chennai, saying the companies would work together “to create mission-critical flight test software.” In 2011, Boeing added Cyient (then known as Infotech) to its list of “vendors of the year” for designing, testing, and developing software for the 787 and 747-8 models from another center in Hyderabad.

Boeing’s competitors also rely in part on outsourcers. In addition to supporting sales (as mentioned above), aircraft manufacturing companies claim that distributed design teams are more efficient because they work around the clock. But outsourcing has long been a sore spot for some Boeing engineers, who, in addition to fear of losing their jobs, say that this led to problems with teamwork and errors.

Moscow errors

Boeing has also expanded its design center in Moscow. In 2008, during a meeting with the chief engineer responsible for the Boeing-787, one of the employees complained that he had sent drawings to the team 18 times to Russia before they realized that smoke detectors should be connected to the electrical system, said Cynthia Cole ( Cynthia Cole), a former Boeing engineer who led the union of engineers from 2006 to 2010.

“Design has begun to turn into cheap goods,” adds Vance Hilderman, co-founder of TekSci, a contracting services company that began to lose orders due to foreign competitors in the 2000s.

According to Hilderman, a thirty-year-old security engineer whose recent clients include major Boeing suppliers, US avionics companies, have moved more than 30% of their software development overseas over the past few years, compared to just 10% of European companies .

A strong dollar was the key to the attractiveness of this model. Engineers in India earned about $ 5 an hour, now it’s $ 9 or $ 10, compared to $ 35-40 for those in the United States on an H1B visa, adds Hilderman. But he explains to his customers that in reality, the low price per hour costs them $ 80 due to the need for control, and says that his company partially returns customers who need to fix bugs.

HCL, formerly known as Hindustan Computers, was founded in 1976 by billionaire Shiv Nadar and has annual sales of more than $ 8.6 billion. According to the company's vice president, Sukamal Banerjee, HCL is a global company with 18,000 employees in the United States and 15,000 in Europe, and has vast experience in computing. And this is why the order from Boeing won, and not at all because of the price. He bluntly states: “We have extensive experience in R&D (Research & Development, research and development work - approx. Transl.)”.

However, when working on the 787, HCL set a remarkable price for Boeing - for free, according to Sam Swaro, an assistant vice president who offered HCL services at a conference in San Diego hosted by Avionics International in June. He said the company didn’t take upfront payments for 787 and started invoicing only on the basis of sales after a few years - an “innovative business model” that he proposed to extend to other companies in the industry.

The Boeing-787 was put into operation in 2011, three years late, and exceeded the budget by billions of dollars, partly due to the confusion caused by the outsourcing strategy. Led by Dennis Muilenburg, a longtime Boeing engineer who became CEO in 2015, the company said it plans to return most of the work on the latest aircraft to its hands.

Engineering Swamp

Boeing-737 Max became a sales leader shortly after it was announced in 2011. But for ambitious engineers it was a bit of a “swamp,” says Peter Lemme, who designed the autopilot for the Boeing-767, now a consultant. The Boeing-737 Max was a 50-year-old design upgrade, and the changes had to be limited enough so that the Boeing could stamp new planes like hot cakes, with minor changes for assembly lines or airlines. “For an engineer, this is not the best job,” Lemm added.

Rockwell Collins, currently a division of United Technologies Corp., won a contract to supply cabin displays for the 737 Max and relied on HCL engineers in India, Iowa and Seattle. A spokeswoman for United Technologies declined to comment on the situation.

Contracting engineers from Cyient assisted with flight test equipment. Charles LoveJoy, a former Boeing employee, said US engineers had to double-check drawings made in India at 7:30 every morning. “We had problems with the Indian team. They complied with the requirements, but we could do better. ”

Numerous investigations, including a criminal investigation by the US Department of Justice, are trying to figure out how and when critical decisions were made regarding the 737 Max software. According to investigators, during the crash of Lion Air and Ethiopian Airlines aircraft, which killed 346 people, the MCAS system pushed the aircraft to uncontrolled dive due to poor data from one sensor.

According to Lemma, this design violates the basic principles of redundancy, which were unshakable for several generations of Boeing engineers. Apparently, no one has ever checked how the software will react in this situation. “It was a resounding failure,” he said. "Not one person, but many people should have thought about this problem."

Boeing also said that shortly after the start of the 737-Max shipments in 2017, they discovered that a warning light that could alert the crew of a sensor problem was not properly configured in the flight display software. A May Boeing statement explaining why the company didn’t inform regulators about it in time said that the engineers decided that this was not a security issue.

“The general management of the company,” the statement said, “did not participate in this audit.”

From the translator:after reading the article, I stopped wondering about the situation in my industry (e-commerce). If the industrial giants responsible for human lives have such a mess with processes, then what can we talk about in smaller offices. Well, I’ll add that Bloomberg certainly juggles (there was such an impression), since their task is hype and views, so what is written should be divided into two.

Messages about errors, typos and other problems are welcome.