NHTSA's serious mathematical errors allowed Tesla to declare autopilot safety.
The National Highway Traffic Safety Administration (NHTSA) is risking its reputation after a devastating report published by a small research and consulting firm called Quality Control Systems. This report is devoted to the analysis of the NHTSA report from 2017, as a result of which officials found out that the Tesla autopilot reduces the risk of getting into an accident by 40% (the diagram above).
The report was published only now, almost two years later. Quality Control Systems had to sue the NHTSA so that, in accordance with the Law on Freedom of Information, to obtain the data underlying the agency’s findings.
In its report, QCS talks about the shortcomings of the NHTSA methodology, which are serious enough to completely discredit the 40 percent figure that Tesla has quoted several times over the past two years, highlighting the safety benefits of the autopilot.
NHTSA studied autopilot safety after the accident with the death of Tesla owner Joshua Brown in 2016. The autopilot, or more precisely the Autosteer lane maintenance feature, was active during the crash, and Brown ignored several warnings to put his hands back on the steering wheel. Critics have wondered if the autopilot can reduce safety by encouraging the driver to pay less attention to the road.
However, NHTSA found out that Autosteer actually reduced the breakdown rate by 40%, so everything is fine. Tesla quoted this figure in her blog, where she argued for the protection of technology. A few weeks later, Ilon Musk even scolded reporters who were "fixated" on Tesla accidents, and did not tell that the autopilot objectively makes driving safer.
As it turns out, the arguments of Ilona Mask could arise as a result of the fatal error of Tesla experts in calculating statistics.
What is the point, if briefly. How did NHTSA evaluate transport safety? The most understandable and logical way. They asked Tesla for complete statistics on all vehicles sold between 2014 and 2016, and calculated the accident rate before and after installing Autosteer.
They chose this technique: they took the number of accidents before the autopilot was installed and divided them by the number of miles traveled before Autosteer was activated. Similarly, they took the number of accidents after installing the autopilot and also divided by the number of miles traveled after activating Autosteer.
Total Tesla provided data on 43,781 cars.
The only problem is that in the table from Tesla for most cars the data is incomplete. For example, it is not indicated at what point Autosteer was activated on the car. It turned out that for 29 051 cars in the database there are no required fields for such calculations.
When calculating, NHTSA counted all these cars as 100% equipped with Autosteer, as if they hadn’t traveled a single mile before activating this function. However, 18 accidents that occurred with the participation of these cars are recorded in the statistics to Autosteer , which is more than 20% of all 86 accidents that occurred with Tesla cars without the Autosteer function!
It is obvious that such a serious bias in the calculation of the final statistics greatly shifted the result in favor of the safety of Autosteer. How much is unknown. We cannot calculate real data for the entire sample, since most of the data is corrupted. However, QCS carried out similar calculations for 5714 cars, for which there was complete odometer information “before Autosteer” and “after Autosteer” without a gap between these two points (this is another shortcoming of the Tesla database: in many cars these two numbers did not match). So, on a normal sample, the calculation showed that the accident rate after installing Autosteer increased by 59%. Although this cannot be considered an objective figure, because 5714 cars represent too small a part of the Tesla car fleet and there is no way to know whether they are a representative sample.
Apparently, NHTSA has long noticed its mistake, just embarrassed to admit it. Back in May last year, the ministry distanced itself from its own report, calling it a “quick comparison”, which “does not evaluate the effectiveness of Autosteer technology”.
Why did they refuse to give QCS initial data for re-examination for so long? After all, the request for information was sent in February 2017, about a month after the publication of the original report. The agency could not immediately provide information because Tesla claimed data privacy.
Only during the trial, the court rejected NHTSA’s arguments on statistical confidentiality and ordered the information to be declassified.