Toyota unintended acceleration and the big bowl of "spaghetti" code (2013)
32 points
10 hours ago
| 9 comments
| safetyresearch.net
| HN
userbinator
5 hours ago
[-]
I still believe that the actual cause was tin whiskers, but all the RoHS lobbying buried the evidence.

http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA...

reply
Aloha
4 hours ago
[-]
I'm a very large believer that aggressive RoHS regulations have created more intractable problems than they have solved.
reply
ta20240528
4 hours ago
[-]
Sure, let's let them put lead back in paint.
reply
Aeglaecia
5 hours ago
[-]
im sure that any reasonably charismatic software engineer could scare the shit out of a judge/jury based on code analysis ... perusing the linked nasa document, recurrence odds of this particular failure mode do seem significantly greater than the odds of a cosmic bit flip ...
reply
thebruce87m
2 hours ago
[-]
> Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month.

https://www.scientificamerican.com/article/solar-storms-fast...

Just to give perspective on the bit flip probability. ECC ftw!

reply
nsoqm
2 hours ago
[-]
Is this was true, wouldn’t any modern computer would be crashing several times a month? And please don’t tell me “oh but it is”, because it is not.
reply
crote
1 hour ago
[-]
How many of those errors vould result in a full system crash, though? And how many of them are just going to cause silent and mostly-harmless data corruption?

After all, was the error in the first line a typo on my side, or a single-bit upset?

A while ago some researchers registered off-by-one-bit domain name typos, which due to physical key positioning were unlikely to be the result of genuine mistyping. I can't find a reference right now, but I recall them getting quite a lot of queries!

reply
ashirviskas
43 minutes ago
[-]
Most of the RAM may not be critical enough to crash the whole system. Just some random app you have open or a browser tab. So even if it is true, most bit flips should not crash a system.
reply
SoKamil
41 minutes ago
[-]
Write a Bit Flip simulator and report your observations ;)
reply
PhotonHunter
6 hours ago
[-]
Was there ever a recall of the ECU, and if not, why did the UA events go away? Were UA events more common at higher elevations, where there would be more cosmic ray activity?

This story is like Baba Yaga, it comes out from the shadows to scare people every now and then, but Barr’s theory has the interesting property that the ECU would be cleared by the error and so there could never be evidence of the event as he postulated.

reply
gnabgib
9 hours ago
[-]
Popular in 2015:

(96 points, 106 comments) https://news.ycombinator.com/item?id=10437117

(152 points, 145 comments) https://news.ycombinator.com/item?id=9643204

reply
qchris
6 hours ago
[-]
Related to [1]; this topic was discussed earlier today (perhaps inspiring this submission?) in a HN thread on C++ coding standards for the F-35 JSF (search "spaghetti").

[1] https://news.ycombinator.com/item?id=46183657

reply
supahfly_remix
8 hours ago
[-]
Does anyone know where one could obtain the firmware for this? It might be interesting to reverse engineer.
reply
altairprime
3 hours ago
[-]
It’s available in various archives of the Toyota TechStream pre-2024 editions, in some sort of weird encrypted file format that can be trivially decrypted; I haven’t tried myself but the ECU I work with isn’t encrypted in-vehicle. I’ve spent five or six years in Ghidra with various hybrid Subaru-Toyota ECUs from 2013-2020 and I wonder what kind of source control practices result in the massive function spaghetti that must have produced in this SH-2A code; I can see where Toyota bolted their direct injection runloop into Subaru’s. So, yeah, if you’re curious, the firmware’s out there, if you’ve got a few years to spare and an absolutely ridiculous amount of patience (and a solid grasp of CAN bus messaging protocols, which you’ll need to identify code blocks and variables and such!)

“The Car Hacker’s Handbook” may be of interest as a first step review, but honestly I just dove in with Ghidra and just .. didn’t ever stop. YMMV :)

reply
pengaru
4 hours ago
[-]
"supply voltage to the electronic control system was purposely lowered and perturbed to simulate bad alternator and/or battery system. The result from the manipulation of supply voltage was rather astonishing. The control systems seemed to work even with the perturbed supply voltage but not correctly. As a matter of fact, it seemed to cause the sudden unintended acceleration repeatedly. The supply voltage to the ECU can be disturbed by minor mishap in the alternator output function and possibly by the overload of ever increasing use of electric devices in the vehicle by the driver. In any case, the current study showed the reproduction of the sudden unintended acceleration when the supply voltage changes abruptly by sudden drop of the alternator output voltage or by overload of the electric devices."

https://www.sciencedirect.com/science/article/abs/pii/S03790...

reply
M95D
1 hour ago
[-]
Couldn't read the article, only the summary, but it sounds like glitching by the description. I expect the ECU lowers 12V to 5V or 3.3V by using a buck converter which includes a filter capacitor. To glitch the CPU, the 12V would need to drop well below 5V to have any effect. I don't see how this could happen. If the battery is weak enough to drop below 9V under any conditions other than a short, that car won't even start. My suspicion is that they glitched the ECU power supply directly, not the 12V input - the summary doesn't say.

My conclusion is that it's mosty (scientific) clickbait.

reply
Glawen
2 hours ago
[-]
So they drop the voltage to mimic an engine cranking, and they are surprised that the ECU behaves like it is cranking. When engine cranks, voltage drops below minimum voltage required by ECU to keep SW running (SW resets). To counter these, ECUs keep outputs to the max. Normally I would expect an electrical loop with the crank signal though.
reply
LanceH
7 hours ago
[-]
Ah yes, where Toyota was found guilty of not being a US company.

The only thing they did in the recall was the same floor mat anchor as so many other cases.

"NASA engineers found no electronic flaws in Toyota vehicles capable of producing the large throttle openings required to create dangerous high-speed unintended acceleration incidents. The two mechanical safety defects identified by NHTSA more than a year ago – “sticking” accelerator pedals and a design flaw that enabled accelerator pedals to become trapped by floor mats – remain the only known causes for these kinds of unsafe unintended acceleration incidents. Toyota has recalled nearly 8 million vehicles in the United States for these two defects." -- transportation.gov

Cosmic rays and other wild theories over the simple theory of driver error. Even with a stuck throttle, the brakes will still stop a car (not to mention shifting into neutral still works).

reply
Denatonium
7 hours ago
[-]
Not to mention that in an emergency, you can always turn the key to kill the engine, and then put it back into pre-igntion (to unlock the steering column). You won't have power-assisted braking or power-steering, but with a bit of adrenaline-fueled strength, it is definitely preferable to being in a car that is stuck accelerating.
reply
PhotonHunter
6 hours ago
[-]
The service brakes of anything short of a supercar are sufficient to stop a car at WOT.
reply
joecool1029
4 hours ago
[-]
Well, they didn’t here, also: https://en.wikipedia.org/wiki/Brake_fade
reply
jjav
2 hours ago
[-]
Brakes will always overpower the engine unless the braking system is severly damaged. This is simple physics. Cars decelerate far faster than they accelerate, which is to say, the brakes can generate far more horsepower than the engine can.

(Apparently the Rimac Nevera, with about 2000hp, can accelerate faster than it brakes. So that one might be the only exception. So unless you're driving a 2000hp car, the brakes will always overpower the engine, that is not debatable.)

Brake fade is irrelevant here. Brakes fade when overheated beyond their operating range, either due to fluid boiling and/or the pads overheating. This is nearly impossible to achieve in street driving, but can be experienced on the race track. None of the claimed acceleration accidents involved extreme repeated braking prior to the incident.

reply
mmooss
3 hours ago
[-]
That's a lot of thought and action in a unexpected and very fast-moving situation. I don't think that's a realistic expectation, except perhaps for trained personnel like airplane pilots.
reply
helterskelter
7 hours ago
[-]
Key?
reply
chneu
4 hours ago
[-]
Long press the start/stop button.
reply
laweijfmvo
7 hours ago
[-]
shift into neutral
reply
ehnto
6 hours ago
[-]
It was a 2005 model, so it should have been possible. However the article isn't super clear on where exactly the software is running, and the transmission controller and engine control unit can be interlinked in various ways. Especially more modern vehicles, it would be entirely possible to write code that disallowed shifting if it was an automatic. We have no idea just how poorly orchestrated this system was and what features were affected.

I don't know enough about 2005 Camry's though, so I wouldn't speculate much further than that.

reply
SV_BubbleTime
7 hours ago
[-]
Ok, but their engine controller was found to have 12,000 global variables and no one could ever say conclusively that the pedal issue was real or not.

The issue was not that no one found the flaw, it’s that no one could prove it wasn’t there.

reply
majormajor
5 hours ago
[-]
>The issue was not that no one found the flaw, it’s that no one could prove it wasn’t there.

Are cars since then required to have formally verified codebases, or is "no one could prove [there are no bugs]" still true?

---

Trying to evaluate what happened based on observation of events alone and stats, in absence of a formal proof of issue or non-issue... the cars didn't just disappear overnight so if there was such an issue... where did it go?

reply
Gibbon1
47 minutes ago
[-]
I'm suspicious that a lot of those 12,000 are constants. Just based on how I think those guys operate.

You and I would change a constant and recompile. They will just splat location 0x239A

reply
behringer
5 hours ago
[-]
Over 9000!!??
reply
jiggawatts
4 hours ago
[-]
Common in some real time systems with minimal or no usage of the stack.
reply
McGlockenshire
6 hours ago
[-]
> no one could ever say conclusively that the pedal issue was real or not

You should ask a mechanic's opinion.

reply
stackghost
8 hours ago
[-]
Safety Research Systems, the author of TFA, is a for-profit company whose income is based on lawsuits.

Make of that what you will.

reply
throwaway81523
3 hours ago
[-]
Philip Koopman was an expert witness against Toyota (which is a bit questionable to me) and he has some stuff on his website about the case too.

https://betterembsw.blogspot.com/search/label/Toyota%20UA

reply
fnord77
6 hours ago
[-]
> Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.
reply
jdlshore
6 hours ago
[-]
My understanding is that global variables are more common in embedded systems because it provides memory determinism.
reply
Glawen
2 hours ago
[-]
Yep that's the standard in embedded on bare metal without memory allocation. There is a mechanism in place to synchronise data during interrupts, so it's not really direct write. Usually also coupled with a two complement variable or similar to make sure memory is not corrupted for safety critical data.
reply
monegator
5 hours ago
[-]
nah, by looking ad other people's firmware it's because most of my embedded colleagues are goats stuck in pre-ANSI C. You can always declare a "global" static, so it's not on the stack or heap, and access that via functions.

Nothing wrong with source-file-level statics, you're bound to use them

reply