I was speaking with a 787 pilot last Sunday, I told him that the week before when I was at an airport there were two pilots sitting next to me talking about how "This is the third bloody 787 rescue we've had this month... I can't believe we had full engine and <I think he said auxiliary?> failure at the same time" - I asked him if this is common and he said "I hear of it, but I haven't had that many major failures, but lots of little things - last time I flew in from <city> a few moments after we touched down we lost auxiliary power from the rear engine, all the cabin lighting went black along with a number of other things, thankfully we'd already significantly reduced speed and were straight and already lost most of the speed we were carrying, so we were fine and taxied to the disembark location, they had it up and flying again within the day - but it certainly was disconcerting to say the least".
I will be slightly paraphrasing from memory there, but certainly was quite surprised how calm he was about the whole thing, there's no way I'd board one of those things.
There is also a RAT at the back that can be deployed to generate some power(~5-10 minutes max) in case of severe emergency in Air. It is what you hear sometimes, when the aircraft is making a very shrill noise flying over your head.
However, if it is not a test flight, a RAT deployment should make you very uncomfortable and worried…
I find it hard to believe that anyone reading this was within earshot of a plane in a severe emergency and heard this particular sound and since turbine engines are already quite shrill I am basically just sorta confused who your audience is for this suggestion.
Would you hear it from inside the plane? Even if it’s not as loud as the main engine, if it’s audible at all a lot of people would notice a change in pitch/tone. At least, I notice when the sounds the plane is making change even though I don’t know anything about the reason.
> After starting the descent, the flight crew made an announcement to the passengers; however, unbeknownst to the flight crew, the noise generated by the RAT (because of its high rotation speed) prevented the passengers and the cabin crew from hearing the announcement.
I feel like it's not the RAT you'll notice from inside the plane, it will be the silence from the engines. That combined with at least a momentary flicker of the lighting (I'm not sure if a RAT on a 787 will run cabin lighting but I doubt it), and you'll know.
Haha I didn’t parse it that way but I can see how you thought that upon rereading. I just want to understand why we would hear the RAT when there wasn’t an emergency overhead. I supposed planes regularly test them?
I'm not going to bother slogging through everything to be able to speak in specifics for every airplane ever built, but:
A RAT provides backup electrical and/or hydraulic power for control surfaces (and other goodies). A RAT would certainly be inspected during a heavy check and likely even during line checks (e.g. an "A" check or equivalent). How often is gonna depend on the airplane. But to suggest that a critical piece of equipment isn't checked regularly is just silly.
Additionally, it's pretty much guaranteed that if an airplane comes with a RAT the RAT is required to be functional for ETOPS flights. That alone means you're gonna be inspecting it pretty frequently. ETOPS certification has three parts: airplane, airline, and humans. You'd want to look at the ETOPS Maintenance Document at whatever airline to be sure.
Outside of Asia (where domestic widebody flights are still common) I'd guess many if not most 787 flights are ETOPS flights.
I thought the guy I was speaking mentioned something about instrumentation but I wasn't 100% sure and that sounded more serious so didn't mention it - but if the aux engine failing would do that - I guess that lines up!
Modern two-engine planes like the 787 have an auxiliary power unit (APU) in the tail. This is a small turbine that runs a generator and a pump for the hydraulics. It’s typically only turned on when the plane is on the ground, or if there’s an emergency in mid-air. It is also needed to start the main engines so if the APU is faulty the plane will probably be stuck where it is. In theory a 787 can take off with just one engine but this is not very safe and wouldn’t be done in all but the most exceptional circumstances.
There are variations on this depending on the plane model, of course. Some older planes can use an external starter for their engines, but I think that’s very rare now.
Aircraft with INOP APUs can generally be "air started" with a ground-based high-pressure air system. It's relatively common and I've been on a plane that had to do the procedure. It was entirely undramatic other than engines being started before the pushback, but I doubt most passengers even noticed.
Now, interestingly, the 787 is a "bleedless" aircraft, so it doesn't use high-pressure air from the APU to spool up the engines. I believe it can use its hefty bank of lithium-ion batteries to start its engines if the APU (and associated electrical generator) is INOP.
Not a pilot/engineer - just an enthusiast. Someone more au fait with the 787 might be able to correct me on the above.
My understanding is that there was a push to modify the U shaped tow trucks they use to position planes to have a battery powered system to start the engines.
The idea being that the APU isn't particularly clean burning, not compared to power plant emissions. It's been a long while since I've heard anything about that plan, for or against.
Interesting! Although it'd (presumably) only be useful for the 787, short of heavy modification to existing aircraft. Even the Airbus A350, an aircraft from the same era, uses a traditional bleed system. If planes continue down the bleedless route I can see it happening.
I’d note that commercial airplanes generally operate with 6-7 9’s of availability. For anyone that’s ever built a system with 5 9’s, this is impressive. In fact it’s impressive enough you probably don’t think twice about sleeping on a flight.
Six 9s would be half a minute of downtime per year.
I don't see how that is possible given the maintenance required for these planes. Even the simple A checks ground a plane for hours every couple hundred flights while D checks take months to complete every 6-10 years.
“We don’t wish to cause any alarm, but is there any one on board who is familiar with regular expressions, cron expressions and parameter expansion rules in bash?”
You joke but… There was an emergency nose high recovery out of San Diego airport where at one point the pilot had every passenger crowd into the first class cabin…
I know a commercial pilot who used that as a joke once and got in trouble. The plane in question had several pilots on it but the rest of the passengers didn’t find it funny for obvious reasons.
We’re just going to see more and more issues like this as more and more software is used in applications like this. I would be willing to bet that a Tesla would also spontaneously crash if left on for hundreds of hours, but they just rarely if ever are left on that long.
Ford F150 Lightning had a similar issue on a cross country road race some YT'ers put on. It died at 13% battery, Ford said it was due to not letting the truck rest.
Scary as it is, is there any reason for a passenger jet to have uptime if more than, say, 24hrs? Wouldn't you just switch it off and on again between every flight, regardless?
If this issue was in a car, we would never know as no one keeps their car running for 50 days straight.
Overnight, planes tend to be plugged in to ground power, to ventilate, keep the batteries charged, for the cleaning crews, etc. Most get rebooted once in a while, but it's always possible one won't be, hence the directive to be certain.
This particular problem has been known for years (the article is from 2020).
Unfortunately, an aircraft has no “reboot”. It is just a violent power cut. A lot of headache is introduced in non-critical aircraft software because there is no “graceful shutdown” or long power duration. Infact, certain hardware has an upper limit(much lower than a week) before which it needs one power cut(sometimes called power cycle) or it suffers from various buffer overflow, counter overflow and starts acting mysterious.
The testing for aerospace is extremely rigorous ... For DO-178C level A (Catastrophic failure that can cause a crash or many fatal injuries) we're estimating 2 years to do MC/DC test coverage metric of a fairly basic software system that has two mechanical backups. And that's above and beyond the extensive unit tests.
The main thing that gets checked is the worst-case timing analysis for every branch condition. And there are stack monitors to monitor if the stack is growing in size.
Look at Rapita System's website for more info ... we don't use them, but they explain it well.
It's amazing that's legal. Like, why do we accept software that does this? It can be done in such a way that these things don't happen.Put another way, why aren't the companies involved being fined and sued out of business? Why aren't their managers facing criminal negligence charges? It's outrageous.
Because there has never been a single commercial jetliner fatality caused by software in its intended operational domain failing to operate according to specification. That makes the commercial jetliner software development and deployment process by far the safest and highest reliability ever conceived by multiple orders of magnitude. We are talking in the 10-12 9s range.
And just to get ahead of: “Well what about the 737 MAX”, that was a system specification error, not due to “buggy” software failing to conform to its specification. The software did what it was supposed to do, but it should not have been designed to do that given the characteristics of the plane and the safety process around its usage.
>“Well what about the 737 MAX”, that was a system specification error, not due to “buggy” software failing to conform to its specification. The software did what it was supposed to do
Exactly: the system was designed to fly the plane into the ground if a single sensor was iced up, and that's exactly what the software did. Boeing really thought this system specification was a good idea.
So what should we make of these issues described in the article? When, not if, this kind of thing kills people will it be a specification error? Will we blame it on maintenance? Surely it can't be the software's fault!
First of all, who got blamed for the 737 MAX? Boeing did. This is one of the few industries where the responsibility does not get easily sloughed off.
Second, 787s have been flying for ~13 years and ~4.5 million flights [1]. Assuming they were unaware of the problem for the majority of that time, their unknowing maintenance and usage processes avoided critical failures due to the stated problems for a tremendous number of flights. Given they now know about it and are issuing a directive to enhance their processes to explicitly handle the problem, we can assume it is even less likely to occur than previously which was already experimentally determined to be ludicrously unlikely. Suing someone into oblivion for a error that has never manifested as a serious failure and that is exceedingly unlikely to manifest is a little excessive.
Third, they should be remediating problems as they arise balanced against the risks introduced by specification changes and against the alternative of other process modifications. Given Boeing’s other recent failings, they should be given strict scrutiny that they are faithfully following the traditional, highly effective remediation processes. It should only be worrisome if they are seeing disproportionately more problems than would be expected in a aircraft design of its age and are not remediating problems robustly and promptly.
> Suing someone into oblivion for an error that has never manifested as a serious failure and that is exceedingly unlikely to manifest is a little excessive.
I appreciate your point of view. The air travel industry is undeniably safe, moreso than any transportation system ever. By a large margin. On the other hand, it is possible to make software systems that do not have the defects described in the article. So how do we get to the place where we choose to build systems that behave correctly? I don't think we get there without severe penalties for failure.
> So how do we get to the place where we choose to build systems that behave correctly? I don't think we get there without severe penalties for failure.
What failure? The planes work. This is puritanism.
>The air travel industry is undeniably safe, moreso than any transportation system ever.
I disagree: the Japanese shinkansen bullet train system has never had a fatal accident, except for a single incident 30 years ago when someone was caught in a door and dragged 100 meters. No fatalities from collisions, derailings, etc., ever, since the 1960s. That's far safer than air travel could ever claim to be.
Even other train systems have better records than commercial aviation, in general. Plane crashes are rare these days, but they still happen once in a while, and the results are usually catastrophic.
Are planes safer than cars? Well of course, but that's a really, really low bar: cars are driven by all kinds of morons who frequently (esp. in the US) have little to no training or testing, are frequently distracted, don't have a copilot who can take over at any time, and are frequently operating in a very, very chaotic environment (like city streets). It's truly a wonder there aren't more fatal crashes. But safer than trains in general? I seriously doubt it.
Actually, the Shinkansen seems to average ~100 billion passenger-km per year [1] or ~60 billion passenger-miles per year. Using that as a overestimate for the last 60 years, that is a grand total of 3.6 trillion passenger-miles.
US commercial aviation averages ~1 trillion passenger-miles per year [2]. So if we compare the last 4 years of US aviation that is a comparable number of passenger-miles.
Over the last 4 years recorded on this dataset (2019-2022)[3] it looks like there were 5 fatalities total. Over the last 4 years recorded on this dataset (2018-2021)[4] it looks like there were 2 fatalities total.
So, while it does not appear to be safer, it is within a few factors on a passenger-mile basis. Furthermore, there are multiple periods of 4 trillion consecutive passenger-miles where there were 0 recorded accidents. It nowhere near obvious that it is “far safer than air travel could ever claim to be” and certainly a much closer race than you believed given your other assertions.
It works fine until it doesn't and people die. At which point the blame falls on the maintenance crew? That's wrong. And where there's smoke there's fire. If the software has this horrible bug, likely the broken culture that created it has written worse, more subtle bugs.
I agree completely with the first part.
But SWA-1380 was a commercial operating fatality in 2018. Not a crash into terrain, but the engine definitely crashed into the fuselage.
Many car's control units continue to run while the car is off. If you want to reboot your vehicle, you need to unplug the 12v battery for at least a minute.
On some cars (recent VWs in particular) when you plug the battery back in you need to twiddle some settings in the computer otherwise the charging circuit will fry the battery prematurely. We've gotten ahead of our skis with this nonsense, time to rein it in.
It's hard to imagine an interpretation of this behavior that doesn't involve manufacturers trying to punish independent mechanics and end users who service their own cars. Like, there's no way it's an "honest mistake", right?
BTW I have an AGM ("advanced glass mat") battery in my 1995 Toyota which has a completely analog charging system, and it doesn't get cooked, so it's not because there's something special about the battery.
My point is there was absolutely no need for the System Engineers to touch the charging system. The normal analog diode rectifier variety that has been standard since the 1960s is Good Enough. No "Innovation" Needed. Take your spacecamp nerds elsewhere.
Some of these planes are constantly flying as long as they're not in maintenance. A plane not in the air is a plane the company bought that's not currently generating profit.
unsure what you mean here. most of the systems go to a sleep state in modern vehicles ev or not. the 12v battery keeps only certain ECU's up - think ECUs that control alarm, lock and unlock state and any communication with the mobile app via LTE... but the rest of the systems are OFF, you don't want an EV battery to hit 0% and 12V to also hit 0% - that would basically make it a brick from what I understand- because EV's have contactors which need to shut for the battery to be 'engaged' the 12V battery controls these contactors.
A car with an enormous rack of high capacity batteries able to accelerate an 8000 pound object to 60mph and sustain that for hundreds of miles generally doesn’t depend on the backup battery for literally anything. It has so much excess energy storage in the form of electricity in the primary batteries it generally doesn’t power down the onboard computers at all.
Indeed when you get close to exhausting the main battery rack it starts selectively shutting down everything. I’ve never personally let mine get to 0% ever - but for instance a Tesla is continuously on, and if you use sentry mode it’s not just on but the GPU is constantly doing classification of the environment to determine if someone is prowling your vehicle.
Low voltage battery death in any EV essentially causes a brick. The only exception is some cars (I think Tesla does this?) keep their contactors closed all the time when the 12v is determined to be failing. It makes the drain at idle much higher, but then at least it can continue moving… as long as you don’t let the HV pack drain…
Very strange, because for me, an aircraft(medium) is never alive for more than 24h. A big one like 787 may be alive for up to 72h(assuming longer routes). 50 days for me would be a dream and a lot less headache but it is very expensive to keep an aircraft powered that long with ground power.
I know someone on the north slope of Alaska. He does not turn his personal truck off all winter. This is even more typical for semi trucks and whatnot around there.
Airlines will run the aircraft as long as possible. As another commenter mentioned, if an aircraft isn't in flight, it's in maintenance. All of these times it's on.
That's not what's alarming to me. What's alarming is that the plane could possibly be in a position to be continuously powered on for 51 days in the first place.
maintenance would happen with the aircraft in 'wheels on ground' mode but that may not mean all systems are turned off. I expect it's like a bug in the SMC on a computer. To really turn it off you have to do some magic.
I've flown with airlines before where there was a cascading delay due to a "plane deficit" at the terminal (not the technical term, that's my own). Not to say it's always uptime, but I imagine there are instances of constant uptime.
They can't just change things up on a dime like that. Even if it's 3 AM and most planes are sitting on the ground they can't just be used for your flight like that because they are all scheduled to take off in the morning rush a few hours later.
In the software world I call this an end user discovered issue. But when the issue involves a plane that is carrying actual souls. That can feel very scary.
I am sure this has been resolved by now since its from 2020.
That depends on how much code was having trouble, and what you mean by "resolved".
The safe option might be to avoid the situation, and I could imagine that even if there is a code update it might just make the plane balk at getting ready to take off after a certain amount of uptime.
Reminds me of the F-22 Raptor crossing the International Dateline error in 2007. They were flying a squadron of them from Hawaii to Japan. They crossed the IDL and all nav/fuel systems went down, as well as some communications gear.
They only made it back because they were flying with tankers at time, who led them back to base.
This is remarkably business-as-usual for airplane electronics.
As a more mundane example: the wifi on planes does temporary [edit: DHCP, not NAT] leases. But the system on many has expiration windows on the order of hours, possibly more than a day... Couple that with the number of passengers planes serve and busy routes can easily exhaust the lease pool.
The solution: there's a button the flight attendants can push to reboot the router, dumping the lease table.
To steelman the choice, the reserved IP /8 subnet is 10.x.x.x and is often used for corporate networks and other larger subnets experience similar usage. People on the plane using WiFi are likely to access their corporate networks via VPN, potentially causing routing issues.
Users VPNing into the reused address space for their own home VPN are probably knowledgeable enough to figure out what is going on and a small enough user base to not care about.
I'm no network guy so someone please explain why using 10.x.x.x. on a plane might "potentially cause routing issues"? It doesn't jive with what I understand about unrouteable address spaces. Is the 10.x.x.x space somehow different than the 192.168.x.x space that millions of people use VPN's out of every day (basically every WFH person on their cheap NAT'd home Wifi)?
Had a similar problem to this many years ago. Happened every 24 days approximately and lost one user setting. Had a logic analyser connected to it for days trying to reproduce the issue in some way. Went to go for a piss and get a coffee one afternoon and came back and there it was triggered!
What happened? Well it turns out there was a timer that no one used that overflowed and caused an interrupt which wasn’t handled any more, the interrupt handler fell through, caused a halt and the WDT fired fire rebooting it and some idiot hadn’t stored that one setting in the NVRAM.
So then we had more problems. 5000 things with EPROMs in that were rebooting every 24 days which were spread all over the planet. Many questions to ask over how the hell it ended up like that.
I hope people are asking these sorts of questions at Boeing.
Edit: also the source code we had did not match what was on the devices. Turned out the engineer who provided the hex file hadn’t copied that code to the file server and had left a year before hand. We didn’t find that until the WDT fired and piqued our interest and could reproduce it on the dev board because the software was different (should have checked that past the label on the ROM which was wrong!)
Not getting it.. yeah the famous 32 bit ms overflow after 49 something days. But why then 51 here? Shouldn't they be required to reboot after 49 days please please? :D
It's possible to run tasks instead of starting every second, starting one second after the previous iteration finishes.
So if you have something that checks the system health every millisecond, and keeps a count instead of a duration, then if it takes a couple microseconds to complete you might get something less than 86 million ticks per day instead of 86.4 million.
The OS used on the 787 has a hard real-time scheduler. Tasks are started up at a specific frequency (set per task), run to completion or to the end of their time slot (set per task) and terminated. We had, IIRC, a strict 100ms slot for our bit of LRU software to do everything and it would be launched every 1s (from memory, that was 15 years ago). Information could be stored between executions so partial completion is something you could handle if needed by storing state information and using it at the start of the next iteration (we didn't need that, our tasks finished in the slot).
You don't base the start of a future task on the end of the prior one, you base it on a fixed clock for these kinds of systems.
This company just can’t stay out of the news. Their planes are trash. Software is straight garbage. Many people have died because of this company and suffered undue stress/anxiety because of the massive dip in quality.
Boeing engineers/builders caught on audio stating they wouldn’t be caught dead in their own planes unless feeling suicidal.
The company definitely can't stay out of the news and it's gone downhill over the recent years but you've picked an interesting post to lament about those on. The news they can't stay out of is over 4 years old in this case. The model of plane it's about (787) has never had a single fatality despite >15 years of operations and >1,000 units operating today. In all, deaths are probably the worst possible metric to berate Boeing on - including every death (e.g. hijackings, not just engineering failures) their popular 747 line has had comes to <6,000 fatalities despite carrying billions of passengers over a period of >50 years.
Despite their ever increasing incompetence on delivery speed, test compliance, and innovation... commercial air travel with Boeing (and other major air manufacturers) has always been one of, if not the, safest mechanisms of travel we've ever executed on. Particularly the last 5 years have been the safest period in terms of air travel deaths or injuries.
None of that means we shouldn't criticize Boeing by any means, just that doing it over perceived death and accident counts because of what news headlines imply is complete nonsense in terms of actual numbers no matter how you slice it. It's important those kinds of things are reported but it's equally important to not get swept up in paranoia over it.
Agreed, my 737 fears were relieved by researching how many of them are in the air at any moment, how many millions of trips they fly each year, how old airframes can get before they get retired, etc. Even the "worse" models are feats of engineering.
I was speaking with a 787 pilot last Sunday, I told him that the week before when I was at an airport there were two pilots sitting next to me talking about how "This is the third bloody 787 rescue we've had this month... I can't believe we had full engine and <I think he said auxiliary?> failure at the same time" - I asked him if this is common and he said "I hear of it, but I haven't had that many major failures, but lots of little things - last time I flew in from <city> a few moments after we touched down we lost auxiliary power from the rear engine, all the cabin lighting went black along with a number of other things, thankfully we'd already significantly reduced speed and were straight and already lost most of the speed we were carrying, so we were fine and taxied to the disembark location, they had it up and flying again within the day - but it certainly was disconcerting to say the least".
I will be slightly paraphrasing from memory there, but certainly was quite surprised how calm he was about the whole thing, there's no way I'd board one of those things.
APU failure maybe? That would be troublesome indeed; with no engines and no APU you'd lose most instrumentation and a lot of the hydraulics.
There is also a RAT at the back that can be deployed to generate some power(~5-10 minutes max) in case of severe emergency in Air. It is what you hear sometimes, when the aircraft is making a very shrill noise flying over your head.
However, if it is not a test flight, a RAT deployment should make you very uncomfortable and worried…
[delayed]
https://en.wikipedia.org/wiki/Ram_air_turbine
I find it hard to believe that anyone reading this was within earshot of a plane in a severe emergency and heard this particular sound and since turbine engines are already quite shrill I am basically just sorta confused who your audience is for this suggestion.
Indeed unlikely to hear RAT deployed due to emergency. But they do deploy it sometimes on test flights after maintenance.
The RAT makes an extremely distinctive sound. You'd recognize it nearly instantly. However the RAT will not power everything.
https://www.youtube.com/watch?v=KzejbxNj1hY
That's cute, it sounds like a little Cessna
Would you hear it from inside the plane? Even if it’s not as loud as the main engine, if it’s audible at all a lot of people would notice a change in pitch/tone. At least, I notice when the sounds the plane is making change even though I don’t know anything about the reason.
It's apparently quite loud
> After starting the descent, the flight crew made an announcement to the passengers; however, unbeknownst to the flight crew, the noise generated by the RAT (because of its high rotation speed) prevented the passengers and the cabin crew from hearing the announcement.
https://asn.flightsafety.org/wikibase/187755
I feel like it's not the RAT you'll notice from inside the plane, it will be the silence from the engines. That combined with at least a momentary flicker of the lighting (I'm not sure if a RAT on a 787 will run cabin lighting but I doubt it), and you'll know.
Username does not check out.
Jokes aside... I'm certainly part of the intended audience: point me at an interesting rabbit hole, and there I gooo.
Haha I didn’t parse it that way but I can see how you thought that upon rereading. I just want to understand why we would hear the RAT when there wasn’t an emergency overhead. I supposed planes regularly test them?
They don’t.
I'm not going to bother slogging through everything to be able to speak in specifics for every airplane ever built, but:
A RAT provides backup electrical and/or hydraulic power for control surfaces (and other goodies). A RAT would certainly be inspected during a heavy check and likely even during line checks (e.g. an "A" check or equivalent). How often is gonna depend on the airplane. But to suggest that a critical piece of equipment isn't checked regularly is just silly.
Additionally, it's pretty much guaranteed that if an airplane comes with a RAT the RAT is required to be functional for ETOPS flights. That alone means you're gonna be inspecting it pretty frequently. ETOPS certification has three parts: airplane, airline, and humans. You'd want to look at the ETOPS Maintenance Document at whatever airline to be sure.
Outside of Asia (where domestic widebody flights are still common) I'd guess many if not most 787 flights are ETOPS flights.
I thought the guy I was speaking mentioned something about instrumentation but I wasn't 100% sure and that sounded more serious so didn't mention it - but if the aux engine failing would do that - I guess that lines up!
Modern two-engine planes like the 787 have an auxiliary power unit (APU) in the tail. This is a small turbine that runs a generator and a pump for the hydraulics. It’s typically only turned on when the plane is on the ground, or if there’s an emergency in mid-air. It is also needed to start the main engines so if the APU is faulty the plane will probably be stuck where it is. In theory a 787 can take off with just one engine but this is not very safe and wouldn’t be done in all but the most exceptional circumstances.
There are variations on this depending on the plane model, of course. Some older planes can use an external starter for their engines, but I think that’s very rare now.
Aircraft with INOP APUs can generally be "air started" with a ground-based high-pressure air system. It's relatively common and I've been on a plane that had to do the procedure. It was entirely undramatic other than engines being started before the pushback, but I doubt most passengers even noticed.
Now, interestingly, the 787 is a "bleedless" aircraft, so it doesn't use high-pressure air from the APU to spool up the engines. I believe it can use its hefty bank of lithium-ion batteries to start its engines if the APU (and associated electrical generator) is INOP.
Not a pilot/engineer - just an enthusiast. Someone more au fait with the 787 might be able to correct me on the above.
My understanding is that there was a push to modify the U shaped tow trucks they use to position planes to have a battery powered system to start the engines.
The idea being that the APU isn't particularly clean burning, not compared to power plant emissions. It's been a long while since I've heard anything about that plan, for or against.
Interesting! Although it'd (presumably) only be useful for the 787, short of heavy modification to existing aircraft. Even the Airbus A350, an aircraft from the same era, uses a traditional bleed system. If planes continue down the bleedless route I can see it happening.
Yeah the 787 can be started electrically but it takes a ton of juice.
https://www.youtube.com/watch?v=1W_RtawHVvw
Previous: https://news.ycombinator.com/item?id=22761395 https://news.ycombinator.com/item?id=33233827
More interesting, a root cause analysis: https://news.ycombinator.com/item?id=33239443 https://ioactive.com/reverse-engineers-perspective-on-the-bo...
The 47 bit timestamp at 32MHz would explain the duration (Though not why it isn't 33MHz?).
Thanks! Macroexpanded:
Reverse Engineer’s Perspective on the Boeing 787 ‘51 days’ Directive - https://news.ycombinator.com/item?id=33239443 - Oct 2022 (55 comments)
Boeing 787s must be rebooted every 51 days to prevent 'misleading data' (2020) - https://news.ycombinator.com/item?id=33233827 - Oct 2022 (140 comments)
Boeing 787s must be turned off and on every 51 days (2020) - https://news.ycombinator.com/item?id=27117320 - May 2021 (42 comments)
Boeing 787s must be turned off and on every 51 days - https://news.ycombinator.com/item?id=27111650 - May 2021 (4 comments)
Boeing 787s must be turned off and on every 51 days to prevent 'misleading data' - https://news.ycombinator.com/item?id=22761395 - April 2020 (152 comments)
I’d note that commercial airplanes generally operate with 6-7 9’s of availability. For anyone that’s ever built a system with 5 9’s, this is impressive. In fact it’s impressive enough you probably don’t think twice about sleeping on a flight.
Six 9s would be half a minute of downtime per year.
I don't see how that is possible given the maintenance required for these planes. Even the simple A checks ground a plane for hours every couple hundred flights while D checks take months to complete every 6-10 years.
Edit: minute not hour
By my math, six 9's is 30 seconds, not 30 minutes?
(1 - 0.999999) * (60 * 24 * 365)
EDIT: This chart agrees: https://en.wikipedia.org/wiki/High_availability#Percentage_c...
You are correct. Forgot what my units were in bc.
If something goes wrong, does it matter whether you are asleep or awake?
Only when a flight attendant is asking on the intercom: "We don't mean to alarm anyone, but is anyone on board a pilot?" and you happen to be one.
It's entirely a different kind of flying
All together
“We don’t wish to cause any alarm, but is there any one on board who is familiar with regular expressions, cron expressions and parameter expansion rules in bash?”
Several overweight men stand up and walk towards the cabin, immediately throwing off the weight distribution and the plane plummets.
You joke but… There was an emergency nose high recovery out of San Diego airport where at one point the pilot had every passenger crowd into the first class cabin…
The flight was saved.
I know a commercial pilot who used that as a joke once and got in trouble. The plane in question had several pilots on it but the rest of the passengers didn’t find it funny for obvious reasons.
Hopefully you didn't have the fish.
woosh!
Airbus A350s had the same issue: https://www.theregister.com/2019/07/25/a350_power_cycle_soft...
We’re just going to see more and more issues like this as more and more software is used in applications like this. I would be willing to bet that a Tesla would also spontaneously crash if left on for hundreds of hours, but they just rarely if ever are left on that long.
Ford F150 Lightning had a similar issue on a cross country road race some YT'ers put on. It died at 13% battery, Ford said it was due to not letting the truck rest.
Scary as it is, is there any reason for a passenger jet to have uptime if more than, say, 24hrs? Wouldn't you just switch it off and on again between every flight, regardless?
If this issue was in a car, we would never know as no one keeps their car running for 50 days straight.
Overnight, planes tend to be plugged in to ground power, to ventilate, keep the batteries charged, for the cleaning crews, etc. Most get rebooted once in a while, but it's always possible one won't be, hence the directive to be certain.
This particular problem has been known for years (the article is from 2020).
Unfortunately, an aircraft has no “reboot”. It is just a violent power cut. A lot of headache is introduced in non-critical aircraft software because there is no “graceful shutdown” or long power duration. Infact, certain hardware has an upper limit(much lower than a week) before which it needs one power cut(sometimes called power cycle) or it suffers from various buffer overflow, counter overflow and starts acting mysterious.
This has to be a joke right ?
You're telling me Aerospace's "real engineering-level" is worse than something a sophomore can cook up ?
The testing for aerospace is extremely rigorous ... For DO-178C level A (Catastrophic failure that can cause a crash or many fatal injuries) we're estimating 2 years to do MC/DC test coverage metric of a fairly basic software system that has two mechanical backups. And that's above and beyond the extensive unit tests.
The main thing that gets checked is the worst-case timing analysis for every branch condition. And there are stack monitors to monitor if the stack is growing in size.
Look at Rapita System's website for more info ... we don't use them, but they explain it well.
> Unfortunately, an aircraft has no “reboot”. It is just a violent power cut.
That’s a reboot.
It's amazing that's legal. Like, why do we accept software that does this? It can be done in such a way that these things don't happen.Put another way, why aren't the companies involved being fined and sued out of business? Why aren't their managers facing criminal negligence charges? It's outrageous.
Because there has never been a single commercial jetliner fatality caused by software in its intended operational domain failing to operate according to specification. That makes the commercial jetliner software development and deployment process by far the safest and highest reliability ever conceived by multiple orders of magnitude. We are talking in the 10-12 9s range.
And just to get ahead of: “Well what about the 737 MAX”, that was a system specification error, not due to “buggy” software failing to conform to its specification. The software did what it was supposed to do, but it should not have been designed to do that given the characteristics of the plane and the safety process around its usage.
>“Well what about the 737 MAX”, that was a system specification error, not due to “buggy” software failing to conform to its specification. The software did what it was supposed to do
Exactly: the system was designed to fly the plane into the ground if a single sensor was iced up, and that's exactly what the software did. Boeing really thought this system specification was a good idea.
So what should we make of these issues described in the article? When, not if, this kind of thing kills people will it be a specification error? Will we blame it on maintenance? Surely it can't be the software's fault!
First of all, who got blamed for the 737 MAX? Boeing did. This is one of the few industries where the responsibility does not get easily sloughed off.
Second, 787s have been flying for ~13 years and ~4.5 million flights [1]. Assuming they were unaware of the problem for the majority of that time, their unknowing maintenance and usage processes avoided critical failures due to the stated problems for a tremendous number of flights. Given they now know about it and are issuing a directive to enhance their processes to explicitly handle the problem, we can assume it is even less likely to occur than previously which was already experimentally determined to be ludicrously unlikely. Suing someone into oblivion for a error that has never manifested as a serious failure and that is exceedingly unlikely to manifest is a little excessive.
Third, they should be remediating problems as they arise balanced against the risks introduced by specification changes and against the alternative of other process modifications. Given Boeing’s other recent failings, they should be given strict scrutiny that they are faithfully following the traditional, highly effective remediation processes. It should only be worrisome if they are seeing disproportionately more problems than would be expected in a aircraft design of its age and are not remediating problems robustly and promptly.
[1] https://www.boeing.com/commercial/787#overview
> Suing someone into oblivion for an error that has never manifested as a serious failure and that is exceedingly unlikely to manifest is a little excessive.
I appreciate your point of view. The air travel industry is undeniably safe, moreso than any transportation system ever. By a large margin. On the other hand, it is possible to make software systems that do not have the defects described in the article. So how do we get to the place where we choose to build systems that behave correctly? I don't think we get there without severe penalties for failure.
> So how do we get to the place where we choose to build systems that behave correctly? I don't think we get there without severe penalties for failure.
What failure? The planes work. This is puritanism.
>The air travel industry is undeniably safe, moreso than any transportation system ever.
I disagree: the Japanese shinkansen bullet train system has never had a fatal accident, except for a single incident 30 years ago when someone was caught in a door and dragged 100 meters. No fatalities from collisions, derailings, etc., ever, since the 1960s. That's far safer than air travel could ever claim to be.
Even other train systems have better records than commercial aviation, in general. Plane crashes are rare these days, but they still happen once in a while, and the results are usually catastrophic.
Are planes safer than cars? Well of course, but that's a really, really low bar: cars are driven by all kinds of morons who frequently (esp. in the US) have little to no training or testing, are frequently distracted, don't have a copilot who can take over at any time, and are frequently operating in a very, very chaotic environment (like city streets). It's truly a wonder there aren't more fatal crashes. But safer than trains in general? I seriously doubt it.
Actually, the Shinkansen seems to average ~100 billion passenger-km per year [1] or ~60 billion passenger-miles per year. Using that as a overestimate for the last 60 years, that is a grand total of 3.6 trillion passenger-miles.
US commercial aviation averages ~1 trillion passenger-miles per year [2]. So if we compare the last 4 years of US aviation that is a comparable number of passenger-miles.
Over the last 4 years recorded on this dataset (2019-2022)[3] it looks like there were 5 fatalities total. Over the last 4 years recorded on this dataset (2018-2021)[4] it looks like there were 2 fatalities total.
So, while it does not appear to be safer, it is within a few factors on a passenger-mile basis. Furthermore, there are multiple periods of 4 trillion consecutive passenger-miles where there were 0 recorded accidents. It nowhere near obvious that it is “far safer than air travel could ever claim to be” and certainly a much closer race than you believed given your other assertions.
[1] https://www.statista.com/statistics/1262752/japan-jr-high-sp...
[2] https://www.transtats.bts.gov/traffic/
[3] https://www.bts.gov/content/us-air-carrier-safety-data
[4] https://www.airlines.org/dataset/safety-record-of-u-s-air-ca...
Because it works fine. A maintenance tech gets one extra line item on the weekly or monthly inspection checklist.
It works fine until it doesn't and people die. At which point the blame falls on the maintenance crew? That's wrong. And where there's smoke there's fire. If the software has this horrible bug, likely the broken culture that created it has written worse, more subtle bugs.
Commercial air travel in the US is incredibly safe. The last fatal crash was in 2009.
I agree completely with the first part. But SWA-1380 was a commercial operating fatality in 2018. Not a crash into terrain, but the engine definitely crashed into the fuselage.
>an aircraft has no “reboot”. It is just a violent power cut
Guess how I typically reboot things :)
By traveling to Mexico and laying out bait along the migratory path of the butterflies?
Many car's control units continue to run while the car is off. If you want to reboot your vehicle, you need to unplug the 12v battery for at least a minute.
On some cars (recent VWs in particular) when you plug the battery back in you need to twiddle some settings in the computer otherwise the charging circuit will fry the battery prematurely. We've gotten ahead of our skis with this nonsense, time to rein it in.
This issue is notorious for BMW cars. You have to notify the ECU each time you install a new battery.
It's hard to imagine an interpretation of this behavior that doesn't involve manufacturers trying to punish independent mechanics and end users who service their own cars. Like, there's no way it's an "honest mistake", right?
BTW I have an AGM ("advanced glass mat") battery in my 1995 Toyota which has a completely analog charging system, and it doesn't get cooked, so it's not because there's something special about the battery.
Don't attribute to malice what can easily be explained by overstressed Systems Engineers trying to resolve multiple conflicting Requirements.
My point is there was absolutely no need for the System Engineers to touch the charging system. The normal analog diode rectifier variety that has been standard since the 1960s is Good Enough. No "Innovation" Needed. Take your spacecamp nerds elsewhere.
Sure, you MUST know better than the BMW engineers who designed the feature we have zero information about.
Ahhh, "program a new battery" $400 please.
Rein. It’s about controlling a horse, not an entire nation.
Thanks, I blame phone autocorrect
It's always champing at the bit
Some of these planes are constantly flying as long as they're not in maintenance. A plane not in the air is a plane the company bought that's not currently generating profit.
I’ll bet you the typical EV stays powered on 24/7 with reboots around OTA updates.
unsure what you mean here. most of the systems go to a sleep state in modern vehicles ev or not. the 12v battery keeps only certain ECU's up - think ECUs that control alarm, lock and unlock state and any communication with the mobile app via LTE... but the rest of the systems are OFF, you don't want an EV battery to hit 0% and 12V to also hit 0% - that would basically make it a brick from what I understand- because EV's have contactors which need to shut for the battery to be 'engaged' the 12V battery controls these contactors.
A car with an enormous rack of high capacity batteries able to accelerate an 8000 pound object to 60mph and sustain that for hundreds of miles generally doesn’t depend on the backup battery for literally anything. It has so much excess energy storage in the form of electricity in the primary batteries it generally doesn’t power down the onboard computers at all.
Indeed when you get close to exhausting the main battery rack it starts selectively shutting down everything. I’ve never personally let mine get to 0% ever - but for instance a Tesla is continuously on, and if you use sentry mode it’s not just on but the GPU is constantly doing classification of the environment to determine if someone is prowling your vehicle.
Low voltage battery death in any EV essentially causes a brick. The only exception is some cars (I think Tesla does this?) keep their contactors closed all the time when the 12v is determined to be failing. It makes the drain at idle much higher, but then at least it can continue moving… as long as you don’t let the HV pack drain…
Very strange, because for me, an aircraft(medium) is never alive for more than 24h. A big one like 787 may be alive for up to 72h(assuming longer routes). 50 days for me would be a dream and a lot less headache but it is very expensive to keep an aircraft powered that long with ground power.
> it is very expensive to keep an aircraft powered that long with ground power.
Why do you say this?
I know someone on the north slope of Alaska. He does not turn his personal truck off all winter. This is even more typical for semi trucks and whatnot around there.
Airlines will run the aircraft as long as possible. As another commenter mentioned, if an aircraft isn't in flight, it's in maintenance. All of these times it's on.
It's another thing on a checklist that can go wrong.
If problems persist after rebooting, you may need to use a giant paperclip to perform a reset.
(2020)
It sounds like my random Raspberry Pi sitting somewhere in my server room that has to be restarted every <x> weeks.
> This alarming-sounding situation
That's not what's alarming to me. What's alarming is that the plane could possibly be in a position to be continuously powered on for 51 days in the first place.
When a minute of downtime costs thousands, why wouldn't you expect planes to be in constant utilization?
> why wouldn't you expect planes to be in constant utilization?
They require weekly maintenance which takes them out of service for at least 12 hours.
What we may of as 'constant utilization' is quite different in a regulated fleet environment like airlines.
maintenance would happen with the aircraft in 'wheels on ground' mode but that may not mean all systems are turned off. I expect it's like a bug in the SMC on a computer. To really turn it off you have to do some magic.
"Constant utilization" means "they aren't sitting idle", not "they aren't undergoing necessary maintenance ever".
The number of flights varies a lot by time of day, so there is nothing close to constant utilization.
There's not much reason to turn them off outside of maintenance. When they're parked, they're connected to grid power.
Airliners are regularly and routinely shut down. "Cold and dark" is a common startup procedure for the first flight of the day.
A parked Aircraft is not kept powered when there are no maintenance or other routine(cleaning/checks/certification/preparation/restocking etc.)
It is very surprising that how a lot of comments here claim the contrary.
Even when parked for next flight, until resupply and cargo routines are declared, it is also not powered.
I've flown with airlines before where there was a cascading delay due to a "plane deficit" at the terminal (not the technical term, that's my own). Not to say it's always uptime, but I imagine there are instances of constant uptime.
They can't just change things up on a dime like that. Even if it's 3 AM and most planes are sitting on the ground they can't just be used for your flight like that because they are all scheduled to take off in the morning rush a few hours later.
In the software world I call this an end user discovered issue. But when the issue involves a plane that is carrying actual souls. That can feel very scary.
I am sure this has been resolved by now since its from 2020.
I don't think airplane software ships updates the way npm packages do. I would be more surprised if this is fixed.
I think from the point of view of Boeing, the FAA and the airlines, "put it in our maintenance checklist to reboot every 51 days" is a fix.
With that framing, this sounds like one of the easiest maintenance tasks imaginable. No wrenches or grease involved.
> I don't think airplane software ships updates the way npm packages do.
I'd ideally like to sleep tonight, thanks.
That depends on how much code was having trouble, and what you mean by "resolved".
The safe option might be to avoid the situation, and I could imagine that even if there is a code update it might just make the plane balk at getting ready to take off after a certain amount of uptime.
Scary would be right.
Reminds me of the F-22 Raptor crossing the International Dateline error in 2007. They were flying a squadron of them from Hawaii to Japan. They crossed the IDL and all nav/fuel systems went down, as well as some communications gear.
They only made it back because they were flying with tankers at time, who led them back to base.
This is remarkably business-as-usual for airplane electronics.
As a more mundane example: the wifi on planes does temporary [edit: DHCP, not NAT] leases. But the system on many has expiration windows on the order of hours, possibly more than a day... Couple that with the number of passengers planes serve and busy routes can easily exhaust the lease pool.
The solution: there's a button the flight attendants can push to reboot the router, dumping the lease table.
Nitpicking here, but you mean DHCP rather than NAT, right?
Yes; thank you.
Even with super long leases, couldn’t they just have a larger subnet? A /8 oughta do it.
But I guess we’re talking about the same people who made the mistake in the first place…
To steelman the choice, the reserved IP /8 subnet is 10.x.x.x and is often used for corporate networks and other larger subnets experience similar usage. People on the plane using WiFi are likely to access their corporate networks via VPN, potentially causing routing issues.
Users VPNing into the reused address space for their own home VPN are probably knowledgeable enough to figure out what is going on and a small enough user base to not care about.
I'm no network guy so someone please explain why using 10.x.x.x. on a plane might "potentially cause routing issues"? It doesn't jive with what I understand about unrouteable address spaces. Is the 10.x.x.x space somehow different than the 192.168.x.x space that millions of people use VPN's out of every day (basically every WFH person on their cheap NAT'd home Wifi)?
Couldn't we spare a single extra /8 for airplanes to use?
Though I suppose it's not worth it when you can hit 'reboot'.
How about "we" use IPv6 instead, and nobody runs out of address space ever again?
Had a similar problem to this many years ago. Happened every 24 days approximately and lost one user setting. Had a logic analyser connected to it for days trying to reproduce the issue in some way. Went to go for a piss and get a coffee one afternoon and came back and there it was triggered!
What happened? Well it turns out there was a timer that no one used that overflowed and caused an interrupt which wasn’t handled any more, the interrupt handler fell through, caused a halt and the WDT fired fire rebooting it and some idiot hadn’t stored that one setting in the NVRAM.
So then we had more problems. 5000 things with EPROMs in that were rebooting every 24 days which were spread all over the planet. Many questions to ask over how the hell it ended up like that.
I hope people are asking these sorts of questions at Boeing.
Edit: also the source code we had did not match what was on the devices. Turned out the engineer who provided the hex file hadn’t copied that code to the file server and had left a year before hand. We didn’t find that until the WDT fired and piqued our interest and could reproduce it on the dev board because the software was different (should have checked that past the label on the ROM which was wrong!)
This was news in 2020. Has it been fixed?
And 4.5 years later, what's new?
51 days * 86400 seconds * 1000
=> 4406400000
2^32
=> 4294967296
the coincidence seems unlikely, it's basically ~~5 hours and a half~~ 30 hours of difference if one has a 1-ms counter increment
Watch Windows 95 crash live as it exceeds 49.7 days uptime https://news.ycombinator.com/item?id=28340101
Must be a northwest washington thing.
It's a day and a half difference, and since 2^32 is the smaller number that would be pretty catastrophic. Pretty likely it's coincidence.
Where did you get 5 hours and a half? It seems to be closer to 31 hours:
from me typing too quickly in bc, apparently :')
Not getting it.. yeah the famous 32 bit ms overflow after 49 something days. But why then 51 here? Shouldn't they be required to reboot after 49 days please please? :D
Possibly cumulative error in the timing source?
It's possible to run tasks instead of starting every second, starting one second after the previous iteration finishes.
So if you have something that checks the system health every millisecond, and keeps a count instead of a duration, then if it takes a couple microseconds to complete you might get something less than 86 million ticks per day instead of 86.4 million.
The OS used on the 787 has a hard real-time scheduler. Tasks are started up at a specific frequency (set per task), run to completion or to the end of their time slot (set per task) and terminated. We had, IIRC, a strict 100ms slot for our bit of LRU software to do everything and it would be launched every 1s (from memory, that was 15 years ago). Information could be stored between executions so partial completion is something you could handle if needed by storing state information and using it at the start of the next iteration (we didn't need that, our tasks finished in the slot).
You don't base the start of a future task on the end of the prior one, you base it on a fixed clock for these kinds of systems.
Or maybe it's aliens and their strontium-89 wormhole collapses after 51 days. At this point we're just making shit up.
Or just ticking every 1.025 ms (e.g. at 975 Hz instead of 1khz)... that brings us to :
so a difference of 1.12 hours with the "51 days" mention.This is even scarier than the base concern.
Maybe it takes 2 days to boot the entire thing?
This should carry a label: 2020. This article is 4.5 years old
Added. Thanks!
this is what happens when you hire based on checked checkboxes and not qualifications.
This company just can’t stay out of the news. Their planes are trash. Software is straight garbage. Many people have died because of this company and suffered undue stress/anxiety because of the massive dip in quality.
Boeing engineers/builders caught on audio stating they wouldn’t be caught dead in their own planes unless feeling suicidal.
The company definitely can't stay out of the news and it's gone downhill over the recent years but you've picked an interesting post to lament about those on. The news they can't stay out of is over 4 years old in this case. The model of plane it's about (787) has never had a single fatality despite >15 years of operations and >1,000 units operating today. In all, deaths are probably the worst possible metric to berate Boeing on - including every death (e.g. hijackings, not just engineering failures) their popular 747 line has had comes to <6,000 fatalities despite carrying billions of passengers over a period of >50 years.
Despite their ever increasing incompetence on delivery speed, test compliance, and innovation... commercial air travel with Boeing (and other major air manufacturers) has always been one of, if not the, safest mechanisms of travel we've ever executed on. Particularly the last 5 years have been the safest period in terms of air travel deaths or injuries.
None of that means we shouldn't criticize Boeing by any means, just that doing it over perceived death and accident counts because of what news headlines imply is complete nonsense in terms of actual numbers no matter how you slice it. It's important those kinds of things are reported but it's equally important to not get swept up in paranoia over it.
Agreed, my 737 fears were relieved by researching how many of them are in the air at any moment, how many millions of trips they fly each year, how old airframes can get before they get retired, etc. Even the "worse" models are feats of engineering.