"All six display units (DUs) blanked with a selected instrument approach to a runway with a 270-degree true heading, and all six DUs stayed blank until a different runway was selected."
Points out how you should use vector coordinates in navigation, instead of trig. All those pesky divides happen in one place, are obvious, and handling them is trivial.
I don't know of a language (but I don't know many) which trigonometric functions use degrees and not radians.
In radians and finite precision, "cos(x) = 0" is false for all x (unless your cos is wrong), thanks to Pi being irrational (for example, cos(Pi_in_f64) is about 6.123233995736766E-17). You have "sin(0) = 0" though.
>angle = arctan(dy/dx) where dx=0.
If dy != 0, dy/0.0 is +-Infinity (if following IEEE754), so angle is +-Pi/2.
Coordinates/physics is among the things I see developers get wrong the most (even excluding units issues), along with threading/locks and instances_ownership/pooling.
Data races and deadlocks issues can be rare, and leaks get fixed by a restart, but these coordinates issues can introduce constant errors and biases often overlooked by tests (in addition to almost systematically unhandled cornercases like these).
Boeing engineers probably were skilled enough to account for zero division errors in their software, considering that the software works for eastward-pointing reciprocals. It's the sign function returning an unexpected value for zero and causing problems down the pipeline.
If I may speculate... the blanking of 5 displays is probably not a bug. Maybe it's a (misguided) feature meant to keep people from landing at certain restricted airports and they coded the logic incorrectly?
My guess is that it's not just any old "Runway 27", but it's runways where the coordinates of the endpoints of the runways have precisely the same latitude down to whatever precision is in the database. It's probably quite rare to have a runway that's this perfectly east-west.
Looking at the coordinates of the runways in the FAA database, they are all perfectly east/west within about 2 feet of deviation N/S. This would round to zero if the calculations were done as lat/long as single precision numbers.
Runway 27 just means a runway that runs east-west and you're approaching from the west. Landing the other way it would be Runway 9, it's a neat little system that gives pilots everywhere a surprising amount of information in a single name.
That's why I said misguided in parenthesis. I have no idea what criteria or scenarios it may have been for, but a commercial airliner could conceivably have an emergency almost anywhere and potentially be cleared to land at the forbidden locations.
Just speculation on my part but clearing all displays seems like deliberate behavior being incorrectly triggered.
I don't disagree, but I think it sort of illustrates the fact that using the term "bug" for documentation mistakes is a bit too non-standard, even though I used it myself.
Why are you so sure the blanking is not caused by a bug? “On error clear screen” seems like a pretty good way of making it very clear to the pilot that something has gone wrong and they need to fall back to manual procedures (rather than looking at displays that may have gone stale or be showing bad information).
I don't know a single airplane that does that and I've flown quite a few types.
It's normal to have error detection and the way to show it is to cross out (red lines, black background) the failed information on the screen. Not blank the whole thing.
The units are fairly independent in a modern system, communicating via data buses. So this sounds like a case where the bug made a unit emit something really bad on that bus to make them all go blank.
I have to assume then that these are the runways that lead down into the secret Illuminati bases beneath the airports. Has somebody taken a closer look?
And if you have an emergency you’re going to put the bird down wherever there is a runway, even if it’s area51. You may be asked to call the phone number when you land tho :/
The world is spherical not square, which means that the farther away from the equator you are the smaller a circle parallel to the equator is.
If you tried to use a map of the northern hemisphere for the southern hemisphere without turning it upside down, you would be stretching the point that is the north pole to cover the entire circle that is the equator, and condensing the entire circle that is the equator to the point that is the north pole.
Been there. Almost nobody uses it. But from what I read in the newspapers, the few times it does get used, it seems to have more than its share of crashes.
While it is easy to blame Boeing these days, I suspect we have survivorship bias here. Dig into Airbus software and all sorts of similar scary bugs will pop up. For instance, one may remember inconsistencies between airspeed measurements that resulted in the loss of AF 447 on June 1, 2009. Overall, it's great that at least some airliner's closed software finally gets more eyes looking at it.
Frozen pitot tubes causing an airspeed disagree is something we train for. The issue there isn't software, it's mechanical, and what the Airbus did is exactly what it's supposed to do according to the manual.
The problem there was the junior co-pilot responding all wrong and the other pilot not noticing because Airbus sticks don't move together. The captain woke up, came to check and realized what was wrong but did so too late to fix it.
Agree, though the investigators also pointed to the lack of a clear display of the airspeed inconsistencies even though the computers had identified them. This sounds more like a UX problem rather than a bug, but still the software issue in the edge case.
I seem to recall Airbus having different engineering teams across EU using different versions of the CAD tool and then when they went to build the plane the wires were too short because one version of the CAD tool accounted for the radius of wires turning a corner while the other didn't.
Boeing had an approach to this issue: Your either running the correct version of the CAD software, or you can't check in designs. This also meant that every time it was updated there was a rush to get engineers who's update did not work back up and working. Fortunately those were a small minority and it was usually disk space issues.
The measurements would have been the same in both cases: therefore it would have been a) designed correctly and implemented correctly, or b) designed incorrectly, but caught in the design phase, looping back into a). This way, you get c) designed correctly, but implemented incorrectly or d) designed incorrectly and the design implemented as specified, in both cases leading to an unworkable implementation.
It is certain that under close scrutiny one can find bugs as well in Airbus airplanes. Bugs are inherent to software production, all processes are just there to minimizes the risks posed by critical bugs.
However, as airbus released the first fly by wire airliner (a320), they have more experience in software over boeing. Boeing was always afraid to give too much power to the software, a boeing plane has more mechanical backup than an airbus.
I sincerely hope that they are not fucking around by following the agile trend from the sw industry, and instead stay focused on old and proven engineering development practices.
I must say that this whole boeing show makes me scared to hop in a boeing, and is another reason why i will now seek flight in a a320 instead of a 737. The first reason is simply comfort, the 737 cabin size is really too small.
Edit : the a350 had a bug where the pilots needed to switch off the plane every 149h, and recently they discovered a bug in the a220 P&W engine (not airbus developped) leading to engine shuting down in flight.
They also found a kind of mcas like bug the a321neo, but I think it happens in extreme manoeuver which can be only replicated in simulators
A Boeing with an engine fire will pop up a dialog saying “flight computer detected a possible engine fire, would you like to shutdown?”. An Airbus with an engine fire will pop a dialog saying “flight computer detected an engine fire, shutting down engines”.
The joke being Boeing pilots have a chance of saving the plane in the event it’s not an engine fire, Airbus will just do what it thinks is best and let the pilot sort out the aftermath. MCAS changed all that
Yet another reason to exercise low-level math functions with a wide range of inputs -- maybe not on every run, but at least in pre-release integration tests
... where "abused" appears to cover most good-faith efforts to implement Agile in practice. When it works it's Agile—despite the fact that these implementations have very little in common. When it fails the mantra is always "Agile wasn't implemented properly". Agile has every characteristic of a utopian ideal that cannot survive contact with reality, or at least the realities of safety-critical software.
Big-A Agile is often described in Utopian terms. Which is unfortunate, but little-a agile is understood to be a thing you grow towards with constant improvements and changes to both the process and the system being developed based on frequent, ideally continuous, feedback. Boeing, in my experience, didn't want to change their processes, they wanted to run the same processes faster. Which is decidedly not agile (and I hate Scrum for this).
The form that I've seen them adopt on programs I've been involved with is heavily inspired by Scrum. So they have sprints, they accept product releases from the subs and will, ostensibly, test it (running the integration lab they have the more complete test infrastructure, in theory). But, in practice, the released code isn't tested for months, but development continues, monthly or bi-weekly releases continue, and no feedback is ever provided to the developers. Finally, 6-12 months later most of the builds are discarded, the most recent one is tested, and it's discovered that, true-to-Waterfall, the wrong thing was built. Patches are quickly developed and applied, tests pass, and something is released to the customers.
Months or years later, the software is found to be defective, the test suite is found to be insufficient (again), and the process repeats.
> ... little-a agile is understood to be a thing you grow towards with constant improvements and changes to both the process and the system being developed based on frequent, ideally continuous, feedback.
Agreed, and I have no objection to the feedback and continuous improvement aspects of agile development. Or anything in the Agile Manifesto, for that matter. In my experience, however, when projects "go Agile" they tend to cargo-cult simplistic, visible changes like Scrum instead.
> Boeing, in my experience, didn't want to change their processes...
Yes, this is generally the issue. On the other hand, those processes are strongly influenced by industry standards such as DO-178 which codify "tried-and-true" practices established in the early days of software engineering—it isn't just a matter of corporate culture. It's not mandatory to follow these processes, but if you do then certification is basically assured, which is much simpler and more reliable than arguing the case for a brand-new process with no established history in the industry. For the most part these standards treat software as just another "component" to be integrated into the hardware, like an extra-complicated combinatorial circuit, rather than what it is: the design of a iterative, stateful process to be carried out by the hardware during flight. Novel development and certification processes are a huge risk; understandably, no one wants to go first, even when they agree that change is needed.
> I'd like to learn how Boeing's software delivery practice has changed. Sounds like there may be some lessons learned we can all learn from.
Does Boeing build its own cockpit displays and avionics? I was under the impression that that stuff was developed by companies like Rockwell Collins and Honeywell.
And magnetic declination changes all the time. Maybe different systems are using different sources of truth?
Maybe when they did initial testing, they included some airports at the extremes, but when the declinations got updated, they were no longer good test cases?
In the automotive and aerospace world, of course tests in the real world cannot cover all cases (cost, time make it impossible).
Even most (not all, but almost none for meaningful applications) computer program cannot be proved to come to an halt [0], so complete testing is impossible by essence. We can only use more restrictive rules for programming but cannot formally guarantee anything.
As those systems are tied to the physical world, a whole lot of complexity is added by uncontrolled parameters.
Yet we love testing things. So a lot of techniques exist, such as SIL [1] and HIL [2].
So you could imagine using a real dashboard hooked up to a plane simulator. Which would enable testing the device in a wide array of conditions.
> Even most (not all, but almost none for meaningful applications) computer program cannot be proved to come to an halt
That is not true. If you can demonstrate that every loop or recursion is bounded, then the program must halt as a necessity. Loops of the form for (i = 0; i < N; i++) are trivially bounded, unless you're resetting i in the loop. If you have containers that are not being modified in the loop, the finiteness of your data structures is usually sufficient to form the lemma that the loop will terminate.
Recursive datastructures (such as cyclic graphs) are much more challenging to prove, and most challenging are fixed point algorithms (do { } while (changed);), as will be noted by them frequently being the causes of actual infinite loops in my experience. But if you had mandatory annotations to declare lemmas for termination, it is doable. With that feature, a programming environment that forbade you from writing code that couldn't be proven to terminate is probably sufficiently feasible to allow you to write large, complex applications.
My POV here was very cautious, as I didn't exclude programs which can be proved. The problem is that even if control flow is "provable", a lot of algorithms use number computations and is much harder to deal with from a formal POV :
"In cooperation with the University of Iowa and Rockwell Collins, this research focuses on the verification of safety properties on Lustre pro-grams. SAT or SMT, based verification approaches such as k-induction give good results on programs with a mostly discrete state space (boolean, bounded integers). However, when numerical computations are involved (real/float computations) the formalization of the property to be proved often needs to be strengthened using auxiliary lemmas to make it inductive with respect to the system’s transition relation. When attempted manually the discovery of such lemmas is time consuming and hinders the efficiency and scalability of formal verification. Automating lemma discovery hence appears crucial to allowing end-users to apply formal verification on industrial cases."
Taken from [0]
The seminars I attended to, from the creators of coq (a formal verification language), didn't disagree with this point of view. Of course, formal verification is not the only thing we can do [0].
In any case, what you propose seem interesting, if the halting problem was the only problem to solve to have a formally proven system.
Given these planes can take off and land on autopilot (I'm not sure that's true, just what I hear) then surely it's easy to run simulations of all those events; so if it was in scope for simulation it could be caught?
There's about 20k commercial airfields, if you ran 20mins of simulation for each one then you're done within the year. Presumably Boeing can run in parallel and above realtime.
Are their watchdog systems for everything, or do they primarily rely on pilots noticing problems?
Wild speculation: Perhaps it's a power issue, something about the conditions causes a power demand that exceeds specs (or makes an LED? voltage drop too low). What else links all the screens?
Actually, an autopilot landing requires CATIII ground equipment...but that could be easier to set up in a simulation.
But that is the nature of testing in multivariate conditions: e.g. if the bug triggers only when the heading is exactly 270.0000, I could see why a testing suite wouldn't catch it.
Apparently another plane went down in Ukraine according to this article. That seems like a much bigger story. Maybe I happened to miss that one yesterday?
Many of the "holes" that were "obvious" in the low-res photos have been shown by better photos to be rocks & pebbles (kicked up by the impact) landing on wreckage.
As suspicious as it seems given the political context, there's no proof the crash involves foul play of any sort.
tbh given the rarity of 737NGs plunging to the ground [on fire, if low quality social media videos are to be believed] shortly after takeoff, the ADS-B signal ceasing whilst the aircraft was still on course and wreckage being spread over a wide area which aren't the usual results of engine failure or improbably poor piloting, and the context of the aircraft flying later than scheduled past military forces on high alert, I think some sort of foul play or military misjudgement is the null hypothesis to be ruled out...
It could be a big story, but unless it's a discussion on a technical failure, it doesn't really belong on HN. Ever since the last big election cycle people have been more strict about not allowing political and world-news posts that aren't related to tech or hacking culture.
In the past big news often made its way to HN. I don't really care about trivial world news so I don't search tons of alternative news sources.
If a nuke was launched and exploded somewhere I'd still expect it to end up on HN despite not being tech news. So the requirement of significance for unrelated news to make it here has gone way up.
When I say "another" I was referring to that previous incident where a passenger plane was shot down in Ukraine and not the Max plane problem. Any passenger plane crashing seems like a significant story. I hope we aren't numb to that already.
A skeptic might suggest that a major plane crash (a very uncommon event), occurring in Iran at the moment of maximum military tension (after Iran had launched missiles at US bases and could be expecting a response), is a pretty unlikely coincidence more likely explained by an accidental shootdown or the like.
I believe dang pruned it due to not enough information being out in order to foster anything beyond the wildest of speculations. Such as people thinking it was another MAX 8 at some points.
While I don't agree with the course of action; I can see the temptation to doing so.
Often at least waiting til the CVR and black box data is found is necessary to really know for sure what ended up transpiring. Though there were apparently some mobile phone videos of things of dubious provenance/quality in the original threads that I was aware of and are still searchable. Of which there are at least 2 currently via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
From reading the article, it seems like it occurs when the runway is precisely running east to west and the airplane is coming in westward. And also the latitude/longitude is somehow involved. You can see one of the airports here. Notice how it is exactly horizontal, i.e parallel of latitude. https://www.google.com/maps/place/Wiley+Post-Will+Rogers+Mem...
This is incorrect, while most runways are aligned with the magnetic heading, that changes when you have a field with multiple parallel runways. Atlanta Hartsfield comes to mind first, with its five parallels in the East-West directions, 8L/26R, 8R/26L, 9L/27R, 9R/27L, 10/28. One would naively expect these to be not parallel by just reading the name, while in fact they are all in line.
The affected runways have fairly similar directions (between RW25 and RW29. Runway names follow compass directions. See https://en.wikipedia.org/wiki/Runway#Naming). I don’t know what the distribution of runways in the USA is, but that feels too similar to me to be a coincidence.
Even if it is, I think it wouldn’t be the whole story, as there must be zillions of such runways in the USA.
I think you are conflating two separate events. Iran shot volleys of missiles at two different bases in Iraq. Separately, but on the same day, aUkraine-bound Boeing 737 800 crashed hear Tehran[1]. Some articles are reporting that it had technical problems. But it may take a while to verify because Iran is withholding the black box from Boeing.
Generally, passenger planes are shot down due to incompetence from trigger-happy idiots, not because anyone actually wanted to attack a passenger plane. So this is entirely plausible.
Boeing and Airbus both source avionics from the same third parties. The 737NG uses Smiths, but everything else in the Boeing fleet (and all newer Airbus airframes that I’m aware of) come from Honeywell. While you can fault Boeing for not finding this, the vast majority of the fault lies with the OEM.
Even if this particular problem is not Boeing’s fault, but Smiths’, the whole 737MAX saga still puts Boeing in a pretty bad light – cutting corners on security, outsourcing their software to dubious vendors, etc.
The Daily podcast (NYTimes) did an episode last week called “Boeing’s Broken Dreams” where they interviewed a former Boeing safety manager turned whistleblower. Definitely sounds like there was a major cultural shift over the past decade or so.
There's plenty of bad light to go around. Say company A builds a faulty part, then a stack of ten subcontractors test it and fail to find the error, and finally company B tests and accepts it. Who is responsible for the eventual failure in the field? They all are, because they all had veto power.
I for one will never ride on a 737 again, even if they manage to get them back off the ground. For every problem that's come to light there are probably two more than haven't been discovered (or disclosed).
Fact is: there are around 10,000 737s around, each if them starts and lands multiple times a day, by far mostly without problems. Risk is similar to you slipping while getting out of bed in the morning or being overrun while crossing a street. (Excluding Max versions, which are grounded)
"All six display units (DUs) blanked with a selected instrument approach to a runway with a 270-degree true heading, and all six DUs stayed blank until a different runway was selected."
cos(270°) = 0