- no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension.
- a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement.
"No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc.
Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake".
Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
People who do not work in hardware seriously underestimate how often counterfeit components enter the supply chain. Even trustworthy distributors like digikey and mouser have had regular cases of their supplies becoming contaminated. It is just way too easy to do and rarely discovered as long as you mix them with enough legitimate components to avoid suspicion. Unless you're unlucky and they happen to reach someone with an affinity for chip photography, the worst case is they'll just think a few components were DOA or out of spec.
It mostly affects low complexity components that are easy to clone so a BMC would be unlikely, but even that is not safe as sometimes used components de-soldered from other products make it back into the supply chain too.
The bigger supply-chain problem there wasn't the fake FTDI chips, which actually worked reasonably well; it was the totally genuine and authorized FTDI driver update which FTDI designed to brick your customers' hardware if they installed it, if you had been so unfortunate as to get fake FTDI chips.
I look at it the other way around, the driver update is a rare case where it was publicly exposed just how widespread of an issue forgery is in the electronics industry. FTDI likely expected the number of fake devices already in the field to be significantly lower than they actually were. If that was the case, it is unlikely it would have become the story it did.
> "And I would have gotten away with it too, if it weren't for you meddling kids!"
No, I don't think that was the thought process. I think they wanted to send a message and I think they wanted to test the waters to see if anyone would hold them accountable for deploying first party malware.
If I remember correctly, in the FTDI case that was very unlikely to happen. It wasn't a case of `if (looks_fake) do_brick()`. Rather, it accessed registers in a way that they knew their implementation supported but that was buggy in a widely counterfeited version.
(And I understand it they did this knowing the effect it would have. It wasn't some accident.)
The converse: they performed EEPROM writes in a way that was ignored by that particular chip from FTDI (only one - it would've bricked other FTDI products) but honored by the clones. They exploited a bug in their own product to make their bricking code only affect the (non-buggy) clones.
The code was blatantly deliberate; to pull this off they had to perform a preimage attack on the EEPROM checksum, etc.
I think ironically it was other way around - they issued write in such way that due to quirk in genuine chip it was no-op but actually worked on the clone.
More specifically: many of FTDI's devices, including the FT232RL, can store configuration data in an EEPROM. EEPROMs are conventionally organized as an array of bytes, but FTDI devices always access the EEPROM 16 bits at a time. The FTDI devices include some commands to read/write the EEPROM. For convenience, these commands use an address counting in bytes, but the official drivers and tools only ever use even addresses (so that the EEPROM addresses accessed never go "across" fields).
The FT232RL is one of FTDI's parts which has an internal EEPROM. An implementation quirk means that any attempt to perform a EEPROM write on this part with an odd address will be ignored. (This quirk is specific to this one part -- most of FTDI's other parts will perform this operation as expected.) This fact was apparently not known by the developers of the FT232RL clone; it will perform the write.
FTDI released a driver update which would, upon detecting a FT232RL, attempt to write zeroes to several odd addresses near the locations used for the device's VID/PID (vendor ID and product ID). These writes were ignored by the genuine devices, but were interpreted by clones as writing zeroes to these fields. This caused the devices to fail to be recognized when next connected to a host.
What's funny is that, as far as documented functionality is concerned, the clones work perfectly. In some regards, they actually work better than the originals!
That's not quite right. The addresses are in 16-bit word units; there are no unaligned accesses or byte offsets. You can write to any 16-bit word on other FTDI chips. However, the FT232RL has a quirk where it expects you to write adjacent pairs of 16-bit words sequentially, and only commits the write on the second one. If you only write the first half (at an even word address), nothing happens.
Usually, when programming the EEPROM in these devices (e.g. with FTDI's tools or open source ones), you write the whole thing at once, so those pairs are always written sequentially and it works anyway. However, if you try to write to some words at even addresses without following up with the next one, the FT232RL misbehaves and ignores the write.
Specifically, what happened seems to be that the FT232RL uses an internal 32-bit EEPROM. So when you write to even addresses, those writes are staged to a buffer in anticipation that you will subsequently write to the next odd address. At that point all 32 bits are written. The bricking code only ever wrote to even addresses, which involved a preimage trick to keep the existing checksum valid, since the checksum is at an odd address they couldn't write to. Instead they calculate what the value for the word before the checksum should be to make the existing checksum valid, and write that instead. Genuine chips would just stage all these writes in a holding register and never get to commit them.
Basically, FTDI decided to use an internal 32-bit EEPROM macro when developing this chip, and came up with this hack to shoehorn it into the 16-bit protocol (in what is arguably a buggy way). The clones just implemented 16-bit writes properly, which is the same thing all other FTDI chips (with external 16-bit EEPROMs) do.
(Source: I'm the guy who first figured all this stuff out back when this happened, including reverse engineering the FTDI bricking code and writing an unbricker).
You’re right - I think I was misremembering the details and it was something like a poisoned supply chain leading to people who thought they had bought the real thing shipping devices that ended up breaking or something.
Well, it was "a poisoned supply chain leading to people who thought they had bought the real thing shipping devices that ended up breaking"; it's just that FTDI was the company that poisoned the supply chain and broke the devices.
While clearly that's a thing that could happen when FTDI goes around injecting malware into the supply chain, I'm not aware of any accounts that it actually did happen. Did it?
Sure, and they would have gotten precious little criticism if they had displayed a big warning message on detecting a questionable part. The problem was that they decided to unilaterally destroy customer property.
Much worse: FTDI's customers are the distributors, whose customers are (usually) board assembly houses, whose customers are electronics vendors, whose customers are electronics users. FTDI decided to unilaterally destroy the property of their customers' customers' customers' customers. Or, rather, the customers of people who thought they were FTDI's customers' customers' customers, but whose suppliers' suppliers were actually cheating them.
The miraculous thing is that FTDI escaped criminal prosecution for this.
It wasn't that the chips were fake, they could only be fake if they claimed to be FTDI. They could have been merely FTDI compatible, at which point did FTDI have any grounds to destroy compatible parts?
They were using FTDI's USB VID. Probably most of them also were marked as "FTDI" on the chip package, but of course that's not what FTDI's drivers looked at.
There is no law against using someone else's USB VID, as long as you don't put a USB certification logo on your product.
What FTDI did, on the other hand, is malware and clearly illegal. It is destroying private property and I guarantee violated multiple laws in many countries.
I don't know the laws elsewhere, but here in Europe trademark holders absolutely have the right to demand counterfeit products be seized and destroyed. Even if the owner acquired them in good faith. Also it would be interesting whether any compatible chips exist that use FTDI USB VID but were not marked with the FTDI logo.
Of course this kind of vigilante justice by FDTI is illegal, but who is going to press charges if that means they will get their devices taken from them and destroyed?
Right, we're talking about clones that don't use FTDI's logo. I don't know if they exist, but they well might. The point is that the VID itself doesn't have any legal protection.
And also, the way FTDI bricked the clones (only) was by exploiting the fact that they also implement the EEPROM write command in the sane way other FTDI chips do, instead of having the write staging quirk of the "genuine" FT232RL that they introduced when they wired the internal 32-bit EEPROM block to a 16-bit interface.
So all around, the clones work better than the originals in several ways.
> Nightmares? They work better than the originals!
Yeah, nightmares. It doesn’t matter if they’re crap if it’s what you have to work with.
I really never understood why the open source world went with FTDI when the CP210X was perfectly available (I’ve been using them for ~12 years now), in fact I found about the FTDI chips later and I was very confused of why would I even pay more for an inferior part.
But whatever appears in hobby boards is bound to appear in some products.
Ahhh, that's the critical piece of context I was missing. Thank you!
Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms.
If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
> would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others?
I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
In software it would be like if you found evidence that one of your trusted developers had outsourced their own job to an unknown person in India, who was controlling their home office desktop PC by some sort of remote desktop tool and adding code/making commits on their behalf.
More specifically, if there is a person in a QA role looking out for counterfeit components, and they miss such a glaringly obvious typo, they may not be paying attention to enough details. And if a language barrier is at fault, that's itself a flag that they may not have full comprehension of the specifications they are supposed to QA for. Perhaps not "stop the line" but "we need to do an audit of this component" - and that audit needs to actually be performed.
I agree. The amount of scrutiny that goes into specifying, procuring, and then validating something as seemingly simple as a label would be eye-opening to someone who hasn't lived it. Not to mention that I spent a solid 40 hours last year writing code to print what was apparently a very simple, but automated, label.
If I saw a misspelling like that it would indeed be cause for hitting the Big Red Button.
As someone from outside the industry, I started reading the article with the mindset that this was about counterfeit parts and was interested in the outcome. Yet the more I read the more I was asking myself what the article was actually about. Eventually I just stopped reading, looked at the comments, and learned that it was about a typo.
Quite frankly, it doesn't matter how much process there is to avoid this sort of thing. It is a big industry and we should expect an error as trivial as a typo to slip through eventually. The reader would have been much better served if the author was upfront about that, discussed why the error was significant, and offered some insight into how the industry normally avoids this type of error. Maybe they did get to that point eventually, but this reader will never know since this reader gave up when the article felt self-serving rather than informative.
Especially an independent black box SOC type of component that has wormed its way into all manner of systems.
There's no proof that tampering has happened but there is proof that not only it might have happened but also that there is no way to tell if it did happen short of, idk, forensically auditing the network traffic of your $10,000 switch with the help of another $10,000 switch and someone skilled enough to pick apart the output over a long enough period to determine if tampering happened.
I'm not paranoid to do something like that but if the US Government isn't paranoid enough to do something like that then something has gone horribly away in government cybersecurity.
A company I worked for ordered thousands of set-top boxes. When we received them they were labeled as containing 'Marcovision' copy protection on the back, instead of Macrovision. Turned out to be a stupid typo
Anyone familiar with electronics manufacturing knows that for most of the people involved in the production of your stuff, English is a second language and roman characters are not their primary alphabet.
> it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
The notion that someone would pause a line over this (even if we were not in the middle of unprecedented component/manufacturing/shipping disruptions) is beyond fucking absurd, much less that anyone would do so until a "re-check of the supply chain" is completed.
Production schedules are tight as hell.
You miss your deadline for getting the board assembled, they don't make it to the line or factory putting the boards into the chassis on time.
That means they've started on another job and now you wait until they have free time on the line.
That means you don't get your container to the port on time.
That means you miss the space you had paid for on the ship.
That means you miss your product launch date. Possibly by months; especially right now, shipping is severely constrained.
That means your competitor takes your lunch money.
If your goal is to crank out the largest quantity of cheapest-unit-priced products as quickly as possible, then yes, absolutely stick with what you just wrote above.
Are you familiar with what happened with the counterfeit capacitor plague?
Keep on cranking out that production line with your suspicious/manufacturer-source-unknown parts with improper labeling on them and end up in a situation with hundreds of millions of dollars of financial damages due to burst capacitors.
Not every manufacturer has its absolute and highest goal set as massive quantity/cheap and shoddy QA/lowest price/highest volume possible.
For something like a 100Gb ethernet switch, standards should be much higher than the PCBs of a bunch of $40 802.11ac wifi routers to put in a shiny box and sell at Best Buy.
I worked at a place where I had ready access to a large lot of $1,000 "blade" PC's that I could use for experiments, tests, etc., as needed.
The reason I had them available is that they were supposed to go into a machine that we sold, but since someone in Receiving missed a step in their inspection process, they all had to be marked as Non-Conforming and could not be used in the machine, even though there was absolutely nothing wrong with them. Since they were a special order, the vendor wasn't about to take them back.
So it was either toss tens of thousands of dollars worth of hardware in the dumpster or try to find a use for them internally.
Also focusing on unimportant, but obvious, issues takes resources away from controlling for not so obvious but critical issues. That and given the fact that checking English spelling isn't a trivial thing for Asians (as it would be the other way round) means this is a non-issue.
Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.
More likely that nobody noticed the typo until it was too late and they didn’t want to halt an entire production run just to replace a sticker that most people will never see.
Stopping the production run and delaying shipments would have a palpable impact on shipment numbers. Most likely it was noticed, corrected for later batches, and they made the call to run this round anyway because it had zero effect on function.
(Source: If you work in hardware long enough, you’ll have to make similar calls like this eventually)
For us older folks remembering what "stamps" are, and that people used to collect those: The most valuable ones are those with printing errors, e.g. typos. If you produce enough stuff at scale mistakes happen. And I would assume the stickers are much less controlled and monitored then the chips / hardware itself.
No idea how it works in electronics, in aerospace so you would do thorough testing on the affected units. And if they meet the spec, someone would make a call regarding relabeling. Considering how expensive relabeling tends to be, it isn't unreasonable to assume the parts just got a concession for the labels. I wouldn't read to much into it.
I work in an industry rife with typos, and can assure you that typos are missed all the time, and it doesn't indicate a greater QA problem or lack of QA. It's just a missed typo.
Where I work, we treat grammar and spelling errors as a sev 2 bug (sev 1 being highest.) It will quickly erode trust in your product even if everything else is working fine. All of our customer-facing text (UI/CLI/API names, labels, error messages, etc.) go through QA and also a separate documentation team for consistency and spelling checks.
Here in the UK, one of the national newspapers, The Guardian was so famous for typing mistakes that it was nicknamed 'The Grauniad' by Private Eye, which is a satirical magazine.
I used to work in incoming inspection department for a medical device manufacturer. If the switch/router company did a "first article inspection" (FAI) and missed this (or missed it on a later incoming inspection), I would agree, but if they found it and called AMI and got the answer it was a typo, then they would note that on the inspection sheet and move on.
The problem is, as pointed out in the example, is that the memo the writer got from Dell that came from AMI informing Dell about the typo contained an image that the own writer sent to AMI. Meaning AMI had no idea about this typo before they were contacted by the writer.
We were super anal about things there, but I always made sure scrap was actually scrap and not recoverable. Your description sounds like a stupid situation and they should have just reinspected those blades.
I'm not so sure. These BMCs are dumb SoCs like any other (just with specialized I/O), their firmware comes from an external SPI flash. I doubt there is anything AMI specific at all in these chips. Looks like it's basically a license sticker some worker is putting on these boards after they're all assembled and tested. I can see how these stickers are there, maybe for contractual/legal reasons, without being a critical part of the BMC board BOM (1x roll of "AMI loicense stickers").
Maybe they're aware of the supply chain interdiction and willing or motivated to let it continue. You'd think that the integrity of their product would be paramount concern. But how might their attitude change if a state decided to intervene? Or what if they have short-sighted leadership making the best decisions for them individually?
> If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
I'm pretty sure "they" that design the stickers are very different from "they" that design the chip... Similarly chip verification has rather a different skill set and attention to detail than someone doing "sticker QA".
Also I doubt anyone was doing sticker QA, I'd guess that this was probably an outsourced mistake (I used to work in printing when I was younger - the mistakes you wouldn't believe).
One of my favorite customers actually took a calorimeter to the LEDs that were in one of our deliveries of router chassis. They felt that the Amber wasn’t quite Amber enough and so they measured it. They were right while it was amber it wasn’t spec amber. So they shipped it all back. We were very embarrassed and supply chain was given a dressing down. They missed a parts quality issue.
A typo like this is extremely indicative of a sloppy organization.
Or a communication issue? How do the people looking inside the assembled product tell the people who design the product that they made a typo?
A few years back, the company I worked for created a landing page with an image of a beautiful young woman who was happy about our deals. It wasn’t my job, I was looking at the design out of curiosity and I noticed that one eye of her was looking in one direction and the other eye to the other direction. I tried to raise the issue with a few people higher up but they didn’t understand or didn’t care as they were excited for the release or busy with other stuff.
To this day, I wonder, was it intentional? Maybe it was a joke or something I didn’t get. The campaign run fine, no one talked about it. Maybe I don’t get graphic design, maybe the eye situation was a marketing message about how the lady was having eye on the numerous amazing deals of ours.
So sales gave Supply Chain a dressing down over out-of spec (was the customers measurement confirmed by in-house Quality?) LED color. Nice way of collaborating, we in SCM love this. Was the color check part of goods inspection protocols? Because it it wasn't nobody within your "sloppy" organization had any chance at catching that error.
I could write a book about customers being extremely picky about "visual" quality issues. Usually as a way of either returning inventory they don't want or to drive prices down. Dressing down someone about that, and indicating that the whole org is sloppy, is not the usual reaction I would expect.
Parent comment probably meant "colorimeter" which will measure colors in either spectral distribution or XYZ color space.
So you could say something like RAL color # 123 +- 5% in XYZ space. Or you can just specify: That color needs to be closer to RAL #123 than to any other RAL number, so then you'd also have an implicit definition of the valid color range.
We publish the relevant wavelength on the data sheets.
This is for interface LEDs on a network device. This was one batch of a few hundred chassis, and they ended up being _just_ _slightly_ more yellow than the rest. Back they came.
It might be regulatory concerns, maybe they hadn't licensed that particular wavelength of orange for local broadcasting purposes, or EMI concerns with regards to the the off-white spectrum, at least if I let my thoughts run without moderation.
Naive question, I don't know much about this industry. But if someone from China or the NSA wanted to make counterfeit chips, why would they risk putting something different on the label? It seems like a weird place to draw the line. Would the NSA really say "oh no, we can't violate anyone's trademark!" and misspell the label and hope their entire operation isn't exposed?
Counterfeit chips aren't just a state-level actor problem, companies do it for profit as well. I think the worry is just, if the person who was supposed to be checking for counterfeits missed this, how would they have any chance of catching a more sophisticated counterfeit?
What the original author isn't aware of is the fact genuine product stickers often will include minor differences (two transposed letters, a line of text being slightly off to one side, a copyright symbol with a misprint, etc) that are used to quickly visually verify the stickers. This works incredibly well because forgers try to get everything right, and often won't detect these subtle differences from the photos they are working with.
The military does the same thing on badges for secure areas. They will have a misspelling of the base name or two shades of the same color on the front/back so guards can visually verify them as legitimate.
Sometimes bandits are sloppy or have Dunning-Kruger, e.g. with pirated DVDs the sleeve art is just a hot mess of random words, but I guess if they were making fake chips they'd be a bit better at it. Or the guy they hired to do the Photoshop says he's real good but he's actually sloppy, and everyone else just sees random glyphs (imagine if you were having to copy an Arabic label, I assume you can't read Arabic).
Although it would be much (much) more elaborate, a counterfeit BMC could have hidden ROMs or bootloaders capable of opening backdoors into running machines.
This is covered by the Wikipedia articles others have linked, but briefly: Richard Nixon (US President from 1969-1974) was forced to resign by the revelation that he had paid for criminals to break into the Democratic Party's campaign headquarters in the Watergate Hotel. The entire affair became known as Watergate.
Later, one of Nixon's former speechwriters, William Safire, propagated the use of -gate as a generic suffix for any type of scandal, notably including very minor ones. It's likely that part of why he did this was to retrospectively diminish the perceived seriousness of the Watergate scandal.
Meta-grumble: I am so tired of "-gate" being added to "controversy names". It continues to lose cultural relevance (as evidenced by your post) yet it hangs on.
Thank you! Christ wish they would have gotten to the point right up front, my disbelief when I saw this blog had actual pages for this article, wow had not seen that in a while
Taking the other side of this: what if this is really a man-in-the-middle attack of sorts on the supply chain? These kinds of typos we are in phishing emails why not here? And it could very well be that AMI is doing damage control by just saying “it’s a typo” — like they can’t spell their own name?
I can tell you that manufacturing businesses are difficult to manage, have thousands of moving parts if not millions, those parts are more often than not human.
It's like a choreographed dance with millions of people simultaneously, somebody is bound to trip over their laces.
What I find funny: the guy who runs STH frothing at the mouth about a typo. To paraphrase Nick Fury: "Pot, kettle."
The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child.
He also is an incredible drama-llama, making mountains out of molehills; I've seen him do this time and time again.
He sees a misprinted sticker and sees supply chain attacks? Dude. This gear is assembled by people in third world countries making wages that amount to a few dollars a day or less. They don't speak English. They may not even read roman letters. I'd challenge him to do QA on any non-roman alphabet...
I bet someone did notice the stickers, but getting them reprinted (assuming it was caught before assembly started) may have meant a delay. Even a minor delay can be a major, major problem since this stuff is scheduled practically down to the hour in the factories; ditto for shipping deadlines. Or if they were already on assembled boards (or worse, inside assembled equipment) the cost to replace the sticker would be astronomical, with exactly zero value to the vendor or their customers. It's cosmetic.
> The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child.
I mean, he acknowledges this:
> I make a lot of typos, so I know how it goes. As someone who is dyslexic, this piece is going to have a few typos when it goes live, so I empathize with this. At the same time, I tend to spell my name correctly on official documents.
In the electronics industry, a typo on a label, and especially in the brand name, is a very strong indicator that the product is counterfeit (AFAIK such mispellings are often deliberate to avoid using a trademark). For instance, fake Apple products are often "Designed by Apole in Colifornia". And while it's true that the products are most likely assembled by non-native English speakers, this sticker appears to include some licensing/serial number information and was probably designed by someone at corporate headquarters for the express purpose of certifying that the chip is genuine.
And the author places the bulk of the responsibility on Dell and AMI for not catching the error, rather than the workers at Foxconn or wherever. Counterfeit parts, supply chain issues, and cut corners are common, so engineers in the US closely inspect a random sample of boards received from the factories to make sure everything was built according to spec, which is where this should have been caught.
Having worked for a hardware company before, they were VERY particular about their hardware quality. They would frequently get engineering samples and do full teardowns on them to evaluate build quality. Those engineers were intimately familiar with the hardware. And these were toys. I would absolutely expect that high-end computer equipment goes through frequent engineering sample review. If they aren't doing that, then how are they ensuring the quality of the company doing the manufacturing?
By counting the number of returned items and service calls: that's way cheaper than upfront checking and in the short run will suffice. That's MBA logic for you.
Without further details, it's hard to tell. What I can imagine happened: Someone realized there is a typo. AMI was made aware of that. AMI did additional Quality Checks. Chips were in spec. Re-Labelling, after the fact, is expensive and time consuming. The batch got a concession, AMI probably paid less for that batch. All good.
In a case like this, it seems like a reasonable course of action. Assuming otherwise is borderline paranoiac.
My Dell Precision 5520 (née XPS 9560) battery came with obvious spelling and grammatical errors.. just like the cheap Chinese clones I bought to replace it after a few years did (because Dell doesn’t sell the battery online). The clone I received was garbage and not up to the advertised specs so I tossed it and called my Dell sales rep and managed to get him to order me a new battery (it’s apparently not considered a user-replaceable part although it really is) and lo and behold the new one came with the same spelling errors as the one I got from China.
Ordered a few hundred laptops with a office 2016 license key. They key came on a printed card with each laptop. About 33% of the cards were printed in the US. Every one of them worked. The other 2/3rds were printed in China. Half of those were already registered under random email addresses (as each had a randomly generated email that appeared to be at yahoo). It was a massive fight with Dell sales to rectify the situation.
Same manufacturer selling out-of-spec batches initially produced for Dell on the grey market maybe? As long as Dell parts are within spec, who really cares? If anything, geting in-spec parts from Dell from, what looks to be, the same supplier while direct orders from the supplier are out of spec should increase confidence in Dell's supply chain.
> This was not caught by Dell, or even the STH team at first, it was a YouTube commenter. If that is how we as an industry are catching the easy plain-to-see stuff, that should scare everyone about what may be hard to see.
And this is why the people shouting that "the direct cause of the log4j bug is our collective failure to give open source maintainers bags of money" are wrong. Don't get me wrong, you should give people bags of money if you want them to drop everything and go fix a critical bug after it's been discovered.
But our collective ability to discover bugs is abysmal.
Which definition of our? Seems like it was found just fine. It didn't get the attention it needed until after it was publicly disclosed, however.
The distinction is that even with a magic 100% infallible bug finding service, if the fixes don't get the attention they need, even with this magic service is, the problem's still going to be around. (While this perfect infallible service doesn't actually exist, fuzzers do find bugs in a semi-automated fashion. Still, getting someone to pay attention to the reports is an uphill battle due to false positives.)
1. If one is going to counterfeit the chip, counterfeiting the sticker wouldn't give you any pause - so it's probably just an error, however counterfeiters wouldn't have the review process that would catch typos.
Oddities in labels are a typical sign of a counterfeit. Yes, it's possible to counterfeit labels with good quality, but often it doesn't happen.
If someone manages to do a good job of replicating the labels, hopefully they did a good job of replicating everything else.
On the other hand, if QA didn't notice the label is misspelled, what other problems did they miss (some of which may be obvious to them if they look, but not obvious to me even)
I recently needed to contact Microsoft support, and looking at the support engineer's email address I literally felt like I was being targeted by fraudsters, easily the most suspicious company name I can think of.
yeah, i find the whole azure support lacking.
We have this server with 20+ disks, some in storage pool and now I have no way mapping them out to see which logical drive maps to which vm disk. the support after 3 months of back and forth just gave up and closed it citing known issue without official documentation...
That's actually not that hard. The LUN numbers map through 1:1 to the guest OS. So the external LUN #1 is the internal LUN #1, etc...
Then you can use the Windows OS storage pool commands to figure out the internal mapping.
However, you do have a point -- the disk WWN or GUID or something properly unique ought be available via the Portal or the ARM REST API! That's far easier to automate robustly.
I thought that's how too, but maybe I got my mental model messed up.
Will give it a proper visit, the last time I tried to downgrade some ssd to hdd resulted in windows boot loop. fun times, had to take a snapshot then fiddle it's registry in another vm so it'd use the last known working registry to get it back up.
I believed it then, and I still believe it now. An "evil" BMC (like the ASPEED AST2600 mentioned in this article) is game over security-wise. BMCs are capable of flashing the BIOS/UEFI, capable of inserting arbitrary disk images as virtual CD/etc drives, capable of arbitrary keyboard/mouse input equivalent to having a hardware keyboard/mouse attached, like a remote evil-maid. If you had to pick one single thing to "pwn" in a server it would be the BMC. There's no way it's just a typo.
Bloomberg didn't just claim "evil" BMC, but very specific things that nobody ever could confirm, and Bloomberg never supplied any evidence about. If the Bloomberg reporting, or any followup, had been anywhere close to e.g. the reporting in this blog post in detail, then they would have a believable case.
Nah, thing is everyone jumped on Bloomberg for a "shitty" story back then Now all we have is one blog post and some typos and the majority opinion is ether bad quality control / sloppy organization / indicator of systematic issues or b) potential supply chain attack. Which doesn't make a lot of sense.
Of course it makes sense to treat reporting that claims a thing and provides evidence of the thing differently from reporting that claims something less likely without providing any evidence.
It's very believable, but that's basically all that the bloomberg story had going for it. None of the actual concrete details (of which there were few) made any sense.
Worth noting that the original post mentioned the Bloomberg story and if you click through to the blog post related to that, the post’s author found Bloomberg’s focus on Supermicro irresponsible even though the general issue is a problem.
It's obvious that an evil BMC would be game over because the BMC is the most trusted component in the system (for normal servers that don't have Titan or whatever). But there's no evidence of compromised BMCs actually happening.
Is there evidence of it not happening? Can I do a reproducible build of a verified AMI BMC firmware and compare it against what comes flashed on my motherboard? They have a "MegaRAC Open Edition" but it says it's only for "OCP compliant platforms": https://github.com/opencomputeproject/HWMgmt-MegaRAC-OpenEdi...
Honestly the industry is a decade away from meeting that standard. And given that AMI is the lowest bidder we shouldn't even expect them to meet current best practices.
As the source of the report and the communication between them and Bloomberg remains undisclosed, we can only speculate.
Perhaps there was a Chinese whispers (no pun intended) style miscommunication and while the original source meant "software component", it became "hardware component" somewhere along. Or the attack was actually developed as proof-of-concept but never applied in the wild. The attack is feasible as other security researchers have shown:
Yeah, but after Bloombergs repeated insistence that they got it right and stand behind their reporting I'm really only willing to give them credit it something that actually matches what they claimed gets revealed. Because "BMCs are vulnerable" is not a big story, they don't get to claim that as evidence for their wild claims.
whatever became of it? I thought at the time security experts would swarm and dissect it to death to give us articles after articles of concrete tampering examples... but it just vanished?
That is an unintended vulnerability which can be exploited. While also bad it's not evidence of a malicious hardware installed before leaving the factory.
Okay, assume that your motherboard/BMC is backdoored. What exactly is the sane way to proceed, given that you can't verify it and you have no reason to believe that any alternatives are better?
At home I have my BMCs on a network segment where they can't talk to each other, to the Internet (so no auto-updating), or even to any of my client machines. When I want to use one I connect a patch cable from my laptop to a designated breakglass port on one of my switches.
I do run DHCP/NTP/etc on that subnet, but those services are running in FreeBSD Jails connected to the host OS with vnet(9) "epair" interfaces given RFC3021-style /31 addresses (and/or the IPv6 equivalent) and firewalled with PF on the host-OS side such that none of them have the ability to explore any other parts of my network if they get popped :)
The idea was that it could just compromise your host OS. It doesn't matter that you jail services, since the attack is affecting the hardware underneath the kernel that underlies them; the theoretical attack was more along the lines of "the motherboard watches incoming packets and if it sees a trigger then that packet never makes it to the OS, and if the motherboard wants to send traffic it can either send packets itself (after reading the route table directly out of kernel memory) or modify already-outbound traffic to exfiltrate whatever it wants".
Buy open-source hardware that doesn't have so many compromise points? Assemble some kind of bootstrap verifier out of NAND gates and use that to run your own chip fab?
Weird. A few weeks ago I turned up to a site that wasn't working after a power cut. They had two S5200's configured in a stack as core switches. Both had lost their firmware and we could only access the BIOS on them. All attempts to install/upgrade the firmware was a bust. Both had to be RMA'd and two days later we installed the replacements and 1 of those had the same issue.
TLDR: Just bad naming from the founders because they wanted to keep AMI initials
*
American Megatrends Inc. (AMI) was founded in 1985 by Subramonian Shankar and Pat Sarma with funds from a previous consulting venture, Access Methods Inc. (also AMI). Access Methods was a company run by Pat Sarma and his partner. After Access Methods successfully launched the AMIBIOS, there were legal issues among the owners of the company, resulting in Sarma buying out his partners. Access Methods still owned the rights to the AMIBIOS. Sarma had already started a company called Quintessential Consultants Inc. (QCI), and later set up an equal partnership with Shankar.
By this time the AMIBIOS had become established and there was a need to keep the initials AMI. The partners renamed QCI as American Megatrends Inc., with the same initials as Access Methods Inc.; the renamed company then purchased AMIBIOS from Access Methods. Shankar became the president and Sarma the executive vice-president of this company. This partnership continued until 2001, when LSI Logic purchased the RAID Division of American Megatrends; American Megatrends then purchased all shares of the company owned by Sarma, making Shankar the majority owner.
*
... can we talk about the fact that it has a copyright year of 2003??? Sure, I totally believe there aren't a billion known vulnerabilities in that OOB management chip's code... the damn thing is old enough to vote!
No, virtually nothing made 18 years ago is secure. In fact virtually nothing made a day ago is secure, but when something has been sitting around stagnating for 18 years that means the world has had ~6575x as many opportunities to find a vulnerability.
What on Earth are you talking about? There's countless things.
Any random chip, let's take a max232, from 1987. It hasn't changed since 1987. They are widely used.
There's countless simple chips there that have no "security issues" such as fram and pram chips that just store data and that's it.
Many programmers are only accustomed to seeing the modern JavaScript systems with 5GB of dependencies to print hello world and sure, fine; a virtual machine on a browser, on a userland, on a kernel, there's lots of stuff there to go wrong.
That's not what this necessarily is.
Systems can be simple. You use 1970s era zilog-80 with code that hasn't changed in 30 years every day when you do things like wait at a stoplight, use municipal water when you turn on your faucet, put food in your microwave, turn on your light switch ... You probably interact with and certainly depend on more 1970s and 80s era hardware and software that's been faithfully reordered and reproduced for decades then you do any modern system.
Motorola 6800 (not a typo) MOS 6502, and Intel 8080 clones are in every car, jet airplane, bus and train made in the past 30 years or so and they are still being used; oftentimes with no real modifications, for decades.
They are used exactly because there's no surprises and everything is known and accounted for. Newness, the thing you advocate for is the vulnerability.
Society entrusts things like interbank transfers (trillions a day) and international carrier switching on these systems.
This strategy works so well that these systems are utterly invisible.
I once ordered a large number of Compaq computers all identical for a credit union in Houston and found an esoteric combination of a specific hard drive firmware with a specific network card firmware that caused the machine to do very strange things when using an expensive piece of software (Hummingbird) to add a protocol to allow Windows 3.1 to be able to communicate with an IBM AS/400 to replace dumb terminals with terminal emulation on Windows. My memory may not be correct about everything I just said, but close enough. The point is that not every identical machine came with an identical hard drive or network card, but all considered comparable enough for Compaq to ship them to fulfill the order.
I was incredibly impressed at the speed that Compaq escalated the issue from a first line support person to having a couple engineers drive out to the credit union to see it with their own eyes to determine a fix (I had no pull or special access, and we're only talking 150 computers). It was convenient that this happened in Houston so the engineers were in close proximity.
This article makes me skeptical that HP would handle it the same way today that Compaq did at the time based on HPE's response to STH. I hate the buying process with both Dell and HP, but they are the brands I buy from when it is a server or workstation, because I love the build quality for both brands on their enterprise servers and high end workstations. I've never owned a consumer device from either brand, but I would not have the same expectations on a consumer device from them.
i used to work for a Dell contracted repair/shipping center. Corporate screw up happen all the time. there is one incident, before winter shopping holidays, Dell ordered thousands of 27" all in one pc for retail sales from a China OEM.
the problem is the all in one is not all in one. the back of all in one is missing a PC! the all in one is now a 27" monitor with a empty lump where the PC supposed to be. Dell screw up. Dell end up selling these all in one PC monitors to Corporation for cheap.
Wouldn't the LCD panel in the all-in-one (missing actual x86-64 PC) be something like an LVDS interface between the intended motherboard and the panel? this would mean you couldn't just sell them off as cheap weird monitors since they would have no external HDMI or displayport connectors or interface to plug a computer into.
Persons intending to use it as a monitor would have to go scrounge ebay for something like an HDMI-to-LVDS interface board and wire it up themselves.
Regardless of the reason for the misspelled stickers, they are so prominent and visible that any (western?) employee to saw the boards should have noticed. At the very least, there should be some QC human or even automated visual software which would identify oddities.
In fact, what's the point of putting a sticker on if nobody is going to look at it?
Marketing? Without seeing the internal quality check requirements, and how any deviations are to be treated and were treated in that particular case, it is kind of pointless in reading to much into it.
It's just a sticker, but a sticker that has a license. In the world where we live I would expect that something like this has to be aproved by a legal team.
Neither Dell nor HPE manufactures switches (the former never has, the latter hasn't for the past few years). So, these are all sourced from an OEM like Edgecore.
And, someone at that OEM ordered a bunch of misspelled stickers. Easy mistake to make, if the latin alphabet is literally foreign to you.
And if you think that sticker is bad? Wait until you see the actual firmware, oh boy... (I had some fun Edgecore LACP bugs take down an pretty sizable network. Things got slightly better once they moved to Linux-based firmware, but never to the point that their kit was, like, entirely reliable...)
Well, for many stickers, they have to be ordered from original vendor / AMI. I guess this is not the case here.
Turns out it is coming from AMI, but AMI Taiwan.
>AMI Taiwan needed to get license stickers for the local market. Instead of using the “American Megatrends” MegaRAC PM sticker template, it decided to make its own that had the misspelling.
edgecore is just a marketing name, the actual company is accton
they're generally a competitor of companies like compal, clevo, quanta. All well known in Taiwan if you're in the business of having 3rd parties manufacture your stuff.
some 15-20 years ago most of the big taiwanese electronics manufacturing companies (top tier x86-64 motherboard makers would be a good example) moved a lot of their factory operations to mainland china, for lower cost labor.
I frequently work with Taiwanese manufacturers in the industrial PC space. Depending on what the client needs, I can specify whether the equipment is made in China (Suzhou) or Taiwan. Having the equipment made in Taiwan costs at least 10 to 15% more.
I think so, but assuming that is the case... what's to stop a shady chipmaker from printing properly-spelled "American Megatrends" stickers? More generally... are there any actual protections offered by genuine stickers?
The article makes this out to be a major supply chain security issue, and that only makes sense if branding stickers are actually reliable for validation purposes. But that seems... nonsensical? Wouldn't stickers be very easy to forge?
But, I don't work in supply chains. Anyone with better expertise in this area able to chime in?
I will admit I skimmed the article, because it is long and overly-detailed for my level of interest, and because it lacks summary sections.
It's not at all that a properly spelled sticker gives assurance that it's not counterfeit... it's just that a misspelled sticker is such an obvious sign of a potential counterfeit that it's basically the #1 thing that any counterfeit/suspect items program teaches people to look for. Most people working on counterfeits don't speak English so it's very easy for these kinds of mistakes to slip through, and on the other hand they're rarely made by the genuine manufacturer which usually has a process to check for this kind of thing even if the engineering work is done in a non-English speaking country (most of all that the logos usually come from off-the-shelf art files from the marketing department, so no one's even typing the name to make a mistake).
Almost any corporate or institutional counterfeit or supply chain security program will explicitly teach you: if anything is misspelled or shows other obvious mistakes, hold the part as a suspected counterfeit. It's a pretty good quality indication.
So of course manufacturers do genuinely make spelling mistakes sometimes, but this context makes it a pretty embarrassing and serious thing to do. It's like your bank misspelling their name in an account notification: sure, in some extremely theoretical sense it doesn't mean anything, but in practice they're giving you exactly the signal that everyone tells you to check for to identify phishing, and it raises questions about their processes that they let it slip through.
Okay so dumb misunderstanding whatever, am I alone in being ONE HUNDRED PERCENT ready to trade ill-gotten lucre in exchange for bootleg MACROSOFT WANDOWS branded streetwear
If I’m spending thousands of dollars on these devices, I expect them to not screw up something like that.
It would not be the first time someone sends a device for RMA to be told it’s out of warranty because it has been tampered, when in reality it was a manufacturing mistake.
It's a sign that there wasn't any QA whatsoever - if they have missed something as obvious as a typo on a label, how can you assume that all the technical parameters of the components are up to spec?
1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches.
2. Author fears supply chain tampering.
3. Author contacts American Megatrends (AMI). Hijinks ensue. AMI eventually confirms that it's a typo.
4. Therefore, there is no supply chain tampering.
5. But author is concerned about what this means for the state of supply chain verification.