We had a similar problem at work in the late 90s. A member of staff reported that their mouse would stop working between certain hours of the day. It had apparently been okay in the morning, stopped working over lunchtime then started again later.
On some days it would work perfectly all day long, but on others it would stop working between those hours.
The biggest clue was it would always work perfectly on overcast days, but on sunny days this strange behaviour would manifest again.
Turns out the problem was related to the mouse being a cheap mouse. The case had very thin plastic.
The mouse was a ball mouse, and it worked by shining an LED into a sensor on each of the X and Y axes. On sunny days the sun would completely overpower the sensor due to the plastic case being very thin and on overcast days it would not. On sunny days the mouse would only work when the sun had moved around the sky to cast a shadow over where the mouse was being used.
Reminds me of a problem that I had (many years ago) with my iPhone 4 - if I tried to boot it in a dark place, it would get stuck on the Apple logo in an infinite boot loop.
Turns out some versions of the Pangu jailbreak for iOS 7.1.x would crash during boot if the reading from the ambient light sensor was below some threshold. To this day I don't know the exact explanation of this bug, but it seems that Pangu included some unnecessary code that messed with the light sensor [1].
If you don't believe me, there is a huge reddit thread[2] with a lot of people confirming this.
That's funny, there exists a similar issue with the LG G7 that a friend of mine ran into several years ago. The fingerprint sensor on his phone just straight-up completely stopped working, and subsequent OS updates did nothing to fix it. At first we assumed it was hardware failure, and he was ready to send it to a repair shop. While investigating it I saw a comment somewhere that it had something to do with the light sensor, and after holding my thumb over it for 10 seconds it "magically" started working again after 4 months of being completely non-functional.
Seems unlikely. I don't have access to the paste but from the comment below it I think it's probably a false positive that Pangu was doing something with the sensor. (Not that I don't doubt that the sensor could be the problem, it's just that the code is not very conclusive.)
I had a similar problem, but in the opposite direction. My cable internet speeds at home were fairly good (for the US, anyway), but sometimes would absolutely bottom out. Not dead, just glacially slow. After troubleshooting everything under the sun, I came to realize that the problems would happen not when it was raining per se, but when it was heavily foggy or misting. Normal to heavy rain was fine.
Called the cable company, tech came out. Everything inside was fine, but the cable from the main line to the house had a tiny cut in one spot, not enough to really affect the connection, but enough for ambient moisture to work its way in and foul the connection.
on dslreports or broadbandreports there's at least two instances of me complaining about two cable companies because, at last, it was figured out there was moisture ingress in the LE (line extender, usually on cable lines on poles). The only common denominator was it happened during prime time, every night, and went away around midnight.
The other common denominator was the cable company refusing to believe it was an issue with their equipment; this meant it took a couple of months of calling them every night until they finally sent a technician and a manager to my house to verify that I wasn't wrong, leaving my house, coming back 15 minutes later to say "it'll be fixed tomorrow, there's a problem with the LE balance up the road" - and then the issue is resolved.
Now this doesn't sound so bad, until you learn that the first time this happened to me, i had only VoIP - so the internet would start to foul, i'd call the cable company, and the tier 1 would reset my modem at some point, and then i wouldn't be able to call back until after midnight (or whatever), when there was no longer a problem. So after a week of this, i would walk 30 minutes - one way - to a pay phone (remember those?) once the internet slowed, call them, explain that i couldn't do anything they wanted me to do physically, since they disconnected my phone line every time i called.
This is what happens with a de facto monopoly.
I will never pay suddenlink another dime, even if they're the only terrestrial provider, for whatever reason.
Interesting, I wonder if I’m experiencing something a little bit similar that Comcast can’t seem to debug.
Almost every day, in the heat of summer, I get one to five 10-minute outages as soon as the temp gets over about 80F. More when it’s hotter, usually. Usually it results in a modem reset, so it’s hard to tell how long the actual outage is.
Been happening for going on 5 years. They replaced the under-street cable from our house to the junction box across the street to no effect. I suspect it’s that junction box, but afaict, none of my neighbors that share that junction box have the same issue. Not very fun to have your WFH day collapse unexpectedly in the middle of the afternoon.
Strangely, for the last month we’ve had several days of 80+ temps with no sign of outage. So fun.
Edit: yes of course multiple modem replacements and inside cable checks, to no avail.
We had the same problem at an old house. There was a cable splitter in the attic that was expanding in the heat and losing connection to the cable. We bought a heftier one and moved it under the insulation in the attic.
Yeah, probably very similar thing if that pattern is true. It's the shift forcing your modem to change speeds, but neither side being willing to accept it.
If you can, try forcing a level at/below the speed you get during the breakage and see if it just rides it out. If it does, shift it back up and plan your coffee breaks around it. Or don't, I'm not your mother
Several day's of 80+ temps, meaning it hasn't dropped below 80? Possibly it is just getting above 80 before you start your work day. And not dropping below until after your work day has ended?
I've experienced something similar except for temperatures below ~32-36 degrees. At this particular location it would result in a ~1hr outage going below that temperature, but not when it went back above it for some reason.
I think you’d get this problem, monopoly or not, whenever cost saving measures are in place (and they always are, for good reason) at the customer-interface level.
Maybe there should always be a hidden option that only people that meet a certain troubleshooting ability threshold get access to when calling in for tech support….
> I think you’d get this problem, monopoly or not, whenever cost saving measures are in place (and they always are, for good reason) at the customer-interface level.
I'm guessing that these scripts that we're all complaining about solve 95% of problems customers call in about. Sure makes things painful for the 5% of cases, though.
I've been a (grudging) Comcast customer for ~17 years, and I have been impressed by how their monitoring has improved over that time. It's been quite a number of years since I've had to convince them that I had an actual problem that their systems didn't automatically detect.
There is a hidden option - you can call it "proof of work", or "proof of determination". You keep calling, and trying ways to escalate, maybe even send a paper letter; eventually, something in the customer "support" process will yield and you'll get through to someone who can actually help you.
A customer's DSL connection dysfunctioned every evening during December - but worked fine the rest of the year... Culprit: interference from nearby Christmas decorations leaking EM all over the place.
A customer's DSL connection dysfunction's frequency increased mornings and evenings. Culprit: the lift's electric motor leaking EM all over the place.
A bunch of DSL connections degrade when traffic increase... Crosstalk in big cables of course !
The sort of fun incidents that take a good while to troubleshoot... I'm glad we are migrating away from DSL to fiber: either it works or not !
It's not like fiber doesn't have its own weird failure modes. Favourite one I heard was shoddy belowground work while crossing a street. No problem with ordinary car traffic, but heavy trash haul trucks could interrupt the link.
An ISP my friend worked at was having weird outages in one area, and it turned out that they had an apartment block built right in the way of their free-space optical link. Surprisingly, it was fine at first because the link went straight through it without obstructions, window to window. But when they started to add window panes, finishing the construction, the link became spotty, and adding the doors blocked the signal completely.
Having had to debug many of such cable issues in the past, it's baffling to me that cable companies aren't proactively monitoring for things like this.
They have all the data available on their end, as far as I can tell! (Unless DOCSIS modems somehow don't have a standard "signal receive report" functionality?)
Telcos used to monitor their copper outside plant for moisture. This was called Automatic Line Insulation Testing in the Bell System. The ALIT system ran in the hours before dawn. It would connect to each idle line, and apply, for tens of milliseconds, about 400 volts limited to very low current between the two wires, and between each wire and ground, measuring the leakage current. This would detect moisture in the cable. This was dealt with by hooking up a tank of dry nitrogen to the cable to dry it out.
Here's a 1960s vintage Automatic Electric line insulation test system at work in a step-by-step central ofice. [1] Here's the manual for automatic line insulation testing in a 5ESS switch.[2] 5ESS is still the major AT&T switch for copper analog phone lines. After that, it's all packet switching.
For fiber, of course, moisture doesn't affect the signal.
This led to an urban legend: "bell tap". While Western Electric phones were designed to not react to the ALIT test signal, many cheap phones would emit some sound from the "ringer" when the 400V pulses came through, some time before dawn.
If you're really into telephony history, the Internet Archive has "The History Of Engineering and Science in the Bell System" (3 volumes) online.
If you have to build reliable distributed systems, it's worth understanding how this was done in the electromechanical era of telephony, where the component reliability was much worse than the system reliability. "Number 5 Crossbar"[1] is worth reading, but hard to follow if you have no idea how telephone switching worked and are unfamiliar with the terminology.
Number 5 Crossbar, in current terms, was a collection of microservices. There was a big, dumb switch fabric, and "markers" which told it what to connect. Other microservices included trunks, originating registers (which listen to incoming dial digits), senders (which sent dial digits to the next switch), billing punches (which recorded toll call data for later billing), translators (which held routing tables), and trouble recorders (which logged errors.) Central offices had at least two of each resource, for redundancy. Resources were "seized" as needed from resource pools, with a hardware timeout and alarms to prevent resource lockup. If something went wrong in setting up a call, it was retried once, using different resources. If it failed on the second try, the caller got a fast busy and there was an alarm and a trouble recorder dropped a trouble card. Markers did not have persistent state. They started each call with a reset. So they could not get stuck in a bad state.
In the entire history of the Bell System, no electromechanical switching office was ever down for more than 30 minutes for any reason other than a natural disaster or a fire. It's worth understanding how they did that.
Not truly related to the post content, but there is something about the way these old manuals are formatted/printed that immediately inspires confidence in the contents.
Maybe because you know that someone spent a lot of time on it before it was published since no adjustments could be made after the fact.
The previous version was a panel of blinking lights
called the "trouble indicator". When an alarm sounded, someone had to go to the panel and record by hand which lights were on. There were about 200 lights. So the trouble recorder, which recorded that info automatically, was added in larger central offices as an upgrade.[1]
We still have a land line. When a call comes through the phone often gives a gentle "peep", then a pause, then goes full-on ring. I've started to react to the "peep".
But every evening, mostly around 21:00 or so, the phone gives a gentle "peep" without then ringing.
Crap electronic ringer, probably. If you put a scope on the line, you should be able to see what's happening. Remember to be prepared for higher voltages, up to 400V.
There are various weird, obsolete signals in analog phones. Ring pulse alerting signal. ALIT test. Polarity reversal. Ring to ground. Ground start. Caller ID (1200 baud FSK between the first and second rings) DSL. Basic talk and ring was standardized around 1900, and everything else is backwards compatible. Ringers are supposed to ignore all that stuff. People who implement Asterisk PBXs are into this.
Here are some actual waveforms, if anybody cares.[1]
In my observation, to a first approximation, cable operators take off-the-shelf equipment, connect it, power it on, and bill customers for it. They don't really have the r&d capability to innovate and create new monitoring solutions quickly.
It might happen that an equipment manufacturer sees an opportunity and builds something, but then they have to go into a long sales cycles to convince operators to use it. Operators are in a duopoly situation in most places, so quality of service is kind of a secondary concern for them - customers may get annoyed, but as long as the competition is not vastly superior, few actually switch. It is not a market prone to innovation.
Common issue in Ireland for DSL customers. Damaged copper cabling would leak water when it rained causing dropouts and lower speeds. Telecom engineers would call out on days when the copper had dried out and be unable to find any fault. Turns out correlating such reports with weather reports is hard. :/
I ran into a similar issue, except internet and phone would get really bad on a cold morning.
Tech showed up around noon, saw I was indeed having a bad connection, went and checked the signal at the junction box for the street (can't remember what you call these) and everything was normal there, so he closed it back up again and double checks the signal at the house again, but it was fine. He walks the lines to double check but everything looked normal.
His best guess was that moisture was condensing ever so slightly inside the junction box that morning, and was let out as soon as he opened it at around noon, which fixed the problem.
Moisture in copper cables is what slowed me down too. It was in a section up the road from me.
However now that fibre is installed, it’s glorious and works in the rain.
I had a similar problem, due to an old line running to my house; liquid getting in, etc. And when it acted up, I'd call the cable company and be like "look, I can show you I'm losing packets right now... I need you to run tests on your end to confirm". And every time, they'd tell me they could schedule a tech to come out and take a look at it. Only, I couldn't "schedule" the problem to occur when the tech came out.. so they'd come out, declare all was fine, and leave. It was infuriating.
Eventually I called so many times and had so many appointments, that the tech lead gave me his direct number and told me to call him directly the next time it happened. When it did, I did, and he ran some tests, and confirmed there was a problem. I don't know that we ever got it sorted out (it was a while ago), but just getting them to agree there was an issue took a very long process.
I have a garage door that will not close on sunny days.
Same sort of problem. The obstruction sensor at the bottom of the door is confused by the strong sunlight and the door stops closing part way and re-opens.
I've tried a toilet-paper tube around the sensor but that isn't always successful. I really wish there was a laser sensor to replace it with.
The garage door obstruction sensors are located inside the garage, so it technically might be an indoor product.
Although, the possibility of a garage being oriented such that the sunlight would directly hit the sensors while the garage door is open seems like it could be a not infrequent occurrence.
Is it a Genie system? My old Genie system had that exact same problem. Even making large sun shields out of Amazon boxes didn't fix it.
I had to replace my opener and door anyway and had a conversation with a tech about it. We decided on a LiftMaster in part because their sensors are very good at dealing with sunlight.
I have this exact problem and (mostly) fixed it by swapping the sensor and transmitter. I just cut the wires and spliced with electrical tape. Now the problem still happens but only sometimes in the fall and spring when the sun's angle is just right. This is with a west facing garage about 41°N latitude USA.
But yeah, why this isn't laser based, or using a light frequency that is less affected by sunlight? Probably cost, or ignorance.
It varies by brand- some brands are better at filtering out sunlight than others. The home builder should know not to use certain brands in garages that face the sun... but they often don't.
It'a not laser based so the sensors don't have to be perfectly aligned. Keeps your garage door working when you kid knocks it with their foot.
I had a VCR back in the day that refused to function if you opened its case. It turned out that instead of using physical switches inside it used pairs of lights and detectors that would give false positive results when ambient light shined on them.
No, there's not any call for high voltages in a VCR, outside the feedback loop of a SMPS if it's fancy. The most common thing to use light sensors is just detecting the difference between tape and clear leader at the ends of the tape (or broken tape).
Your mouse story makes me think of the day the CI system at work turned out not to be robust to vibrations.
One day we started having flaky tests, seemingly out of nowhere. We quickly identified that the issue affected tests involving graphical X client applications, but then we struggled to make further progress. The issue was just impossible to reproduce in other conditions... Well, as it happens, the CI jobs were running on some desktop machines we had installed somewhere within our premises. It turned out that some gentleman had plugged a mouse into one of the machines, and left it lying around on the shelf. Since then, when one of the machines was under a heavy load, the fans would spin faster, causing more vibrations, in turn causing the mouse to move, ever so slightly. And for ungodly reasons, this had side effects on tests.
Fun fact: the machines were not on my site, I managed to diagnose this over SSH. I was quite proud :-)
When it’s sunny, my wife’s car can’t open the garage door, and my car requires getting extremely close. Once the sun goes down, we can both open the door from the street.
It turns out our solar panels (or the optimizers, or the inverter) emit radio frequencies that interfere with our garage door opener. When the sun is out and they are producing energy, the interference is stronger than the homelink garage door opener.
A few years ago the garage door openers started working fine. It took a few days to realize it was because the inverter had failed.
I’m fairly certain there are some FCC regulations that would require our installer to fix it, but that relationship soured during installation and I’d rather deal with an unusable garage remote than dealing with them for warranty work.
If they had a HAM in direct neighborhood, I imagine said HAM would already pay them a visit - the interference from the inverter is likely not constrained to the ISM band.
I always saw it written as either "HAM" or "ham", and I assumed the latter is the "young generation doesn't give a damn about spelling or punctuation" spelling, and therefore that the former is the correct one.
I used a coax cable to move the antenna closer to the exterior wall and didn’t see an improvement, however, I might not have grounded the shielding properly. I’ve had to replace the control board since then and didn’t replace the antenna on the new board, but may try that again.
How old is the garage door opener? Older ones used frequencies that are more susceptible to interference from certain sources. It's possible to buy new receivers to connect to your existing door opener.
The first was a VDSL connection I had at home. It worked great (fast, for the time) except when it didn't. It always failed in the evening. Techs would come out, bless it as being good, and leave -- because of course it worked while they were there. Unless they showed up and it was broken and then they'd declare that it was an outside problem, and that they'd have to get someone else to fix it (because the residential techs can't do overhead work).
I made lots of (very polite) phone calls, which results in more refunds and more service calls. More than once, my driveway and the street in front of my house looked like an AT&T convention.
This went on for months.
I had direct numbers and emails for tier 3 support and the local manager who oversaw this plant. We were all getting to know eachother too well, and there were boots on the ground addressing this problem as many as three times in week.
I eventually noticed that as the days got shorter so did the evening outages...and that if it was a cloudy day, then that day was often outage-free.
I had an epiphany: The problem might correlate with the angle of the sun, and the duration of exposure!
I checked my logs and the past weather, and sure enough: It lined up.
So I reported my findings, even though they seemed like nonsense as the words came out of my mouth, and they sent out some crazy-haired guy with bluejeans and an untucked shirt who was clearly not used to wearing a uniform, and who was also obviously not normally customer-facing.
"I know exactly why they can't find the problem," he said after I reiterated what I'd learned. "Your neighborhood still has old lead-sheathed overhead lines, and nobody knows how to work on that anymore."
"But I'm certified on that. I'm going to go back to the shop, pick up a bucket truck and get your line fixed. It will take me most of a day to do this, but I will be back when I'm done."
And it was getting pretty late, but he did come back to let me know that he found some things and fixed them. And I don't know what those things were, but it was fine after that -- and it stayed fine.
Thermal expansion letting cosmic rays leak into copper pairs wrapped in paper, tar, and lead? Who knows. I certainly don't know.
I've never encountered that stuff professionally (and it isn't your grandfather's 25-pair cable) and as this dude said, "nobody knows how to work on that anymore."
On some days, exclusively in the morning hours, the printer would fail to detect the start of a new label, printing over several labels.
After connecting remotely and checking the usual (queue, network connection, drivers etc), I asked my colleague to call me, as soon as it happened again.
When I went there, I saw that a ray of sunlight hit the printer.
The windows had shutters, but there was a gap.
Label printers detect the gap between labels using a laser.
And for some reason, the printer's case had a clear window at the top.
I printed an empty label and stuck it on the little window.
I'm amazed an IT department would troubleshoot deeply enough to figure out it was the thin plastic letting in interfering light on sunny days.
I would have guessed they'd shrug at the first sign of trouble, swap it out with a known-working mouse and mark the ticket resolved... unless all the replacement mice were thin plastic too, I suppose.
I'm assuming it was a trackball mouse from the description. The OP said it was cheap so I don't think cost was an issue but from my experience some employees are very particular about their peripherals (don't blame them one bit!). If they're important enough (or maybe just nice enough) I could absolutely imagine spending the time to make sure their preferred device is working properly.
For myself, I can't remember a time when a mouse cost more than an hour of an IT guy's time.
I suppose a good office-computer mouse in 1990 would cost $100 ~ $200 (say $350 today). In that case, yes troubleshooting it for a day would make sense, especially if it's not an isolated problem.
> For myself, I can't remember a time when a mouse cost more than an hour of an IT guy's time.
This is not a sensible comparison to make. Support staff have a lot of free time. They have to, because they're support staff -- if they were always busy, then whenever a problem arose, it would be impossible to get support.
So to have the IT department playing with the mouse is unlikely to cost the company anything. If something comes in that's more important, the mouse problem will be put aside. If they have nothing better to do, they can play with the mouse.
Your car repeatedly doesn't start so instead of taking it to the shop, you... write a letter to the CEO of Pontiac who not only actually reads the letter but also personally dispatches an engineer to waste a week going out for ice cream? And Pontiacs have a known vapor lock design flaw that only you, the letter writer, are experiencing? And you've only experienced it on your ice cream runs? And you've never got the vanilla ice cream but took a little extra long so the vapor lock dissipated and disproved your cute theory about vanilla ice cream?
Seriously, this is one of those dumbass stories that come from your boomer relatives with the subject line "fwd: fwd: fwd: fwd: fwd: fwd: fwd: fwd: re: fwd: fwd: fwd: fwd: fwd: fwd: vanilla ice cream"
Nobody actually believes this story is true, right?
While I appreciate your point, I think anyone who has spent sufficient time troubleshooting complex systems has dealt with similar types of problems, and can grasp the _spirit_ of the story.
In fact, I'd argue the quaint style of the story does geeks a favor: if it's appealing to normies, maybe they'll appreciate us technical folks' perspective a little more.
actually the story is detrimental to helping people understand how technical systems and troubleshooting work, because it's so poorly invented.
lots of people, both technical professionals, and non-engineers who are observant and have an appropriate level of belief in causality, troubleshoot transient failures like this all the time. a difference in the amount of time between shutting down the engine and starting it up is one of the first things that someone like this would test, or control for. it's beyond implausible that the second time the guy got vanilla (after riding along for 4 trips, two long and two short), the engineer didn't raise the question of how long he was in the store.
people troubleshoot things like this by being able to separate causes which are plausible, although unlikely and surprising, from things which aren't remotely plausible. the 500-mile email story and the stories above about sunlight interfering with sensors demonstrate this.
if you're the sort of person who believes that the type of ice cream you get might affecting your car's ignition - the type of person who buys ice cream often but never thinks about how long the errand takes, you simply never get the point of being able to make a pattern between those two things. the second time your car doesn't start, you blame it on the scratch-off lottery ticket you won $2 on which used up your supply of luck for the day. the third time, you conclude that the car ignition knew you were late and likes to choose its failures to cause maximum annoyance. and the fourth time, you realize that your mother-in-law gave your car the evil eye that morning.
the story as told, especially when presented as a real parable about engineering rather than an amusing myth, is frankly insulting to the other type of person. the untrained, not necessarily educated person who cares about machines and believes in material reality. the person who starts checking their watch each time they go to the store and a couple of weeks later is telling their mechanic friend "if it's more than 3 minutes or so, it's fine. but if you try and start it before 2 minutes, then you have to wait another 5 before it's ready to go".
And having a stranger along for the shopping trip doesn't affect the timing more than the extra walk to the back of the store?
And the engineer is sitting in the car on the first night and it "wouldn't start," which signals the end of the episode for the day. Were they stranded? Did the car start after a few tries, which would have given a huge hint about the root cause? Surely the engineer who had reproduced the issue would quickly narrow it down by running diagnoses on the car itself.
But the family dynamics are the most improbable part here. How does the family have this predictable routine and not simply stock up on ice cream? The family has enough kids that the consume a whole $unit of ice cream per day. So with that much chaos in the house, how does the dad justify going out for a drive after dinner when the chaos of family multi-tasking (cleanup, chores, homework, bedtime) is at its peak? "Oh, look. Out of ice cream again. I'll be back in a few!"
Aisle caps commonly feature a product that would ordinarily be found alongside a bunch of closely-related products somewhere else in the store. But I don't think I've ever seen a refrigerated aisle cap.
Obviously, they'd put the most popular flavor way at the back of the store, and the least popular flavors at the front. The store is not in the business of maximizing throughput of customers - quite the opposite, they want customers to spend more time walking and getting lost between shelves, as this maximizes the amount of wares moved.
I believe the aisle caps are bid for, and then arranged by, the manufacturers. The store just sells them the space.
The manufacturer is unlikely to see a problem with putting a display of very popular products right at the front of the store where people can't help but see it.
So you think Vanilla Inc, the company that only makes vanilla flavor ice cream, is paying for a whole chiller that they fill with vanilla, since all the other flavors are not as popular?
Or perhaps, unbeknownst to everyone in the world who uses the word 'vanilla' to mean 'mundane', vanilla ice cream actually enjoys far higher profit margins than all other ice cream.
No, I think Dreyers, the company that makes ice cream, uses the limited space available in the aisle cap to showcase one or two of their most popular flavors. (Or one or two flavors that are seasonally relevant.) That's a totally normal use of the aisle cap, just like how a Triscuits aisle cap is all original Triscuits and the many secondary flavors that Triscuits come in have to be found in the crackers aisle.
It would be literally impossible for an aisle cap to feature every ice cream flavor available - there are so many that each flavor would have very little representation in the display, and the concept would fall apart as soon as anyone bought something from it. At that point, you're paying a bunch of extra money to send the message "check out our least popular flavors".
It's unclear whether you're claiming 1. the vanilla ice cream is sometimes kept in a separate chiller must nearer the entrance and this obviously made-up story is perfectly true, 2. at least some stores exist where the vanilla ice cream (and only the vanilla) is kept in a separate chiller must nearer the entrance, or 3. the vanilla being kept by itself in a separate chiller has elements of plausibility, and can not be dismissed out of hand, even though it's possible that this specific set-up has never existed in any actual store in real life, ever. Which is it?
If you read my comments before deciding you needed to respond to them, it would be pretty apparent that I am claiming none of those things. I specifically noted that a refrigerated aisle cap is implausible.
What's not implausible is the idea that one flavor of a product with several flavors might be located far away from all the other flavors. That happens all the time.
The most popular flavors, by definition, benefit from advertising the most. The goal isn't to achieve equality of popularity between every product you sell. It's to sell the greatest number of products!
The advertising costs the same whether you advertise popular flavors or unpopular ones, but you'll get a lot more sales by advertising the popular ones.
Go check out a grocery store, see whether the aisle caps slant towards popular or unpopular product varieties.
Another similar anecdote I heard before was related to a wireless device, and some employees flying a drone during their break, generating interference.
My parents have some old Gateway amplified computer speakers. Came with the 386!
They still work perfectly... except for a regular pop of noise every few seconds that would intermittently show up, that scaled with the volume setting.
It turned out, their portable phone (read: landline with short-distance wireless RF handset) would ping from the base station to the handset, if it were off the cradle, which was being picked up by the unshielded line-level audio cable and amplified.
Moved the base station further from the cable, pop disappeared.
you can still hear cellphone and wifi noise in crappy amplifiers - i have two sets of active muff hearing protection - the ones with microphones on each earpiece, and if you get inbetween a beamforming WAP or near any wifi antenna, or near a cellphone, you get "brrz bz bz bz bzzzzzz tiktiktiktiktik".
But this is different than the old <2g/edge phones, which wouldn't interfere unless you were about to get a call - because the tower said "where's this phone?" and your phone would max out it's tx and say "here i am!" and that's what you heard. This is probably incorrect, but based on my observations this is what occurred.
Remember the doodads you could put on your startac style phones on the antenna bit, with LEDs in them - they'd light up when you were about to get a call, as well, by design!
We did a lot of wireless (2.4GHz range) sensor development at my last job. It was a rule of thumb to avoid any testing at lunch time since the microwave generated so much interference, everything would fail when someone wanted to heat up their meal.
Ugh... Mom and the microwave were a scourge back in the days of yore when I hosted servers for friends in various games. I was hardwired into the router as "the keeper of the hardware", but the wireless would just get schlocked every time she heated up her coffee.
And Mom ran on coffee. Lotsa coffee. Sometimes I think she just did it because she didn't hear enough complaints coming out of the speakers.
I rejoiced the day that thing died. She's now got an even beefier one, but it doesn't interfere on the 5ghz bands at all, and I'm not testing the 2.4 out of respect to the spirit of that ole menace.
reminds me of testing out a pinewood derby track in my backyard for our cub scouts pack. It had an IR sensor that would detect the cars at the finish line. When someone was standing next to the finish line, it worked flawlessly. Otherwise, it was very flaky and randomly would trigger without the cars trigger it. I pretty quickly surmised it was interference from the direct sunlight, so we put up a pop-up shade over it and it worked flawlessly (without someone standing nearby, coincidentally casting a shadow). The other dads were amazed that I figured it out, but it's just one of those things you learn from experience (and some background knowledge).
This happened to me in my first job and it took me weeks to figure it out. The penny dropped and I put tape around the thin gap in the casing where the top joined to the bottom and it fixed it immediately.
This is a secondhand anecdote, but it’s pretty funny. Back in the days of server rooms, a friend’s server for his company would reboot every day around 5pm. They checked everything they possibly could with the OS, they would be logged in and running checks on it and it would spontaneously go offline for about 5 minutes and reboot every day. Finally they decided to go stand in the presence of the server around the time it goes down every day. They watched a cleaner come into the room, unplug the server rack, plug in their vacuum and vacuum around the servers, and then plug the server rack back in.
Oh I have a similar one. This one is first hand, I was in the room when we were debuging the issue.
We were developing a smart camera product which were counting traffic on a road. So for example a city council would install this camera somewhere on a road and it would generate statistics of how many lorries, and passenger vehicles, and motorbikes used that road.
One of our cameras exhibited a problem where it restarted every day around roughly the same time. It wasn't exactly the same second though, in fact there was a clear pattern to it. One day it would restart at 19:12:10 and the next day two second later, then again the third day two more seconds later. (not the real timestamp and i don't remember the real time deltas either, but there was a clear progression)
After much debuging we learned that the issue was that as the sun was settling some street furniture projected a shadow in front of our camera. Our software wrongly concluded that it is a vehicle and started collecting information about it for classification. But of course shadows creep a lot slower than real vehicles so it run out of memory before the "shaddow vehicle" has passed out of the frame. And once we run out of memory the system froze and then got restarted by a watchdog.
Turns out the pattern we have seen in the timestamps was caused by the angle of the sun changing which made the shadow trick our algorithm just a little bit later every day.
I mean it actually happens. Not powering down the actual machine, but in my last house the landlord had a cleaner visit every fortnight. Occasionally I came home to find downloads broken or my remote connection would drop randomly in the afternoon. We lived in an old house and had a wifi extender to reach the upper floors (and not enough sockets). The cleaner would unplug the extender to vacuum the kitchen and half the house would go offline.
The ice cream story reads too much like a dramatisation to be truly believable, but accidental and repeated unplugging is common I expect.
I think the vacuum story gets retold because it's so close to real-life stories.
The UPS on a server I manage would trip once a week around the same time. The old story came to mind and sure enough, once a week it was time to vacuum and someone would plug in a vacuum cleaner into the same circuit (700W vacuum cleaner on a 100V/15A circuit), causing enough voltage dip to kick the UPS into gear.
I've never heard that one. It has many more characteristics of a modern urban legend than the simple "janitor unplugs the server then plugs it back in" story. Urban legends are essentially modern day folk tales. They'll often involve fear, horror, or even humor, and they're frequently cast as cautionary tales. They tend to be reasonably detailed (which helps keep a listener's attention), but at the same time they frequently not pinned down to a specific time or place. They're almost always told as secondhand stories (e.g. it happened to "a friend of a friend"), and tend to fall at the edge of plausibility.
But, the key thing is that urban legends always reflect some sort of underlying societal belief or worry. "Janitor polishes off patients" is about the fear of death, and how it can come for anyone at any time, and there's nothing they can do about it. "Janitor unplugs server every night at 5" is probably more about the idea that strange things happen sometimes, but it usually turns out that there's a good explanation.
Essentially all these stories are apocryphal. Even this vapor lock story.
Sorry but even before today, having a automotive engineer sent to a random person's home over what clearly sounds like a quack letter seems implausible to me. The dictates of capitalism, human resources, and the politics of the workplace would make this difficult if not impossible. Even in the past when there was more human capital in support positions and more of a sense of customer service.
Way, way too many suspicious stories involve high-level people being involved in trivial issues. I just find it all pretty suspicious. Real stories tend to start with poor customer service at the dealership and being mocked by managers and mechanics. Not some unrealistic ideal white knight manager sending off engineers to people's homes. Imagine how many weird letters a place like pontiac gets. They don't have the manpower to do this if they actually chose to do it, and engineers might balk at the idea of doing at-home support too.
Pretty much any "idealized Americana" business story should set off BS alarms in us. "Oh a trivial problem with your car? No problem ma'am, I'm sending our top engineers over tomorrow," doesn't happen because its costly and unsustainable. Instead ask anyone who has odd car problems. Its endless painful calls and visits to dealerships and mechanics. There's a reason we have lemon laws for cars. Its because whats described in this story doesn't actually happen and people demand restitution.
I don't doubt that someone had a famous vapor lock shopping story (ive heard different versions of this story, usually about a housewife picking up her child from a nearby elementary school), but over the years these stories get modified into memetic structures based on dishonesty because most people are social capital seeking and having a humorous story provides them the immature ego boost they need. So "wow my car had vapor lock when I make quick trips" became "So the CEO of Ford came to my house to look at my ice cream car..." The latter is just more interesting in the market of storytelling.
That is to say, the ONLY reason this story is here is because its been modified to be memeticly attractive. A "boring" (i find old technology faults interesting, personally), but a "boring" story about vapor lock wouldn't make it to places like HN or reddit, which are memetic responders (upvote/downvote mechanisms) and lowest-common denominator (by this demographic) popularity machines. But dress up that boring story and now everyone is repeating it, often times claiming its their story and they know the people in it! The same way the comment you're responding to probably doesn't actually know the famous "unplug the server at 5pm" person.
I used to work for General Motors as a field engineer. Basically, I was the "mechanic of last resort" for some issues. The most certain method of getting something fixed was to write a letter to someone at the top, or very close to the top. When the request rolled downhill to "fix this", no one knew if this was a whim or something serious (like it was the CEO's neighbor). So those service requests got absolute priority.
I remember one incident where that "fix it" letter came from high enough that I drove out to the customer's house and swapped out their radio in their driveway. At night.
When GM bought EDS, Ross Perot ended up on the board. He'd do all sorts of silly things. Like when something went wrong with his car, he'd take it to a dealership. And report back to the board how that went. The first few times, they just told him to hand the keys to the valet at the executive garage and tell them what needs fixing. The plant I worked out of made radios. Other branches of the division that I worked for made engine computers and instrument clusters. If someone at the executive garage had a radio problem, one of my tasks was to go out to the assembly line, grab a radio, test it, then get it to the Kokomo airport for the GM jet to pick up. FedEx (tagline: when it positively has to be there the next day) wasn't fast enough. That fleet of jets would carry parts from plant to plant. And the executive garage was the best equipped and best staffed GM dealership on Earth.
By coincidence, I have witnessed this just last night. No details will be provided to protect identity, I'll just say you probably heard of the company.
You don’t happen to be the guy who overnighted a bunch of network cables for $5,000 because the boss had to have them overnighted and the company didn’t care what it cost I read about recently? lol
Delco! I now live in one of those small factory towns in Indiana. It’s fascinating to me how so many little towns in the Midwest existed just to make one small part for Detroit.
I've had the CEO of a municipal water company at my house looking at what his contractors did to my property frontage and potentially my well. Also, my other story in this thread is documented in posterity on a forum, you can see i didn't embellish any of it, and that involved a manager or supervisor having to drive quite a long time to my house at 10 PM with a technician to source a problem i had had for weeks or months.
So, in essence, "it depends". Good stories, in the memetic sense, will have hooks to ensure that the moral or point of the story is remembered; in a great story, the memetic hooks are so great that you can repeat the story nearly verbatim to other people, after hearing it yourself.
Well, CEOs do look at these letters quite frequently. At least, I did when I was the CEO of a networking company.
We were only selling equipment in the US at the time but some had found its way around the globe. This one particular Indian gentleman had been engaged for some time with support claiming that his unit wasn't working right and exhibiting all kinds of strange behavior.
I had a habit of looking at the cases with the longest open history, which is how I found it. I continued to monitor though I didn't send anyone out to India or anything like that.
Eventually it came out that the sleek looking aluminum unit had some kind of stocky packaging material stuck to it, so the customer had put it in the washer to clean it off.
Without vast knowledge, many unreasonable things are seen as reasonable. How much of our history's truth is based on what was told to the gullible majority? Should we not talk about Mythology because a skeptic questions it's authenticity?
Comedians make up stories all the time to entertain audiences. These stories don't require accuracy, they are more about delivering specific results; a laugh, a story to share, confirmation bias, etc.
Many people lie, believing they're telling the truth. I think you will have a hard time truth policing people who don't and won't care, but focusing on truth and validity is probably a useful skill for you in many parts of your life.
If I tell you something I believe to be true, but actually isn't, I'm not lying, just wrong. Lying involves an intent to decieve, so if I don't know the truth, it's not a lie.
That is true, in addition, outcomes are not defined by intentions. It was my hope to draw attention to why we perpetuate this behavior (results).
There is a lot of gradation, the differences between stating misinformation you believe is true vs stating what the majority believes to be true due to laziness vs an intentional lie, etc.
Your lack of faith is disturbing, specially when there are first hand accounts of [Apple/Microsoft/Insert Co] sending engineers to people's houses to diagnose unique problems. This could be a perfectly real story, a different thing would be that you choose not to believe it.
during the PPC 603 era of Macintosh Performa series, we had a bad motherboard, and apple sent an engineer to our house to replace it. This is around the time that the story of apple sending engineers to houses to remove motherboards, raise them 2 feet above a flat surface, and drop them was being passed around. Something about reseating chips that one of their pick and place machines was misaligned on or something.
When I worked my first job in retail in the mid eighties - we were selling Atari ST and there was an official Atari bulletin to do exactly that. It would reseat the socketed chips.
interesting; i doubt my memory is that faulty, so it's possible the story went through some iterations, making it an apocryphal story about apple, instead. Or, possibly, the major pick and place manufacturer had a slew of alignment issues in the 80s!
I thought the apple "lift and drop the computer" story was about the Apple 3. It had no vents, so it was always overheating and "unseating chips" like the Xbox 360 heat issues.
And so it doesn't become apocryphal but stays a real story, there 100% were people who "Fixed" their RRoD Xbox 360s by wrapping them in towels for an hour so they cooked themselves even more, or they put the mobo in the oven on a low temperature. That fix was also 100% used for fixing certain graphics cards unseating in the mid 2000s as well.
I've got an HP Laser Printer from about 15 years or so. About once a year, the printer stops responding and I have to remove the board, put it in the oven for 10 minutes or so to revive it.
What even is the theory on how that could possibly work? I believe you, it's just that I don't have any mental model of how baking a circuit board makes it work right for a year, and how it keeps working to fix the issue.
Laser printers have parts that can get very hot. If their "baking" reflowed the parts, but just barely, maybe those hot parts would just cause the exact same problem all over again.
I've reflowed a GPU in an oven before, personally. with 5 little balls of aluminum foil as standoffs and monitoring the temp closely. It fixed it. I probably still have that - working - card somewhere.
Those might be field engineers, but I have heard of the WiFi teams at computer companies going out to people's houses to test especially weird situations.
It's a fun story to read, whether it's fact or fiction. We are not too concerned about whether a large company would really send an engineer to investigate a seemly absurd claim, in the same way that we are not too concerned about whether one single shoe is able to uniquely identify Cinderella and no one else.
It's not just any memetic structure though. Urban legends have a number of components to them that make people want to re-tell them. See, for instance, my other comment here: https://news.ycombinator.com/item?id=37596325
One of the things I talk about is that these stories often ride the edge of plausibility. Even with this one, dispatching an engineer to someone's house over a "quack letter" seems like a thing that could happen or could have happened in the past. Having high level people involved is a thing that others have talked about, but I'd also like to point out that people have literally emailed jeff@amazon.com before about issues and gotten resolutions that way, so it's not completely off the mark.
The real clue here that we're dealing with an urban legend and not an actual incident (though it may be based on one) is the lack of specifics. Urban legends are often not really pinned down to any specific time or place.
Do not mistake cynicism for intelligence. It is actually negatively correlated [0, + other studies]
Just listing a basket of "I doubt..." and "No one does..." does not make it all false.
Of course Just So stories exist, but that does not mean that all cool stories are fabricated Just So stories. In this case, it is the cynicism that merits a [Citation Needed] tag.
I personally had a similar situation. We had an accounting firm whose servers we were maintaining, and occasionally the print server would reboot, always between 9 and 10 am. So I sat in there for a week between those hours and noticed the light would dim and occasionally the reboot would happen.
It turned out that the lawyer next door would come in, turn on his PC, printer and coffee pot simultaneously because they were all on the same power strip, and the drain was causing an undervoltage on the circuit the server was on during startup. We had it on a UPS, but it turns out that at the time consumer grade UPS systems only handled outages.
I measured drops as low as to 85 volts, in practice anything under 95 or sou would reboot.
1999, the town hall of a small municipality s/e of Stuttgart Germany. For the 3rd time in half a year me (IT apprentice) and senior developer are onsite to fix one MS SQL db's broken indizes. It's the early afternoon of friday and humid hot, a thunderstorm is on it's way. We toil on and are almost done when at about 16:00 the chief officer turns up and informs us that we have to hurry because at 17:00 all electricity will be cut in the building!
Senior developer and me share a shocked glance.
"Is this every friday?"
"Why yes! We conserve lots of energy that way."
Senior and me share another glance and on impulse I speak up.
"Isn't the weekly db cleanup scheduled for 16:30 PM on fridays?"
The senior developer nods. We spend the remaining hour to fix the problem and then cancel this weeks schedule and manually shut down the DB server for the time being.
Every friday these guys cut the power to the building, causing all kinds of production issues with not only their Netware Server but also the DB server for their collections software (our product). As per maintenance contract we were required to repair the DB everytime which was a hassle and a waste of ressources on our behalf.
In the end it took another month or so until they relented and rescheduled the hard power down until saturday morning so that all servers could shut down properly even AFTER the backup jobs could complete (backup was also totally broken but they never did check or even perform a restore so they never actually noticed).
I think it's more like a common experience. My wife and I both work, so we have cleaners come in every two weeks for a deep clean. We've used several different companies, and apparently it's standard practice to unplug things when a power outlet is needed. They don't unplug computers that have monitors, and they don't unplug things with visible clocks that would need to be reset, so they do take some care not to inconvenience us, but they'll unplug anything else, including NAS appliances, DVRs in the middle of recording shows, etc. When we hire a new company, we make sure they mark down a special request that they not unplug anything, pointing out that we have ample outlets and can help them find one or free one up if necessary. I also replace any network connectors that lose their little plastic locking tabs, because they're likely to slip loose when things get jostled around during cleaning.
It is a common story and sometimes those get put in the collective blender and we get apocryphal stories out of it. Here's two stories of my own:
Back in the mid 90s, I built out a system that gave every school in a district their own webpage that was carved out of some government funding for providing internet access. There was no budget for hardware though, so it ended up running on a repurposed workstation in somebody's office. One Tuesday even the cleaners unplugged it to vacuum and it didn't power back up after being plugged in. On Wednesday somebody helpfully stuck a piece of paper saying "don't unplug" to it, which seemed to solve that problem until the whole project was mothballed.
In the late 90s, I worked at a company where we started getting complaints from the staff about machines being getting slower over time. Nobody took it seriously until there was an inventory of machines taken and we found that a large amount had significantly less memory installed than they should have, somebody was stealing half the memory sticks from each. Hidden cameras were installed in the office and it turned out that somebody on the cleaning crew came with a screwdriver and ESD bags and knew how much to take to leave the machines working.
I would expect the repetition generally comes out of necessity. If he's selling the parts to feed himself or his family he's that much less likely to choose to stop if it means giving up some source of income, however ill-gotten.
It's more likely from "I got away with it, I'll get away with it again" - but I've not done deep research into thefts, but the ones I know were along the lines of "I need more money for more drugs, this will get me some money".
funny thing, something like this happened to me. we were on site doing some implementation and went back to the hotel at about 3am.
less than an hour later, we get an email from nagios (i think it was nagios? it was a GOOD while back) complaining the server was offline. we got into a cab and went back straight up (this server was not supposed to be offline ever).
guess what? a maintenance guy turned the server off by mistake while cleaning up the server room -- even worse, he was not even supposed to be there!
this triggered a bunch of security checks and the company found out that most employees had access to any room in the building.
Similarly, we were just finishing up a project for a client and we’re just going through and connecting everything up.
The hardware consisted of 6 color terminals (Wyse 350s I think), the server, and 10 PCs with large, 20” tube monitors. We had these all in my bosses office powered through assorted outlets and power strips. The room was wall to wall machinery.
It was after hours when the janitor came in and plugged in his vacuum. I looked at him, he looked at me, I looked at my boss, who looked back. We then both turned back to the janitor, who looked at us.
Not a word was said as he fired it up. 2 seconds later, the power was out, we blew the breaker. Took us a half hour to get it back on. But it was funny at the time.
Pretty much, server rooms still exist in offices but they'll mostly be for the network infrastructure, with most workloads including email and storage having moved to the cloud. Office365 is a really tempting offering, compared to operating your own servers + staff.
I had a similar experience: I have a permanent dynamic lighting system installed in a food hall / night club. One day about a year after installing the owner called to tell me a fixture was on the blink. The following day, another, the following day, more, but different ones. This was worrying as all the hardware is custom and would be very hard to source replacements for. We investigated lightning strike or other power surge related to construction on the block, water damage, etc. Long story short we found out that the new cleaner, who was VERY VERY thorough, was pulling out the server rack to mop behind it nightly and straining the cat cable terminations.
In the US the "winky" receptacles that go (-|) or (⊣|) - except the T faces the other way - are 20 amp outlets, 120 volts. every receptacle that differs from the "shocked face" means something. Our standard shocked face receptacles are 15A. In hospitals, you need a guarantee that your monitors or assistive devices aren't going to trip a breaker somewhere if someone plugs in a vacuum; making it impossible to use the receptacle for something else is something that only matters after it's too late, i'd think.
This reminds me of a GM minivan that my youngest brother-in-law drove, back in the '80's. He'd gotten it from his father, who was a career GM automotive engineer - and complained that, occasionally & randomly, it would not start. It seemed like the minivan's whole electrical system was dead...
Brother-in-law was known to be "not so good" with cars - so his automotive engineer father didn't take the complaints seriously.
Complaints and emotions escalated, until brother-in-law convinced his dad to swap vehicles for a month, so (he hoped) his dad could experience the problem for himself.
After the problem manifested in the parking lot of the GM Technical Center, and the whole crowd of GM engineers surrounding the vehicle couldn't figure out why the heck the electrical system seemed to be dead, my brother-in-law felt pretty vindicated.
I have a 2009 Mercedes SLK that had the same symptom. It was absolutely fine 99.9% of the time, but one time in a thousand it wouldn't even try to start. No clicks, no indication that it even knew I was in there turning the key. Just dead. And then seven or eight hours later... it would start fine, as if nothing ever happened. Couldn't correlate it to anything.
Had it towed to a service center a few times when this happened. Every time, by the time they got around to trying it (a few hours later)... it would start fine, with nothing to diagnose.
Then, it was 99% of the time it was fine. I was with a group of folks car-camping off road to fly a human powered airplane for a couple days, and... no start. Finally started -- with no sign of any problems -- around noon the next day, and I high-tailed it out of there a day before the rest of the group, because getting stuck there /after/ the rest of the group would have started pushing my comfort level. So at this point it's actually interfering with my life.
I've tried all the usual stochastic troubleshooting (swapping out fuses, light to moderate percussive maintenance, alternate keys) and nothing. Finally it fails to start in my driveway, and I get it towed to an independent mechanic. It's short tow, and it fails to start when it gets there! So now he's seen the problem, and is as puzzled as I am. Of course, when he tries again the next morning, it starts fine.
He proposes two possible fixes: replacing some ECU module, or replacing the fuse box itself (under the theory that it's the connector or connection into the bottom of the fuse box that is having some moisture ingress or intermittent connection). Of course whatever we choose, I won't know if it was right or not until the next time I'm stuck. The ECU is multiple thousands of dollars, and the fuse box is < $200 with labor, so I make the easy choice.
This was six or seven years ago, and that car is still my main and only car. Hasn't had a single mechanical issue since swapping out that fuse box. A good independent mechanic and a good guess!
Finally figured it out after months when I moved a wire and it started, then move the wire close to another wire and it would no longer start. These were wires that would typically be close, so my guess is one of the wires was now generating enough noise that it was bleeding over into some other system and causing it to fail. As the car bounced around the proximity of the wires changed and lead to the random behavior.
EMF can do some seemingly crazy things. I built a kiln controller and the initial version would sometimes randomly lockup, reboot in the middle of an operation, or do other seemingly "crazy" things. Sometimes even the hardware watchdog would stop functioning.
Turns out contactors pulling in and out a 5000w load generates some strong EMF and sometimes that EMF is enough to cause random glitches to the CPU or other hardware.
Switching to high power solid state relays completely solved the problem while keeping the system compact. The actual silicon transistor was so big you could have drawn the mask by hand and it was attached to a heat sink half the size of an adult fist. I was initially worried about reliability but (knock on wood) 8 years later the system is still working without issue.
Yeah. I wrote the code for a controller that managed the temperature of a lubricant. The heater was a propane-fired burner that was mounted on the same skid. Every so often we'd get a random reboot. Finally, when it was very quiet, I heard a soft 'tic' just before I noticed that the CPU had rebooted again. EMI from the spark gap that ignited the propane was coupling back into the I/O lines and would occasionally reset the CPU.
One of those things where if it had happened 100% of the time we'd have figured it out quickly. But it was so infrequent that no one thought of that as a cause.
We had one access point (unifi) in our datacenter which was consistently failing.
fail, RMA, fail, RMA, probably replaced that thing 5 times. It was also incredibly unreliable.
Meanwhile all the other access points (probably 10 of them in total) had 0 issues.
Eventually realized that the cable for that AP was running perpendicularly over the conduits which fed power into our suite, so, about 1MW of power. Relocated the cable so it was farther from the conduit and it never had an issue again. Makes you wonder about the effect of working in such environments.
Most likely this solid state relay has a builtin flyback diode that was not in the circuit of the contactor. The diode should be as close as possible to the load to reduce the EMF.
I know an older guy at church, whose kids all graduated college - except for one.
His "failed" son is the top mechanic at a Mercedes dealership. He does some supervision, training, etc. But the reason the dealership is paying him $200K/year is his skill at figuring out and fixing problems like that, for the dealership's most desirable and profitable customers.
(That I've heard, none of the mechanic's "successful" siblings are making that kind of money.)
I know a story of a certain large engineering firm, dating back to the second World War. They had a senior engineer who habitually came to work drunk and slept through meetings. Every once in a while, they'd wake him up, and he'd save them a couple of million dollars. He had a gift for finding clever solutions.
He probably would have had a better life if he'd gotten his act together. But knowing how to fix subtle issues, or how to design good processes, can be a ridiculously valuable skill.
That definitely sounds like a loose connection, not an ECU problem. Usually it's a bad clamp on the battery terminal. Often it's the ground clamp in particular.
The easiest diagnosis is to rotate the cables on the terminal several times to rub off oxide build up, then leave them in an orientation so that the natural tension of the cable forces the clamp into good contact.
A "dead" (low voltage) battery will still cause some indicators/lights to come on when you crank, while a bad connection usually acts like zero volts.
I was about to say the same thing. My car (nowhere near as nice as a Merc) had the same issue. There would be oxide deposit on the contact points and the minuscule amount of rust and crack on the clamp would render it useless on a bumpy road randomly (1 in 10 times approx). It was scary the first few times but when I figured out the symptoms, it took me less than a minute to get it to start.
Today's car mechanics are a different breed. The ability to diagnose and triangulate problems without advanced tools made car mechanics quite intuitive and resourceful in the past. I take my car to one such mechanic. I always leave with a bill lower than any other estimate I get and he always finds and fixes the problem.
Wow, it's crazy to see this comment because I drive the same type of car and recently experienced a similar problem.
The car was dead for long enough that I could get it towed to the mechanic and they educed it was a problem with the EIS (electronic ignition system).
The EIS computer was sent off to mercedes for diagnosis, they reported that the computer itself was fine but it was an issue with power. The mechanics traced it back to a bad connection of a wire somewhere.
It would have required removing and replacing the box itself... the fuse box has a connector going into the back side, and then a separate harness inside the box that splits out the individual fuse sockets; that connector seems to have been the problem. At various points I swapped/reseated every fuse in there trying to fix things, never occurred to me to remove the box itself.
I have seen similar behavior on vehicles that have a dying alternator. The issue is that one vane of the rotor (or maybe stator?) has shorted and no longer functions. If when the car is shut off and the rotor stops in a particular position the alternator won’t work. Give it some time and with heating/cooling moving things around just a bit plus the act of repeatedly trying to turn it on potentially making it move just a bit, and it works fine.
Alternators aren't needed for starting. Perhaps a similar issue with the starter motor? But even then you'd hear the thunk of the starter solenoid. Any audible clue definitely makes root-causing simpler.
I have a crank sensor that is going out, and though it still works most of the time if the crankshaft stops in just the right position it becomes an absolute bear to start, and runs in limp mode when it does.
This reminds me of an old F250 my dad had. He towed it from Colorado to Missouri and about halfway he noticed his gas mileage got worse. But, you know, coasting down mountains until you get to plains would also have a similar outcome, so he didn't think about it.
Once he got to Missouri he noticed the transfer case (I think) had somehow bounced out of neutral and was partially engaged, which chewed many teeth off the flywheel. To start it we had to get a pipe wrench and manually rotate the flywheel to a spot with enough teeth to catch the starter and turn over. That got harder and harder (as it kept chewing off more teeth) so eventually we had to push start it and pop the clutch. Thankfully it was a manual so it still worked!
My dad had an old diesel Benz - the generator (that old, didn't have an alternator) went out and for a couple of months we just made sure to park it on a hill and drive during the day. Electricity is overrated!
Seen things like this before. Often a bad connection somewhere, possibly involving material that expands/contracts significantly in response to a change in temperature (turning the module into an inadvertent temperature controlled switch).
I don't know if Alec Proudfoot counts as a normy, but DaSH PA flew a bunch of times, with many different pilots. That particular event, which was an attempt to try some records, was a bust; there had been rain recently and the dry lake beds weren't. Landing gear was a recurring problem (light weight and robust don't go together), and even some last-minute attempts to build a runway out of 4x8 plywood sheets was unsuccessful.
I have some very, very fond memories of flight days at Moffett, though. I had gotten to work on the Moffett runway for work previously (so cool out at the bay end of the runway at night, totally silent), and getting back there to help run ground ops (including assembling DaSH PA before sunrise so first flights would hit the calmest possible air) was just lovely.
Heisenbugs are manifestations of whole-system design failings, where projects are not engineered to facilitate troubleshooting, subsystems are strongly coupled, and everything is just barely held together with baling wire and bubblegum.
That GM vehicles from this infamous era would suffer from maddening, mysterious electrical glitches makes perfect sense.
A formative early experience of mine was learning valgrind to track down a Heisenbug for a C project I was working on (which turned out to be an invalid read in a dependency). I’m indeed thinking of this anecdote when generalizing about whole-systems failings, since troubleshooting memory errors is so difficult.
I think there’s an analogy to be drawn when designing large systems on top of unsound foundations.
please expand to include more explanation for why you think this is a memory managed lanugage problem. I used C and C++ professionally for years and ran into all sorts of issues like this. Interaction between subsystems doesn't care at all what language you're using inside each component, they care about design patterns and architecture.
GP is correct that I generally prefer memory managed languages, I just think it’s right to emphasize that this preference is informed by experience. I’ve spent large amounts of my career writing C code, and now when I have a choice I’d prefer Rust for systems projects.
The higher-level point is that Heisenbugs are an emergent phenomenon of complex systems when fundamentals are lacking.
* C systems are lacking because the language is very old and we’ve learned that we need additional infrastructure to avoid memory errors.
* 1980s GM systems were lacking because of a management culture which didn’t value reliability, leading to inevitable issues in e.g. poor grounding and electrical isolation.
It’s my belief that many contemporary tech companies have management cultures similar to 1980s GM, and subsequently waste tremendous resources when troubleshooting complex systems which are not designed to facilitate troubleshooting. That’s why the original article resonates strongly with me.
GM cars are notorious for (sometimes) developing strange problems that would have you think they are possessed with the devil. (Despite post-1990 GM cars being near peers to Japanese cars for reliability overall)
I learned a lot of things from my father. Unfortunately a lot of them were what not to do. Don’t buy cheap tools that you’ll have to replace three times in the lifespan of one that costs 50% more. And don’t buy GM.
Mechanically they may be reliable, but 90’s GM forgot how to make paint stick to metal and had to pay to repaint a massive number of vehicles that simply pealed if parked outside for too long. How?
And there is absolutely no forgiveness in my soul for the Chevy Citation. I joked when I moved to Seattle that the main problem is since there is no salt, there are still Citations on the road and that is unnatural. Their place in the natural order is the junk yard.
After owning my 1988 Chevrolet Beretta for about a year, the paint came off in one big sheet one morning when I was brushing a light dusting of snow off with my wool toque.
The first call to GM revealed that it was the acid rain (acid rain in the 1980s was what global warming is today -- the cause of all evil). Exposing my car to rain voided the warranty.
The second call to GM revealed that ultraviolet light destroyed the bond between the paint and the primer. Exposing my car to sunlight voided the warranty.
I spent hours researching this issue through the trade mags and published court filings. Plenty of legal findings about implied warranties of fitness for purpose. Evidently GM had an unpublished policy that it would pay the cost of a repaint to dealers for this situation, and the dealer was expected to provide the work for free.
Of course, the greasy grin of the dealer as he quoted full price while knowing he would collect the same from GM was enough to make me drive the car with no paint for the next 10 years so everyone could see, and recommend nobody purchase any GM product ever again.
I'll say this though: that primer sure prevented rust.
I wonder if the irony of the situation is that some chemical engineer invented a new primer that sticks to metal extremely well and it turned out that it may stick to metal like glue but it doesn’t stick to paint as well as the old stuff.
But based on what peeled, I’d say thermal expansion or UV damage were involved. The former could still be the primer’s “fault”.
>I joked when I moved to Seattle that the main problem is since there is no salt, there are still Citations on the road and that is unnatural. Their place in the natural order is the junk yard.
This was the fate of many British Leyland cars, even the ones that people genuinely liked such as as the classic Mini and the MGB were practically hygroscopic.
I will say I was somewhat delighted by the number of good looking mustangs and british roadsters there were. And so many Beetles. My roadster had a little too much bondo for my liking.
I didn't realize the Countryman was bigger than a midsized sedan until I parked next to one the other day. Walking back to my car something seemed off. Wait, that's a Mini??
None of the 'Mini's' are Mini. They're just BMW that bought the brand and builds cars loosely inspired on the original, with sizes ranging from 1.25 times the original all the way up to definitely non-Mini crossovers. Our company had one of the smaller ones as a lease car and even though it drove ok'ish it definitely didn't feel or drive like a small car. It was a way for BMW to appeal more to women (with that demographic BMW never really caught on) who loved the original Mini to bits. And then wealthy women want a larger but still trendy car.
BMW is hyper aggressive about the brand, including going after each and every use of the brand for the original cars, even when used in the context of spare parts.
I think about eight, ten years ago they stopped being mini when collision safety laws changed. They had to raise the hood for pedestrian safety (hitting legs and spinning heads into the windshield is not allowed) and raise the door height for side impact safety. The whole car got normal sized because of it and then they just kinda said fuck it.
Of course there are: "a mention of a praiseworthy act or achievement in an official report, especially that of a member of the armed forces in wartime" Don't focus on the North American usage of "a traffic citation". Citation is almost a contranym, which is a word that has at least two meanings that are opposites of each other, i.e. bolt, bound, buckle, cleave, clip, consult, ...
I think it's funny how GM came out with the "Cavalier" to compete with Honda's "Civic". Or that matter, there was a Chevy Cobalt (e.g. a "Kobold" is a demon that causes mine accidents) or an AMC Gremlin.
Sure, the name is derived from "kobold", but that's like saying you should never call anything good "terrific", because it derives from the root "terror". Etymology isn't destiny.
> The statement refers to a popular anecdote in international business and marketing about a supposed blunder made by American automaker Chevrolet with the car model, "Nova."
> According to the story, when Chevrolet tried to market the Nova in Spanish-speaking countries, the car reportedly did not sell well because in Spanish, "no va" translates to "doesn't go". This led people to joke that a car named "doesn't go" wouldn’t be a popular choice.
> However, it's important to note that this is largely a myth. In reality, the Chevrolet Nova was relatively successful in Spanish-speaking markets. "Nova" as a word is understood to mean "new star" in Spanish, and it's unlikely Spanish speakers would naturally break up the term into "no" and "va", just like English speakers wouldn't naturally break up "notable" into "no" and "table".
> But the story remains popular as a cautionary tale of the consequences of not considering linguistic and cultural differences when naming products for international markets.
Technically Chevrolet did but the entire thought process for that vehicle was questionable so the name is IMO a harbinger. This is not a place of honor.
A few other GM vehicles have this issue, Chevy in particular. A well known example is the market failure of calling a car Nova (No-Va) in South America.
You know, I've heard that rebuttal, but I've been told this anecdote of the car's notoriety by family members from Columbia and more recently from a friend from Argentina. So perhaps "no va" and "Nova" are pronounced differently, and perhaps the car did sell well, but the Spanish-speaking peoples most certainly did find the term "no va" in the car's name.
You should know what they say about the Mitsubishi Pajero, too!
Sure, we have these in my language as well. For instance, the car Kia should be written like "vomit" - so we have a convoluted spelling and say the name slightly wrong to distance the word from "vomit". But it's still clear to everybody how the name should be pronounced and spelled.
Doesn't seem to affect sales, though, probably like the Chevy Nova. Those Kia are everywhere.
> Mechanically they may be reliable, but 90’s GM forgot how to make paint stick to metal and had to pay to repaint a massive number of vehicles that simply pealed if parked outside for too long. How?
Could be an older component stopped being available. Like when Apple switched to environmentally friendly lead free solder, but then the NVidia laptop GPUs got so hot they unsoldered themselves.
> 90’s GM forgot how to make paint stick to metal and had to pay to repaint a massive number of vehicles that simply pealed if parked outside for too long. How?
Dodge Ram Trucks had exactly the same problem. I also have no idea how two different companies with 100 years of history simply forgot how to paint cars.
i learned that these things typically are grounding issues. it usually came down to a single source of ground being a loose connection which is why it was intermittent. as a personal anecdote to this, as a first car as a teenager, i drove a GM/Chevy S-10 that one day started to have issues where all of the gauges on the dash would just go crazy and the lights would go on and off, and then suddenly just start working again. after taking it to the shop my dad recommended, a mechanic walked out to greet me. after i told him the symptoms, he stepped back to look at the truck model, asked me to confirm the year model. he promptly opened the driver side door, reached under the dash, located a specific screw, hand tightened it as a test, and everything worked. he came back with a screw driver to properly tighten in before sending me on my way free of charge. he told me that specific model was notorious for the screw holding the ground wire to come loose. it would cost him more in time to write up a sales slip to charge me.
I think independent mechanics tend to opt for earning goodwill in this situation rather than charging fair value for a quick and easy fix. Objectively, he provided more value than someone who might have spent several hours of troubleshooting time, but customers don’t necessarily think logically about it. So it ends up being better for the mechanic to earn that goodwill rather than appear “greedy”.
Some do the freebies because that’s who they are, and the free publicity is just a secondary effect. Some do the freebie because they know the publicity as the primary reason. I feel like this experience was the former.
> Despite post-1990 GM cars being near peers to Japanese cars for reliability overall
American carmakers really needed that kick in the ass from Japan. Around 1990 was when my parents went from being protectionist, "buy American" to never buying another domestic car again in their lives. They were angry, angry at the reliability difference and angry knowing that domestic carmakers could have done better but instead relied on people like them to buy the flag.
I had a 1989 Chrysler LeBaron that developed insane electrical issues. Windshield wipers would randomly turn on, the radio would change stations by itself. It really did seem possessed.
When I was a kid my (not wealthy) family had a Lincoln Towncar that was probably purchased used and fixed up and it ended up with some freaky electrical problems like you describe- most notably (because it freaked me out as a small child) I remember the automatic door locks would start locking/unlocking themselves rapidly, and they were big chunky metal switches that pop up and down and made an awful sound when this happened
I dunno. When my son needed a car to get to work we found a 1996 Buick Park Avenue in almost perfect condition at 100,000 miles. He didn't like the look of modern cars, didn't want a touchscreen, besides a 2010 Japanese car can cost almost new prices and they wanted $9k for a Chevy Sonic with 180,000 miles (didn't know they went that far!)
Granted the traction control and anti-lock brakes occasionally fail to boot up, but it seems to be a pretty good car, but it was close to the top of the line. Gets 27 mpg which is not bad for a big ass car. I like how it has a lot of the feel of a 1970s boat but it has airbags, OBD II and most of the good features of a modern car... And we didn't need to get a loan to afford it. Driving home though I was looking in the mirror and seeing it dwarfed by today's XXL trucks and SUVs.
I mean...it's possible they've improved a lot compared to where they were 30+ years ago, but to call GM's cars "near peers to Japanese cars for reliability" just doesn't hold up.
On Consumer Reports' list of car brands by reliability[0], none of GM's brands even crack the top 10. GMC and Chevy are 20 and 21, respectively, out of 25 brands. (The top 5 include, unsurprisingly, Toyota, Lexus, and Honda—your classic reliable Japanese brands.)
Plenty of cars that intermittently won't turn on in this thread, but I once had a car that intermittently wouldn't turn off... or at least the headlights wouldn't. I assume a relay somewhere was overheating and that was making it stay closed, but I never debugged it, and nor could my cousin who was an auto mechanic. We never tried too hard, though: I would just take the fuse out if it happened and put it back the next morning.
True story: I had that happen to an airplane. I was a student on a long solo flight and on the last two legs of that flight, the engine wouldn't shut off using the usual method of pulling the throttle back to idle and turning off the magneto.
It's been long enough that I don't remember what I did to get it shut off (maybe I toggled a circuit breaker?), but when I got back to base, I made a note in the airplane log about the problem and also left a note for my instructor, who was out that day.
When I came in for my next lesson, instructor mentioned that the next person to use the airplane, also a student on a solo cross-country, got stranded 100 miles away because when the engine wouldn't shut off, he panicked and pulled the throttle hard enough to rip the cable through the firewall. Airplane had to be put on a flatbed to get it back home.
That's probably what I did. Thinking back, the first time it happened, it just took about 10-15 seconds to shut off. The next time, when I was back at my home airport, it wouldn't stop running at all and I think I turned the fuel cutoff to get it to stop.
The radiator fan in my Jeep did that in Sudan when it was 48C. Better to have it stuck on that stuck off in those temps!
I just tapped the relay with a wrench and it un-stuck and turned off.
Funny enough that was almost 5 years ago, and it hasn't done it once since in more than 80,000 miles of driving. Not in -35C, not in +45C. Odd little relay.
I'm going to armchair and guess that your brother needed a new battery.
My (GM) car gets really funky startup behavior when the battery gets old. It will often turn the starter fine, but the electronics can get stuck in weird states until I disconnect the battery (essentially a hard reset).
The dealers wanted a ton of money to diagnose the issue, even though I suspected it must have been the infotainment system draining the battery. I just ended up replacing the infotainment system with a cheap CarPlay compatible one and the problem went away.
Problem is I cannot get money from the class action settlement since the original infotainment system is already out and I fixed it myself.
But I cannot prove it without reinstalling the original infotainment system. That is many, many hours of work for little gain.
What I do not understand is how the settlement did not require Subaru to compensate all Subaru owners for the faulty design. Why was there proof required, other than the purchase of their car with the faulty components?
I worked tech support that included a device for software developers that had Ethernet connectivity. I learned very quickly to say "Try reseating the Ethernet cable" because if I said "Is the Ethernet plugged in" about 1/3 of the people would respond very negatively (e.g. "I'm not a fucking moron") but by having to reseat the cable, they would sometimes discover it's not plugged in.
The classic way to get the consumer to power cycle the hardware is to ask them which color ring is at the base of the DC plug. It doesn't really matter what color it is, what matters is that it was removed and the hardware definitely restarted.
I once has this happen with my father's new standing desk. He, PhD with an Electrical Engineering degree, couldn't get it working. Turned out the C13 power cable was only 90% inserted, and felt firmly in place but did not have enough connector to power the desk.
A friend who is an automotive engineer shared once that most colleages were not so great with cars. Engineering new cars is one thing; fixing them is another. It's like asking a programmer to do system administration - two different jobs.
This reminds me of a strange recurring experience I used to have with electronics in my house.
In that house,on certain days of the week at midnight, something that sounded like recordings of political speeches from WW2 would be audibly (but faintly so) from all of stationary electronics. It was so faint, that it was usually hard to pinpoint any one source, and the words used weren't understandable, so it usually sounded like it was coming from everywhere all at once, and like it was an old recording.
This would happen even if they were unplugged. Even if my power went out.
I though I was either going crazy, or my house was literally haunted by hitlers ghost.
Well one day I was in my car and recognized something on the radio that reminded me of this spooky problem I had.
It was the signon of a Catholic am radio station that opened up with Gregorian chanting, and a sermon. This signon happened at the same time on the same days of every week.
Turns out, the wiring in that house was somehow functioning as an am radio receiver, and some common components would vibrate out the audio encoded in the radio signal.
One night, after a particularly bad thunderstorm, our phone lines started picking up an AM radio station before you placed a call. My parents swore up and down somebody had left a phone out of the receiver and after unplugging every station in the house, we realized the problem. Now I can still pick up that same radio station if I hold the male end of a cable plugged into a guitar amp. At least the station plays decent music!
Near my work, there is a roughly 25 foot stretch of road in front of a factory where seemingly every FM station is overpowered by Muzak. I don’t know what the heck is going on in this factory but I have seriously considered contacting the FCC.
> I speak with the Comcast attorney about my call to State's Attorney's office. She says she was unaware that State's Attorney has denied the request to drop the charges, and that she will write another letter to ensure that the situation is taken care of. She also asks about whether the video trap has been installed and is working correctly. I tell her that everything is blocked except for 2 religious channels, 1 Spanish language channel, and the video portion of E! TV. She says something along the lines of, "I guess the signal of the Lord manages to find its way through somehow."
My son's laptop screen kept shutting off while he was playing American Truck Simulator. His truck would drive off the road while the screen was black.
Every time I played on his laptop, this did not happen. He swore he was cursed.
This went on for many days, with many instances where it would happen for him but not for me. Then one day I just sat and observed him while he played, looking for any difference. That's when I noticed his watch band is metal with a magnetic clasp. The position of his wrist on the laptop was tripping the hall sensor, making the laptop think the lid was closed.
Him and I (and his mother) were glad to find out he is not cursed. :-D
While screens of earlier MacBooks could be turned off with a single magnet, my 2020 Intel MBP requires both the left and the right sensor (around the tab and the enter key) to trigger at the same time to consider it closed. It would be nearly impossible to trigger that accidentally. For starters, you'd need to wear two watches..
Maybe this has been fixed in recent models, but ten years ago it was easy to experience baffling laptop sleep states if you happened to set a working macbook down on top of another, similarly-sized macbook with its lid closed.
If it was Tesla they would have forgone any sensor done computer vision on the webcam feed to determine if the lid is open or closed. I'm looking at you, terrible automatic wipers.
This used to hit us sporadically at my university's help desk. It'd also happen if you have one MacBook open on top of a closed one. The magnet could easily line up and it'd shut the screen off.
These are the kinds of stories that made Car Talk so much fun. It was so much more than just hearing about mechanics repairing cars. It was the fact that the situations were so odd and unusual that the stories were interesting. It was also fun hearing how these mechanics had been around so long and seen so many of these unusual situations that they became normal to them. It didn't hurt that they were good story tellers
these days it's "just buy a new carburetor" - i remember rebuilding a few like 20 years ago - on small engines, but now a rebuild kit is, say, $12 and a new carburetor is $20. For an extra $8 you only have to remove two bolts and fasten the new one in, rather than take a carburetor apart and replace gaskets and floats (or whatever, it's been 20 years).
word to the wise - never use the shutoff switch on anything with a carburetor - use the fuel shutoff valve. This prevents gasoline from varnishing your carburetor bowl.
I probably won't use that word again for a year, now, heavens.
I acquired a generator from a family member, and unknown to me when I needed it to power the house the thing wouldn't run unless the choke was held closed with a wire tie. You can imagine how well that ran...
So, that sounds like a carb rebuild, right? Come to find out that for about $17 shipped a seller on eBay with something like 200K feedback will sell me a brand new carb. Fitted that, it starts on first pull. There's absolutely no reason to mess around with a rebuild on something like that.
As a small engine mechanic, you you lucky. Usually those cheap carburetors will do the job, and usually the engine won't run quite as smooth. The rest of the times it's so erratic you have to buy an OEM one.
I will always clean/rebuild carburetors before replacing. And usually when replacing I will explain to the customer and get permission to spend their money on an OEM one.
Thanks, that's good to know. This is a basic, older Generac GP5500 and I couldn't figure out how to get the carb apart, it really seemed pressed together. Maybe that generator already had it's replaced, as the replacement I got looked identical.
Looks like the OEM part is only $50-some, so worst case I'll get one of those down the line, but the generator was running at a friend's house for three days straight a few weeks back and no issue, so hopefully it'll keep being good.
The emulsion tube might be pressed in on that carburetor. 99% of the time there's some deposits built up from old gas and that's why it doesn't run / doesn't run well. Use thin wire or a carb cleaner kit or a torch tip cleaner kit to poke the junk out of all the parts you can remove/access, and use a can of carb cleaner to spray through all the channels you can't otherwise access.
Stated another way, distinguish observations (facts) of a story from the inferences--don't dismiss the observation/facts when rejecting inferences. This is analogous to an XY problem.
I have the role of "chief troubleshooter" in my engineering group. I have a bit of a nervous tic that forms when I hear absolutes like this.
Don't assume the customer is always an idiot. Don't assume ANYONE is always an idiot. It limits you as an engineer.
Listen to everything. You don't have any need to abide any of it, but listen to it all.
I've had some of the least-qualified people throw out something which absolutely ended up being insightful, although sometimes in a way they didn't expect.
Indeed. A timing problem was my first thought when I read the title which was obviously provocative, maybe that's because I deal with timing problems often so I got lucky there, but I rejected the idea, thinking the user would have noticed it too. Nope.
I used to do some help desk as part of my dev job, and from my experience, users easily assign any random fact as the source of the problem. Often things like "correlation is causation" or *post hoc ergo propter hoc" (after that, therefore because of that). Good as heuristics, but bad when they are substitutes for reasoning.
Users cannot diagnose at all because they have no idea of how the thing they are using works (which often normal - we are the engineers, they are the users) (in this regard, users that think they know "that stuff" are among the most difficult to deal with). One cannot properly diagnose something one doesn't understand well.
My wife's car right now is low on windshield wiper fluid. It warns us at a particular spot while we are driving. Not a particular distance from home, a specific geolocation. First thoughts were that it was the amount of time as she went to something and came back. But it does the on screen and audible warning at that location every time, no matter what was driven before that, whether it was 5 minutes, 5 miles, an hour, or 60 miles. The only additional clue is that exact same spot is a dead zone for most phones/carriers.
We got the wiper fluid filled, so the mystery is in remission, but I'm wondering if all warnings will pop up right then. I'm guessing it has something to do with the telemetry of the car being nudged in that spot, waking up and saying something.
Absolutely flat. There is a mild bend there, but nothing compared to corners and the traffic circle. For that matter, we can get out and drive all afternoon and it said nothing until that spot.
A lot of those sensors average the reading over quite some time to avoid false positives. It's possible that the last few miles of road have just enough average slope or contain just enough curves that the sensor averages "low" right at that spot with the speed of traffic.
I have a similar situation but in my case it's very obviously the gentle bend in that location combined with the relatively high speed that makes the fluid hug one side of the reservoir. I'm pretty sure that the sensor is mounted to the opposite side.
Sounds like an area with strong EM interference. Maybe a stop-light with those underground sensors? or under a high-power line? or near a power transfer station? or high-power radio antennae?
Could be giving just enough weird EM interference to bump the sensor from "enough fluid" to "low"
(I know I sound like Scully from x-files but could just be?)
Experiencing this same issue acutually these days on a 2019 VW Passat, it triggers when you accelerate or break or you are doing a turn for a longer period. It's just the fluid being moved around in the container. I think today I cleaned my windshield for a bit just to make the car keep the low signal light always on.
Long slow bend on completely flat road, 40mph. No braking or acceleration for a quarter mile each direction.
The road is like that for miles with some lights, traffic circles, etc... Only warnings in one spot.
I really didn't believe my wife, or thought maybe it was happening a few times when she made a regular trip like getting drive thru coffee and coming right back. But then we ran a bunch of errands all afternoon and got one warning there on the way out, and one warning on the way back.
If I were designing a wiper fluid warning, I would use some sort of fluid level sensor and I would denounce it aggressively: the indicator would only light up if the sensor detected a low level for more than a couple seconds. That way traffic circles, bumps, etc would not cause many false positives. I might even couple it to some kind of acceleration sensor so a warning would not turn on during or shortly after any heavy vibration or acceleration.
A long slow bend would cause a prolonged, steady centrifugal force and/or sideways acceleration due to a banked road, which would defeat these mechanisms.
The question then is does it only happen in one direction? If the answer is yes, than I think you've solved it. If the answer is no, it might still be the problem but the sensor might be top or bottom mounted and not side mounted from the center I suppose.
Maybe it's a combination of the "bend" and the "long slow" part.
If you are making a turn the centrifugal force will push the fluid away from the axis of rotation. There is likely a level sensor only on one side of the tank, so the turn might push the fluid away from the sensor enough to trigger a warning.
The sensor likely has a time component to avoid triggering every time you make a turn, but if this bend is long enough, maybe the fluid is displaced long enough that it overcomes that minimum time.
Could be related to terrain a certain distance/time before that spot in the road. The car may only trigger the warning after the fluid level has stabilized which a flat road would contribute to.
Perhaps it's long and slow enough to make the fluid move to a certain point and stay there long enough for the light/alarm to come on. If it's electronic vs solely mechanical, I'm sure they've got some smarts so that certain changes are ignored for a period of time (incline/decline, sharp turns, short stops).
Maybe you found the sweet spot at a certain point in that long, slow bend?
Is it a Subaru? Mine is currently doing that when I go through a long turn in a highway interchange. I'm sure I need to add fluid but it's on the edge where it only shows up when the force of gravity is pulling the remaining fluid a certain way.
My thoughts too. I would often induce a quick g-force to get the last few drops out of the same fluid tank. I could see the same triggering a sensor under similar circumstances.
One night while in the computer science lab doing a Java assignment, my professor for that class happened to walk by the lab and quipped to me, "oh, good luck on _that_ machine."
He explained: once upon a time, the machine refused to run _any_ Java programs, and would spectacularly crash and burn instead. C++ fine, python fine, anything Java was a hard nope. He didn't believe this at first until his program also started crashing the machine.
It took a tech, him, and another professor about two weeks to work out that the JVM happened to allocate the same RAM address to the integer 12 on that particular machine every time the JVM started. The actual chip of RAM that contained that hardware address was faulty, so whenever the machine tried to allocate to that address, it would crash.
Swapping out the bad RAM stick immediately solved the problem.
> In causal inference, a confounder (also confounding variable, confounding factor, extraneous determinant or lurking variable) is a variable that influences both the dependent variable and independent variable, causing a spurious association. Confounding is a causal concept, and as such, cannot be described in terms of correlations or associations. The existence of confounders is an important quantitative explanation why correlation does not imply causation. Some notations are explicitly designed to identify the existence, possible existence, or non-existence of confounders in causal relationships between elements of a system. / Confounds are threats to internal validity.
Here is a sketch of a statistical model that shows a confounder (a variable affecting both the dependent and independent variables)
S = f(H, I, T)
`S`: car starting or not (dependent variable)
`H`: how hot is the car engine (independent variable)
`I`: ice cream type chosen (independent variable)
`T`: time taken to buy the ice cream (a confounder)
Explanation: `T` influences `S` because a shorter time leads to `H` (a hotter engine, which is prone to vapor lock). And `T` also influences `I` (type of ice cream chosen) because the placement of vanilla ice cream allows for quicker purchase. Voila, now we have a spurious relationship between `I` and `S`.
This is an interesting explanation, but wouldn't `I` influence `T` rather than the other way around? Since the type of ice cream determines the amount of time taken in the store.
My comment did two things (but they were somewhat muddled). It: (a) laid a particular model; and (b) offered {explanations/claims} of causality. But unfortunately it said nothing about (c) experimental design.
I'll start with (c). Attempting to talk about a model in isolation from its experimental design can be misleading, as it ignores the context that gives the model its interpretive power and validity. In this case, a good experimental design must include a sufficiently diverse sample of people to account for variation.
Regarding (b), depending on the person, the influence could flow either way between `I` and `T`, to varying degrees.
- Example of `I->T`: One person might come into the store strongly preferring one type of ice cream (`I`) and be willing to take time to look for it (`T`)
- Example of `T->I`: Another person might come into the store in a hurry and be motivated to procure the closest ice cream flavor.
Regarding (a), no model is 'true' but some are better than others for particular purposes.
- To the extent that prediction is the key goal, confounding variables don't usually matter.
- But to the extent that _statistical inference_ is the key goal, there are many techniques for teasing apart influence.
Unfortunately, too often in machine learning contexts, the word "inference" refers to the process of using a trained model for _prediction_. Yikes. This contrasts sharply with the term's use in statistics. The field of statistics got this one right, even as ML techniques have taken off spectacularly.
When I moved into my last house, I had a fun electrical problem. We moved in October and I would end the day by taking the trash out. Now, my garbage can was outside the door from my garage to the backyard. One night the light outside that door didn't go on. It was late, so I figured I'd look at it the next day. Worked fine all day, then at night it didn't work again. It was a new house, so I called the builder.
The electrician came out to check it and gave me the most incredulous look when I told him it didn't work at night. But he went to take a look. Came back later and said that the wire was barely touching the light fixture. So, at night, when it go colder it would slightly pull back and no longer be touching. During the day it would warm up, expand and would work just fine.
my wellhead does this when it's below 20F outside. Oxidization on the wires throughout the year mean that any shift of the wire causes a disconnect.
haven't figured out a solution, i just run a drip that is loud enough i notice if it stops so i can go out and reseat the wire, otherwise the stuff outside the well can freeze and that costs money to replace.
Sounds like you just need a better mechanical connection. There are silicone-filled waterproof wire nuts available. Maybe if you clean the wires and secure them with those all will be good?
I was trying to fix the flaky Bluetooth connectivity on my Subaru. There was one post on the forums that said "Just slam the glove compartment closed really hard."
Knowing that this was bullshit, I tried everything else to no avail. I finally caved and slammed the glove compartment. To my surprise, Bluetooth has been solid ever since...
Simpler but funny tech support anecdote: at some point, someone called me over and said their keyboard would print the letter 'e' repeatedly. I tried another keyboard of course, and it still did it! Rebooted the computer, still 'e's all over.
Finally, I looked at her mouse: it was half of a wireless mouse-keyboard combo. Turns out, the keyboard hadn't been turned off before it went in the pile, and other keyboards on top of it were pressing its keys, or rather just the one key.
Bluetooth devices are the worst for this. I had a very similar case where the letter "z" would randomly be pressed. Same case of random Bluetooth keyboard in a pile. More obnoxious though was a Apple Mouse that kept trying to connect to one of our work stations. It over and over would pop up a dialog saying the mouse wants to connect. It would happen usually in the morning and last a few minutes during which you in essence couldn't work. There was no way in macOS to have it remember the decision to not connect. When we tried to just pair with it too get rid of the dialog it would fail. We never solved this one. It just stopped happening one day.
Like all prefunded startups we rented a crappy apt to work out of. Outside of the neighboring fire station which woke us at all hours, everything worked fine.
A month into summer, the internet started to die at around 1pm… then magically restore itself in the early evening before the tech would arrive. This went on for a solid week or more.
Eventually the provider agreed to send a tech immediately after we called. On first inspection, everything looked good. Thankfully he was diligent and found that an old pin based IC was likely expanding in the heat and every so slightly unseating itself in its socket. Properly seating it and adding some hot glue solved the issue.
Never underestimate heat dissipation in product design.
This reminds me of the "my monitor blinks every time I sit down in my office chair" turning out to be EMI spikes from the gas lift affecting the signal traveling on monitor cables.
A DisplayLink KB article even mentions it (and the associated white paper about the issue), stating:
Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue.
This actually happened to one of my colleagues a few years back! Except it was someone in the next row over sitting down causing his monitor to turn off. Took them ages to figure it out.
Direct whitepaper link (warning: PDF): http://www.emcesd.com/pdf/eos93.pdf - if people prefer to search themselves and not use my direct PDF link - it is entitled "A New Type of Furniture ESD and Its Implications" by Douglas C. Smith, from 1993
Reminds me of a short book I like that talks about these sort of tings titled "An Engineer's Guide to Solving Problems." It starts with a similar situation - "The Dog Barks When the Phone Rings." Eventually, you get to a section called “If I Could See it, I Could Fix it!” which discusses the importance of understanding the problem before attempting fixes.
Went to a friend's house years ago, and knocked on the front door. Loud running, then BOOM door moves as dog jumps up onto other side. After some finagling, my friend finally gets the door open and says "Use the doorbell next time, he hasn't figured it out"
Let me guess, the dog is chained to a metal post, the metal post happened to splice the telephone line buried in the garden, and the dog was getting 100V of ring voltage every time.
I find it incredible that not only a family would eat ice cream every, but that they would travel to get it everyday instead of just having it on hand in the house.
Not to mention that the car company would send someone out not only to their home, but in the evening and not only once but for weeks. I'd even have a hard time believing this if they loved next door to the companies HQ. Either the world has changed a lot or this is totally made up.
Or that a store would put it's vanilla ice cream in a separate case in the front and all it's other flavors in another in the back, instead of putting all it's frozen foods together in the same area for logistical reasons.
all (and i mean all 6) convenience stores around me have a separate "good humor" chest freezer in the front, directly adjacent to the entrance, and the rest of the ice cream (here it's blue bell invariably) is in a vertical case in the rear of the store. I can go take pictures, if HN doesn't believe the pervasiveness of this sort of thing!
I did not even imagine the possibility of shopping for ice cream, daily, at a convenient store due to the extra costs from convenient store pricing. But I guess it is in the realm of possibility.
the good humor company pays the convenience store to have their freezer in that location in the store, and are the ones responsible for keeping it stocked, unlike the rest of the store.
My local grocery store has a selection of ice creams in an end-cap cooler at the face of an aisle. The complete frozen goodies selection is down at the back end of the store. I almost always snag what I want from the up front cooler rather than making the trek to the back.
Plus the hand-held frozen goodies, like ice cream truck fare, are usually in their own cooler separately from the big stuff.
This reminds me of my village story. Two brothers married and lived next to each other. The wives hated each other and competed on everything. If brother 1 got something brother 2 had to buy the same thing but bigger.
Over time they ended up building homes in this manner too and one of the brothers purchases the largest available plastic watertant on roof. Offended the other brother decided that he will build even bigger water tank. So they built a massive concrete water taken on the roof.
However the problem with this tank was that it would simply not retain the water which was pumped into it via a small motor pump which pulled the water from nearby well.
This resulted into the brother accusing the brother of "black magic" and engaging in daily fist fights and abuse.
Eventually someone figured out that the person who connected the intake pipe connected it at the bottom of the tank. So wehn the tank was full the water would simply go back to the well.
I was debugging a Heisenbug once, developing embedded FW for a mobile phone.
After some time, I noticed that the phone seemingly only crashed in one area of the open office floorplan where I was working.
I started walking around the office testing this theory, not really believing it. But after a while, I had hard evidence that the bug would only manifest once I entered that part of the office.
When I came to terms that I wasn’t hallucinating, I realised what the problem was. There was poor reception in that part of the office, causing the phone’s modem to switch from 4G wideband to narrowband (glossing over details here), which triggered the bug.
Easy to see with hindsight, but I was very confused there and then
We had a problem like this too at a satellite company. A customer claimed their service was cutting out every day between 3-4pm. There were no service outages, and it only seemed to affect them, but we did see their modem drop consistently each day. Turned out it was a very low elevation location to where the dish was virtually horizontal. They received UPS packages almost daily around that time, and the truck was blocking the signal long enough to lose service momentarily.
Oh man, I'm currently fighting a problem with a mid-70s coupe that's driving me equally batty.
Randomly during longer trips, the car will just die for no discernible reason. It's the Car of Theseus at this point with how much I've replaced, but the issue persists, and the nature of these intermittent problems makes debugging a nightmare. More puzzling still that the car starts up fine after a short nap.
I'm sure you've tried a bunch, so my comments might be talking into the void.
I had a similar issue that turned out to be a slightly loose battery connection. While the battery clamp was making contact with the battery terminal, I didn't tighten it enough and it made poor contact.
Is it correlated to temperature at all? If it is, I wouldn't be shocked if something, like a relay, is building up heat and increasing resistance to the point of operating incorrectly. A short nap might give that component enough time to cool off.
I had a Toyota pickup that would die randomly, sometimes at highway speeds. Although in those cases, the engine would restart after a second so it felt like you hit a brick wall then kept going. A friend suggested a bad fuel pump. OK, replace the pump and filter (it was cheap and an easy R/R) but no change in behavior.
After a while, I correlated the problem to very high humidity: usually happened during heavy fog or rain. So, it's probably an ignition problem, right? Replace spark plugs. Nope. Distributor cap/rotor. Nope. High performance plug wires. Nope.
Drove me nuts for about two years. Then one day I'm in my garage looking for something and I move my timing light out of the way. Hmmm, didn't think about that...
After two years, problem turned out to be timing slightly out of spec. Fixed in five minutes!
Cracked fuel line? Might let air into the system in certain operating points; might make the fuel pump unhappy (refuses to prime) in some situations.
I figured mine out when it finally broke completely: All the other issues went away after replacing it.
Wiring harness issue. Reseat any and all connectors from the battery to the distribution box (there may be two of those depending on the model year) and reseat all of your fuses, if any of them go in too easy then use a small screwdriver to force the contacts to be closer together (or slightly twist the fuse tab, not the most elegant solution but sometimes those contacts in the fuse box are so far recessed that you can't get at them). Good luck!
I can't vouch for the veracity of this story, but an older engineer who worked on a particle accelerator once told me this story from the 1980s:
The particle accelerator would start overheating every day right after lunch time. They eventually figured out that enough people were using the bathroom after lunch that it was affecting the water pressure in the cooling system!
Is this the new 500 mile story (or the second-place replacement?)
I don't know whether this is true or not, but either way: incredible. Love this, and love the company for actually sending someone on company time (!??!?!?!?) to check it out. <3 :'))))
> Vanilla, being the most popular flavor, was in a separate case at the front of the store for quick pickup.
This feels very contrived; who organises a supermarket like that? "Where's the ice-cream?" "Which flavour? We keep them separately, to make an anecdote work."
(Like, the moral of the story still works, but the specifics feel very dubious...)
But I guess that would make the timing part be obvious and kind of ruin the "fun" of the story. Though I fully agree, the difference between "front of store" and "back of store" icecream couldn't be that much and also doesn't make much sense.
Or I could imagine something like a Sonic drive-through, where you often pull up in to a slot and turn off your car, but only until your order gets brought to you by a bellhop:
- works when I order a burger (burger takes a few minutes to cook)
- doesn't work when I order a hot dog (those are hot and ready to go)
or something like, works when I order a chili dog (because they have to heat up the chili)
> Or I could imagine something like a Sonic drive-through, where you often pull up in to a slot and turn off your car, but only until your order gets brought to you by a bellhop
.... Wait, that's still a thing? Not American, and my visits there have only been to urban areas, but I had kind of assumed that this was something present in films from the 1960s, not, like, 2023.
Hell, I'm skeptical vanilla is the most popular flavor, although that may change by year.
At the very least, chocolate is neck and neck and on the only sales chart I could find (which I think was for the UK?) chocolate sold more than vanilla.
Could be also that it was certain large shipment they wanted to get rid off. My local hypermarket does have freezer sometimes with icecream next to check out lanes with lot off other stock they want to get rid off.
Fabricated anecdotes like this miss the point: debug is a multi-layer approach, not a think-it-through logic puzzle. In reality, the driver would have encountered vapor lock in multiple situations, not just shopping, and as a result I think this example does more damage than good by promoting a furrowed-brow approach.
So this guy never ever restarted his car after a short interval except for when he bought vanilla ice cream? Additionally, he never varied the time intervals in the shop when he was buying aforementioned ice cream?
Is it is an understatement to suggest this is a highly unlikely circumstance?
You don't think it's likely that the regular 5 minute stops were the first warning sign and most common time he restarted his car after a short interval? Especially vapor lock on a hot day when one is likely to buy ice cream? Maybe he would've noticed it in more places if it got worse. And since he was going for ice cream, he was unlikely to dawdle around waiting for the product to melt?
He probably learned how to reproduce the problem, experimented with it, and when the engineer came, he was proud to reproduce it and embellish his story a bit to make it seem perfect.
A lot of men are extremely efficient shoppers. They know what they want and where it is. They don't browse around at all. They have the cash or card ready. Walk in, pick up the ice cream, pay, walk out. Could be done in 30 seconds if there are no other customers.
A few years back, I had the opportunity to work on a project in rural India. Our team was responsible for creating a new patient booking system for a community hospital. We designed a web-based application that could be accessed through the hospital's intranet.
Everything seemed to be going well, but then we hit a snag. Our server began restarting unexpectedly at random intervals throughout the day. Initially, we were baffled. We assumed that the application was crashing because it could not handle the load, and we spent hours digging through code and logs.
But the real culprit turned out to be something entirely different. The server was located next to a storage cabinet, and its power supply line was inconveniently blocking the cabinet's door. Whenever the nurses needed to access the cabinet, they would simply unplug the server, get what they needed, and then plug it back in, oblivious to the chaos it caused in the system.
It taught us the importance of considering all variables, even the human ones, when troubleshooting technical issues. And so, we moved the server, cleared the path to the storage cabinet, and things were back to normal :)
Someone in my extended family had a Toyota Highlander which would randomly decide "you don't need window defog today", leading to a near inability to drive the car if fogging was an issue. The dealer didn't believe him until one day it actually happened while they were watching. They had no idea what to do or how to fix it. Solution: Trade the thing in against a Honda Passport. He's had that for quite a while now and I haven't heard any complaints.
On the other hand, my previous Honda Civic with its dual voltage electric system (computer thinks the battery needs charging? Generate 14.4V. Computer thinks it doesn't? Generate 12.6V) caused us considerable grief until we just started driving around with the headlight switch on all the time (this forced it into the higher charging voltage). This "feature" is not well known/understood even by mechanics and has probably caused untold numbers of alternator replacements.
Current Civic is so automated that even the headlight on and high/low beam is under the computer's control. Hopefully no weird chronic computer bugs in this one.
I had a single user whose machine would have intermittent disconnections while compiling his codebase. Not every time, but some of the time and every day, at that. This was back in the day when compiling the linux kernel was a 1hr+ task.
The key piece of evidence was when we moved a printer into his office, and it had the same disconnects, at the same times. Every day around midday. Not one disconnection, mind you - starting at about 11 the disconnects would get more and more frequent. From noon to one it was every few minutes. In the afternoon and evening it was just fine. It just so happened that the engineer would work in the morning, and his first compiles if the day were around that time. So, compiles were a red herring.
In the end the problem was the kitchen one floor down from his office, where a microwave was heating up peoples' lunches and causing enough EMI to disrupt (in those days rather brittle) wifi. We bought them a new microwave, and the disconnects stopped.
I have one of these to share. My first job out of college the office had an issue where at ~3pm every day, the network would go down for a minute or two, then be slow for a while after coming back up. This as a digital marketing company, so lots of file/image/video uploads happening all the time. We ultimately determined it was the router hitting max load, crashing then rebooting and having to manage all the pending uploads. From everyone realizing it was getting close to the end of the day, and making sure their video uploads were started. The company had grown a lot in the last year, but was still using a home grade router/wap combo. I replaced it with a UniFi AP and an UniFi edgerouter. And the problem went away!
My DSL internet would get incredibly slow at certain hours of the day when web traffic would be expected to be heavy (evening, as people are getting home, and around 10, as people are watching Netflix). AT&T refused to believe that the cause was congestion. No no, they said, it must be that there's a sprinkler that's going on at the same time each day, or a microwave that was causing a problem with your internet. Eventually they sent someone out and discovered that it was, in fact, a congested circuit that I was on. I couldn't believe they were so steadfast in claiming that it was some crazy-sounding external cause.
Ah, reminds of my first car. I got a new Corolla and went on my merry way. But on the third day I found the steering stuck and extremely hard to maneuver. Scared the heck out of me, so I went back to the dealership.
They inspected the car and informed me the next day that there was nothing wrong with it and asked me to pick it up. I drive out of the parking lot, and again the steering gets stuck, and I just circle around and give them the car back, showing the live problem.
They inspect it another day, and aren't able to find anything wrong, and ask me to pick it back up.
Sure enough, I haven't even gotten to my home yet and the problem reappears, so I drive it back to the dealership.
This time they open up the car, the steering wheel and changed somethings in it and asked me to pick the car back up. Took me 4 mins of driving to land back in the same problem after which I made them refund my money.
I decided to shelve a few extra bucks and purchased a Camry from the same dealership. The second day while I was parking my new Camry, I found the same issue of the steering wheel getting stuck all over again, with this new car.
So this was surely a me-problem, and not the car problem. After some analysis I relaized that I was turning off the car at red lights, without shifting to park. So when I turned it back on, the steering would lock itself. But if I shifted to park first then it would work perfectly!
I intentionally caused a problem like this for a co-worker in 1996.
He was a flamboyant character and had a habit of talking loudly on the phone for an hour or so a day. I modified his mouse driver to give me remote control on demand and installed it when he was away from his computer.
For the next few weeks, I would wait for his daily call and gradually move his mouse as he was trying to use his computer. He was visibly frustrated and eventually caught me when he went on a rampage during his conversation and he caught me laughing.
When I was a student of mechanical engineering many years ago I was very lucky to do an internship at a major car supplier in Japan. We did the engine application for a car on the Chinese market and being unexperienced as I was, I was more of a burden than help to the engineer who I was assigned to.
However, when driving around with our test car, from time to time the engine stalled, or stalled an restarted. This seemed to appear randomly, and since the engineer was busy with doing the engine application, he asked me if I could try to take care of the issue.
I noticed that the problem occured only during rain, then I noticed that it only occured when the windscreen wiper was set to interval. With some help from engineers from the main office, I was able to find the relais controlling the interval mode for the windscreen wiper as the root cause. I was sending voltage spikes into the cable tree, causing a reset in the central control unit. A decision was made to change the supplier for the relais (this was not supplied by our company), and the problem disappeared and never occured again. Everybody was happy and I felt really proud that I was actually able to contribute to the success of the project.
engineer, being a logical man, refused to believe that this man's car was allergic to vanilla ice cream. He arranged, therefore, to continue his visits for as long as it took to solve the problem.
the moral of this story, the engineer being a logical man refused to try reproducing the problem and devoted infinite time to watching the user shop every night. that way he gets paid to hang and eat ice cream
What is most incredible about this story is that the president heard about the story himself and that an engineer was sent to investigate. Now days if a car has problems under warranty it is often assumed the driver is at fault and of course problems never happen when you are showing the mechanic.
A couple of weeks ago I got back from a remote site, the only diagnostic info we had beforehand for one of the issues was that the mobile equipment would trip out with a CANbus error. The operator would isolate the equipment, then it'd come good. I was struggling to find the software fault since, if a simple off-and-on-again fixed it, clearly it was a software problem.
I got to site and found a loose connector on an IO module under the operator's movable arm rest. It was the connector that carried the CAN comms. I plugged it in. No more dramas.
I can only surmise that the process of them slamming the arm rest up, tromping down the stairs, flicking the isolator off and on, tromping back up the stairs, and slamming the armrest down was enough to re-seat the loose connection temporarily.
I lived in N. California in 1996, and I had an impressive 486DX66 tower machine. That summer, I would go to work during the day, and my girlfriend would use the computer as much as she liked. I'd use it at night and on weekends.
She began to complain that it was locking up. It happened to her over and over, but not to me, so I wasn't sure why.
One day I had occasion to remove the case and look inside. The CPU fan had become dislodged, and would spin uselessly without cooling down the fast 486.
I owed an apology to my girlfriend. Many years later, I was able to apply this elementary knowledge to help out my father, whose notebook always locked up. He put it on a cooling pad, which was enough to allow the vents to work, even without running those external fans.
This reminds me of a problem I had a few years ago. Whenever I sat down on my office chair, my monitor turned black for a few seconds.
One day I started to understand what’s happening when I touched the aluminum Apple keyboard while being electrically charged. I was wondering how it was possible to get an electric shock since my Mac Mini had no connection to ground via its IEC-60320 C7/C8 connector. I learned that the Mac Mini grounds itself via DVI.
Turned out the pressure cylinder of the chair caused some kind of electromagnetic pulse, which interfered with the DVI signal and forced the monitor to resync every time I sat down.
Yeah, that isn't how this works. Popular things are closer to the back of the grocery store -- milk, eggs, meat, produce, higher-volume soft drinks -- in order to force you to walk past everything else to reach them.
Ice cream is not that popular, so it could reasonably be stocked near the front of the store. But all the flavors are stored together, as you'd expect.
Engineers who make house calls are also not a thing in Detroit. You might see that happen in Japan, where airline CEOs have been known to make pilgrimages to call on the families of crash victims. But I'd be (pleasantly) surprised if the car companies have ever done anything like that.
People say this a lot. I think it's mostly not true.
Milk and eggs are at the back of the store because they're high-turnover items that require refrigeration. In literally every grocery store in my area --- Pete's, Whole Foods, Caputo's, and Jewel --- produce and high-volume soft drinks are both in the front of the store; in fact, the major soft drinks are stocked outside the store at Pete's, in the vestibule where the shopping carts are.
In no grocery store I have ever been in have the bulk of the milk and eggs been in the front of the store, despite the fact that stores compete with each other, and are a low-margin business.
Grocery store layouts are certainly optimized. But I don't think the "milk and eggs are hidden so you'll buy snacks" narrative makes much sense. The stores are loaded from the back, and easiest to refrigerate from the back, and having clerks constantly trucking milk from one refrigerator in the back to another in the front of the store seems pretty suboptimal. I think dairy placement is a constraint the stores work around, not an evil scheme.
> Popular things are closer to the back of the grocery store -- milk, eggs, meat, produce, higher-volume soft drinks -- in order to force you to walk past everything else to reach them.
> Ice cream is not that popular, so it could reasonably be stocked near the front of the store. But all the flavors are stored together, as you'd expect.
This part is actually more plausible than you'd expect, especially if you assume that details are slightly garbled by the N'th retelling.
Last year, we went to a convenience store to pick up some ice cream treats after a big hike. There was one of those mini freezers with a set of frozen treats right next to the entrance, facing the cashier. But there was also a line of full-sized freezers containing all the usual frozen goods (including frozen ice cream treats) in one of the aisles. Depending on which frozen treat you wanted, you'd either be picking it right up by the front door, or having to wander down an aisle to find it.
Yep, same with the grocery stores around here, they all have last-minute snacks and drinks at the checkout stands.
Ice cream tends to be somewhere close to the middle of the store. But if you're after staple foods in larger-than-snack-size quantities, you'll be doing some walking.
It’s an engineering allegory, not a literal historic account. You’re supposed to learn from the engineer who diligently takes notes until they discover the obscure correlation that clues them in to the real problem. The true problem is not always what we assume it to be, but you can’t dismiss the existence of a problem because it seems unlikely.
The included details of the narrative are deliberately fanciful to make it obvious that it’s not intended to be taken literally.
Of all the details people are trying to pick apart, I’m surprised nobody mentioned how strange it was that they drove to the store every single night for ice cream rather than just buying a few large containers and putting them in the freezer. :)
It's a law in my family that no matter how many containers of ice cream we buy - or how large - they will always get eaten in the next 24 hours. So we end up doing the same thing (though not every day).
Many years ago I was at a small group meeting with Terry Myerson who ran the Windows phone org and had just assumed responsibility of the entire Windows org. I had a Nokia 1520 (the giant phone) and he asked me what I thought of it. I said I loved it, and then complained about how it kept on dropping bluetooth with my car's head unit, a new Subaru STi.
The next day there was an engineer sitting in the back of my car with a bunch of test devices capturing traces of the BT comms with the head unit. Apparently Subaru didn't sell enough units to warrant its own certification process for BT so this was the first time engineers had looked at it. IIRC it did get better a few updates later; it was maddeningly unusable out of the box.
Ford has been known to send corp engineers to dealerships to help diagnose and resolve recurring issues in specific vehicles (has happened to a coworker of mine, and the engineer did end up fixing the issue!) -- I wouldn't entirely doubt.
My mother has a Jeep Patriot with a water leak where it kept getting into the ceiling lights, drip on the interior, or on the driver even.
After 3 or 4 trips across the county to the dealership she threatened to invoke the Lemon Law on the new vehicle and wouldn’t you know it, Jeep sent out an engineer to Beaufort, SC and he spent a week on this vehicle. Was fixed and never had a leak again.
Some versions say it's the dealership, not the manufacturer, being contacted, which seems more probable.[0] (It's also from the 70s, which also helps it be a little more believable, imo.)
I once sent a message via linkedin to a VP of an error I was getting on their company's website. He answered and had one guy on their team to look into it. They fixed it the next day. Sometimes people answer their messages!
It does happen, sometimes. Sometimes it's a case of who you know.
When I was at Amazon, Jeff sent out a question-mark email after a complaint from a customer, something he used to do when a direct email complaint caught his attention in some particular way. The complaint was that some amazon hardware was having difficulty with this customers wifi network. I want to say it was a 1st gen Echo, but it has been close to a decade since, and I only saw the report that was produced from the situation, that I honestly forget the particulars.
That question mark email ended up with Amazon sending some senior engineers around to go figure out what was going on at the customer's home, and figure out what that meant for the device and how it might be mitigated. It ended up being some weird combination of physical properties of the house, the wifi arrangement, and some suboptimal behaviour in the Amazon device that was fixed via a subsequent software update.
I miss Car Talk with "Click and Clack the Tappit Brothers". Not sure if this was on the show, but seems like it could have been. Definitely check it out if you enjoyed this anecdote.
Hold my beer. I once had a fairly large CRT monitor which most of the time worked fine, but sometimes would show a warped, distorted image. After a while I noticed that warping was happening only on rainy days. I opened the case and discovered gooey stuff smelling cat urine on the monitor's electronics. Washing it with pure alcohol solved the problem. I have no idea what was going on there, but I consider my cat was an extremely lucky dude as I almost never turned that monitor completely off.
Before you blame your cat, consider the possibility of an exploded electrolytic capacitor. Among many other foul comparisons, I’ve heard people say they can smell like cat pee after they die.
Vanilla consistently ranks as the most favorite ice cream flavor. A search for "most popular ice cream in the world" has as the top result: "According to statistics, vanilla ice cream is the most popular ice cream in the world. From the USA across to China, its appeal is universal."
Same reason grayscale cars are very common: even if a family likes bright colors, they may not like the same bright colors, and may not trust they’ll be able to sell it to someone else later.
Vanilla accounts for about a quarter of all ice cream sales in the US. Chocolate is a close second but I think vanilla wins as it is also the most common to serve with other desserts. Apple pie a la mode is always with vanilla ice cream for example...
It's the most popular flavor in the same way the Settlers of Catan is the most popular Euro board-game. It's rarely anybody's favorite, but very few people hate it.
It is, certainly would have been at the time anyone would be driving a Pontiac. But it is dubious that the store would have a separate case for it. And making a trip for it every night after you've had dinner? And an engineer spent days working on the problem in person? THe story is weird.
> And an engineer spent days working on the problem in person?
This is the only slightly credible bit IMO; companies sometimes do this! If you have a really weird problem in something you've sold a few million units of, you really, really want to know what that problem is, before more people start complaining.
The supermarket layout is clearly contrived to make the story work, though, and doesn't otherwise make any sense.
Why did the car only fail to start in front of the ice cream shop? Surely there are more occasions where you turn off the engine and turn it back on again within a short time?
Because the story is fiction. There are so many variables that would change the amount of time the engine is stopped: where you parked, how long the line at the register is, whether you had exact change or not. Also the ambient temperature would change both the likelihood and duration of vapor lock.
Yes. There are so many details in the story that are unbelievable. I will possibly believe that there was some original true story with some subset of these circumstanced but it got rearranged and exaggerated for effect.
I will never believe that GM sent an engineer to troubleshoot this problem. Maybe in a small town a local dealer's mechanic spent some time troubleshooting but he probably figured out it was based on the short shut off time. Then he speculated that it took less time to get Vanilla than the other flavors? I doubt that is true - most likely it was random. It is amazing how often a random sequence will have an extended sequence that our brains won't believe is random.
Not sure I even believe there was any genuine observation of correlation from the owner. These type of apocryphal technical support stories are almost always formed backward from a problem and someone tries to think of the most insane way it could have manifested. But people love to believe them.
I currently have a Raspberry Pi that consistently loses its network stack at some point in the early morning.
The likeliest correlation is that I have the Pi dump backups at that time, and it may be crashing the network stack due to unexpected hardware output because running a hard drive and the internal wifi simultaneously under-volts the system. But it sure does look like it just gets visited by demons in the pre-twilight hours.
I had a Dodge truck with a similar problem but it was never resolved. If I went to an ATM or convenience store on a muggy day when I left my truck transmission stuck in 1st gear. It was obviously the short time stopped and start up again but no clue why. The dealership looked a few times even taking it for a drive with a test tool but nothing happened. It did it until I sold the truck for about 17 years.
I have an 83 Land Cruiser and knew immediately what the problem must have been related to from the outset. It actually has a fan that will kick on when the engine is turned off and the car parked to blow on the carburetor and cool it off.
Incidentally, I still don't believe a word of this story (at least as it's told here). The short delta of time difference between walking _further_ into and out of a store would not have enough impact on the cooling of the engine to make a such a substantial difference as to it starting or not. It simply will not bleed off that much more heat unless this store is a mile long and it's an additional 20 minutes to get a different flavor.
The only reason I express the doubt over it is because it makes the story a contrivance, which makes it pointless. If the person's different activities _actually_ resulted in a significant difference of time the car has been sitting, it's likely the owner themselves would be able to easily deduce what could really be the issue. By pretending the issue introduces some very small delta of time, it arbitrarily masks the true cause (which is the entire point of the story).
"If the person's different activities _actually_ resulted in a significant difference of time the car has been sitting, it's likely the owner themselves would be able to easily deduce what could really be the issue."
Oh, I wouldn't even remotely bet on that. Even in my professional sphere of programming I've been caught by what I call "cognitively available" theories of what the problem is that turn out to be entirely wrong in the end, because the real problem is something I wasn't even remotely considering before hand, and possibly even would have dismissed if it had crossed my mind.
If you don't even know what "vapor lock" is, and I assure you this will describe the majority of car owners, why would you think "time in store" is the difference?
What is cognitively available to this person is that they buy different sorts of ice cream and that causes the problem. It puts the spotlight of cognition on that factor to the exclusion of other things. Even the engineer trying to solve the problem was probably slowed in his investigation by such an appealingly available issue being proposed first; again, I've certainly experienced this in my own professional sphere where someone proposes some explanation that ultimately turned out to be completely spurious, and it takes actual effort to get both myself and my team off of that line of thought.
Is that what that's for? My Mini Cooper does the same, sometimes. I assumed it was something to do with the engine space being so small and cramped, so the fan had to continue for a bit after shutdown to prevent residual heat from getting to the electronics or something.
I’ll add to the crazy anecdotes with something that happened to me a few months ago.
I had a brand new AC/heater hvac system installed last year. Encountered this problem, the AC would just turn off. First time, called the AC company and they come out. After a few visits, they pinpoint the problem as the condensation pump, it’s just a small water pump that’s down in my basement to push water out a little PVC pipe. When the water fills up in the reservoir, a pump kicks on and moves water. If the water gets too full, a sensor turns the AC off. After the rd visit they decide that “the grade isn’t gradual enough and the pump turns off”. They replace the pvc. Fourth attempt they replace the pump, pump must have failed. Fifth visit, they replace the pump with a different brand. Sixth visit they disconnect the automatic shutoff of the AC, my basement floods with water. Seventh call they change the pvc pipe again. Eighth call they are baffled, say everything is working as it should, give it time.
After they leave, my wife goes “whenever I turn the basement light on, I hear a weird noise”. I immediately turn the switch off and walk down into the basement and test the outlet. The light oddly enough controls a random outlet the pump was connected to. Every time a tech came over, they turned the switch on.
I still wonder how many more phone calls it would have taken before they figured it out.
I used to have wireless infrared headphones. Sometimes, the audio would go crazy, but only in a certain area of my office. As it turns out, the IR blaster on my TV was interfering with the signals. It was really interesting to hear the clicks of data being sent over IR, though.
How about a bunch of iPhones in a medical facility being disabled because of some amount of helium being leaked when they were charging the MRI machine?
My parents have an early '50s MG, which they rarely drive. For most of my adult life, I've lived about an hour away from them so I'd sometimes take it out if I was there. Needless to say, it had a never-ending list of quirks due to being so old and also rarely being driven, but it's a lovely old car all the same.
One day, I dropped by to say hi, but they were both out. I decided instead to take the car for a spin around the local beachfront to give it a turn over. A few minutes into the drive, I started to lose engine power, so I pulled over. The engine then completely died on me, so I let it sit for a moment before trying to kick it over again.
The fuel pump is such that when you first turn the key to power the accessories, you hear it go tickticktick tick tick.. tick... tick...... - the ticks slow as the pump builds up fuel pressure. It's a good audio cue as to when you can then turn it to ignition and kick the engine over. In this case though, the ticking wasn't slowing - just the same tickticktickticktick. I tried to kick it over several times, but no matter which deity I invoked, no luck.
Empty fuel tank then. I checked the fuel level (walk to the tank on the back and poke a special bit of wood in to see how full it is) but lo and behold, plenty there. So I let it sit for a few minutes more and try again, hoping it may be to do with a flooded carburetor after my several attempts to restart it. The same: tickticktickticktick.
I gave Dad a call, describe the problem, tell him what I'd done to solve it and that I'd concluded the fuel pump must be cactus. I asked if he was going to be home soon to come give me a tow home. At this point, I learnt that he was some hours away and Mum was overseas, so no luck there. As I'm mentally preparing to push the car home for over an hour, he interrupts - "open the bonnet, grab the spanner out of the toolbox there and give the pump several hard whacks".
"...what?"
"Beat the bejeezus out of the fuel pump a few times."
So I did, and I turned back on the accessories. Ticktickticktick ticktick tick tick... tick.... tick.......
Dad then explains that this problem's been around for decades. Very occasionally, a bubble of air will end up in the fuel feed line. It then blocks the pump, which can't clear it, but a bit of suitably percussive maintenance consistently dislodges it and the pump can draw fuel in again.
As they say, old cars definitely have character, and I think that comes about largely because people can understand, fault-find and fix these sorts of analogue issues that arise. New cars are much more reliable and don't face nearly as many random faults, but those that do happen are almost impossible for Joe Public to resolve on the side of the road.
Funny thing the first time I heard this anecdote was by a priest (he was formerly an electrical engineer). The conclusion was similar: how many times a seemingly illogical issue has a very logic explanation (even if it looks illogical to you at first).
As an old engineer. Stories like this are great to teach new engineers never to ignore information. Take in all the crazy information people say, or things they think are happening. It is all valuable.
Well, I believe that we won't have funny customer support histories like that from our age to post 40 years from now. At least not from google et caterva.
Can you imagine an engineer turning up to your house to solve a problem with your car these days? Manufacturers now would just tell you to buy a new one.
Thank you for your comment. Your comment should be the topmost as it offers a necessary correction. I liked the story so much that I favorited it and wanted the story to be true. It made me sad initially to know that the story is not true. But I am happier now since I that I know accurate information. Thanks much.
I call shenanigans. There is no way this not an apocryphal story. There is no way anyone could not have realized that it was the time spent in the store and not the flavor of ice cream that was the relevant aspect of the story.
A decade and a half ago (or maybe two?), I was working at a software service shop. This shop has been in charge of support of a built-to-order native Win32 application (in Visual Basic or Delphi or something like that) initially developed by another previous company that had long disappeared.
One day a coworker got a support call from a user. Far from mythical outbursts of irate customers, an understandable the voice at the other end of the line remained extremely calm and eager to help resolving the issue, although you could feel a gleam of despair in the back of her voice.
The woman at the other end of the line had to input troves of data daily in a specific form in the application, basically taking a stack of whatever field-ridden paper forms they received and entering them into the system via the equivalent digital form manually.
She was adamant that every time she attempted to input a specific, optional date the form would close on its own, losing all of the input data. To make matters worse the specific date was the last one to be entered, thus all the already-repetitive hard work of inputting an entire form was getting even more annoying.
To make matters even more complicated, she was working part-time, in rotation with another person covering for days she was off. The other person never encountered the problem!
So their solution for years had been that the woman who was reaching out to our support would input everything but the affected date field, save but not validate the form, and place the paper form in a special basket; the second person would then go through the paper stack again and fill the missing dates.
They would go to the extent of telling people who submitted the form that the affected date would lengthen the form processing by a few days. Somehow this flied quite well as people would understand that this was a case where the date would imply more work (as in, actual work, e.g cross-checking more stuff or more internal paperwork or whatever) on our customer's side.
In any case, this solution would stop working soon as the woman was going to go full time again and the other person in rotation would move to do something else. The customer had put up with this for an absurdly long time but now they were painted into a corner so they were left with hardly any option but to reach out for support.
We couldn't hardly believe this was happening, but the person was very articulate and came up with a quite complete description of the situation that in most support cases you only dream of having. So my coworker dove into the code base, which was not ours and only received the occasional bug fix or small development to keep the software compliant. He could not find anything in the code, he could not reproduce anything locally, the date field - of any of the other fields for that matter - looked like any other standard field, but there must have been something.
Confounded, the only thing we could think of was that it only happened on the user machine somehow. He asked to remotely connect and try it on the user's machine. No dice. He asked for the person to do it, anxiously watching as the user cautiously moved the pointer towards the date field when suddenly the form vanished! He went at it again and did the exact same thing, down to when things are clicked or typed and whatnot, but nope! But the user could still reliably reproduce it, over and over again.
Could it be that doing things through a remote connection affect the ability to reproduce? Out of options, he asked for the other person - which likely was at the office that day - that used to input the form to come over and input the same thing directly, watching the screen as fields filled up. It looked _exactly_ the same as when the usual user was doing it, but it worked.
We now could not see any other option but to move the investigation on site. Two people were tasked to that end to cover all possible grounds and brainstorm on the spot.
Arriving on site, they were greeted by this very amenable woman affected by the bug. They entered a small, poorly lit office, one taking a seat next to the user and the other standing due to the lack of space, and watched the user as she powered up an absurdly small, overaged CRT screen, then booted the ancient PC. As the user started typing on the $5 keyboard, moved the creaky mouse around they could not help but feel compassion as they saw the daily stack of paper forms that had to be entered every day, all day long, using such poor equipment and inhumane conditions when the customer company was throwing seemingly endless and voluminous amounts of money at our employer.
Focusing on the actions on the screen, the team member leading the investigation went through all stages of bewilderment and despair as the user arched forward to aim at the impossibly tiny date field on that tiny old screen, and just like that, the window vanished. His mind helplessly racing, in the stillness of the moment a voice raised up from behind:
"Can you do it again?"
Said the one standing up.
So, with unabated calmness only zen masters can achieve, she nodded, proceeded with moves repeated a million times over, and the window closed again.
"Yup, I got it".
You see, one of the requirements for this digital form was to match the paper form in layout and wholly fit in the screen. To achieve that, the font size and UI element dimensions were dialled down. On the user's small, poor, aging screen in a badly lit environment and with an impossibly bad mouse this made UI elements spectacularly hard to interact with, especially the date picker drop down. So when reaching to that specific optional date picker in a specific place of the screen the user had to lurch forward a bit to take a good look and lurch a little bit further and to the right to extend the arm to mover the pointer over there and click to make the date picker appear. As it turns out the affected user was a bit overweight and with this very movement her - ahem - "sizeable" right breast ever so slightly brushed against a key on the right side of the old, mushy keyboard, in a way that made it register a keypress without any audible or haptic feedback. That keypress - which I can't recall which it is - turned out to be what closed the form.
We could have patched the software to prevent that from happening. We didn't. Instead a report was produced that in no uncertain terms - but not pointing at the actual mechanical details - a combination of hardware and poor, non-ergonomic work environment was identified as the root cause; and not just that, but through small talk it was learned that the reason she was working part time was medical and largely caused by these terrible work conditions, so we added a note "as a courtesy" they could maybe end up being liable for any health issue that would "hypothetically" come up buy working in such environments. She got a new computer and display, a proper keyboard an mouse, a better desk and chair, and, I seem to recall, an additional light. The customer complied, reported the issue as fixed, and as far as we know the issue never occurred again.
Ah, gather 'round, me dear friends, and let ol' Uncle Colm regale ye with a software engineering tale that'll have ye chucklin' and admirin' the wonders of technology!
'Twas back in the early days of the internet, when dial-up connections ruled the land and floppy disks were the height of data storage. I found meself workin' as a young software engineer for a company that aimed to revolutionize the way folks communicated. They called it "email."
Now, ye might think email's as commonplace as a pot of tea these days, but back then, it was like magic. Me team and I were tasked with creatin' the very first email client for personal computers. We had grand ideas, but, oh, the challenges we faced!
Our office was a chaotic mix of wires, half-eaten sandwiches, and programmers hunched over their keyboards, with cups of strong coffee never more than arm's reach away. We'd spend hours debuggin', tweakin', and scratchin' our heads, tryin' to make sense of the code.
But there was one particularly ornery bug that had us stumped. Every time a user sent an email, a gremlin in the system would gobble it up, and it never reached the recipient. We dubbed this foul creature the "Email Bandit."
We tried every trick in the book, but the Email Bandit remained elusive. The boss was near despair, and me colleagues were ready to throw in the towel. But ye see, I had a plan. A plan so bold, it was downright cheeky.
I reckoned that if the Email Bandit was stealin' our precious messages, maybe he had a hankerin' for a particular type of email. So, I composed a message that read, "Dear Email Bandit, we've got a special treat for you. Please don't eat this one."
We sent that message out into the digital wilds, and we waited. Lo and behold, within minutes, the Email Bandit struck! He gobbled up the message and vanished into the ether.
Now, remember, we engineers are a clever bunch. We'd embedded a little tracker in that email, and it led us straight to the Email Bandit's hideout – a rogue line of code that no one had noticed before.
With the Bandit cornered, we rewrote that line, patched up our software, and celebrated like we'd won the lottery. From that day on, our email client worked like a charm, and we'd slain the Email Bandit once and for all.
So, me friends, never underestimate the power of a bit of creativity and a dash of cheekiness when it comes to solvin' software engineering conundrums. And remember the tale of the Email Bandit, a reminder that sometimes, the most elusive problems have the simplest solutions, hidden in plain sight. Sláinte to the world of software engineering!
I read a story about a university IT guy with a professor complaining that he couldn't access any websites hosted more than 100 miles away. (Or thereabouts, I forget the exact numbers.) Which is obviously nonsense; computers don't know "miles", only network relays.
Long story short, an idle timeout variable had been set to 0 milliseconds, so any connection that took 1+ ms failed, so you could only connect to systems within 0.5 light-milliseconds of the university, which is about 100 miles.
I'd actually read the story years and years ago, thus the poorly-remembered details. Total coincidence that it was posted here today. Unless someone here read one article and was inspired to post the other :)
On some days it would work perfectly all day long, but on others it would stop working between those hours.
The biggest clue was it would always work perfectly on overcast days, but on sunny days this strange behaviour would manifest again.
Turns out the problem was related to the mouse being a cheap mouse. The case had very thin plastic.
The mouse was a ball mouse, and it worked by shining an LED into a sensor on each of the X and Y axes. On sunny days the sun would completely overpower the sensor due to the plastic case being very thin and on overcast days it would not. On sunny days the mouse would only work when the sun had moved around the sky to cast a shadow over where the mouse was being used.
Perfectly logical but baffling at first.