Vaguely related - this reminded me of a support call I had where similarly the real world apparently merged into the digital world.
I was doing IT support for a small Australian company back in '98. A guy called me from a remote office, and after a few pleasantries he explained that the screen saver had fallen off the monitor of his dumb terminal, bounced on one of the keys on his keyboard, and now terminal was locked up. He wanted to know what key to press to unlock the terminal.
Eh?
I knew the guy, and although he wasn't trained in IT, he knew his way around the basics, he wasn't completely clueless.
I asked him to explain the problem again as I wasn't sure I'd understood. He repeated exactly what he'd said the first time.
I replied "What do you mean the screen saver fell off the monitor, that's impossible? Besides, it's a dumb terminal, they don't have screen savers."
After a little more fumbling around this weird upside-down world he was presenting me with, it suddenly clicked. He was talking about the physical CRT anti-glare screen filter [0] that used to be common around then, that literally hung in front of the screen. This has come unstuck and hit the scroll lock on the terminal. He called this a screen saver.
Since then the phrase "Screen saver" seems to have now morphed to mean what I used to call a desktop wallpaper, but that's a separate topic.
I love this kind of stuff. When you're sure the thing that seems to be happening couldn't possibly be happening, and then you find out that literally the speed of light is coming into play.
We had a similar problem at one of my first jobs where I was a programmer and backup network support guy. One employee was having a problem with his CRT monitor flickering. It was very subtle, but just enough to drive him nuts.
So we replaced the monitor with one that worked fine on another machine. Same problem. We tried replacing cables, power cords, and did a bunch of other troubleshooting things. Problem persisted. Eventually we replaced his entire computer. Same problem.
Finally I put his computer and monitor on a cart with an extension cord and wheeled it out into the hallway. The problem went away. It turned out to be bad electrical shielding in his office.
> The problem went away. It turned out to be bad electrical shielding in his office.
My Commodore 64 started "typing" of its own accord. We sent it to be repaired twice only for it to work perfectly when they tested it. Turned out after we got a bigger TV, we kept it too close to it, and the static electricity eventually caused the effect.
By the time the repair people got to it, it'd presumably discharged enough for it to stop, and it worked fine for a while.
Heh, reminds me of a cursed user I was trying to help in the mid 90s.
Sold a person a computer, they said it bluescreened when they used it. So we picked it up and tested it. No problems. Sent it back, bluescreening again. So they came to the office with the computer. I set it up and used it for 30 minutes with them there, not a single issue. The moment they touched the mouse the computer bluescreened. Replaced the mouse and the problem went away.
I once worked in a lab where all computers had its own electrical stabilizer, but they were so poor that probably they did more harm than good. When someone turned on a stabilizer, the nearest CRT monitors would distort for a second, then flicker and colors would be degraded.
Luckily, my place was by the wall, so the effect was diminished, but it gave me big headaches. I lasted only 6 months in that company this being the biggest reason.
The stabilizer was triggering the degausser on the CRT. Turning on speakers or putting cellphones where a call was coming in would sometimes do this too.
I knew this existed, and I was looking for it a few weeks ago; it's an interesting page to browse through every once in a while. But I just couldn't remember the name until now.
Having a "highlight" people can't find doesn't seem much of a "highlight" to me?
Edit 2: I feel like adding an "I feel lucky" link on there that would give you a random sample from the list. It's in reverse chronological order and it would take quite a while to scroll back through all of the comments (there are over 400 at present).
> I feel like adding an "I feel lucky" link on there that would give you a random sample from the list. It's in reverse chronological order and it would take quite a while to scroll back through all of the comments (there are over 400 at present).
How about a link with reverse chronology of random highlights, one per year of HN history?
This could inspire nominations for the highlights back catalog.
I was wondering how I could have browsed to /lists and /highlights and eventually found the first one in the footer of the page. May I suggest to add /highlights there as well? Maybe just after FAQ and before Lists?
+1, After two years from joining HN I’m still learning about it. This is the first time I heard about highlights section! I couldn’t find it in lists nor on any other part of the site yet still interesting to read some comments there that do not show up in best comments section. How exactly this works?
/bestcomments (or whatever name may become in the future) had a drawback, I believe. Technically speaking (and generally), the comments in this section are the ones that belong to stories from /best (or highest-upvoted). These stories obviously get more exposure and attention, so their top comments get more upvotes. For instance, a story with 2,000 points might have a comment with 200 points (which is tenth the score of its parent story) that will obviously hit the /bestcomments, while a comment of, say, 50 points on a 150-point story (which is a third) may never have the chance to hit /bestcomments.. Just some thoughts about my daily social network :)
There is probably some sweet spot of n between 0.5 and 2, but not near 1, such that the quotient of comment score^n and thread score^n gives good results.
We do in fact have down votes, you just need 500 (I think?) karma to be able to use it. When you see comments that are not "dead" but are greyed out, they've been downvoted more than they've been upvoted.
Yeah, at the time of writing I kinda defaulted to assuming it was "just" a tag for posts, which one can use the normal search on. But a while later realized that it very likely isnt.
But reading through the list, you made dang good choices in highlighted comments.
I know Trey. He's one of the most honest and forthright people I've ever met.
To a fault, probably.
And as the former comp.mail.sendmail FAQ maintainer, I can confirm all sorts of weirdness can and will happen if you mix versions of the binary versus the config file.
He certainly was honest and forthright about how much he embellished and... "took license" to spin this yarn. After reading the FAQ, I wondered if there was even a kernel of truth to be found in the story, because he successfully poked holes in practically every detail that made it interesting.
The story would be better, if they had kept the real numbers, and not added fake numbers afterwards (maybe they didn't keep notes, and forgot the real numbers).
At the end of the story, that 3 millilightseconds is the one-way distance, and that can't be correct.
Yes, that part doesn't add up. The time from sending SYN to receiving SYN+ACK would be six milliseconds assuming lightspeed between the source and a destination 500 miles away.
That said: I know the ending, and by now the details about SunOS and sendmail aren't too interesting, but the "This is the chairman of statistics" line always gets me laughing out loud.
why would anybody want to do that? I thought karma points above a certain low number don't offer benefits and the number is not visible to anybody but you.
EDIT: I just realized it's visible in profiles :facepalm:
Every September (in the 90s) a new cohort of University students would start and gain access to the internet, such as it was. Hence the reference to timing
As the Wikipedia article states, it's more about the specific period of late 1993–early 1994 when some large online service providers in the US started providing access to the greater internet, the USENET in particular, and masses of new (and usually clueless wrt established norms and netiquette) internet users flooded newsgroups and forever changed and disrupted their character and culture.
Yeah, but that was given the name 'Eternal September' as a reference to the literal Septembers of the preceding years, when there would be a regular influx of newbies.
I'm pretty new to this community and this was my first exposure to this story. It's definitely going into my "anecdotes from the internet" mental repository.
Somewhat related but not really. Back in 2007 when I worked for a large ISP as a second line support technician for various services, ADSL was very much still in vogue. And the technology, over copper wire, had a max distance of where it would be stable. Some clients were on a special plan that tried to up this distance by a bit, maybe 2-3 more km but really it was still quite unreliable but still usable for browsing the internet, generally.
But during the summer I received a call from a client that had been unable to use his IPTV service during the day for almost a month without hickups and disturbances and his internet was slow as a glacier from time to time and as I was measuring the equipment, packet loss and all the usual stuff it struck me that he was very far away from his nearest telephone station. After some back and forth with a technician and lots of measuring we came to the conclusion that since it was so hot out during that summer the line just expanded over to a distance that was just far away enough that the line would become unstable during daytime when it became hotter outside.
We could not really do anything to help him. I do not miss the copper net.
Typically they wouldn't do these jobs, the copper net was already on it's way out by that time in favor of fiber optics and the entire landline system was on maintenance at that point.
I left shortly after and these days, now they have removed almost all of it already.
tangentially related: 15 or maybe 20 years ago i worked at a repair shop and someone brought in a TV that they said switched to spanish every night at 5pm.
they were watching over the air channels and there was only a setting in the tv for menu language. sure enough though, at 5pm that night we watched as the tv started speaking in spanish. we tried a few more channels and found that all but one or two were in spanish.
as it turns out, some stations broadcast audio in multiple languages and some tvs allow you to change the preference. sadly for this person, the used tv they bought came from a spanish speaking country and didn't have anyway to change that preference.
My Bluetooth speaker switched to Chinese after a few years of use. I have no idea of how it did it and no idea of how to revert it. There is no reference to it in the manuals.
A few days before, I brought a robot vacuum home. It was made and purchased in China. When I started it for the first time, it bumped into my server and unplugged it.
Therefore a state-sponsored cyberattack is not out of the question.
I have a pair of Bluetooth headphones that indexes through 3 languages (Mandarin, Korean, English) on startup when you also hold down some other button besides power (play/pause IIRC). I don't think it's documented. Maybe your speaker is similar.
At lunch today I was just talking about Sendmail, which I can assure you is a rather rare occurrence. I was talking about the first time I set up sendmail, back in '91 or '92. I was using the bat book and nearly tore my hair out over a week getting that first setup working. I eventually came to understand and appreciate the m4 config, but I ended up moving to qmail and postfix in the mid '90s and never looked back.
That's on purpose—we want good articles to get multiple chances at attention, and we want the 'classics' to pop up periodically (just not too often), so newer cohorts of users get some exposure to them. So you did well :)
The purpose of linking to past threads is not to imply anything bad about the repost (if it were a bad repost, we would handle it differently) - rather, it's to point curious readers to other discussions on the same topic that may interest them.
I love this feature of HN. Between this, the onion salesman, C+- and a few other favorites I’m on a roughly 2 times a year cadence of getting to reread old favorites!
I've wondered how feasible it would be to do something like this to have a website that could only be accessed when a client is within a certain physical proximity of the host. Could make for a fun CTF!
I’ve played CTF challenges where the latency to the host was a key factor in determining if you could get a flag or not. For those, I’ve often found it useful to spin up a cloud machine in a datacenter near the target (or, better yet, in the same datacenter if we can figure it out).
A very common case is when the challenge has a short timeout but requires a lot of interaction, e.g. you only get ten seconds but you have to perform 10000 queries for a heap spray or something.
The most insidious case I remember was a read() call that didn’t check the result, causing it to return short if the fragments of the input didn’t arrive fast enough.
So... if you're referring to a challenge that did that during one of the DDTEK years of DEFCON-CTF, that was one of mine.
The expectation wasn't to buy time in an adjacent cloud, but to use out of order ip fragmentation or tcp segments, having the servers network stack reassemble the packets such that the read was coherent in one go.
My goal was to teach competitors to model real world challenges of exploitation.
Cool! I don't believe I was referencing any specific challenge. We've had a number of challenges throughout the years that have had these issues. The most common culprits are challenges hosted by Asian organizers, as the ping times from America to some parts of Asia tend to be quite bad sometimes.
The challenge you're describing sounds cool - I wish I'd played it! By the time I started playing DEF CON CTF though, it was with LegitBS as organizers.
Didn't do that, but one of my earliest "dynamic" websites ca mid 90's would have a CGI try to ping the client with a short timeout, and if we got an answer that indicated a leased line or something rather than dialup and we'd serve up a heavy animated version of our logo instead of a static image... But could be used as a vague indicator of distance too.
Trickiest part of doing that today is so many fewer hosts are reachable via icmp, so you'd probably be better off serving up an initial response with some JS to measure more accurately.
(Another silly little thing we added was a link back to a users own ISP from the top ten or so of our competitors based on net block - got us a worried phone call from one of them who thought we'd been hacked and wanted to make sure we didn't think he was responsible)
My quick hack would be to establish a websocket connection, and send a random stream of numbers to the client. If the client didn't return the number within a ping threshold, block their access.
Sure, but then you have the really interesting challenge of trying not to block your next door neighbour who’s accessing your service via tor/satellite.
Depends on the satellite; the ISS orbits at 211 miles, which means that it would be close enough to send a 500-mile email if it's flying directly overhead. :P
i think about this story often and i find that the person who figured out that it was 500 miles actually deserves more credit than they get in the story. have to really think out of the box to figure that one
Every time I read this story the part that always surprises me again is the units command. Converting from 3 millilightseconds to miles is brilliant, and I am delighted every time that the units command can do this.
And Trey Harris's "500 mile email" story is what clued me on to GNU units and its capabilities.
Reminder: if you're on MacOS, or one of the BSDs, your default units is from BSD, not the GNU version, and is far less capable. GNU units can be installed on MacOS through Homebrew. The package is "gnu-units", the command is "gunits" once installed.
GNU Units is also available in FreeBSD ports, just type "pkg install units" (or go to /usr/ports/math/units and "make install"). It likewise gets the binary name "gunits"
Anyone who likes the units command should plan an evening where they can sit in a comfortable chair with an appropriate beverage, and read all the comments in the data file in the source. It is like a novel about the history of measurement.
He made the story up. When confronted with the fact that units didn't convert some of the units from his story his response was essentially "Oh yeah, well of course I always supplemented my units definitions on all my machines with obscure units.".
These potential anachronisms are easy to insert when retelling a story. It doesn't really mean he "made it up", but reverse engineering how the diagnostic was determined might lead one to use tools/definitions that didn't exist at the time.
It's also entirely plausible that a local definitions file was used to provide the unit that didn't come in the default install. Hence the anachronism is merely "potential" instead of "definite"
I did a startup making a mp3 player that was attached to whole house audio distribution systems.
We got an angry email from a customer saying he woke up to ABBA playing full blast at 3:00am. While it was likely an integration/timer issue, he was wondering 'why ABBA? what is the player trying to tell me?"
The control system was sending 'play' with nothing else, which was more of an edge case based on our UI, and so it started at the beginning of the list of artists, and ABBA was at the beginning of that list.
Other players might have started at the beginning of the list of songs, but for some reason (25 years ago) we chose the beginning of the list of artists.
Later on it was configurable - random, favorite playlist, etc.
Hey thanks for linking my site! I was wondering where the extra traffic was coming from :)
If anyone has any stories that I should add, please send them my way!
Also, I have been slacking a little with the newsletter because of life events. Since there have been several sign ups since this post, I'll be more active in getting the weekly newsletter out.
That's terrific. I was worried that it was going to end at the terse OP, but the follow-up post has just the kind of play-by-play details that I can't get enough of in debugging stories.
Hey I'm enjoying your site! It would be cool if there was a way to just get the stories. For example, Twitter is no longer capable of loading tweet replies on my machine for whatever hilarious reason, and so I could only read the teaser for the Twitter link.
Anyway, I understand that's probably not the kind of site you had in mind. Thanks for the cool site!
Thanks for the suggestion! Maybe I should add archive links from The Internet Archive for stories in case links die. In the case of tweets, I’d probably need to screenshot them or something.
Realistic. And, believe it or not, I know of at least one organization that plans to convert an entire literal skyscraper of office space from routed networks to a single, flat switched network for all the employees of all the subcompanies. In 2023.
Obviously everyone with a bit of braincells left tells them to not do that because it's utterly dumb, but hey, strategic decision by the holding company to save on costs...
At least they're not using hubs. (For the younger generation: a hub is an Ethernet device that takes any packet it ingests in one port and sends it out to all other ports, with no consideration at all if the device that the packet is destined for actually is on that port - something a switch does, by maintaining a mapping of MAC addresses to ports. Extremely dumb devices, but used to be way faster and especially cheaper than switches in the 90's/early '00s)
I still keep an old 4-port hub in my junk-box because that way I can diagnose/snoop on network traffic... Although so much of it is encrypted these days that it's harder to see what's going on.
P.S.: Yes, modern alternatives would be to to buy a switch and that can be configured to "mirror" packets onto a chosen port, or a smalls Ethernet network tap unit... But why buy more stuff if I don't really need to?
"In a "switched" network, when Device A wants to send data to Device B, the switch directly connects these two devices so they can chat. Think of it like a train switcher that directly links Track A to Track B for a specific train, instead of sending it through a maze of tracks where other trains are moving.
In contrast, a "hub-based" network is like a party line in old telephone systems. When Device A talks, EVERY device hears it, but only Device B cares and listens. This is less efficient and can be slower because all devices get the data, which clogs up the network.
Another option is a "routed" network, where a router decides the best path for the data. This is like GPS choosing the best route based on current traffic conditions. It's more flexible but can introduce more delays because the data might go through multiple steps to reach its destination.
It's called "switched" because the switch acts like a railroad switch operator, making a direct track connection from one device to another for each piece of data. It "switches" the pathway specifically for that data to make the communication as direct as possible."
Does this add up? If the connection timeout is 3ms, then that means there's 3ms for a roundtrip, 1.5ms each way. So the maximum distance would actually be roughly 250 miles. But even then, packets don't actually travel at the speed of light in fiber optic cables. It also assumes that the cables are laid as the crow flies, which they aren't.
Fun read! Along the way I was trying to guess the cause and my best guess was TTL-related. However I don’t quite understand the actual cause! If the connection timeout is 3ms in practice, shouldn’t that be for a packet round-trip? So ~250 miles? And wouldn’t we expect at least a small delay on the remote SMTP server to process the packet?
I love these sort of debugging stories! It sounds like that timeout would be based on the round trip travel time to the remote host rather than the one way distance, wouldn’t that make a 250 mile cutoff?
I don’t get why this story is so popular here. None of the technical details adds up.
Multiple back and forth due to protocol handshakes and router delays would add enough latency to prevent any connection from happening at all if the timeout was set to 0 as stated.
I guess people just like the “it was the time light took to travel” vibe.
Definitely not the same level of WTF, but I worked on one of my favorite bugs I've ever seen super early on in my career.
I joined a tiny digital agency maintaining wordpress sites and about a month in one of our customers files a ticket that their website was broken. Just a white page, no error, no nothing. I ask my boss if I should switch to work on it and he goes, "Nah, this customer does that every few months and there's never anything wrong. It's something with their hosting or something (they were self hosting a site we built and maintained). Just take a look at their site when you have time to say you did and close it as can't reproduce."
Two days later I have some spare time so I take a look and sure enough, everything is working as expected. A few months pass, same ticket from the same customer, I pick it up a day or two later and everything is working as expected. A few months later, same thing, but this time I have nothing I'm working on so I pick the ticket up immediately and sure enough the website is broken. I immediately show my boss and he's like "well, I'll be damned" and then tells me to fix it. I poke around for a few hours, but can't figure it out, so I call it a day. When I get in the next morning, things are working as expected, so we're both like "wtf?"
I don't remember how exactly I ended up figuring this out or even all the details of the bug, but the root of the issue ended up being that who ever wrote the code for the site had some code that revolved around the current date and they'd hardcoded that there was always 30 days in the month. When ever the current date was the 31st of the month this code broke and took down the website, but by the time the 1st rolled around the code worked again.
I have a story from my current workplace which threw me and the other IT staff for a loop.
Get a support ticket from one of the staff saying that the screen of their laptop keeps "going black" when they are using the computer. So I set up a time to take a look and ventured away from my desk to see what's going on.
It's worth noting that this is a laptop, less notable is that it is a shared device as different people work this shift, regardless, all the staff that share it are adamant it only happens with this one user. So I asked them to demonstrate, they open a browser and start typing, sure enough after a sentence or two the screen goes black and the machine locks / goes to sleep.
I ask to take a look at the laptop and log back in, open notepad and start banging away on the keyboard like a cracked out chimpanzee, nothing happens. I hit every key on the keyboard, double checked hotkey / function keys, can't get it to happen. Now I had written a small exe for our staff that locks their workstations when unplugging their security keys and killed it just to make sure that wasn't causing issues, nope that's not it.
Then they tell me they even swapped laptops and the problem follows this user... So I blow out the user profile and log them back in and let it sync again, issue still persists. Now I am watching them type like a hawk trying to see if I can spot what key sequence or wizardry is happening to cause this issue.
I can't find any reason that makes sense, when I check the event logs it just says "Machine sleep due to lid closure or button press". Happens on 2 different laptops so it's unlikely to be a faulty lid switch or loose / sticky button. I notice a bracelet on this persons wrist and joke about it not being a super magnet or something and she laughed and said no, but it does use a magnet to hold it together.
I didn't think much of it at the time and ventured back to my office defeated and confused. I am going over the issues with one of my co-workers and mention the machine sleep due to lid closure or button press, then something clicked. We grabbed a spare nametag with a magnet and started waving it around the same model laptop surfaces and sure as shit the screen goes black and it goes to sleep.
The magic spot on this laptop was to the right of the trackpad, the user wore the magnetic jewelry on their right wrist and when they arranged their hands just right it triggered the lid closure sensor which on these models uses a magnet in the lid.
In real life this issue persisted for a few weeks before we figured out what was going on, I thought I was going crazy though when it was happening.
"You waited a few DAYS?" I interrupted, a tremor tinging my voice. "And
you couldn't send email this whole time?"
"We could send email. Just not more than--"
"--500 miles, yes," I finished for him, "I got that. But why didn't
you call earlier?"
"Well, we hadn't collected enough data to be sure of what was going on
until just now." Right. This is the chairman of *statistics*. "Anyway,
I asked one of the geostatisticians to look into it--"
"Geostatisticians..."
"--yes, and she's produced a map showing the radius within which we can
send email to be slightly more than 500 miles.
Pure gold. I love that the stats department put in such rigorous testing before submitting the ticket.
Sometimes you just need to gather all the young'uns around the campfire and regale them with the tales of old, like The 500-Mile Email, or The Story of Mel.
I was doing IT support for a small Australian company back in '98. A guy called me from a remote office, and after a few pleasantries he explained that the screen saver had fallen off the monitor of his dumb terminal, bounced on one of the keys on his keyboard, and now terminal was locked up. He wanted to know what key to press to unlock the terminal.
Eh?
I knew the guy, and although he wasn't trained in IT, he knew his way around the basics, he wasn't completely clueless.
I asked him to explain the problem again as I wasn't sure I'd understood. He repeated exactly what he'd said the first time.
I replied "What do you mean the screen saver fell off the monitor, that's impossible? Besides, it's a dumb terminal, they don't have screen savers."
After a little more fumbling around this weird upside-down world he was presenting me with, it suddenly clicked. He was talking about the physical CRT anti-glare screen filter [0] that used to be common around then, that literally hung in front of the screen. This has come unstuck and hit the scroll lock on the terminal. He called this a screen saver.
Since then the phrase "Screen saver" seems to have now morphed to mean what I used to call a desktop wallpaper, but that's a separate topic.
[0] https://dylbs6e8mhm2w.cloudfront.net/productimages/500x500/E...