While we all (HN audience) know roughly what kind of things are going on, seeing it in all the delicate details is fascinating.
Just thinking that all these bytes and everything that's happening have 6 more layers below all the way to physical electrical/electromagnetic flow, all routed potentially through many different ISP backbones and possibly international routes, all happening in roughly less than 1/10 of a second is even more fascinating.
It reminds me of being a teenager in the late 90s and the early days of the internet. I discovered entirely on my own that I could telnet to port 80, type "GET / HTTP/1.1\n\n" and the server would send me the headers + page content. Shortly after I discovered the same worked for SMTP.
I was very far from the first person to have this revelation but it was definitely an eye-opening "there is no magic, it's all just software" moment for me. It fundamentally changed the way I think about computers at every level and inspired me to look "under the hood" at everything... how CPUs implement superscalar OOO execution. How atomic operations work at the CPU level. What a syscall actually is. How subroutines are called (and calling conventions). How dynamic linkers work.
You don't have to be an expert at all these things but it is like a superpower to understand the basics of every layer.
> It fundamentally changed the way I think about computers at every level
I had the exact same revelation also in the late 90s! (~1998 for me). I was already telnetting into servers a bunch and was getting into running an Apache server. I remember the moment I typed "GET / HTTP/1.1" into port 80 so clearly because it suddenly turned the "web" and "HTTP" into something comprehensible with composable parts I could understand at a deeper level.
In our current world of SSH and HTTPS, it seems less likely the new generation will have the same experience. But we also have browser developer tools nowadays which make it so much easier and motivating for someone to start to learn the about the web and JavaScript. In the 1990s I had to use proxomitron to painstakingly modify a webpage's JavaScript or HTML, but these days it's dead simple.
`openssl s_client` is underused, and it's not as clunky since they added support for host:port syntax over separate args. Encouraging thinking of that as the modern replacement would help.
Wow thanks for showing me about `openssl s_client`. I just have this a try and you're it was quite easy!
openssl s_client -connect example.org:443
[TLS details...]
GET / HTTP/1.1
Host: www.example.org
<newline>
<newline>
[headers and HTML response!]
The one thing tricker nowadays is almost all of the time you need to send the `Host:` header for things to work. Took me a sec to realize that since in 1998 it was almost never necessary.
Glad to spread the knowledge, and glad you gave it a shot! It really is that easy, and host headers have been pretty regularly required anyway.
Slightly more interesting is using it to access internal sites, and setting up your own TLS roots and chains for personal or corporate infrastructure. In practice, while useful for internal use, I generally recommend everyone use LetsEncrypt and public names for even internal APIs when they cross team boundaries, because it's just easier.
> In our current world of SSH and HTTPS, it seems less likely the new generation will have the same experience.
Pushing encryption into the transport layer a la QUIC could solve this, if not for the spurious dependency on user-hostile TLS instead of a simpler PKI. SSH would become telnet over a QUIC stream, which could be used with a QUIC-enabled netcat (say). HTTP/3 could have been either 1.1 or 2, just over QUIC, but this wasn't pursued.
I think this is one of the most fundamental things a person can learn about software engineering: there is no magic. If something happens, it was part of the operation of written code and an exchange of data somewhere.
My daughter is starting to get to an age where she’s inquisitive about how magical things work, and I usually respond by asking “how _could_ it work?” And we talk a lot about what actually does what.
Being even somewhat conversant in some of the more popular protocols is also like a superpower that the newbs don't really have, too. The ability to use telnet or nc to answer the question "aside from any other piece at any level in the stack that could be going wrong, can I even talk to the HTTP server" helps you eliminate a lot of possibilities about what's going on when troubleshooting something.
Same! I didn't even know anything about telnet or programming. I was using Klik&Play or MultiMedia Fusion to make games, and some component supported TCP, so I opened a port using that and just for fun connected to it with a browser. Then I saw a request, so I used that the other way around on a real web server and it worked. Same thing with SMTP.
That's a program I haven't heard of in a long time. I wonder if you can still get it running in a VM since it was just shareware with a nag screen if you said you were using it for "educational purposes"
ive only regarded it as a literal system call, the lowest possible level, “language agnostic” api that does a thing in the OS. do you have some deeper insight?
At the most basic level a system call is: loading the arguments into the ABI-specified registers then triggering an interrupt. Some architectures have a specific syscall or syscall-like instruction that is more optimized than a generic software interrupt but conceptually it is similar.
The syscall/interrupt instruction transitions to supervisor/kernel mode and moves the execution pointer to the configured location in the kernel.
If this sounds kinda like switching threads or processes you would be right. But if you had to pay that context switching cost 2x on every syscall it would kill performance. Most OSes use a split address space as an optimization here: every userspace process has the kernel's memory mapped in the upper half, but with protection bits that make it inaccessible to userspace. That is so when a syscall is issued there is no need to change the active page table entries or flush the TLB: the kernel is already mapped only now in supervisor mode those kernel pages are accessible.
The CPU decided what code got control by the interrupt table which itself can only be configured in supervisor mode. That is what prevents a userspace process from hijacking the CPU. User mode code doesn't have permission to modify the register that points at or the memory containing the interrupt handler tables. Thus by definition any syscall/interrupt will jump to kernel code.
The kernel entrypoint then often has a COPYIN/COPYOUT process that will treat certain register values as pointers and copy the data into the kernel's address space when required (or copy it out to a caller provided buffer).
For reference pre-emptive multitasking is related. The kernel's scheduler configures a hardware timer interrupt. The configuration of this timer can only be done in supervisor mode. So once the current thread's timeslice is up the timer fires and the CPU changes the instruction pointer to the kernel's configured timer interrupt handler. User mode code can't prevent the timer from firing nor change what code the CPU will jump to. The scheduler routine saves the current context to memory, loads the next thread's context (registers, instruction pointer, page tables, etc), updates the timer's next deadline, then "returns" from the interrupt... only the instruction pointer is now in a different thread (or different process with different memory entirely) so the CPU "returns" to a different piece of code. If all goes correctly it "returns" to the next instruction beyond the one that completed when that thread last got pre-empted so from that thread's POV execution was continuous.
The difference between a syscall and a library function call is that a syscall crosses protection boundaries. Implementations differ, but where a library (even the lowest-level OS library like libc) runs in the context of the application and can be invoked with a regular "store pointer and jump" method call, a syscall usually involves transfering control to the kernel through a software interrupt.
I was lazier and just did "GET / HTTP/1.0\n" and saved one character :P
Edit: I am probably wrong about "1.0", might have been that I just did "GET /" and saved 8+ characters. I was just trying to make a funny remark about "Single line request" vs "Multi-line request"
The craziest thing is, if you’re using Ethernet, there’s a very high chance you have an actual physical data connection with the other computer on the other side of the world. A giant physical web blanketing the Earth.
Well, that's not new with computers or the internet. We had that ever since the analog telephone system. Even telegraphs before that for a smaller set of points and routes.
Really no. The RF world has nothing on fiber and copper for density.
There is almost unlimited desire for bandwidth in the access layer and over subscription ratios are still very very high.
The transition to 800G ports and then post-800G is not even warmed up yet. You can run the entire country of Australia on a backbone of two dozen routers today because of the lack of endpoint access bandwidth.
We are at a place where bandwidth is once again plentiful in the core … not so much on the edge. Sort of like 2003 when oc768 created that situation.
In what sense? What's a physical data connection? There's a shit ton of routers and switches in between, it's not like there's an electrical connection.
You could run your finger across from my keyboard to yours, without a single break in the continuous physical connection between them. Connector clips included, of course. ;)
I understand that you are being poetic, but just in case someone reads this as fact: you are describing a dedicated circuit - which is what telephones used. The internet works on packet switching, so there are numerous little breaks between the signal and receiver as your data is routed along a "connection".
No, I'm talking about the physical layer of the OSI model, and including the mechanical connections between those physical interfaces. You're talking about the link layer.
Unless your backbone/computer has a wireless hop, a literal, uninterrupted, physical chain of physical electronic devices, physically connected to one another with wires/cables, goes from my keyboard to yours. This is literal, not poetic. I'm not saying a galvanic connection. I'm saying a physical connection where, if nothing was bolted down, and high tension cables were used, I could pull and you would feel it.
AT&T had wireless microwave towers for phones and tv, so I imagine there was a period near the end if its life where some dial-up connections weren't physically connected:
Working for a Midwest dialup isp in the early 2000s, we definitely served some of our smaller POPs with PTP wireless backbones, thanks in part to vast expanses of flat land with fairly tall structures dotted throughout.
Yes, and if the comment implied a purely electrical connection, it is likely not the case either, as there is electrical to optical and vice versa transitions throughout.
I don't know, I alternate between thinking that's remarkable and not so much. I could run that connection via the power grid too, or the water supply (if we were in the same city).
Edit: I concede it's actually pretty remarkable. It is like the nervous system of the planet. Sure there's not a purely electrical continuity, but neuron synapses don't have an electrical nature either.
Are you sure? As far as I'm aware a single city might have its electricity network split in a few macro areas, and not really connected to each other. The Internet, on the other hand, needs all nodes to be connected somehow (and I bet the vast majority of connections are physical).
This is true but on a much larger scale than the city. For example, the United States has only three electrical grids (East, West, and Texas). Within those areas, not only are all electricity users in some sense connected, they are all /synchronous/, meaning that the 60hz coming out of your wall peaks and troughs at the same exact time as the 60hz at your cousin's house two states away.
The grids also have interconnections with one another, but they happen via DC to avoid needing to synchronize the whole continent.
In a literal sense, modernity is about using energy to coordinate human activity across vast distances at minute time scales. We normally think about this in terms of transportation, telegraphy, telephony, print, broadcasting, and the internet but it's also true of the motors in our washing machines and factories.
Assuming artificial superintelligence is possible, imagine how would its "experience" be in contrast to ours - it would be orders of magnitude faster in both thinking and execution. We would be like some weird plants to them, that take a year just to move from one location to another.
Just thinking that all these bytes and everything that's happening have 6 more layers below all the way to physical electrical/electromagnetic flow, all routed potentially through many different ISP backbones and possibly international routes, all happening in roughly less than 1/10 of a second is even more fascinating.