At ${PREVIOUS_JOB}, I ran a stratum-1 NTP server that got a timecode and PPS signal from a GPS clock. I could tell if someone had left the door to the lab open by what PLL frequency "ntpdc -c kerninfo" reported on that server — the various oscillators in the server would drift above/below their ideal frequencies depending on how hot/cold they were, and the room would get a few degrees colder with the door open.
There was no need, really. The university's networking/telecommunications group ran their own pair of stratum-1 NTP servers, plus four stratum-2 NTP servers, so my stratum-1 wasn't really needed. I ran my stratum-1 NTP server simply because the hardware was available and I had an interest in it. You see, the GPS clock (and its predecessors, a WWVB clock and a pair of GOES clocks) were relics of a time long past...
[insert wavy flashback transition here]
The lab I mentioned was a seismological observatory. In the days before cheap, high-resolution A/D converters and computers with massive amounts of storage existed, almost everything was analog. Gloriously, unashamedly analog.
Seismograph stations sent data from their seismometers back to the lab over a "dry loop" — a leased line with no dial tone or voltage on it — from AT&T. To do this, the seismometer's output signal was greatly amplified, then frequency modulated onto a relatively low frequency carrier (1–2 kHz). The signal then traveled through AT&T's network all the way to the lab, where the signal was demodulated.
Okay, great, we're getting the signals back at the lab, but how do we store these waveforms? The answer is a giant drum, a motor, some paper, and a pen or stylus — a drum recorder.
"Helicorder" was Teledyne Geotech's brand name for their line of drum recorders, but it was so popular that "helicorder" has become a generic name for "drum recorder" in the seismological community, much like "Xerox" became synonymous with "photocopier." (If you ever look at earthquake records on-line, check for "heli" in the URL.)
A piece of paper is wrapped around the drum. A pen/stylus rests on the paper and deflects side-to-side depending on the polarity and magnitude of the input signal. A big positive voltage makes the pen move really far to one side, and a small negative voltage makes the pen move not so far to the other side. The pen is also attached to a threaded shaft that rotates, slowly moving the pen from one side of the paper to the other. The drum itself also rotates, and the rotation speed of the drum and the shaft was usually selectable — most people had recorders with small drums set to record ~24 hours of data per piece of paper, with wider drums set to record for a proportionately longer amount of time. (The higher the drum speed, the better the record quality, but then you had to change the paper more often.)
So we can record the signal onto paper, but we're missing a very important thing — time. We need to know exactly when stations saw ground motion in order to locate earthquakes and other seismic events. Enter the GPS clock (and the WWVB clock and GOES clocks before it). The GPS clock received a very accurate time signal and was configured to output a very simple timecode known as "slow code." Slow code works as such:
• At exactly the start of the 0'th second of every minute, generate a voltage pulse for some amount of time, usually 2 seconds.
• At exactly the start of the 0'th second of the 0'th minute of every hour, generate a voltage pulse for some longer amount of time, usually 4 seconds.
• At exactly the start of the 0'th second of the 0'th minute of the 0'th hour of every day, generate a voltage pulse for some even longer amount of time, usually 6 seconds.
This slow code would be added to the signal being recorded by the drum recorder, adding precisely timed "bumps" to the record. When the paper was changed on the drum every 24 hours, someone would write or stamp several pieces of information on the paper: the seismograph station's name, the date, and the time of the first time mark:
Note the column of time marks between the stamped dates. The narrow marks are minute marks, the slightly wider marks are hour marks, and the widest mark (five lines above the little earthquake) is the day mark.
[insert wavy flash-forward transition here]
Eventually, everything at the seismological observatory went digital, and the seismograph stations were upgraded with digitizers that had their own GPS clocks for timestamping data. The WWVB clock and GPS clock sat unused until I cleaned them up and reconfigured them to serve up time for ntpd to consume.
Quartz clocks can be inaccurate and can vary greatly with temperature. Beyond temperature-compensated crystal oscillators (TCXO) which are designed to be reasonably stable over a prescribed temperature range, there are oven-controlled crystal oscillators (OCXO)[1], that are held at a stable temperature in a small can with a heating element controlled by something like PID control. For anything requiring affordable stable timekeeping (this includes high data rate/quantization multimeters like the HP 3458A), an OCXO is a pretty good way to go. GPS is a fine reference to start one from and with specialized hardware GPS itself can be used as a MHz clock. For everything else there are TCXOs, and if you really don't care on-die RC oscillators can be used (this is the internal oscillator on an Atmega, for example, the default until you add an external crystal). If money is no object, atomic clocks are of course also an option. Past press mentioned that Google uses GPS-referenced atomic clocks to keep their data enter clocks in sync [2].
As an amusing aside, my kitchen oven clock has a poorly-chosen TCXO, such that it is not stable over the temperature range normally experienced by an over control panel. It's a great example of not designing to the application. Whenever I bake something, time accelerates on the oven clock. It's like a cookie-making time machine.
If you are interested in learning more about the hardware of crystal oscillators, this application note from HP is terrific:
The article mentions corrections beyond 500ppm. I worked on time synchronization (using PTP instead of NTP), and when we are talking about "cheap oscillators" we mean those specified at 50ppm (50us per second) which you find in every Ethernet card. In reality we measured +/- 8ppm. Yes, the drift changes over time, and you can see in the log when a door is opened in the morning. But NTP can only reach milliseconds precision anyway, so there is plenty of time to react. Whatever the problem is with those PCs mentioned in the article, using a temperature-controlled oscillator would be overkill.
> First of all, to be clear, I'm not talking about the hardware clock. That really only matters when you read from it, and you generally do that once: at boot.
That's absolute time that's read at boot, from a battery-backed real-time clock (RTC). Relative time can still drift. Crystal resonators with multipliers and PLL are used to supply the CPU with the signal distributed by its clock network. The crystal is a voltage-controlled oscillator (VCO), so it is tightly controlled (could be by voltage controlled by propagation delay within the chip, for example), but it can still drift over time and with temperature. As the article mentions, this can be responsible for system time error, "Your clock's pace will always vary a tiny little bit for different reasons". Operation of the computer shouldn't be impacted since things sync to the clock, but absolute timekeeping can be.
One additional thing that wasn't yet mentioned is that freqency depends also on gravitation, so orientation of your motherboard could matter (for the curious - https://www.youtube.com/watch?v=zILwgQhjC_Q ). I'm not really sure it's big enough to matter though.
You missed one: Your clock is not monotonically increasing at all times. That one doesn't have to happen often to totally screw you up when it matters. Monotonicity is a super important property of a clock.
Am I missing something? Wasn't this covered by Rachel in the following:
S5: your clock ticks fast, slow, normally, and everything else in-between, and changes constantly. Sometimes it gets more than 1 second per second, other times it gets less. Even then, the degree by which it runs too quickly or too slowly changes unpredictably.
Yeah. The ticks just determine when the system increments its time to the next fraction of a second.
There absolutely exist bugs that will hop the clock backwards in time. This does wonders for certain daemon's health and will invoke colorful language from anyone reading system logs.
Glad to see that people appreciate time, one of my favorite engineering challenges. We took it on at rapt.fm in order to have the beat and other events start in sync.
A distributed tick was used on game servers and in web clients via websockets. Max Seiden deserves credit for the work, it involved subscribing to a custom tick emitter on the server-side, and in each subscriber, an adjustment mechanism and local failover.
I did the javascript client work that ended up in the final release, basically a mechanism wherein events could be scheduled a few seconds ahead, and these would get triggered on the same tick. This way, even with the obvious latency in the video feed, at least countdowns and music cues happened simultaneously for everyone participating (well, when it wasn't buggy).
rapt.fm does not use this, but Precision Time Protocol is the ultimate way to make time sync work. I intend to get to know it better for future projects.
At work I implemented a sort of NTP-over-ajax to help guarantee browser/server time synchronization for displaying signals properly in realtime. I'd much rather use websockets, but for various reasons I'm holding off on doing that until that part of the codebase is cleaned up more.
It's proven invaluable especially with wireless devices and also people wanting to hold up two displays next to each other and not notice a difference.
The most straightforward thing is to just run the CPU with an accurate clock frequency. You'll have to replace the Quartz next to the clock-generation chip next to your CPU/Chipset by something which is very accurate.
For the cost, it's about $150 or so for a used GPS with precise 10 MHz reference frequency output, there's a different PLL IC (different from the ClockBlock module) from Silabs which is available from Adafruit for <$10, but I don't know if the xtal there can be exchanged for a 10 MHz frequency. The C-revision of the used IC can be, though... https://learn.adafruit.com/adafruit-si5351-clock-generator-b...
That's straightforward if you're a hardware hacker, it wouldn't slow me down much but I think that for most software oriented people building your own time-server and verifying that it works is out-of-scope.
GP probably meant to hook this straight into their PC and the CPU you refer to is actually the CPU of a separate board rather than the main CPU (where thermal drift will usually make the oscillator less than reliable over longer periods of time).
But I don't know what functionality they actually provide available over the pci-express port, if one is unlucky, it's only the serial port to the GPS receiver ;-).
It's quite easy to build an accurate ticker, that monotonically increases at a rate that is within a few PPM of a known frequency. This is basically the "add finest cheap Casio digital watch technology to motherboard" option. If you fill your datacenter with these, they will slowly drift apart from each other and from UTC. UTC is generally what you want but then you have to account for leap seconds.
There is also huge scope for argument about what is meant by "very accurate".
Reminded me of [0]. Whatever aspect of time you're concerned with, it's always a thorny issue, however easy it may look like at first. And we're not even talking about fleet of relativistically affected nodes such as with GPS!
