Hey guys, I'm a fellow developer of distributed systems here.
First of all I think what you are doing is great.
My question is what's the point of clocks at all? The current time is a very subjective matter and I'm sure you know this, the only real time is at the point when the cluster receives the request to commit. Anything else should be considered hearsay.
Specifically the time source of any client is totally meaningless since as you say further in the discussion that client machine times can be off by huge margins.
If you accept that then one has to accept the fact that individual machines within the cluster itself are prone to drift too, although one can attempt to correct for that I appreciate.
Wouldn't you think though that what is more important is that the order is more based on the bucketed time of arrival (with respect to the cluster).
I don't see how given network delays anyone can be totally sure A is prior to B, atomic clocks or not.
What is important is first to commit.
[edit] Yes would love to talk privately about this topic @irfansharif
When a single system is receiving messages, you pick an observed order of events that meets some definition of fairness, and you stick with it all the way through a transaction. By pretending A happens before B (even if you're not entirely sure) you can return a self-consistent result. And once you have that you can simplify a lot of engineering and make a lot of optimizations, so that the requests aren't just reliable but also timely.
You throw three more observers in and how do you make sure that all of them observe the requests arriving in the same order? Not even the hardware can guarantee that packets arrive at 4 places in the same order, even if the hardware is arranged in a symmetrical fashion (which takes half the fun out of a clustered solution).
> Specifically the time source of any client is totally meaningless since as you say further in the discussion that client machine times can be off by huge margins.
distributed systems like cockroach shouldn't use the client's conception of current time for anything at all, except possibly to store it (_verbatim_, don't interpret it) and relay it back to the client or to other clients (and let the client interpret it however they want).
Hmm, I'm not sure I completely understand your question or your source of confusion here but unless I'm grossly misunderstanding what you're stating I think we might be conflating a couple of different subjects here. I'm happy to discuss this further over e-mail (up on my profile now) to clear up any doubts on the matter (to the best of my limited knowledge).
Why not simply have the cluster sync a time between themselves? First node in the cluster gets the time, and as the new nodes come online they set their own internal time via the cluster? So in a world where there is not NTP or atomic clocks the system could continue to operate.
This doesn't take into account when clocks on different systems run at different speeds, or when clocks jump, especially on VMs and cloud instances, which happens all the time.
[1]: https://www.cockroachlabs.com/blog/living-without-atomic-clo...