I have a theory that simple sets can become difficult to understand not because of anything in their nature, but because of how they are desired to be consumed.
In other words, if you have a simple to understand thing, then it might become hard to understand if it needs to be fed into something that is itself hard to understand. (Or even if the way in which you feed it into the other thing is hard to understand).
Additionally, if you have something which is easy to understand that is fed into multiple incompatible things (either the things are easy to understand or hard to understand), then the whole system becomes harder to understand.
Finally, if the starting point that is fed into all other things is not actually easy to understand but in fact hard to understand ... things can get pretty bad.
Even if time was easy to understand by itself, then I still think it would be pretty hard to deal with because it is used for multiple incompatible domains (astronomical vs how long a day is on earth for example).
Yes, the use-case and domain set the constraints. Using the wrong measurement type or abstraction either increases the complexity of the exceptions necessary to make it fit the constraints or increases the inaccuracy/errors... or both.
With regards to time specifically, I recall many years ago talking with an economics professor I had that was from Zimbabwe and he mentioned the frustration that western industrial nations had when attempting to establish business in Africa. Industrial nations long ago became dependent upon a higher resolution and accuracy of time and so when they say "2PM" they expect accuracy within a minute or two. But the African work force would often show up at 3PM and when confronted by this "lateness" they would reply "3 is the friend of 2." An agrarian or pre-industrial society has no real need for accuracy to the minute, but instead can operate quite well with human observation of the position of the sun. This impedance mismatch caused no end of frustration on both sides, since neither could really fully understand the other's conception of timeliness.
In other words, if you have a simple to understand thing, then it might become hard to understand if it needs to be fed into something that is itself hard to understand. (Or even if the way in which you feed it into the other thing is hard to understand).
Additionally, if you have something which is easy to understand that is fed into multiple incompatible things (either the things are easy to understand or hard to understand), then the whole system becomes harder to understand.
Finally, if the starting point that is fed into all other things is not actually easy to understand but in fact hard to understand ... things can get pretty bad.
Even if time was easy to understand by itself, then I still think it would be pretty hard to deal with because it is used for multiple incompatible domains (astronomical vs how long a day is on earth for example).