This is very similar to the functionality provided by tlsdate (https://github.com/ioerror/tlsdate). They appear to have eschewed tlsdate's default approach of using the timestamp from the handshake in favor of using the `Date:` field, which tlsdate also supports. It would be interesting to see whether the randomization of TLS timestamps in modern implementations of TLS might mean that tlsdate's default mode is no longer useful. Either way, it's really cool to see this sort of functionality being included in ntpd by default!
openntpd has been nothing but trouble for me, but when I switched to djb clockspeed instead, it made things better. Here's a script that runs on GFiber devices, which uses tlsdate securely for the initial timewarp, and djb clockspeed thereafter. Since switching to this we have had extremely accurate timekeeping.
Which version were you using? If you were using the portable version, I heard openntpd-portable wasn't updated for quite a while, and fell behind...missing out on some really big improvements from more recent versions.
The portable tree has apparently recently been picked up again by a new maintainer.
If you don't mind elaborating a bit more on the troubles you had, I might be able to help get things fixed for a later release.
I have had some trouble on Solaris getting the adjtime olddelta value to settle quickly, but haven't heard of any other issues. Even if you're happy with clockspeed, it might help other users to identify the problem.