It's not just about time-to-sleep, it's about sleep quality.
My only suggestion would be to fully go off of caffeine and rate your sleep quality off of caffeine.
I went years of drinking caffeine. So much so, that I had forgotten what deep sleep really felt like until finally removing caffeine from my daily routine.
It's a massive difference in sleep quality. With caffeine, the sleep is superficial. I get up to urinate a lot more. Lighter sounds can wake me up.
Without caffeine, I find myself sleeping so deeply, that my body stays in the same position, and I can wake up to tension in certain areas because I haven't moved for hours. The deep sleep feels amazing.
How old are you? Caffeine has affected me much more strongly as I’ve aged to the point where I just do freshly ground decaf now. Some of the methods to remove caffeine don’t completely ruin the flavour these days so if you love coffee it can be worth trying…
It could be that it's more complex than a toggle on/off, and more like a polygenic trait.