AFAIK you don't need to check historical timestamps (beyond sanity checks) during initial block download (IBD).
What you need to consider is whether an attacker can manipulate historical timestamps to get you to follow the wrong blockchain. The answer is, they can't.
The only thing manipulating historical timestamps would accomplish is allow the attacker to change the historical difficulty of their alternative blockchain. But that doesn't enable them to add additional proof of work. The amount of total proof of work they can add to an alternative blockchain is not dictated by difficulty, but merely by how much hashrate they have poured into the chain.
Since total proof of work for a chain is the metric by which the "correct" blockchain is established we can conclude that manipulating historical timestamps doesn't make it any easier for an attacker to get you to follow their chain.
That's for IBD. Obviously you need to verify timestamps after IBD, since timestamps are used to adjust difficulty and that's a consensus critical rule.
What you need to consider is whether an attacker can manipulate historical timestamps to get you to follow the wrong blockchain. The answer is, they can't.
The only thing manipulating historical timestamps would accomplish is allow the attacker to change the historical difficulty of their alternative blockchain. But that doesn't enable them to add additional proof of work. The amount of total proof of work they can add to an alternative blockchain is not dictated by difficulty, but merely by how much hashrate they have poured into the chain.
Since total proof of work for a chain is the metric by which the "correct" blockchain is established we can conclude that manipulating historical timestamps doesn't make it any easier for an attacker to get you to follow their chain.
That's for IBD. Obviously you need to verify timestamps after IBD, since timestamps are used to adjust difficulty and that's a consensus critical rule.