You need to fork at low difficulty if you want to significantly lengthen the chain from that point, because creating a high difficulty, long chain that is valid is hard.
But-- there's nothing to preclude you making big steps up in difficulty at the end of the chain. It means that one evaluating the length of the chain for authenticity really needs to integrate the difficulty over the entire chain and not just look at the number of blocks.
Suppose I'm a new node and want to verify the blockchain. How do I verify that each block was mined with the correct difficulty?
I'd need some record about the actual real-world timestamps for each block. Then I could say something like "duration between block x and block x+1 was > 10 min, so the down-adjustment in block x+5 is justified".
But if those timestamps were stored on-chain, an attacker could simply lie about them and keep difficulty artificially low on its alternative chain.
On the other hand, if we had some un-forgeable record of block timestamps, wouldn't this solve the double-spend problem all on its own? Would we even need PoW at this point?
But-- there's nothing to preclude you making big steps up in difficulty at the end of the chain. It means that one evaluating the length of the chain for authenticity really needs to integrate the difficulty over the entire chain and not just look at the number of blocks.