Yes, Ed25519/Curve25519 has weird cryptographic choices that make it difficult to do certain things. I wouldn't use it for anything other than the specific use case it was designed for: key agreement and single signature authorization/verification.
And yet people have used it for ring signature authorization schemes (CryptoNote/Monero) which has directly led to inflation bugs (permitted double-spends) and an inability to have key derivation for secure long lived wallets.
That’s due to Curve25519 having a non-unit co factor. There’s a half dozen other weird properties of the curve chosen for smal efficiency gains or whatnot which also might lead to security holes in any application which is not the single signer or 2 party key agreement use case.
