Planar graphics allowed some interesting effects, but Doom’s shortcuts for perspective calculation weren’t compatible.
When VGA came out, it became difficult to see a future for the Amiga - the hardware was just too tied to TV frequencies to match PCs in graphics, just because they didn’t have to be compatible with anything else.