Crypto software also has the property that "internal implementation details matter". Nearly all other software does not have this property - you don't care how your hardware/software multiplies two numbers; only that it does it correctly in a reasonable amount of time. That property does not hold for crypto software - the nature of your multiplication may actually leak information, depending on how it is implemented. So in a way crypto software goes against the entire edifice of modern software engineering, which says that you can build software against an abstract lower-level "stack" and generally not care how the lower parts work.
Yeah, that's a great point, security requires that you understand the insides of any dependencies, and treating any part as a black box is a bad idea.
I think performance critical code sometimes shares this particular side of engineering and it's inability to fit into the easy patterns and abstractions we usually prefer.
