Calculus isn't that complicated, at least not what's done in backprop.

How do you propose calculating the "general direction" ?

And, an example "advanced optimizer" - AdamW - absolutely uses gradients. It just does more, but not less.