Yah I mean that's the rub with SGD... you need to spend a non-trivial compute budget on hyperparam tuning, which sometimes beats Adam.

Adam, on the other hand, generally gets you pretty good results without futzing too much with hyper params.