Haha, this is how I feel too. I had the mistaken assumption in middle school that "if this is all they're teaching me, there must not be that much to know". Then when I got to college I realized that everything taught at the undergraduate level and below constitutes a negligible fraction of everything that has been discovered.

Instead of stretching mundane material that could have been taught in a week or two (calculus, etc.) over a period of years, I would have much preferred taking all of my engineering courses then. That would have left plenty of time in undergrad to learn topics as diverse as representation theory, semidefinite programming, quantum information theory, lambda calculus, etc. Instead, I have to learn these things on my own as a grad student in an ad-hoc fashion. Occasionally, courses on these topics become available, but they pop up at random semesters and you can't take all of them.

Seems like you did the best thing you could have, in the circumstances. When I was bored in class, I didn't study the textbook, I read science fiction novels.