I feel like what we should be working towards is an integrated "problem-solving"-based curriculum. The test-based stuff has us stuck in "memorize these things" mode.
For most kids, math is way more interesting if you can move past generic formulas and into real-world use cases. I remember continually asking math teachers "And what would I use this for?" and never getting an answer which left me super-frustrated and uninterested.
Instead of breaking them out into different "tracks", integrating math, logic, and CompSci into existing courses might produce some interesting results.
It's just like you said. Math frustrates the hell out of most people, and probably makes them feels stupid.
The education system in North America is just not ready for this.