Can't speak for OP, but I started learning algebra at 5. It happened that I really liked playing with calculators at age 2-3, and that helped give me an extremely solid mental arithmetic foundation.
Essentially, the special thing about me probably wasn't some super-human neural circuitry for understanding math. It was the fact that I was wired, somehow, to enjoy playing with a calculator. I had more arithmetic practice by age 4 than most kids have by age 11.
I feel like I can relate to this (a little). I'm no prodigy, but even as a young kid (6 or 7) I would write down algebra problems all day and solve them. I did a ton of mental math, too, writing down several dozen 2-digit numbers and adding them all in my head.
Haven't really thought about that in a long time. Playing with a calculator also got me into programming (TI-Basic on the TI-83 in middle school).
Essentially, the special thing about me probably wasn't some super-human neural circuitry for understanding math. It was the fact that I was wired, somehow, to enjoy playing with a calculator. I had more arithmetic practice by age 4 than most kids have by age 11.