You can argue that Time is an illusion. You can argue that our intelligence allowed us to break free from the harsh laws of nature and evolution too soon, so we never acquired the necessary sensors and abilities. Maybe if we had to compete against multiple equally intelligent species, we would have acquired them. However, I think that using a "vision of Time as it really is" sensor correctly would require a far more complex and energy-hungry brain, so it's probably not "worth it" from an evolutionary perspective. Or maybe it's out of reach of carbon-based organisms.

However Mathematics are different. It's not a perception. It even allowed us to discover things we could not perceive, and explore things we cannot perceive. It is independent from us and the universe. And if you still want to throw away Mathematics, you'll also have to throw away logic, which is kind of an issue. The only thing one could seriously argue, is that something better than Mathematics exists, like General Relatively is "better" than Newtonian mechanics. But maybe that is out of our reach, too.

Meanwhile, repeat after me: I believe in modus ponens, set theory and monads.