We know - how do we know? From experiments. And did we have to assume things about time in order to make experiments? You bet we did. We wouldn't get very far with experiments if we didn't take as axiomatic that causes precede their effect in time, for instance.

I haven't got into the depths of Bergson's argument, but I think I see the point in the headline that time belongs in philosophy, and I can easily imagine he was right that Einstein was smuggling in philosophical assumptions in the particular choice of words he used to describe the implications of his findings.