I successfully removed it years ago because Typescript replaced everything I was using Babel for, things even build way faster and I even got Intellisense autocompletion and refactoring as bonus.
I doubt typescript is faster than babel with all the static analysis it does. In fact it's typical for projects to use typescript as a check in CI and actually build with babel, for speed. Fast alternatives to babel are things like esbuild and swc