I also use Swift for large projects (many modules, lines of code, complex generics etc). Yes, Swift and Xcode are far from perfect. But the fluent language, and the tools that Xcode provides far outweigh any stability issues I've run into, especially recently.
There's some tools you can use to diagnose build times, often slow builds are just a line or two that seem to mess with the compiler and can easily be broken apart or written in a different way.
Also, have you tried the Android emulator/Android studio? Last time I did it made me appreciate Xcode and the Apple dev ecosystem a whole lot more.
I wonder if there's a cognitive anchoring effect. I've found Android Studio to be the best part of Android development, while Java and particularly Android itself being quite painful. XCode, on the other hand, I've found to be particularly painful, while iOS is much easier to work with, and Swift is one of the most pleasant languages I've used, so XCode looks bad in context, while AS looks much better.
There's some tools you can use to diagnose build times, often slow builds are just a line or two that seem to mess with the compiler and can easily be broken apart or written in a different way.
Also, have you tried the Android emulator/Android studio? Last time I did it made me appreciate Xcode and the Apple dev ecosystem a whole lot more.