The problem I have with subtitles is the same reason I could never enjoy comics / graphic novels. I can either read and process the text, or look at / process the action (pictures). For me it is the same thing as trying to pay attention to two conference calls at the same time with a phone in each ear (is it just me that can't do this?). And it isn't just the mental processing, the rest of the screen sort of disappears from my perception when reading subtitles.
Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).
This is really interesting, as I've not heard many people say this. I wonder how common your experience is. Perhaps some people have a more narrow field of vision, or I wonder what else might cause it.
I can only say that I absolutely prefer subtitles 100% of the time because the original performance is usually better. Good voice actors are not cheap, besides the fact that the sounds don't match the lips when it's a different language speaking.
Maybe it's better if you grew up doing it. I grew up watching Chinese action films with subtitles, and it's pretty easy for me to watch the screen like normal and read the subtitles in my peripheral vision without looking directly at them at all.
I figure most people could probably do it with enough practice, though.
I think this is why people mostly talk about watching foreign-language animated works and soap operas/dramas: dramas will have either action or talking heads, but not usually both; and animation can be paced stylistically, so that each shot lasts long enough "by design" to read the subtitles, even when it would be over in a split second in reality.
But I can't imagine watching, say, a foreign live-action sketch comedy show subtitled. I'd have exactly the bad experience you're describing.
At least in Japan's case, I highly doubt the pace is «paced stylistically, so that each shot lasts long enough "by design" to read the subtitles».
If anything, they are usually at the other end of the spectrum, in that most local releases don't even include japanese subtitles (To be frank, I'm quite surprised this is even legal, considering the impact on deaf people)
So more than for artistic reasons, I think it's more often in order to lower the number of quality key frames required, so budget reasons.
I also can think of several shows that achieved significant fanbase in contexts where keeping an eye off the subtitles can be hard, while still being appreciated for their visuals: Tatami Galaxy, Monogatari Series...
Even without the subtitles, it's a stylistic choice in the way comic panels with dialog are screen-written / storyboarded / directed / performed by voice actors.
"Screen-original" films and TV series are written in more a "top-down" style, with scenes being planned for their pacing, and the dialog in a scene liable to get cut down or rewritten to keep said pacing.
"Adapted from a non-visual source medium" films and TV series also almost always get their scenes cut down, just by necessity (a novel won't fit in 1.5hrs.) Though only "almost", because sometimes a director sets out to adapt a work into a really long treatment that can fit all the source material and then some — like the Lord of the Rings movies, or Game of Thrones. But even then, the director is still someone who is immersed in the top-down, "we get to knit it together how we want" zeitgeist of regular Hollywood film&TV production.
"Adapted from a visual source medium" films and TV series, though, are — if not heavily budgeted for adaptation polish — written mostly "bottom-up", starting with a treatment and storyboard that keep every line and shot from the source, and then letting pacing slip to keep in all the stuff that existing fans of the source material expect to see (and because the producer wants as many episodes as possible, so why not stretch one chapter out into five episodes if you can?)
This can lead to some extremely long shots, where there was dialog in a comic-book panel that was clearly not originally written with the intent of having it performed while holding/panning on the exact panel shown; but where a lazy "reading" of the source by the cinematographer leads to exactly that.
You can also call this the "make a radio play / audio drama out of the book; lock it down; and then add visuals to it" adaptation strategy.
(This is also common, as it happens, to specific genres that started out in the radio era. True crime, whether it be on TV or on YouTube, mostly consists of podcasts with the visuals almost being an afterthought. This is why so many people put it on in the background: the way it's done, you don't have to look at it to follow what's going on.)
I’m the same. Reading subtitles takes my attention almost entirely away from the scene, and after switching back and forth between subtitles and the scenes rapidly a few times, all sense of immersion is lost. It’s a distraction and chore.
Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).