This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of.
Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wonderful to have an 'autodub' option! But the times where one would use it seem quite narrow.
In my country (Sweden) subtitles are the norm for imported content, with most of that content being in English.
I believe that reading the Swedish subtitles and hearing the English words simultaneously have played a large role in helping Swedes to learn English.
However, it does help a lot that the grammar of Swedish and English is relatively similar.
I have been trying to use watching anime to get better at Japanese, and the different word-order and clause-order in the spoken Japanese to the subtitles have often tripped me up — especially when longer sentences have got broken up into multiple subtitles.
> In my country (Sweden) subtitles are the norm for imported content, with most of that content being in English.
That was the norm as I was growing up in Israel, with the exception being children's programming (usually animation) that was dubbed. The dubbing was usually quite good, and done by actors prominent in children's programming.
OTOH, being bilingual myself, the poor translation of subtitles was constantly driving me up the wall. I just couldn't ignore the mismatch between the dialog in English and the subtitles in Hebrew, but the rest of my family wouldn't agree to let me cover the bottom fifth of the screen.
Semantically, the translations were generally accurate except in infrequent cases where the translator obviously misheard something or translated a phrase overly literally (more often with non-RP British or Australian English than American), but stylistically, the translation was always bland and "correct" (eg. "sure" and "yeah" and "you bet" would all inevitably be translated as "yes").
I am very surprised more people watched the dubbed version of Squid Game over the subtitles. I saw a friend watching the dubbed version and to me it didn't "preserve the original performances" as you say. Surprised subtitles are a barrier, I enjoy them personally, but I guess they have the data to know.
I have tested out services that translate your "own voice" to another language and it was pretty cool, although I couldn't tell if the translation was quality. Automated translation isn't a solved problem, and this seems more of an issue than the tone of voice.
When I've moved to New Zealand, I was taken aback at how many people just could not stand subtitles. This and your mention about 'barrier' reminded me of Bong Joon Ho (director of Parasite) calling this out: "Once you overcome the 1-inch-tall barrier of subtitles, you will be introduced to so many more amazing films."
The problem I have with subtitles is the same reason I could never enjoy comics / graphic novels. I can either read and process the text, or look at / process the action (pictures). For me it is the same thing as trying to pay attention to two conference calls at the same time with a phone in each ear (is it just me that can't do this?). And it isn't just the mental processing, the rest of the screen sort of disappears from my perception when reading subtitles.
Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).
This is really interesting, as I've not heard many people say this. I wonder how common your experience is. Perhaps some people have a more narrow field of vision, or I wonder what else might cause it.
I can only say that I absolutely prefer subtitles 100% of the time because the original performance is usually better. Good voice actors are not cheap, besides the fact that the sounds don't match the lips when it's a different language speaking.
Maybe it's better if you grew up doing it. I grew up watching Chinese action films with subtitles, and it's pretty easy for me to watch the screen like normal and read the subtitles in my peripheral vision without looking directly at them at all.
I figure most people could probably do it with enough practice, though.
I think this is why people mostly talk about watching foreign-language animated works and soap operas/dramas: dramas will have either action or talking heads, but not usually both; and animation can be paced stylistically, so that each shot lasts long enough "by design" to read the subtitles, even when it would be over in a split second in reality.
But I can't imagine watching, say, a foreign live-action sketch comedy show subtitled. I'd have exactly the bad experience you're describing.
At least in Japan's case, I highly doubt the pace is «paced stylistically, so that each shot lasts long enough "by design" to read the subtitles».
If anything, they are usually at the other end of the spectrum, in that most local releases don't even include japanese subtitles (To be frank, I'm quite surprised this is even legal, considering the impact on deaf people)
So more than for artistic reasons, I think it's more often in order to lower the number of quality key frames required, so budget reasons.
I also can think of several shows that achieved significant fanbase in contexts where keeping an eye off the subtitles can be hard, while still being appreciated for their visuals: Tatami Galaxy, Monogatari Series...
Even without the subtitles, it's a stylistic choice in the way comic panels with dialog are screen-written / storyboarded / directed / performed by voice actors.
"Screen-original" films and TV series are written in more a "top-down" style, with scenes being planned for their pacing, and the dialog in a scene liable to get cut down or rewritten to keep said pacing.
"Adapted from a non-visual source medium" films and TV series also almost always get their scenes cut down, just by necessity (a novel won't fit in 1.5hrs.) Though only "almost", because sometimes a director sets out to adapt a work into a really long treatment that can fit all the source material and then some — like the Lord of the Rings movies, or Game of Thrones. But even then, the director is still someone who is immersed in the top-down, "we get to knit it together how we want" zeitgeist of regular Hollywood film&TV production.
"Adapted from a visual source medium" films and TV series, though, are — if not heavily budgeted for adaptation polish — written mostly "bottom-up", starting with a treatment and storyboard that keep every line and shot from the source, and then letting pacing slip to keep in all the stuff that existing fans of the source material expect to see (and because the producer wants as many episodes as possible, so why not stretch one chapter out into five episodes if you can?)
This can lead to some extremely long shots, where there was dialog in a comic-book panel that was clearly not originally written with the intent of having it performed while holding/panning on the exact panel shown; but where a lazy "reading" of the source by the cinematographer leads to exactly that.
You can also call this the "make a radio play / audio drama out of the book; lock it down; and then add visuals to it" adaptation strategy.
(This is also common, as it happens, to specific genres that started out in the radio era. True crime, whether it be on TV or on YouTube, mostly consists of podcasts with the visuals almost being an afterthought. This is why so many people put it on in the background: the way it's done, you don't have to look at it to follow what's going on.)
I’m the same. Reading subtitles takes my attention almost entirely away from the scene, and after switching back and forth between subtitles and the scenes rapidly a few times, all sense of immersion is lost. It’s a distraction and chore.
I started watching Squid Game with my brothers' girlfriend and gently suggested that we should use subtitles and the original audio. It took a couple of episodes, but some of those dubbing performances are pretty rough and when she did decide to switch she immediately agreed how much better it was.
The only downside is you have to actually _watch_ the show rather that staring at your phone most of the time...
I tried both seasons of Neflix's Ragnarok in original (Norwegian) and dubbed. The first season had a decent dubbing (I still kept the original+subtitles) but the second season the dubbing was simply unbearable, bad voice acting, echo chambers, incredible... So the experience can be very mixed even within the same show.
I watched Squid Game with subtitles and I... kinda agree but I'm pretty certain a lot of scenes have ADR, so even the "original performances" aren't the original performances, but people re-reading lines in a recording studio.
I mean there's some undeniable fact that the original text is the original thing, and low-quality translations often just remove entire parts of sentences or details just because it's "annoying", and you are missing out on lots of fun little details. Not super relevant in paint-by-numbers TV shows but for shows with a lot of flourish in the dialog...
Knowing what was intended to be said is great, and subtitles, for whatever reason, seem to convey this better than dubbing. Also, I feel like I can hear the emotion in the original actors' voices better with subtitles and no dubbing, despite being incapable of understanding the language. In that way, I agree it preserves the original performances.
However, moving pictures will always be an artform that are heavily reliant upon the picture part. With subtitles, I lose my ability to focus on minor but important details in film. During dialogue, this includes subtle facial expressions. I also pay less attention to all other visuals, which I find difficult to do without subtitles off entirely. I feel like I miss out on a lot of potential beauty in cinematography with subtitles enabled. (Although, to be fair, there is usually less dialogue in parts that emphasize on cinematography / setting.)
Anyway, just throwing in my two cents. I used to think people who disliked subtitles were expressing some broader dislike of foreign films, so I found their opinion perplexing or even stupid. But I've come to that side of the aisle myself, not because I dislike foreign films, but because I realize that reading a small bar of text at the bottom of the screen will always handicap my ability to take in the original experience as intended by the director of the film (in 99.9% of cases).
For me, reading the subtitles takes away from the experience much more than any kind of loss in authenticity does. Keeping my eyes on the text takes away from the visual.
I wonder if that’s something you could get better at with practice? When I watch subtitled movies or shows I watch the show and read the text with my peripheral vision. But I watch a lot of foreign media, so I may have gotten used to doing that. I can say for sure though that my eyes are not focused on the text while watching.
Absolutely! I grew up in a country with fairly high rate of literacy, which derived subtitles usage almost 100%, and reading them while enjoying the original voices behave second nature, and what's never fully translatable led to learning foreign languages specifics. Even today, so many years later, I like my movies with subtitles, and even when spoken language is same with the subtitles one.
Most people are fine with them. My dad has hearing problems so we always watched movies with the old fashion “white on black bar” closed captioning. As an adult I just get distracted by them and it prevents me from watching the movie.
Heck, they're not even limited to foreign languages. I put them on in English because I'm starting to lose my hearing now in my late 40's and if I turn up the volume loud enough that I can hear the dialogue ok, the sound effects would be so loud they'd set off the neighbor's car alarm.
And I refuse to watch movies/series without English subtitles, even if the original language is English. I too often miss something the actors say and get majorly annoyed. Could well be something about my hearing.
It's easy to tell which transcription services have access to the original script as a reference and which don't. I have subs on most of the time to catch subtle or barely-audible dialogue (or to decipher accents). There's sometimes when I can tell "oh gee that's not right at all but I can see how you misheard it." And there's other times when dialogue is coming from offscreen or a TV/radio and it's too far in the background to understand unless you are reading along with the script.
If you can afford it, buy a speaker system with at least a dedicated center channel. Dialog is almost always mixed to the center channel in 5.1/7.1 mixes. Which means that you can just turn up the gain on the center channel speaker, while leaving everything else down. The result is magical — it's like turning down the "background music" slider in a video game :)
You don't even need to set up the surround speakers. Having a 3.1 setup has improved my enjoyment of TV and movies substantially. My understanding is most if not all receivers will downmix the surround into your L and R channels.
And very helpful for slang too. I speak Spanish well, but I always add subtitles in English or other language in case I miss a word of two from another Spanish variety.
Coming from a non english country I can assure you that dubbing is the worst. When we were kids we would actively avoid watching anything that was dubbed as soon as we could read.
It’s actually funny seeing how frustrated with subtitles my American girlfriend gets when we watch anything foreign. “You mean this basic skill every 7 year old has is hard for you, an educated super smart 30 year old? Bwahahaha”
I still prefer watching everything with subtitles. Actors mumble a lot.
I have complicated feelings about this. I also watch most things with subtitles these days, because either (or probably both) my hearing is going a bit or sound design/editing in films and tv shows has gotten worse, but there is a non-trivial difference in how I perceive things with or without.
With subtitles my eyes are drawn more often to a part of the screen where nothing visual is happening. They aren't glued there, but it does create some bias. I have occasionally rewatched a scene without subtitles and noticed a lot of subtlety in the performance I had missed the first time though.
That said, a bad dub is at least as distracting. Probably much more so, if anything. I would rather watch a bad sub than a bad dub, though I have occasionally enjoyed dubbed things as well. I don't think it's an all or nothing thing, but "never watch dubs" is the hard line heuristic that makes sense to me if you're gonna go that way.
At least with subs the performance is there for you to see (and hear), even if you sometimes miss it. With dubs, half the performance is removed completely.
I share the same experience of preferring subtitles over dubs, coming from an English country. This is most popular with Japanese media like anime (like movies like 'Your Name'/'Kimi no wa na' or Attack on Titan). It also fits for foreign language films (e.g. preferring the original German for The Lives of Others/Wings of Desire).
Crucially, outside of preference, subtitles are occasionally necessary for translating text on the screen (e.g. on building signs, newspaper front pages, posters, chalkboard writing), as information is conveyed solely through writing without spoken dialogue.
For example, consider the chalkboard scene in "Komi san can't communicate": https://www.youtube.com/watch?v=JoH8I-NC5tM . It's key to the story that a character has trouble communicating verbally, so subtitles are essential to understand the written communication on the chalkboard. (You can then compare to a more extensively subtitled version that translates all the text on the board; good subtitles make a difference to storytelling: https://old.reddit.com/r/anime/comments/qdjgba/the_blackboar...).
> I still prefer watching everything with subtitles. Actors mumble a lot.
Yep I have no problems hearing but newer movies and shows have uneven mixing where it can get really hard to hear over the background noise and subtitles come in clutch constantly for that
As an english first language speaker, I even find english subtitles on english language media very distracting. I consume a huge amount of subtitled films/tv because most good things aren't in english, but subtitles keep my eyes focused on the bottom of the screen.
I don't like dubbing because it's generally much lower quality than the original performance, but I get why people who don't have a problem reading could prefer it.
Me too. I find subtitles are the main distraction for me to enjoy what’s going on in the screen. I understand that a lot people actually need them. That’s why I am very thankful that subtitles can be turn on/off if needed. I actually hate any videos that have subtitles embedded inside the video (part of each frame)
I love subtitles, but for me they tend to spoil the timing of the lines being delivered, especially with comedy. Unfortunately subtitles are necessary even when I'm able to play audio as loud as I want, because it's almost always hard to understand at least 10-20% of the words that actors are saying.
Maybe it would be nice to have subtitles that appear a word at a time as they are spoken, but I imagine that's not very practical for various reasons.
I would also love to a more responsive subtitle format! I've seen some really wonderful custom jobs where people use different subtitle colors for different speakers, place subtitles near the speaker in the frame, etc. I think there is a ton of room for improving the expressiveness of the medium.
That's a pure text format; no bitmaps involved, just markup and TTF fonts. It can do vector shapes and perspective positioning. These are somewhat overdone examples, but one more common nice use case is literally translating signs on buildings, objects and such, seamlessly, while preserving the image style. The whole thing is rendered in real time and you can toggle the subs off and see the original frame.
Again, that's all markup rendered on the video in real time from fonts. Yes, really.
It is the standard in the anime fansub community, but has seen little adoption elsewhere. Partially this is because it seems commercial operations rarely have much interest in subtitle quality or technology.
Incidentally, even YouTube supports an undocumented subtitle format that can support a decent amount of styling (YTT). Unfortunately, support is inconsistent across web/mobile versions, and it looks best when viewed in a browser. I've used it to do things like this (turn on captions):
A pet peeve of mine has been subtitles which flip the order of words, particularly when the sentence is broken into two lines. Like the spoken text in foreign language is “A murderer is... Joe” but it gets translated to “Joe is a... murderer”.
The other day I happened to see live content with closed captions, and I thought I would love the option to see subtitles like this, where words are translated precisely when spoken.
But translation doesn't really work like that. There are significant tradeoffs between verbosity, cadence, meaning and subtext when translating. The different contraints are why the forieng dubbibg and foreign subtitles are usually quite different.
You generally only see word for word matches with subtitles when the subtitles are in the same language as the original lines spoken by the actors.
And the GP's example is the perfect illustration. "A murderer is Joe" sounds really weird. No native English speaker would ever say that. They would say, "Joe is a murderer" or "The murderer is Joe" or even "One of the murderers is Joe." But never "A murderer is Joe." Maybe in a Victorian novel, but not in a movie.
There are certainly tradeoffs, which is why I said I’d like the option.
Since I’m moderately familiar with the other language, I really just need the translation for words I don’t understand or mis-hear.
There is such a thing as a real-time translation. I’ve heard it in news broadcasts when a politician speaks a foreign language. It also happens all the time at the UN and other international negotiations. It works well but has downsides such as occasional latency while the translator needs to hear a few more words before they can translate.
It's a browser plugin that provides advanced subtitle functionality. You can have dusl sets of subtitles up, or just the original language subtitles with the option to pause and mouse over individual words for the definition.
Unfortunately it's difficult to translate sentences between languages in a way that preserves word order and is grammatically correct in the translated language (your example is a pretty good example of this).
Different languages have very different grammar, and even the best translated subtitles won't be able to handle a clever usage of the original language's grammar.
This is one of the many tradeoffs you make with translated works. You always lose something (sometimes a lot) in the translation.
> I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of.
I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions.
I don't watch any language dubbed movies, but I wouldn't chide anyone who did. Not everyone wants to spend the whole two hours glancing up and down across the screen, nor are they necessarily wanting to learn a language passively, etc.
So I think there's absolutely a perceived problem, it just doesn't apply to all viewers.
> I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions.
Well, they are more pure than dubbed versions. There is so much nuance and tone in the original language that is either left out or changed in the dub. Not to mention word order (subject-verb-object in English/Chinese, subject-object-verb in Japanese/Korean) that further changes the timing of impactful words.
I am not looking forward to having to watch the dub versions of Totoro and other Miyazaki films when my daughter is between 2-7 years old...
There's often a lot of nuance lost in translated subtitles as well, in my experience, often because dialogue written for one language can't be coerced into natural dialogue in a different language without significant changes.
Keeping the original audio track does preserve the direct relationship between the actor's tone and expressions, of course, and it could also be something closer to what the original makers intended even if the voice actors in a dub also do a good job.
The audio track also feels somehow more right and authentic when someone who's supposedly Japanese speaks Japanese, or when a supposedly French person speaks French, or when people in a show set in California speak English rather than somehow magically speaking your native language.
Then there's of course the question of dealing with scripts that have multiple languages spoken in them. Something would surely be lost if multiple languages were dubbed into the same third language. It's often easy to recognize the different languages being spoken even if one does not understand them.
Very good point. I do get frustrated when watching Korean media with my wife and needing Korean subs for myself and English for her. I notice that the subs frequently mistranslate, or convert to a Western equivalent, which drops the nuance. ie. Korean honorifics, or differentiation between older and younger female/male siblings, which can get totally lost when translated to English.
Though this is much less of an issue when watching fansubbed media (for Japanese content, anyway) because the fans have translator notes and do a much better job than Netflix or media studio translators.
> Then there's of course the question of dealing with scripts that have multiple languages spoken in them. Something would surely be lost if multiple languages were dubbed into the same third language.
I've sometimes thought that languages could be indicated by phonetic rendering of an accent (an audio version of this was done on the excellent and clever show 'Allo 'Allo where dialog in 'French' is English with a French accent, dialog in English is RP, British characters badly speaking 'French' use RP with funny word substitutions, and so on), similar to how comics often indicate other languages in word balloons with conventions such as <angle brackets> or italicization, but that's probably too high a bar.
(It should go w/o saying reading this is a small spoiler and you should most definitely see this film first if you haven't. I happen to think the American remake is trash, but my dad loves both it and the original. In any case, it is among a small handful of films that I consider perfect.)
When I was a kid, my dad was stationed in Japan. We watched all kinds of stuff, including Totoro on Japanese TV. No subtitles or dubs. Not a clue what they were actually saying but we all enjoyed it anyways. So you could always try it out and see how it goes.
I don't know why I didn't think of this before... thanks! I will definitely try that out, especially since at such a young age she won't be able to sit through a whole film anyway.
I have been watching dubs of Ghibli with my kids. It’s fine. It would be better to watch the subs for me, but I’m not going to get bent out of shape about it. They usually have pretty good English voice casting. I kind of like Kirstin Dunst as Kiki.
Personally I would like if someone could use AI to enhance the voices of the actors, I find a lot of movies like to use loud ominous music while the characters are casually talking in a soft voice. Really drives me insane, and placing subtitles helps this problem but it creates a new one, I tend to get distracted by the subtitles and find myself not watching the movie
This could be an issue with your audio mixing in your particular speaker setup. Often times trying to play audio meant for surround systems (5.1, 7.1, etc) on stereo channels results in a poor audio mix.
I don't think I've yet encountered something with an English 5.1 track that doesn't also have an English stereo track mixed in with whatever foreign language tracks are present but who knows what features any given app has.
It also helps when what you're watching has atrocious audio mixing (looking at you Nolan), or you have shit speakers, or actors mumble or whisper, in which case it's helpful to have them as an aid when something's not clear.
Do you remember the name of a cartoon short involving a rotund orange cat walking around the city doing engaging in various hijinks before it arrived home unnoticed? It's been bugging me for ages. It played between shows in place of a commercials in-between shows circa 2006.
I don't know. I'm not sure it's really possible to overestimate how many people hate reading and view it as a chore. Even without that not-so-charitable interpretation, there is still the "TV is on but I'm focused on something else" use case. Subtitles don't work at all for that.
It would be hilarious to see famous actors deepfaked into speaking foreign languages in a feature film.
I suspect it simply won't work - translating accurately is really hard, but translating elegantly is frequently impossible. Some things are just easier/punchier/funnier to say in language A than B
Laurel and Hardy would refilm many of their shorts in different languages, where they were speaking the foreign language. I always thought that was pretty cool.
In my experience people who dislike subtitles are mostly native English speakers living in English-speaking countries, who are used to the fact that almost all the media they consume is produced in their native language, and so they’ve never developed the ability to read subtitles comfortably.
When the occasional rare case arises where they need to watch a foreign film/show, they find it straining to both read the subtitles and watch the film, and so they conclude that subtitles are bad or “not for them”. In practice this stems from lack of practice — the perceived conscious effort of having to read subtitles while watching a film disappears entirely after you have done it a few times with sufficient regularity, and becomes completely effortless, subconscious, and automatic. But this is a skill that needs to be developed. Saying that subtitles are bad because they distract from the film is like saying that riding a bicycle is bad because you fall all the time, when in reality you just haven’t learned to ride one.
I watch a lot of foreign films and dubbed version misses a lot of nuance - complexity of spoken language, inflection, depth.
So maybe I watch the kind of stuff where that kind of performace is required. Maybe it has certain genres or mediums where dubs work well like Romcoms or Marvel movies.
Then you're simply not thinking about it deeply enough. There are plenty of shows that someone may want to watch but may not be able to invest 100% of my attention watching that show, therefore with a show dubbed in their native language, they can still have the show in the background while being able to listen to the audio.
With subtitles one has to constantly be looking at the actual show itself. Furthermore if one makes use picture in picture such as on a mobile platform or a tablet device the subtitles are so correspondingly small as to be basically hieroglyphic at that point.
This use case is totally foreign to me. If I can't concentrate, I stop the show and continue some other time. Someone wants to check their phone or visit the loo? Pause the show. Linear television is history after all.
That said, other people can obviously have their own preferences. Just sounds so weird to me.
After a decade of training their model Youtube can't even produce meaningful subtitles for ordinary words in English (let alone my native language, french) and we're supposed to trust these fucked-up AI overlords are going to produce any better dubbing?
I have no doubt these solutions will be popular, because many people have difficulty reading and will prefer audio when given the option. I'm just confident they won't reach any measurable quality and the translations will be full of non-sensical/anti-thetical statements.
Agree. Dolby Atmos releases tend to have a lot of noisy background sounds all around you, and then people silently mumble on the center channel. My English is really good, but some movies are still difficult to watch without subtitles. For example, Hawkeye jumped in between "painfully loud effects" and "barely audible dialog".
That's a cute sentiment but there is no evidence that merely being exposed to a foreign language necessarily allows you to learn that language. I think you're confusing the movie 13th Warrior for reality.
I have friends who religiously watch subtitled anime only and they are no closer to learning Japanese in any real sense of the word than a person who would watch the dubs, except for picking up a stray random word here or there, but knowing that kawaii means "cute" doesn't give you even a superficial understanding of Japanese. For that, you need to at least pick up hiragana and katakana.
Learning a language requires deliberate work. There is no shortcut.
I speak Spanish. Subtitles have helped me a lot when languages are similar. I watched three seasons of a Catalan TV show subtitled in Spanish, and now I can follow conversations and read it. Before that I wasn't able to understand spoken Catalan.
Subtitles won't really help with sentence structure or conjugations, but it does help a lot with building vocabulary once you have the basics of sentence structure.
I’m in the “pro subtitles” camp, too, but they too have disadvantages. By changing part of the screen, they don’t preserve the original performance, either (but IMO, that’s a lesser problem than dubbing) and their timing often is off. You’ll read a joke before or after it’s made, may know somebody gets shot before it happens, etc. I think aster readers will have a larger problem there.
They (obviously) also work less well for slower readers.
It might seem to you but producers care about data and I do not find it hard to believe people prefer dubbing to subtitles. I believe most people prefer audio input to text because it requires far less of their attention. You would be surprised how many people watch / listen to shows while doing something else and thus not giving the show their full attention. I'm guilty of this with podcasts.
That's the most profitable system. It certainly doesn't produce the most quality. Just like making a tool that lasts a lifetime is not as profitable as making something that breaks. Maybe crowdfunding media content is the way to go for quality but I can certainly understand how public companies make decisions in their search for infinite profits.
How would society prevent companies from producing mediocre content? I don't think society is a useful model here, that is why I asked about producers and consumers. As long as consumers keep consuming mediocre content then there will be producers for this type of content. People vote with their wallets so "society" is at fault and also we as consumers of quality content can't force people to not like mediocre content
I would much rather we use AI to improve subtitling. Better auto speech recognition, per-word millisecond synchronization, positioning based on speaker/screen content-- all these could improve the subtitled experience and would be way less creepy than deepfake voices.
Yes! My perennial problem is that most Chinese TV content has excellent subtitles, but they're burned in to the videos. So if I want to watch a Chinese show with a friend who doesn't read Chinese, there's no option even for auto-translated subtitles. I've often thought of writing a script to generate subtitle files using text recognition, but haven't gotten around to it.
Ah, but one thing I miss about the burned-in subtitles was how, on VHS at least, one could fast forward like 2-3x and still be able to read the captions. I double-lazied my way out of more than one high school book review where a boring book was made into a boring movie.
There's also the problem of content which doesn't have subtitles to begin with---YouTube's automatic subtitle generation is great, but could be improved and expanded to languages other than English.
It is expanded, at least to Japanese. What's also fun is you can then have it auto translate the auto generated subs. The results are... not typically great, but the potential!
A long time ago, I tried the "autosub" feature with anime and it replaced understanding with unintentional (dry) humour. The output had phrases that resembled a news transcript, which is probably what they were training the system on.
Both are happening. Google meet has live auto subtitles that are pretty accurate. Sometimes I find even as an English speaker I am able to understand through the compression better with the subtitles on.
Completely besides whether this is possible (I think it is; it's an offline learning problem with huge resources/economic potential and it's getting attention) I actually prefer subtitles...
Coming from Germany, this wasn't always the case, but nowadays watching dubbed content feels so weird and impoverished, I don't even want to imagine what it'd be like knowing that the voice is fully synthetic.
I grew up consuming movies and TV shows with subtitles so it has never bothered me. My theory is that if you're not used to them, then they are really annoying. Most of my American friends do not like them.
There is something about watching a movie/show in its original language that you cannot understand. I think its awesome to be immersed in a different world, getting the sounds, inflections, etc from a different language of yours. I will always prefer subtitles to dubbed versions if possible.
I understand some people don't, but I think it's just a matter of being exposed to it. And probably the younger, the better.
Yes I think it's being a matter of being used to watching things with subtitles. Some people find it very distracting, I hardly notice at all watching something with subtitles since it is how i grew up. The only time I notice is when I want to do something else at the same time as watching the movie/show, then I quickly realize oh I can't do that I dont understand the audio. Dubbed things suck it is always the same voice actors too so they always seem not quite right for each new character.
This article is a thinly veiled promotion for Veritone's MARVEL.ai which doesn't even do translation into foreign languages. It's a text-to-speech encoder that has a variety of voices trained for different languages. The actual translations and subtitles need to be done most likely still by human workers or human-supervised machine translation. The article never really makes the point of the title which is that foreign-language subtitles will be ending any time soon.
There should be a disclaimer on this advertorial piece.
AI is also able to create sub titles. So, people will have a choice. Many people prefer subtitles.
Dubbing movies is common in big countries where the locals speak nothing else than their own language. Here in Germany, you have to pay attention if you go to the cinema but they do have original language movies as well. Lots of Germans go to see those instead of the badly dubbed versions. I tossed out my TV ages ago because I don't speak German very well and don't like watching dubbed content. Especially if that content is originally in English which I actually can understand.
One annoyance with the local netflix here is that while they have most content with the original sound track, they often swap out a lot of the text in these movies with German text. E.g. movie fragments showing a news paper or somebody's text messages, or the famous rolling text in star wars. And of course a lot of American movies or series sometimes feature people not speaking English. Netflix solves this by offering German sub titles. And sometimes those are actually baked into the movie.
Par of the issue is that there's a big dubbing industry that is incentivized to keep on dubbing everything whether people like it or not. AI will decimate how profitable that business is.
> Netflix solves this by offering German sub titles.
This is my pet peeve about streaming services. Why would they have only German subtitles in Germany? The licensing costs of all major languages can't be prohibitive. I only watch content that has English subtitles -- Netflix has thankfully improved on including them, but otherwise you're mostly out of luck. Even Blu-rays need to be imported if you want to have English subtitles on English movies!
Netflix doesn't ship all the languages even with their own content, so it can't be the only reason. But good point, never thought of that, and truly sad if access restrictions again is the reason for legal services delivering less than piracy.
If you change Netflix's interface language you'll get access to more languages for both subtitle and audio tracks.
I suspect there's some estimation going on to figure out what languages someone is most likely to know given a certain interface language and location. This is a somewhat common annoyance language learners run into because it's not obvious that you need to go through these hoops to get access to the language you'd like to use.
I like listening to the original audio of foreign movies and it's partly how I learned English, and how I pick up random words from languages I can't really speak.
I also respect actors and actresses enough to appreciate their performance includes their original voices.
I'm a fast reader and subtitles are like a second nature to me, to the point I don't "remember" having read them (likewise I sometimes don't remember if I read a novel in English or Spanish unless a particular turn of phrase caught my attention).
About the only thing I agree with is that dubbing (barring some honorable exceptions) sucks. I did listen to a part of La Casa de Papel's English dub -- the horror!
I have got pretty serious dyslexia, and thus I have problems with reading subtitles fast enough. As a child this basically meant no subs for me. So I definitely see the value in dubs.
These days I have gotten a bit faster at reading and I got things like more readable fonts to help me (also, some subtitle systems are just terrible, I often torrent stuff that is on netflix despite having an account because their subtitle system sucks).
Yet, I am from the Netherlands, and here, with basically nothing except media for children being dubbed, I have gotten better at reading them and used to it. And honestly, despite the effort I have to go through, I totally agree. It is totally worth the extra effort. I'll always prefer the original language, no matter the quality of the dub.
Language is just an important part of the culture of the original content.
I dont understand Spanish or Portuguese, but I think a lot of emotion that is conveyed in the way the languages are spoken is lost in the dubbing. Reading the subtitles and just hearing the actors say the same in Spanish conveys it better for me.
1. AI might be able to prevent bad dubbing/subbing, but it will not be able to compete with _good_ dubbing/subbing in our lifetimes. There's an artistry there that no AI can match.
2. I love subtitles and prefer original audio + English subtitles in anything foreign-language. Besides the voices matching up to the movement of the actors' lips and thus not breaking my immersion, I also adore hearing how familiar emotions are reflected in languages I do not know. It makes me feel more connected to humanity.
Few in this thread mentioned immersion as an issue. Subs are fine for comedy and drama, but I don’t understand how I could watch an action spectacle or travelogue and feel like I’m there when I’m looking away for floating words. Same reason I don’t want a movie audience to draw my attention and remind me the screen is fake.
You just aren't used to them. When you were a child, you probably weren't used to understanding the words you heard and paying attention to the scenes.
I'm partially deaf so I watch everything with subtitles or captions, and don't miss a thing.
It's easy. Blind test humans and ask them which one they find more "artful". Once the average person can not tell between an AI and human creators, the AI is complete.
I think it's a great option but their is an art to translations. This applies especially to books (I'm reading an English translation of https://en.wikipedia.org/wiki/Zorba_the_Greek which is very well done) and while you do have less dialog with movies, I wonder if some subtleties will be missed, particularly around humour.
This is something that bothers the crap out of me watching recent shows on Netflix or similar that come with official subtitles, and that is that they're often crap. You're spending millions on actors and sets and directors and visual effects and editing and everything else around the production, but the subtitles are some kind of shitty after-thought doled out to the lowest bidding subcontractor?
In Sweden, everything foreign is subtitled, and as a result of the vast quantity of subtitles produced, the quality is high, so I've grown up expecting good quality subtitles. And then I watch something on HBO or whatever, and the subtitles are just bad. Clearly not proofread, clearly done by someone who didn't watch the thing, and just translated the written material as is.
And the art side of subtitles is often neglected or plain just not understood. Timing is really important, which people understand when it comes to for example dialogue timing to make a joke land. But then the subtitles are shit, and ruin the joke because the subtitle timing is off, when you could have had good subtitles that worked with the material, preserving the timing!
Why aren't studios and networks investing as much in the subtitles as the rest of the production?
Yes, 95% of subtitles are easy to make, there's a single unambiguous way to translate what's being said into your target language, and by syncing it to the audio it's gonna be good enough. But some things are hard to translate, because there simply doesn't exist a 1:1 mapping between every language, despite Google assuming the world works like that.
But YouTube's auto-caption thing is garbage. It's enough to sort-of kinda make out what is being said, but it's often complete garble, and there's zero understanding of good timing or the art side of subtitling.
My wife is a translator and based on our conversations and her experience, subtitles are needed because people want to have the original audio + text, foreign subtitles are needed for obvious reasons, and human translators are needed for the "tricky" parts like new words and expressions, cultural context expressions, stuff that can't be directly translated, among other things.
That's why in many countries a translated piece counts as a new/original piece, and the translator has the rights to claim some of the books IP.
> subtitles are needed because people want to have the original audio + text
I'm a non-native English speaker, and I like watching English-language content with closed captioning in the original language.
There's often a lot lost in translation, especially on the screen (written content might have different issues), because it's often hard to get the rhythm and timing right without changing the contents in non-trivial ways when the source and target languages are completely different. Since I can understand English fairly fluently and nearly always get the nuance that would be lost in a translated version, I prefer the original language, even for the subtitles.
Since I'm not 100 percent, though, having the subtitles or closed captioning usually lowers my cognitive load. It also means I don't have to worry about missing something because of an unfamiliar accent, a very quickly spoken line, or some other random thing.
> More Netflix subscribers watched dubbed versions of "Squid Game" than subtitled versions.
The default of you load it is dubbed, right? That’s what it started for me. So this stat probably has more to do with the default than anything else. They’re making a weak case that this problem even needs solving.
- The hard part about subtitles is not translation it's contextualization. Languages are not variable replacements and words have different meanings based on the context of the sentence. Languages have nuance, phrases, and slang unique to their cultures which further adds difficulty.
- coupled with the above, permutations can get large with translated dubs. Which subtitle are you supposed to show? Most translated subs are done off the original audio track and the dubs are done separately. So for example with an English show, the English subs match the English track, but the translation will be off for the Spanish audio track. And the Spanish dubs will not match the Spanish subs either.
- lip sync on dubs is a hard problem for many reasons ( full face and body emotion, how words are said in the native tounge, etc..) and will significantly improve the viewing experience. You can do creative things like have fewer full face shots to help with this. It's also important to get the audio mix right so it doesn't sound like an extra track. Keep in mind you now need more videos to cache and store and the ability to swap video streams when you swap audio streams.
- animated content offers other approaches than leveraging nn/deepfakes because the assets are already rigged. The use of game engines offers some really interesting opportunities here.
This will not happen for many, many years to come.
That is, companies will try it, there will be some bad and ridiculous results, there will be pusheback and the tech will largely be ignored for a decade or so.
Main problems will be intonation and inflection. What is a question to an English speaker is a regular sentence to a Russian. And so on. "Petabytes of data from primarily English-speaking libraries" mean jack shit if you wnt to actually do voice translation into/from foreign languages
This is live in production today - business owners are getting calls from robots, and you can trigger it yourself with Google Assistant.
[Disclaimer: I work(ed) on Google Assistant - I'm in the process of transferring out - and have listened to some presentations about how Duplex works. It's pretty crazy technology.]
It's still not very good. The state of the art vocoders ( StyleTron et al ) are years away from getting intonation right. But then if you could get intonation correct, you still need to be able to get pausing and pitch correct, let alone timbre and a few other things I'm forgetting from my research.
You can install it on your phone and it'll subtitle your life if you can find a good place to put it down. It's meant for the deaf and hard of hearing (hence features like visual alarms) but in my experience it'll transcribe stuff that's being said pretty well as long as the mic catches the audio you want to hear. Perhaps it would be even better with a quality (bluetooth? 3.5mm?) external microphone.
Well now you've got me thinking. That has to be very feasible for any content you own the file for, but I wonder how steaming services currently handle subtitles. Are they embedded into the video stream or are they their own channel? Could a third party pull them out and pass them along to a separate device like that?
Definitely not embedded into the video stream, at least for Netflix or Youtube. There's actually extensions that take the subtitles and manipulate them (even put them outside of the video stream) so you can use them for various language-learning purposes. Look at stuff like Language Reactor (https://www.languagereactor.com/), previously Language Learning with Netflix. There's specific browser extensions like subs2srs and yomichan (https://animecards.site/yomichansetup/) which are very common in Japanese language learning communities for people to use on subtitle lines to get direct translation/dictionary entries of certain words and be able to send them directly to Anki (SRS flashcard software).
I second the importance of subtitles. I grew up in a country where dubbing is a huge industry and everything is dubbed. I’ve also lived in another country where almost nothing is dubbed. After moving to the US and seeing many of the films I had previously only known in their dubbed version, I can no longer stand dubbing. I can tell that something is dubbed without having to look at the screen. I’d never want to not have subtitles — I’d rather not watch at all…
That being said, as I understand the article this marvel ai thing is supposed to replicate the original performance using the original voice in another language? Of course that would be fantastic a solution, but I doubt that the technology is anything near what it wants to do…
Whilst this is good, as long as it is an option and the user still has the choice to have the original language with subtitles then great. Many enjoy films in other languages they don't fully understand for many reasons, learning the language, nuance of the language and tonality, many things. After all, a Shakespeare love scene in Klingon is just not going to cut the ambience like many other languages.
But one thing that bugs me, why is it that even today - we have to have programs with a hard-overlay of sign language interpreter in a you have it if you want it or not and the selection is limited. For those who depend upon such sign language, wouldn't it be great if they could watch anything and some AI would do the sign language for them. You would of thought that such a task would be easy, and the market is there but yet, it seems to elude us.
So my hope is that whilst this is useful, it may well open up avenues like the above in which these tools can be leveraged in more practical ways for a group of users who in effect have what they can and can not enjoy, somewhat overly limited as they are at the mercy and whims of the broadcasters who in many instances, only do them to quota tick some regulation put in place to at least not totally shut out a group of people who can't hear at all.
Because synching subtitles to the video isn't even a goal.
Most people read more slowly than they can listen. Many other people can read faster than they can listen. But you need subtitles that stick around long enough for the first group, and really, you want a comfortable margin so that people aren't frantically trying to finish reading every subtitle before it disappears.
I watched the show 大江大河 ("Like a Flowing River") on Viki, which has excellent community-generated subtitling. The second season isn't available on Viki - only YouTube. (Actually, Viki seems to have lost the license to the first season by now, too.) And the subtitles are abominably bad, bad enough to make me stop watching the show. But they're perfectly synched - every subtitle on Youtube is exactly matched, millisecond-to-millisecond, with a Chinese subtitle which it attempts to translate.[1]
[1] Ignoring the timing, which is much too fast for the English subtitles, the fact that each subtitle is translated independently is another huge problem. It leads to nonsense when one sentence is split across multiple subtitles, because the English and Chinese do not naturally present the same information in the same order.
With tools like alass[1] (using it to synchronise against the original language subtitles) it is about as close to solved you can get.
All of the attempts I've seen of using audio information to synchronise subtitles have been awful. One issue is that some languages subtitle everything, even screams and incoherent shouts (such as Japanese) while others only subtitle dialogue and often rework dialogue for the purposes of making the subtitles short enough to be readable easily. It feels like you need too much domain knowledge to know how different languages subtitle things and that subtitles that match the general meaning of what is being said should be matched up.
Every movie, every show ever produced? Subtitles are required even for domestic markets for the hearing impaired, even if we disregard the audience that prefers to have subtitles.
I speak a minority language that is primarily in spoken form by native speakers, very limited in number. I wonder, how AI technologies can be utilized to enhance the content or auto generating speech. I find it hard. Plus, the economic incentives for doing this work is highly limiting because of a small / limited user base.
The notion that some people prefer dub over sub is not new, but it baffles me that it is to the level of urgent business concern. From what I see in other comments, is it an American thing?
We should invest in making people’s ear accustomed/used to hear other languages. It has great intercultural benefits.
In the case of English, the orthography is difficult, a large proportion of people are functionally illiterate, and even more have a tough enough time reading that subtitles just go by too quickly.
I have a difficult time imagining that Russian speakers prefer the dubbed content they're offered. Most of the Russian dubs I've seen have one male voice actor doing all the characters (or with an additional female voice actor doing all the female characters, if you're lucky) with no attempts at timing, emotion, or intonation. The dub track is played on top of all the other audio, instead of being properly mixed in to replace the original dialogue track.
This would be interesting if it worked - I recently re-watched The Expanse in German with subtitles on but it wasn’t very helpful as a lot of the time they simply gave up and used the English terms. Some are a bear to translate - try saying “donkey balls” in German, “eseleier”? The captions reverted to English and I didn’t really pick up what the voice track said but wasn’t even close.
I had much better luck with Star Trek TOS, very little deviation between the English and German dialog, and the subtitles matched up well with the spoken dialog. Much better for learning - though “er ist tot Jim” doesn’t quite have the same punch IMO.
I feel the same way about this as I do about reading poetry in translation, or listening to the (fortunately rare) opera sung in translation. There's something integral about the native qualities of the original language that I can't imagine would survive the AI process.
I’m highly skeptical about AI-replacing-subtitle-as-a-whole. We all know what’s the problem: the quality. We already know that translation is difficult. We already know that acting is difficult.
The only feasible outcome here is deepfaking the voice from the recordings from localized voice actors to the actual movie actors. Perhaps, in the future, a voice actor might be doing all of (1) translation (2) voice acting (3) deepfaking in one shot. One person might be able to dub the whole movie - alone.
But, for sure, we won’t be able to remove human from the process.
The article seems more against manual dubbing and related difficulties than actual subtitles. The best subtitles I've seen included relevant context on cultural jokes I would have completely missed had there not been explanation text. One example would be recently while watching a Korean drama with dubs that didn't address cultural items and beauty standard, the content itself came off very differently than the intention seemed.
As a programmer I think that this is awesome. But... I'm also thinking that we may be missing opportunities.
For example, I prefer to see subtitled movies in their original language (I'm a Spanish native speaker). But there are some dubs that are awesome, even better than the original: for example, Shrek. I'm concerned that eventually studios may just throw the script through Google translate and call it a day.
The article would be far more convincing if it argued that AI could end foreign-language dubbing. Distaste for poor dubbing only partly explains the demand for subtitles. Film aficionados would scoff at the idea of abandoning the original audio, and even casual appreciators of film recognize the value of hearing the original tones and accents.
Oh come on everybody, this is a PR pseudo-advertisement for Veritone. The piece neglects to acknowledge that auto-dubbing is already a thing. No longer research has highly accurate voice synthesis of any specific voice with very limited source material. Realistic voice synthesis is very actively researched, and is already well into auto deep fake territory.
An adjacent concern: Ads bad, yeah, but I appreciate the growing trend of subtitling them. I'm surprised there isn't some existing regulation on that, in the same way that streaming services are required to include them. The lower the cost of accessibility features, the more inclusive the media we consume can become.
On a tangential note: As far as training data for text and speech driven machine learning models go, isn't srt files truly remarkable?
They are time stamped hence annotated precisely, hash and file matching makes media content to subtitle matching exact and overall there are language variations.
They are very artisanal, though. I used to do timing for anime fansubs when I was a teenager, and there's a lot of subtlety going on, specially when you don't speak the language (you're guided mostly by sound and phonetics, as the translations come more or less untimed).
Timing is much more automatized nowadays, but still doesn't feel as high quality as handicrafted timing does.
It is striking to me that this article doesn't discuss the negative impact such a technology will have on actors and translators. In fact, the author makes the surprising claim that there is a shortage of translators and voice actors, which is very hard to believe.
I'm prefer subtitles as opposed to poorly voiced and dubbed versions that are prevelant. Also how will marvel.ai add nuance that is missing in subtitles? The missing nuance is not always about tonality but the context that is missed by machine translation.
Oh. This would be terrible. Not only from a loss of cultural and language preservation, but also losing the actor’ performance. I really dislike watching dubbed shows. If this somehow became a thing it would be sad
Goupon founder Andrew Mason's new startup Descript does a fantastic job editing podcasts from text. This same sort of tooling is absolutely going to make it into film.
This sort of stuff is already trivially possible with zero effort:
In ten years, we'll be able to reposition actors and fix flubbed takes. We won't have to visit set, use expensive lenses, or do many of the things that make Hollywood the domain of the rich and well funded.
In thirty years, we'll just tell the machines what we want to watch.
The problem is it's not very good and there is a vast uncanny valley. Phoneme / voice conversion is an incredibly difficult problem and not likely to be solved with any techniques known today, least ways not cheaply. Transformers show some ability to get it right , but if you wanted to use e.g. a Seq2Seq model you'd be hard pressed to come up with enough data that would generalize.
AI can generate voices similar to the original actor, but the script will still have to be translated and proofread by expert translators. No AI can fully replace them.
Most people here said it already; that would be a very bad development imho. But one that will happen as, at least in my distributed developer team (all continents, many countries) everyone young hates reading; chat, email, books, subtitles; they all hate it and want to listen and talk, not read & write. So it makes sense. Hope there will be choice always; I grew up with English movies with Dutch subtitles which taught me English and saw, at the same time, how awful it was to watch Rambo saying 'Ich bin John Rambo' with a flinty German dubbed voice. The horror.
Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wonderful to have an 'autodub' option! But the times where one would use it seem quite narrow.