Why do these video generation ones never become usable to the public. Is it just they had to create millions of videos and cherry pick only a handful of decent generations? Or is it just so expensive there's no business model for it?
My mind instantly assumes it a money thing and they're just wanting to charge millions for it, therefore out of reach for the general public. But then with Meta's whole stance on open ai models, that doesn't seem to ring true.
There are a few available to the public. runway.ai and kling are a couple that I see heavily used on Twitter.
I pay for runway right now for experiments and it works. The problem is that maybe 1 out of 10 prompts result in something useable. And when I say useable I have pretty low standards. Since the model pumps out 5 or 10 second clips you have to be pretty creative since the models still struggle with keeping any kind of consistency between shots. Things like lighting, locations, characters can all morph within/between cips.
The issue isn't quality exactly, it is like 80% there. When it works, it is capable of blowing your mind. You can get something that looks like it is a bonafide Hollywood shot. But that is a single 5 second or 10 second clip. So far there is no easy way to reliably piece those together to make even a 1 minute long TikTok.
The real problem is the cost. Since you have to sometimes do 10 prompts to get a single acceptable shot it is like a 10x multiplier on the cost per second of video. That can get very expensive for even short experiments.
Hi zoogeny (and anyone else here) — you can try our new app Nim to address the Runway problems you describe https://alpha.nim.video
We offer both image-to-video (same situation as Runway, need a few attempts to make something awesome) and video-to-video (under the name "Restyle 2.0") - this is our newest tool and is highly reliable, i.e. you can get complex motion (kissing, handshakes, boxing, skateboarding, etc) with controllable changes to input video (changing outfits, characters, backgrounds, styles).
Unlike Runway and Kling, we currently offer a smiple UNLIMITED plan for just $10/mo. Check it out! https://alpha.nim.video
Kling’s new one 1.5 model is WAY better than anything else I’ve tried. Makes runway look terrible. Really good temporal consistency and even gets hair and clothes and stuff right.
They also just added the ability to do lip sync to a moving head and it gets the lighting right too - runways lip sync breaks if there’s any movement at all.
I’m gonna stop pumping Kling on this comment thread now - until they start paying me to advertise!
GTA IV Real Life - Runway Gen 3 AI shows the potential to turn low-fidelity source to something life-like https://youtu.be/FGBSzSO8k6A it would be really cool to this to work locally at playable rates
At “the public” Internet scale, if a hundred million people click Generate, imagine if Meta ends up paying a million dollars instantaneously.
- How many clicks of Generate are budgeted for?
- How many clicks should each user’s quota be?
- How much advertising revenue will be earned per click?
- Why should they give away a million dollars?
Right now, AI costs for this are so high that offering this feature ‘for free’ would bankrupt a small country in a matter of days, if everyone on Meta used it once. It doesn’t particularly matter what the exact cost is: it’s simply not tolerable to anyone who owes payment for the services provided.
This is also why the AI industry is trying to figure out how to shift as much AI processing as possible to devices without letting users copy their models to profit off of the training research spend.
Meta owns their data centers, so I don't think that framing is quite right. Increased traffic might cost marginally more in terms of electricity usage, but I think mostly what would happen is the service would degrade.
The hardware serving web requests on Facebook is very different from the hardware used to generate these videos. It’s different kit, that is currently quite expensive and power intensive.
Facebook absolutely does not have a fleet of GPUs idling that could suddenly spring into action to generate a billion of these videos, nor do they have power stations on standby ready to handle the electricity load.
Right, my point is that "paying a million dollars instantaneously" isn't something that Meta would face the way a company with a public cloud infra would, and as a result their motivations / concerns are probably more along the lines of bad user experiences (due to performance bottlenecks) hurting public perception rather than runaway costs bankrupting the company.
Having recently seen cost analysis for hosted enterprise generative AI, we’ll continue to disagree on this point. You certainly are describing valid concerns but Meta never struck me as being particularly worried about how people think of them; and, I am certain this doesn’t have the ’degrade’ capability at the billion users scale — it would have work queue lengths measured in weeks or more, which is useless for social media.
I'd guess 1 in 10 model demos turn out to be useful product, at best.
This and Sora are particularly annoying, though, for how they put together these huge flashy showcases like they're announcing some kind of product launch and then... nothing. Apparently there's value in just flexing your AI-making muscle now and then.
When I see lists of URLs like that I can only wonder what a future post archeologist, coming upon this long dusty thread half a decade from now, will find when they try to go to those sites.
Came here to say this... These companies all want patted on the back for how cool their video models are but we're still waiting on Sora since like last year. More and more publish these "look at us" papers but don't publish the models or even give us access to them.
They do exist, Luma AI DreamMachine is pretty cool. As well as Kling, Minimax, etc. But they aren't anything like Sora or this appear to be. They work but these, while likely cherry-picked, are still a whole new breed of video generation. But who knows if we'll ever actually get to use them or if we're just supposed to reflect on them and think about how cool and impressive Facebook and OpenAI are.
My mind instantly assumes it a money thing and they're just wanting to charge millions for it, therefore out of reach for the general public. But then with Meta's whole stance on open ai models, that doesn't seem to ring true.