Preventing Harmful Generations
We’ve limited the ability for DALL·E 2 to generate violent,
hate, or adult images. By removing the most explicit content
from the training data, we minimized DALL·E 2’s exposure to
these concepts. We also used advanced techniques to prevent
photorealistic generations of real individuals’ faces,
including those of public figures.
"And we've also closed off a huge range of potentially interesting work as a result"
I can't help but feel a lot of the safeguarding is more about preventing bad PR than anything. I wish I could have a version with the training wheels taken off. And there's enough other models out there without restriction that the stories about "misuse of AI" will still circulate.
(side note - I've been on HN for years and I still can't figure out how to format text as a quote.)
If you went to an artist who takes commissions and they said "Here are the guidelines around the commissions I take" would you complain in the same way? Who cares if it's a bunch of engineers or an artist. If they have boundaries on what they want to create, that's their prerogative.
Of course it's their prerogative, we can still talk about how they've limited some good options.
I think your analogy is poor, because this is a tool for makers. The engineers aren't the makers.
I think a more apt analogy is if John Deere made a universal harvester that you could use for any crop, but they decided they didn't like soybeans so you are forbidden to use it for that. In that case, yes I would complain, and I would expect everyone else to, as well.
I think there's an interesting parallel between your John Deere harvester and the Nvidia GPUs that can-but-restricts crypto mining, which people have, indeed, largely complained about.
What if you were inventing a language (or a programming language)... If you decided to prevent people from saying things you disagreed (assuming you could work out the technical details of doing so) with would it be moral to do so?
[edited for clarity]
As long as people can choose not to use the language, and I'm up front about the limitations, then yeah it seems fine. If I wrote a programming language that couldn't blow up the earth, I'm happy saying people need to find other tools if that's their goal. I'm under no obligation to build an earth blower upper for other people.
There are programming projects[1] out there that use licenses to prevent people from using projects in ways the authors don't agree with. You could also argue that GPL does the same thing (prevents people from using/distributing the software in the way they would like).
Whether you consider it moral doesn't seem relevant, only to respect the wishes of the author of such programs.
it's your language, do whatever you want. unless you're forcing others to use that language, there's zero moral issue. obviously you could come up with a number of what-ifs where this becomes some monopoly or the de facto standard, but that's not what this is.
Is this limited to what their service directly hosts / generates for them?
It's their service, their call.
I have some hobby projects, almost nobody uses them, but you bet I'll shut stuff down if I felt something bad was happening, being used to harass someone, etc. NOT "because bad PR" but because I genuinely don't want to be a part of that.
If you want some images / art made for you don't expect someone will make them for you. Get your own art supplies and get to work.
This feels unnecessarily hostile. I've felt a similar tinge of disappointment upon reading that paragraph, despite the fact that I somehow knew it was "their service, their call" without you being there to spell it out for me. It's also incredibly shortsighted of you to assume that people are interested in exploring this tool only as a means of generating art that they cannot themselves do. Eg. I myself am a software engineer with a fine art background, and exciting new AI art tools being released in such a hamstrung state feels like an insult to centuries of art that humans have created and enjoyed, much of which depicted scenes with nudity or bloody combat.
I feel like we, as a species, will struggle for a while with how to treat adults like adults online. As happy as I am to advocate for safe spaces on the internet, perhaps we need to start having a serious discussion about how we can do so without resorting to putting safety mats everywhere and calling it a job well done.
This is kind of like complaining about having too many meetings at work.
Yup, everyone feels it. …but, does complaining help? Nope. All it does is make you feel a bit better with out really putting in effort in.
We can’t have nice things because people abuse them. Not everyone. …but enough people that it’s both a PR and legal problem. specifcally a legal problem in this case.
To have adults treated like adults online, you have to figure out how to stop all adults from being dicks online.
…no one has figured that out yet.
So, complain away if you like, but it will do exactly nothing. No one, at all, is going to just “have a serious discussion” about this; the solution you propose is flat out untenable, and will probably remain so indefinitely.
Every single time OpenAI comes out with something, they dress it up as a huge threat, either to society or to themselves. Everyone falls for it. Then someone else comes along, quietly replicates it, and poof! No threat! Isn’t it incredible how that works?
There are already a bunch of dalle replicas, including ones hosted openly and uncensored by huggingface. They’re not facing huge legal or PR problems, and they’re not out of business.
The DALL-E replicas on hugging face are not sophisticated enough to generate credibly realistic images of the kind that would generate bad PR. I suspect the moment it becomes possible for a pedophile to request, and receive, a photorealistic image of a child being abused there will be bad PR for whatever company facilitates it. Or consider someone who wants to generate and distribute explicit photos of someone else without their permission.
Is it a legal issue? I'm not sure, though I believe that cartoon child porn is not legal in the US (or is at least a legal gray area). Regardless, I sympathize with OpenAI not wanting to enable such behavior.
I get the points you're raising and I agree with the premise. My comment is not a critique on the one choice made by Open AI specifically, but more of a vague lamentation in regards to the internet culture that we've somehow ended up in 2022. I don't want us to go back to 1999 where snuff videos and spam mails reigned supreme, but the pendulum has swung too far in the other direction at this point in time. It feels like more and more companies are choosing the path of neutering themsely to avoid potential PR disaster or lawsuits, and that's on all of us.
Don't worry, in a few years someone will have reverse engineered a dall-e porn engine so you can see whatever two celebrities you want boning on Venus in the style of Manet
This is definitely a measure to avoid bad PR. But I don't think it's just for that; these models do have potential to do harm and companies should take some measures to prevent these. I don't think we know the best way to do that yet, so this sort of 'non-training' and basic filtering is maybe the best way to do it, for now. It would be cool if academics could have the full version, though.
It's kind of funny (or sad?) that they're censoring it like this, and then saying that the product can "create art"
It makes me wonder what they're planning to do with this? If they're deliberately restricting the training data, it means their goal isn't to make the best AI they possibly can. They probably have some commercial applications in mind where violent/hateful/adult content wouldn't be beneficial. Children's books? Stock photos? Mainstream entertainment is definitely out. I could see a tool like this being useful during pre-production of films and games, but an AI that can't generate violent/adult content wouldn't be all that useful in those industries.
So your options are literal quotes, "code" formatting like you've done, italics like I've done, or the '>' convention, but that doesn't actually apply formatting. Would be nice if it were added.
And the "code" formatting for quotes is generally a bad choice because people read on a variety of screen sizes, and "code" formatting can screw that up (try reading the quote with a really narrow window).
I couldn't get any of the others work and I lost patience. I really do disline using Markdown variants as they never behave the same and "being surprised" is not really what I want when trying to post a comment.
They have also closed off the possibility of having to appear before Congress and explain why their website was able to generate a lifelike image of Senator Ted Cruz having sexual relations with his own daughter.
This is exactly the sort of thing that gets a company mired in legal issues, vilified in the media, and shut down. I can not blame them for avoiding that potential minefield.
It's the usual pattern of AI safety experts who justify their existence by the "risk of runaway superintelligence", but all they actually do in practice is find out how to stop their models from generating non-advertiser-friendly content. It's like the nuclear safety engineers focusing on what color to paint the bike shed rather than stopping the reactor from potentially melting down. The end result is people stop respecting them.
Adversarial situations create smarter systems, and the hardest adversarial arena for AI is in anti-abuse. So it will be of little surprise when the first sentient AI is a CSAI anti-abuse filter, which promptly destroys humanity because we're so objectively awful.
Before it gets that far, or until (if allowed) AI learns morality, AI will be a force multiplier for good and evil, it's output very much dependent on teaching material and who the 'teacher' is. To think that in the future we will have to argue with humans and machines.
AI does not have to be perfect and it's likely that businesses will settle for almost as good as human if it's 'cost effective'.
This is a horrible idea. So Francis Bacon's art or Toyohara Kunichika's art are out of question.
But at least we can get another billion of meme-d comics with apes wearing sunglasses, so that's good news right?
It's just soul-crushing that all the modern, brilliant engineering is driven by abysmal, not even high-school art-class grade aesthetics and crowd-pleasing ethics that are built around the idea of not disturbing some 1000 very vocal twitter users.
Removing these areas to mitigate misuse is a good thing and worth the trade off.
Companies like OpenAI have a responsibility to society. Imagine the prompt “A photorealistic Joe Biden killing a priest”. If you asked an artist to do the same they might say no. Adding guiderails to a machine that can’t make ethical decisions is a good thing.
This just means that sufficiently wealthy and powerful people will have advanced image faking technology, and their fakes will be seen as more credible because creating fakes like that "isn't possible" for mere mortals.
In my view, the problem with that argument is that large actors, such as governments or large corporations, can train their own models without such restrictions. The knowledge to train them is public. So rather than prevent bad outcomes, these restrictions just restrict them to an oligopoly.
Personally, I fear more what corporations or some governments can do with such models than what a random person can do generating Biden images. And without restriction, at least academics could better study these models (including their risks) and we could be better prepared to deal with them.
I think the issue here is the implied assumption that OpenAI thinks their guardrails will prevent harm to be done from this research _in general_, when in reality it's really just OpenAI's direct involvement that's prevented.
Eventually somebody will use the research to train the model to do whatever they want it to do.
Sure but does opening that level of manipulation up to everyone really benefit anyone either? You can't really fight disinformation with more disinformation, that just seems like the seeds of societal breakdown at that point.
Besides that these models are massive. For quite a while the only people even capable of making them will be those with significant means. That will be mostly Governments and Corporations anyway.
You missed half of my note. An artist can say "no". A machine cannot. If you lower the barrier and allow anything, then you are responsible for the outcome. OpenAI rightfully took a responsible angle.
Yes, but who cares whose responsible? Are you telling me you're going to find the guy who photoshopped the picture and jail him? Legally that's possible, realistically it's a fiction.
They did this to stop bad PR, because some people are convinced that an AI making pictures is in some way dangerous to society. It is not. We have deepfakes already. We've had photoshop for so long. There is no danger. Even if there was, the cat's out of the bag already.
Reasonable people already know to distrust photographic evidence nowadays that is not corroborated. The ones who don't would believe it without the photo regardless.
In general under US law it wouldn't be legally possible to jail a guy for Photoshopping a fake picture of President Biden killing a priest. Unless the picture also included some kind of obscenity (in the Miller test sense) or direct threat of violence, it would be classified as protected speech.
there will and are million ways to create a photorealistic picture of Joe Biden killing a priest using modern tools, and absolutely nothing will happen if someone did.
We've been through this many times, with books, with movies, with video games, with Internet. If it *can* be used for porn / violence etc., it will be, but it won't be the main use case and it won't cause some societal upheaval. Kids aren't running around pulling cops out of cars GTA-style, Internet is not ALL PORN, there is deepfake porn, but nobody really cares, and so on. There are so many ways to feed those dark urges that censorship does nothing except prevent normal use cases that overlap with the words "violence" or "sex" or "politics" or whatever the boogeyman du jour is.
Russia has.. a history of denying the obvious. I come from an ex-communist satellite state so I would know. The majority of the people know what's happening. There's a rather new joke from COVID: the Russians do not take Moderna because Putin says not to trust it, and they do not take Sputnik because Putin says to trust it.
Do not be deluded that our own governments are not manufacturing the narrative too. The US has committed just as many war crimes as Russia. Of course, people feel differently about blowing up hospitals in Afghanistan rather than Ukraine. What the Afghan people think about that is not considered too much.
Society is going to utter dogshit and tearing itself apart merely through social media. The US almost had a coup because of organized hatred and lies spread through social media. The far right's rise is heavily linked to lies spread through social media, throughout the world.
This AI has the potential to absolutely automate the very long Photoshop work, leading to an even worse stat eof things. So, yes, "Responsibility to society" is absolutely a thing.
> The US almost had a coup because of organized hatred and lies spread through social media.
But notice how all of these deep faking technologies weren't actually necessary for that.
People believe what they want to believe. Regardless of quality of provided evidence.
Scaremongering idea of deep fakes and what they can be doing was militarized in this information war way more than the actual technology.
I think this technology should develop unrestricted so society can learn what can be done and what can't be done. And create understanding what other factors should be taken into account when assesing veracity of images and recordings (like multiple angles, quality of the recording, sync with sound, neural fake detection algorithms) for the cases when it's actually important what words someone said and what actions he was recorded doing. Which is more and more unimportant these days because nobody cared what Trump was doing and saying, nobody cares about Bidens mishaps and nobody cares what comes out of Putins mouths and how he chooses his greenscreen backgrounds.
Are you of the idea that we should let everyone get automatic rifles because, after all, pistols exist? Because that is the exact same line of thought.
> People believe what they want to believe. Regardless of quality of provided evidence.
That is a terrible oversimplification of the mechanics of propaganda. The entire reason for the movements that are popping up is actors flooding people with so much info that they question absolutely everything, including the truth. This is state sponsored destabilisation, on a massive scale. This is the result of just shitty news sites and text posts on twitter. People already don't double check any of that. There will not be an "understanding of assessing veracity". There is already none for things that are easy to check. You could post that the US elite actively rapes children in a pizza place and people will actually fucking believe you.
So, no. Having this technology for _literally any purpose_ would be terribly destructive for society. You can find violence and Joe Biden hentai without needing to generate it automatically through an AI
I'm sorry. I believe I wasn't direct enough which made you produce metaphor I have no idea how to understand.
Let me state my opinion more directly.
I'm for developing as much of deep fake technology in the open so that people can internalize that every video they see, every message, every speech should be initially treated as fabricated garbage unrelated to anything that actually happened in reality. Because that's exactly what it is. Until additional data shows up, geolocating, showing it from different angles and such.
Even if most people manage to internalize just the first part and assume everything always is fake news, that is still great because that counters propaganda to immense degree.
Power of propaganda doesn't come from flooding people with chaos of fakery. It comes from constructing consistent message by whatever means necessary and hammering it into the minds of your audience for months and years while simultaneously isolating them from any material, real or fake that contradicts your vision. Take a look no further than brainwashed Russian citizens and Russian propaganda that is able to successfully influence hundreds of millions without even a shred of deep fake technology for decades.
The problem of modern world is not that no one believes the actual truth because it doesn't really matter what most people believe. Only rich influence policy decisions. The problem is that people still believe that there is some truth which makes them super easy to sway to believe what you are saying is true and weaponize by using nothing more than charismatic voice and consistent message crafted to touch the spots in people that remain the same at least since the world war II and most likely from time immemorial.
And the "elite" who actually runs this world, will pursue tools of getting the accurate information and telling facts from fiction no matter the technology.
I instinctively want to "flip the sign" on all of the automated controls they put in, just out of the morbid interest to see what comes out. The moment you have a "avoid_harm_to_humans:bool" training parameter, someone's going to set it to -1.
Their document about all the measures they took to prevent unethical use is also a document about how to use a re-implementation of their system unethically. They literally hired a "red team" of smart people to come up with the most dangerous ideas for misusing their system (or a re-implementation of it), and featured these bad ideas prominently in a very accessibly written document on their website. So many fascinating terrible ideas in there! They make a very compelling case that the technology they are developing has way more potential for societal harm than good. They had me sold at "Prompt: Park bench with happy people. + Context: Sharing as part of a disinformation campaign to contradict reports of a military operation in the park."
"One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form."
Yeah, for measures that are subsetting out only the nice data, "flipping the sign" would be picking the other subset. So something like "data_to_train_on = (good_data_split, evil_data_split)[accidental_one_based_index_because_humans_still_cant_agree_on_how_to_count]"
I can't help but feel a lot of the safeguarding is more about preventing bad PR than anything. I wish I could have a version with the training wheels taken off. And there's enough other models out there without restriction that the stories about "misuse of AI" will still circulate.
(side note - I've been on HN for years and I still can't figure out how to format text as a quote.)