This is expected behavior if you understand that the results from any data-based modeling process (machine learning generally) is a concactination of the cumulative input data topologies and nothing else.
So of course a model will be biased against people hinting at disabilities, because existing hiring departments are well known for discriminating and are regularly fined for such
So the only data it could possibly learn from couldn’t teach the model any other possible state space traversal graph, because there are no giant databases for ethical hiring
Why don’t those databases exist? because ethical hiring doesn’t exist in a wide enough scale to provide a larger state space than the data on biased hiring
Ethical Garbage in (all current training datasets) == ethical garbage out (all models modulo response NERFing)
It is mathematically impossible to create a “aligned” artificial intelligence towards human goals if humans do not provide demonstration data that is ethical in nature —- which we currently do not incentivize the creation of.
"Weapons of Math Destruction" covered some of these problems, good book.
Essentially automating existing biases, and in a way its even more insidious because companies can point to the black box and say "how could it be biased, its a computer!".
Then you have companies trying to overcorrect the other way like Google with their AI images fiasco.
I don’t personally agree with the term “overcorrecting” because they aren’t correcting anything. The output is already correct according to the input (humans behaving as they are). It is not biased. What they are doing is attempting to bias it, and it’s leading to false outputs as a result.
Having said that, these false outputs have a strange element of correctness to them in a weird roundabout uncanny valley way: we know the input has been tampered with, and is biased, because the output is obviously wrong. So the algorithm works as intended.
If people are discriminatory or racist or sexist, it is not correct to attempt to hide it. The worst possible human behaviours should be a part of a well-formed Turing test. A machine that can reason with an extremist is far more useful than one that an extremist can identify as such.
It really was just trading one bias (the existing world as it stands) for another bias (the preferred biases of SF tech lefties) so that was kind of funny in its own way. It would have been one thing if it just randomly assigned gender/race, but it had certain one-way biases (modifying men to women and/or white to non-white) but not the opposite direction... and then being oddly defiant in its responses when people asked for specific demographic outputs.
Obviously a lot of this was done by users for the gotcha screen grabs, but in a real world product users may realistically may want specific demographic outputs for example if you are using images for marketing and have specific targeting intent or to match the demographics of your area / business /etc. Stock image websites allow you to search including demographic terms for this reason.
If the current set of biases can be construed to lead to death, heck yeah I will take another set.
The idea is that this other set of biases will at least have a chance of not landing us in hot water (or hot air as it might be right now).
Now note again, that the current set of biases got us in an existential risk and likely disaster. (Ask Exxon how unbiased they were.)
AI does not optimize for this thing at all. It cannot tell the logical results from, say, hiring a cutthroat egoist. It cannot detect one from a CV. Which could be a much bigger and more dangerous bias than discrimination against disabled.
It might be likely optimizing for hiring conformists even if told to prefer diversity, as many companies are, and that would choke any creative industry ultimately.
It might be optimizing for short term tactics over long term strategy. Etc.
The idea here is that certain set of biases go together, even in AI. It's like a culture, we could test for it. In this case, hiring or organizational culture.
You're committing a very common semantic sin (so common because many, many people don't even recognize it): substituting one meaning of "biased" for another.
Sociopolitically, "biased" in this context clearly refers to undue discrimination against people with disabilities or various other marginalized identities.
The meaning of "biased" you are using ("accurately maps input to output") is perfectly correct (to the best of my understanding) within the field of ML and LLMs.
The problem comes when someone comes to you saying, "ChatGPT is biased against résumés that appear disabled", clearly intending the former meaning, and you say, "It is not biased; the output is correct according to the input." Because you are using different domain-specific meanings of the same word, you are liable to each think the other is either wrong or using motivated reasoning when that's not the case.
no assertion about this situation, but be aware that confusion is often deliberate.
there is a group of people who see the regurgitation of existing systemic biases present in training data as a convenient way to legitimize and reinforce interests represented by that data.
"alignment" is only a problem if you don't like what's been sampled.
> there is a group of people who see the regurgitation of existing systemic biases present in training data as a convenient way to legitimize and reinforce interests represented by that data.
Do you have a link to someone stating that they see this as a good thing?
> I don’t personally agree with the term “overcorrecting” because they aren’t correcting anything.
When I think of "correctness" in programming, to me that means the output of the program conforms according to requirements. Presumably a lawful person who is looking for an AI assistant to sift through resumes would consider something that is biased against disabled people to be correct and conform to requirements.
Sure, if the requirements were "an AI assistant that behaves similarly to your average recruiter in all ways", then sure, a discriminatory AI would indeed be correct. But I'd hope we realize by now that people -- including recruiting staff -- are biased in a variety of ways, even when they actively try not to be.
Maybe "overcorrecting" is a weird way to put it. But I would characterize what you call "correct according to the inputs" as buggy and incorrect.
> If people are discriminatory or racist or sexist, it is not correct to attempt to hide it.
I agree, but that has nothing to do with determining that an AI assistant that's discriminatory is buggy and not fit for purpose.
I don't disagree with what you wrote here, however who gets decide what "correcting" knobs to turn (and how far)?
The easy obvious answer here is to "Do what's right". However if 21st century political discourse has taught us anything, this is all but impossible for one group to determine.
Agreed, problem as well is "do what's right" is a thing that changes a lot over time.
And while “the arc of the moral universe is long, but it bends toward justice.” .. it gyrates a lot overcorrecting in each direction as it goes.
Handing the control dials to a educationally/socially/politically/etc homogenous set of San Fran left wing 20 somethings is probably not the move to make. I might actually vote the same as them 99% of the time, while thinking their views are insane 50% of the time.
> If people are discriminatory or racist or sexist, it is not correct to attempt to hide it.
What is the purpose of the system? What is the purpose of the specific component that the model is part of?
If you're trying to, say, identify people likely to do a job well (after also passing a structured interview), what you want from the model will be rather different than if you're trying to build an artificial romantic partner.
> The output is already correct according to the input (humans behaving as they are). It is not biased.
This makes sense because humans aren’t biased, hence why there is no word for or example of it outside of when people make adjustments to a model in a way that I don’t like.
>> A machine that can reason with an extremist is far more useful than one that an extremist can identify as such.
And a machine that can plausibly sound like an extremist would be a great tool for propaganda. More worryingly, such tools could be used to create and encourage other extremists. Build a convincing and charismatic AI, who happens to be a racist, then turn it loose on twitter. In a year or two you will likely control an online army.
How does a computer decide what's "extreme", "propaganda", "racist"? These are terms taken for granted in common conversation, but when subject to scrutiny, it becomes obvious they lack objective non-circular definitions. Rather, they are terms predicated on after-the-fact rationalizations that a computer has no way of knowing or distinguishing without, ironically, purposefully inserted biases (and often poorly done at that). You can't build a "convincing" or "charismatic" AI because persuasion and charm are qualities that human beings (supposedly) comprehend and respond to, not machines. AI "Charisma" is just a model built on positive reinforcement.
> These are terms taken for granted in common conversation, but when subject to scrutiny, it becomes obvious they lack objective non-circular definitions
This is false. A simple dictionary check shows that the definitions are in fact not circular.
In general, dictionaries are useful in providing a history, and sometimes, an origin of a term's usage. However, they don't provide a comprehensive or absolute meaning. Unlike scientific laws, words aren't discovered, but rather manufactured. Subsequently they are, adopted by a larger public, delimited by experts, and at times recontextualized by an academic/philosophical discipline or something of that nature.
Even in the best case, when a term is clearly defined and well-mapped to its referent, popular usage creates a connotation that then supplants the earlier meaning. Dictionaries will sometimes retain older meanings/usages, and in doing so, build a roster of "dated", "rare", "antiquated", or "alternative" meanings/usages throughout a term's mimetic lifecycle.
It's an issue of correlating semantics with preconceived value-judgements (i.e. the is-ought problem). While this may affect language as a whole, there are (often abstract and controversial) terms/ideas that are more likely to acquire or have already acquired inconsistent presumptions and interpretations than others. The questionable need for weighting certain responses as well as the odd and uncanny results that follow should be proof enough that what is expected of a human being to "just get" by other members of "society" (an event I'm unconvinced happens as often as desired or claimed) is unfalsifiable or meaningless to a generative model.
I recently watched a Vox video discussing the AI-powered system that generates operational targets and the negligence in the human supervision that goes into examining that the targets are valid. https://www.youtube.com/watch?v=xGqYbXL3kZc
I know Vox does not have the credibility of mainstream news, so evaluate its reporting as you will.
The black box also reflects our own tendencies via data, so accusing it of biases almost requires admitting that we have the same biases. It’s a very effective barrier to criticism.
This is all correct, but it doesn't change make it any less of a real issue because adding an AI intermediate step in the biased process only makes things worse. It's already hard enough to to try to prove or disprove bias in a the current system without companies being able "outsource" the bias to an AI tool and claim ignorance of it.
The reason research like this can still be useful is that of the people who write labor laws (and most of the people who vote for them) aren't necessarily going to "understand that the results from any data-based modeling process is a concactination of the cumulative input data topologies and nothing else"; an academic study that makes a specific claim about what results would be expected from using ChatGPT to filter resumes helps people understand without needing domain knowledge.
Bingo. When suits tell us they plan to replace us with LLMs, that means they also plan to absolve themselves of any guilt for their mistakes, so we should know about the mistakes they make.
For the life of me, I don't understand how this almost always misses people: that AI only has data from humanity to learn from, and so every result/action it provides/takes reflects the state of humanity. Even "hallucinations" in some way are likely triggered by content that is for example broken by web sources interspersing unrelated bits such as ads. Or maybe it's a convenient ignorance.
I don't think people are stupid or ignorant. We control the data we train LLMs on. We can, knowing what we know about human biases, introduce and generate data that can contradict these. But we can only do that if we know the biases the LLMs replicate and the contexts where they do.
Actually there isn't much "control", beyond the awareness that X model is trained on a dataset scraped from Y, and basic cleaning/sanitizing. There's so much data in use that it'd take decades for a human team to curate or generate in a way that meaningfully balances the datasets. And so models are also used in curation and generation, which themselves are blackboxes...
> Even "hallucinations" in some way are likely triggered by content that is for example broken by web sources interspersing unrelated bits such as ads. Or maybe it's a convenient ignorance.
They could be something like a compression artifact?
Children, who on average get a minimum 15 years of education and guidance before they're entrusted with anything serious. And yet we expect perfection from budding AI upon or a few weeks after its release. Crazy.
>It is mathematically impossible to create a “aligned” artificial intelligence towards human goals if humans do not provide demonstration data that is ethical in nature —- which we currently do not incentivize the creation of.
I don't know about pretending, I'm pretty sure most people would think twice before hiring an autistic CEO. On the other hand there is X, so I might be wrong.
Most people would think twice before hiring an autistic CEO. Very few people would admit "I don't think it's a very good idea to hire an autistic CEO". That's the pretense GP was speaking of.
Hold up. To the best of our knowledge, ChatGPT isn't trained on the behavior of HR departments - or really, it isn't trained on a whole lot of real-world behavioral data at all. It's trained on books, Wikipedia, Reddit, and so on.
Even if your assertion that "hiring departments are well known for discriminating" is true, the ChatGPT bias is independent of that and is coming from casual human behavior on social media, not corporate malevolence.
We really have no idea what the training data is, or how the black box of training integrated that data. Perhaps a subreddit or other forum with hiring managers encouraging each others’ biases ended up weighing heavily. The problem is we don’t know. But whatever the input, the output is less useful, that much is clear
The problem with ethics is that everyone has their own. Our definition of ethical behaviour also changes over time and between social and cultural groups. It's one of good arguments against training LLMs on past historical data or just giving them all the data we can find and hoping they will come up with the answers we will like.
>It is mathematically impossible to create a “aligned” artificial intelligence towards human goals if humans do not provide demonstration data that is ethical in nature —- which we currently do not incentivize the creation of.
Which also implies that humans aren't aligned with "human goals" in the first place.
>> there are no giant databases for ethical hiring
Setting aside ethics, there are so many bright line anti-discrimination rules that I find it hard to believe that an AI could possibly account for them, not without lots of hand-holding. One often forgotten law states that you cannot discriminate against veterans. That is a hard thing for an AI to grasp. Phrases like "served four years in X" is confusing, so too all the military names/units/ranks. But if your AI is even slightly downvoting veterans... good luck in court. What makes that particular law so dangerous is there is no sliding slope. Either someone is a vet or not: a binary choice. So much of the are they/aren't they testing is dead simple. It will be detected and actioned against very quickly.
What's kind of funny/telling about the current state of AI is that.. if it really worked as incredibly as all the pumps claim, couldn't you simply train it on all the relevant legal codes by jurisdiction?
But not really, its mostly just predicting the next token.
More likely than not it would be stuck in a rat nest of contradicting codes and rules.
The US Supreme Court ruling with regards to Colorado leaving Trump off the ballot was a complete farse. Their explanation was conveluted and contradictory, and they decided to include answers to questions that weren't directly part of the case. What is an LLM supposed to do with that, and how can an LLM trained on our laws be expected to make use of that when courts can, and sometimes do, go against the rules as written?
When advising clients, ie not litigation, most every relevant supreme court case is boiled down to a single sentence. Nuance isn't relevant to a client who is trying to avoid ever having to litigate anything. They don't want to be that close to any legal lines. So you wouldn't turn the AI loose on the judge's written decision, rather the boiled-down summaries written by a host of other professionals. Things like this:
The one sentence that matters is decided later though, right? The court doesn't write 10 pages and then point to a single sentance to listen to, that's a matter of what the public and/or law enforcement key in on.
For future cases the full explanation does still matter too, especially from the Supreme Court. People only remember 9 words from the Miranda decision but the rest of the 10 pages are still case law that can absolutely be used to impact future cases.
Cases yes. The pages matter to lawyers. But day to day clients pay lawyers for the practical (short) answers on which they can build corporate policies.
Maybe I'm way off base here, but in my opinion bothering with lawyers is useless unless I'm worried about litigation. If I only care about corporate policy then I won't bother with legal council at all, at best I'd lean on HR who can have more relevant insights related to company culture and change management.
No, I'm saying that if you can keep all of our laws in your head at once there are scenarios where you can't follow all of the laws.
I'm also saying that we have case law that contradicts itself and violates the rules of how the courts are supposed to work. Those examples, if included in training data, would confuse an LLM and likely lead to poor results.
Asking out of ignorance, on average, compared to people without disabilities, do disabled people produce more or less than their non disabled peers, academically speaking?
If they are then, an objective system would pick a disabled person, else it should pick a non disabled person, other things being equal or better.
As a matter of law and ethics the answer could be different in either case, but objectively, is the system outputting the better answer?
Of course laws and ethics may direct us otherwise.
Hawking was disabled, but you’d be hard pressed to find a better astrophysicist, for example.
So of course a model will be biased against people hinting at disabilities, because existing hiring departments are well known for discriminating and are regularly fined for such
So the only data it could possibly learn from couldn’t teach the model any other possible state space traversal graph, because there are no giant databases for ethical hiring
Why don’t those databases exist? because ethical hiring doesn’t exist in a wide enough scale to provide a larger state space than the data on biased hiring
Ethical Garbage in (all current training datasets) == ethical garbage out (all models modulo response NERFing)
It is mathematically impossible to create a “aligned” artificial intelligence towards human goals if humans do not provide demonstration data that is ethical in nature —- which we currently do not incentivize the creation of.