This is really the limits of statistical inference. A prime example is a cancer detection AI was really just detecting rulers in the photos [1].
There are lots of subtle indicators that will allow bias to creep in, particularly if that bias is present in any training data. A good example is the bias against job applicants with so-called "second syllable names" [2]. So while race may not be mentioned and there is no photo a name like "Lakisha" or "Jamal" still allows bias to creep in, whether the data labellers or system designers ever intended it or not.
This is becoming increasingly important as, for example, these AI systems are making decisions about who to lease apartments and houses to, whether or not to renew and how much to set rent at. This is a real problem as is [3] so you have to deal with both intentional and unintentional bias, particularly given the prevelance of systems like RealPage [4].
This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.
Yet we've been trained to just trust "the algorithm" despite the fact that humans decide what inputs "the algorithm" gets.
> This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.
99% of humans just follow tradition and couldn’t explain why they do the things they do other than that’s how everyone has always done it even when circumstances have changed and the original reason no longer applies.
I agree, but the legal system is designed that we can set up rules for humans, and punish them. It's hard to imagine how to introduce similar rules for AIs.
My prefered route is I can sue the company if their AI misbehaves, but we are already seeing cases of companies saying "Oh yes, the chatbot said X, and the chatbot is the only way to communicate with us, but that's clearly just the AI being wrong so we will ignore it".
Hopefully some cases will go to court, and side with consumers against companies and their black-box AIs, but I'm not hopeful.
True. I think other humans are better able to point out those cases, because they share context - they have, for instance, witnessed other humans following those detrimental traditions - and know, or collectively create, methods to push back against them. We have legal regimes and cultural mechanisms adapted (not perfectly, I'll grant!) to overcoming harmful equilibria. Humans, as a species, and over many many thousands of years, have learned (not infallibly, for sure!) to deal with human foibles and lapses of judgment.
We have no similar intuitions for dealing with AI "reasoning", and attendant biases. To the extent that AIs are intelligent, they are alien to us. We have no (or very few) valid instincts about them, and they are impervious to our empathy. In fact, empathy - the engine that drives human-to-human cultural progress - is an active detriment in dealing with AI. As a species, we are maladapted to an AI future.
> This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.
Or just don't have the magic box make free-form decisions. Limit it to extracting specific data points (and the RAG stuff that eg bing does seems pretty ok at attributing assertions to where it found them), and then feed those into a traditional explicit calculation.
> This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.
This is basically not possible with deep-learning. Perhaps an alternative is to require organisations using AI systems like this to define policies around how they make their decisions, and then allow consumers to hold them to their policies.
i.e. a policy of not discriminating based on race, and then checking that they don't, and punishing them if they do. They can still use an AI system, perhaps even a racist one if they control for it correctly.
Mandating technological details rarely works, is hard to police, and doesn't keep up with technology. Mandating the outcomes however can work.
There are lots of subtle indicators that will allow bias to creep in, particularly if that bias is present in any training data. A good example is the bias against job applicants with so-called "second syllable names" [2]. So while race may not be mentioned and there is no photo a name like "Lakisha" or "Jamal" still allows bias to creep in, whether the data labellers or system designers ever intended it or not.
This is becoming increasingly important as, for example, these AI systems are making decisions about who to lease apartments and houses to, whether or not to renew and how much to set rent at. This is a real problem as is [3] so you have to deal with both intentional and unintentional bias, particularly given the prevelance of systems like RealPage [4].
This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.
Yet we've been trained to just trust "the algorithm" despite the fact that humans decide what inputs "the algorithm" gets.
[1]: https://www.bdodigital.com/insights/analytics/unpacking-ai-b...
[2]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...
[3]: https://www.justice.gov/opa/pr/justice-department-secures-gr...
[4]: https://www.propublica.org/article/yieldstar-rent-increase-r...