It depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that.
If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.
I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text.
But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.
What do you think of Bolukbasi's approach that's mentioned in the article? In short, you let a system learn the "current state of the world" (as reflected by your corpus), then put it through an algebraic transformation that subtracts known biases.
Do you consider that algebraic transformation enough of a "morality system"?
I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.
> put it through an algebraic transformation that subtracts known biases
> Do you consider that algebraic transformation enough of a "morality system"?
I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.
Yes, every step of machine learning has potential bias, we know that, that's what this whole discussion is about. Nobody would responsibly claim that they have solved bias. But they should be able to do something about it without their progress being denied by facile moral relativism.
If we can't agree that one can improve a system that automatically thinks "terrorist" when it sees the word "Arab" by making it not do that, we don't have much to talk about.
A black box neural network attempting to draw inferences from a human-biased dataset - potentially even more biased because it can't understand subtexts - and then verifying that conclusion through an ad-hoc set of "morality checks" entirely independent from how it reached the conclusion sounds like a recipe for disaster.
That's even before the marketing people get involved and start claiming the system is free from human biases...
I'm not sure what your objection is with regards to the independence aspect. Why would having the morality checks integrated into the "learning about the world" part be better?
If you had an unwavering moral code which dictated that men and women should be treated equally, for example, why would it matter which facts are presented to you, in what order, or how you process them? Your morality would always prevent you from making a prejudiced choice, in that regard.
Frankly, I'm not sure the "men and women should be treated equally" instruction is even possible if the data isn't processed in a way which specifically controls for the effects of gender (some of which may not be discernible from the raw inputs).
Sure, it's theoretically possible that an algorithm parsing text about medics' credentials that (e.g) positively weights male names and references to all-boys' schools and negatively weights female names and references to Girl Guides will be on average fair after an ad hoc re-ranking of all its candidates to take into account the instruction to treat male and female candidates equally. It's just unlikely to achieve this without completely reorganizing its underlying predictive model
[1]there's an interesting parallel to ongoing human arguments about how a machine should follow its "morality checks" should do this: does it ensure the subjects are "treated equally" in terms of achieving 50/50 gender ratio irrespective of the candidate pool (thus potentially skewing it massively in favour of the side with the weaker applicants), does it try to weight results so gender balance reflects historic norms (thus permanently entrenching the minority)? Or does it try to be "gender blind" by testing all its inputs for whether they're gender biased and normalising for or discarding those which are, which is basically learning everything again from scratch...
Even if a categorization is true in a trivial sense, what generally isn't reported and thus readily inferred from fairly naive text-parsing algorithms is significant. People generally don't bother stating a perpetrator (or indeed a victim or possible witness law enforcement hopes to contact) is $majorityrace in most countries' crime reports, for example.
Suppose that the system "learned" that marriage consisted of one man and one (or very rarely more) woman. Would that be "reality"? (In fact, I'd rather bet that it did, and I congratulate the authors on the wisdom of not advertising that fact.)
Various behavioral accidents can easily become embedded in culture, laws, and, yes, programs, at which point it stops mattering if they represent reality or "reality"; the real world will happily follow the cultural construction.
The machine can learn that someone named "Jamal" is more likely to be associated with the word "perpetrator", particularly in corpora centered on American news.
This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.
It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.
Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.
Not answering for the parent: a fact is any instantaneous snapshot of reality. A stereotype is misapplying properties of specific and limited context to a universal scope.
If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?