I think there are two problems with requiring explainable models:
Explainable models will be more easily gamed, and they are likely to be less accurate.
The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A and B. They go and fix reason A and B, but in the mean time, so have many other credit seekers, diminishing their predictive value. In that time, the modelers create a new, different explainable model, that no longer uses A and B, but somewhat related predictors C and D, which haven't yet been used, and so haven't yet been gamed, which the original seeker does not meet?
Explainable models, being a subset of all models, are likely to not contain the most accurate models. I don't anything about the domain of credit scoring (maybe state of the art models are actually small and understandable?), but in speech recognition, for example, models are constantly growing in complexity, their individual decisions are way beyond explainable to anyone in reasonable amount of time, and they are only getting more powerful as they get larger and more complex. In speech, models are already many gigs. In credit scoring, less accurate models mean higher rates, so there is an aggregate loss.
A fair point, but as a society, we have decided that racial discrimination is not a valid mechanism for banks to profit by. That does result in everyone paying a bit more in interest as the risk pool is larger, but a acceptable tradeoff.
In terms of gaming, verification is just as important as scoring If the data you have going into to the system is rigged, and income is not being properly validated, bad things will happen.
As a society we have directed banks to make bad loans to blacks and charge non-blacks extra to make up the difference? I'd be surprised if even 10% of people know this decision was made.
Also, what makes it acceptable to engage in this form of surreptitious wealth redistribution on racial lines?
Not being a racist makes it acceptable to not take race or a surrogate for race into condition for a loan ;-)
(Please don't take that the wrong way. I am not accusing anyone of racism. Simply stating that at some points our ideals is more valuable then an additional point of profits for the bank).
Disparate impact and it's use in credit scoring is mostly governed by Equal Credit Opportunity Act (ECOA), but most of the banks I am aware of go steps further in ensuring that disparate impact does not occur.
This is only important if you believe that racial stereotypes are true, but should be ignored. That if you control for education, income, region, etc, differences between races are still significant.
Disparate impact goes well beyond removing race as a feature. You sometimes can't use features that correlate with race, even if they are highly predictive. E.g. education.
It also has nothing to do with the profits of banks. Better prediction algorithms for loans mean less interest rates for people that are good borrowers, and less crippling debt to those that aren't. It has huge benefit to the economy and society.
Explainable models will be more easily gamed, and they are likely to be less accurate.
The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A and B. They go and fix reason A and B, but in the mean time, so have many other credit seekers, diminishing their predictive value. In that time, the modelers create a new, different explainable model, that no longer uses A and B, but somewhat related predictors C and D, which haven't yet been used, and so haven't yet been gamed, which the original seeker does not meet?
Explainable models, being a subset of all models, are likely to not contain the most accurate models. I don't anything about the domain of credit scoring (maybe state of the art models are actually small and understandable?), but in speech recognition, for example, models are constantly growing in complexity, their individual decisions are way beyond explainable to anyone in reasonable amount of time, and they are only getting more powerful as they get larger and more complex. In speech, models are already many gigs. In credit scoring, less accurate models mean higher rates, so there is an aggregate loss.
[1] https://en.wikipedia.org/wiki/Campbell%27s_law