Rather than explaining how the machine learning system works as an "explanation", perhaps auditing could be added to machine learning algorithms so that you very much could produce a very long but accurate description of the process, a la pages full of "X was compared to Y, X was larger, and thus we move on to step 261". A bit like disassembling machine code.
Of course, in machine learning, the datasets backing up the comparisons could not be shared as they contain variations of confidential and personal data, but you might still end up with a legally tolerable record of the algorithmic steps involved in the decision making process even if they're not useful to see.
I see what you’re getting at, but this would be a very lengthy and arduous process. Not to mention, many ML and DL algorithms are incredibly mathematically complex that describing them with literal step-by-step detail sounds like hell.
It would be if you had to be involved, but I'm suggesting algorithms could have some sort of instrumentation so such "explanations" could be automatically generated and thrown into a data warehouse for possible future use. (This is all a cynical attempt to meet legal requirements rather than anything actually useful for the user, of course.)
Of course, in machine learning, the datasets backing up the comparisons could not be shared as they contain variations of confidential and personal data, but you might still end up with a legally tolerable record of the algorithmic steps involved in the decision making process even if they're not useful to see.