It is a good point but I think one part is a bit backward in an ironic way. One reason AI hasn't replaced radiologists is because radiologists are typically very good physicians, and specifically do not look at images in isolation but review the record, talk with the physicians, sometimes the patients too, etc. So its actually backwards (in some cases) -- AI struggles because it looks at the image in isolation, while the radiologist is looking at the patient more holistically.
Radiologists don't talk to patients (usually), so there's no reason why AI cannot be given all the same patient data. ...although reading doctors' notes is probably a whole 'nother AI program.