How about send them to a psychiatric hospital which is equipped to deal with their situation, rather than locking them in a box? If someone is so mentally unstable that they cant handle the human contact of prison gen pop, maybe a psychiatric hospital is where they actually belong.
I think that'd be a good move, but it wouldn't address some of the violent offenders, and currently our mental health network isn't at the scale it should be to handle the problem. We absolutely should do that over the long term, but it requires a fundamental policy shift, and a willingness for governments to spend money on a group of people who basically have no political voice out of the recognition that it's the right thing to do.
It's not the only component of the problem of prison population control, but deinstitutionalization under Reagan absolutely punted the mental health issue to the prison system.
I worked in the criminal justice system in an inner city for a medium sized metro area. We'd have cases where a guy would repeatedly break into a business, not to steal anything of value, he was just schizophrenic and felt he needed to be in there some nights. Turning him back on the street perpetually screws over the business owner, incarceration without treatment risks ruining his already fragile life. There weren't an abundance of good options, and everyone in the courtroom, prosecution, judge, public defender, they all recognized it.
This was an extreme example, but mental health issues weren't atypical. It wasn't the majority, but a substantial fraction (guessing here, maybe 10-20%?) of the cases I saw involved some underlying mental health issues.
The criminal justice system is just at its worst with these defendants too. For serious crimes, juries just never find them sympathetic, even if their mental health issue is not sociopathy, but something like down's syndrome, it's basically impossible to get a sympathetic jury, despite the fact that it's supposed to be a "jury of peers."
We really need mental health courts. We need to recognize, like we did with drug courts, that some of these cases just aren't a normal criminal justice issue. They have their own special issues that require a special set of expertise and special policy options. We also need mental health facilities or in home monitoring to handle the problem.
That all means massive amounts of money, but it would be money well spent on social justice, and some costs would even be recouped by lowering the burden on the rest of the system, which is an ill fit for this social problem.
On the other hand, there's only so far you can go with this. The Supreme Court has found that prisoners have a right to refuse treatment (with certain exceptions), and many do. Other violent offenders don't have any particular mental illness, just take every opportunity to defy prison rules. Some prison rules are arbitrary, but some aren't.
I still think guards need a robust range of tools at their disposal to control behavior, but I'd be all for much greater oversight and access to appellate review of disciplinary decisions. I'm glad cops in California are experimenting with body cameras, maybe guards will start being required to wear them too. Less wrongful liability for the prison, less behind the scenes violence against inmates, maybe everyone wins.