It's not the Internet Archive's job to be the first one to try to resolve fair use questions like this, and attempting to do so put the organization's orginal core mission at risk.
They should have let someone else take the risk, and continued archiving the internet. That is all that most of their supporters expected of them, for good reason. Their attempt to pivot toward being a generic, universal library was bad scope creep and should have been stopped when it started.
I'd bet that most people who care about this issue are primarily worried about losing the Wayback Machine. If that goes away, a lot of internet history will be lost forever. All those copyrighted works that IA was lending out won't disappear in the same fashion -- they will still be available from other sources.
IMO the Wayback Machine has always been the primary product, and most valuable part, of the Internet Archive. If IA wanted to do other things they should have done them via a separate legal entity to protect the Wayback Machine.
You'd be betting wrong. Even a casual skim of the Archive's blog will show that the Wayback Machine isn't their primary focus, and hasn't been for ~20 years. It may be their most important aspect to you, but the Archive is comprised of a large network of people working together to make all kinds of information available on the internet that wouldn't otherwise be accessible. To those people, those repositories are far, far more valuable than the Wayback Machine (and I promise they care a lot about this issue -- possibly more than HN does, given the timbre this topic has received here).
What I'm saying is that the things the people who work at Internet Archive care about may not be the same as what end users care about. I think the end users care mainly about the Wayback Machine.
Seems to me like the Internet Archive can determine for itself what it's job is, and they've determined that this is in fact part of their core mission.
Your perception that their purpose is solely to archive the Internet is at odds with their actual demonstrated activities for many years (the home page says: "Internet Archive is a non-profit library of millions of free books, movies, software, music, websites, and more"), as well as their public mission statement: "Our mission is to provide Universal Access to All Knowledge."
I've talked to the founder of the IA who told me he doesn't believe the lawsuit puts the IA at risk of having to shut down.
They can, but that doesn't mean they are right. Controlled Digital Lending is not an "obvious" mistake, and contrary to what people on HN keep saying, the emergency library program is not what this case is about. It also does it seem to be the situation that their entire mission is at risk.
They're an archive on the internet, not an archive of the internet. I disagree with your gradualist approach, because while you complain about scope creep private capital subjects itself to no such restraints. The IA is an institution, not a tool.
They should have let someone else take the risk, and continued archiving the internet. That is all that most of their supporters expected of them, for good reason. Their attempt to pivot toward being a generic, universal library was bad scope creep and should have been stopped when it started.