Why would an organization with the goal of being a digital "library" care about robots.txt?
Would you expect public and private libraries to discard arbitrary books of a publisher from their collection because the author put a file called libraries.txt on their website?
If your public library found your journal laying on the floor would you expect them to put it on their shelf, even if you had a message inside the cover that said "not for reproduction"? Would they eschew blame if you'd shared this journal with friends?
The answer is no. You wouldn't. Some people put things on the internet but don't want them slurped up by some mindless machine, and some people post both public content and content which they don't want handled by a machine. It's a fair ask that the IA respect long-trodden patterns of the web.
The is no mindles machine, only computers automating work for humans. If you don't want things on the public web then noone is forcing you to put them there. Once you put things online accessible to anyone then things will get archived and shared. Deal with it.
Why would an organization with the goal of being a digital "library" care about robots.txt?
Would you expect public and private libraries to discard arbitrary books of a publisher from their collection because the author put a file called libraries.txt on their website?