Right but do you feel you have to read, to pick a slightly less fraught example, the white algorithms book? Or Patterson and Hennessy? Like TAOCP, those work perfectly well skimmed or read in bits and parked on your bookshelf or in your bathroom.
GEB (or Moby Dick) are intended to be read in entirety.
I assume you mean CLRS when you say "the white algorithms book" - if so, I'm not sure I necessarily agree that any of these books are meant as reference works or "not intended to be read entirely". They're textbooks, of course, so there's something of an expectation that you'll read them over the course of months or years, but I do believe that all three of your examples are _meant_ to be read cover to cover rather than used as a reference book like a dictionary or a thesaurus or even a cookbook; if you pick up any of these books and start at a random chapter (much less a random section) you're likely to be lost. You might use them as references once you've finished them, but you are expected to start on page 1 and finish on page 1,892 (or whatever insane page count these books have now).
This 'reference' business really seems to stick in people's craws for some reason. I don't mean it disparagingly nor do I feel 'reference' is limited to the style of cookbooks and dictionaries. Nobody writes a multi-volume (or monolith-sized) technical work without the intent it also be used as reference. The work might have other purposes and even a different stated primary purpose - that doesn't make it not-a-reference.