There's a difference between observing how a comment was generally received and remembering exactly how each person reacted. I'm not saying nobody does the latter, but I don't.
And the other way, one can notice how a person reacts to a comment without necessary needing to remember the source of it. Both of these are going to scale linearly (obviously).
I might pick up on something like "A seems to respond to B uniformly negatively" and make a note of that, but remembering one notable thing doesn't require me to "allocate" memory for all of the other non-memorable things. I'm not literally building and labeling a graph, like some commenters are implying. Or if you insist, it's a sparse data structure.
