And then he'll say, that's not what I meant, you're being pedantic, you're addressing individual points instead of the whole issue holistically.

Or maybe it's a personal fight, and you're throwing his words back in his face, and now he realizes he can't trust you with his feelings because you always take him so literally. He's sorry he's not a poet, that he's only human and has trouble expressing his emotions sometimes, but this just isn't going to work between us.

The only person who's ever looked bad when I've used someone's words against them is me.

There is a difference between using someone's verbatim statements when they have been made in an emotionally heightened state of mind (e.g. in a fight), and using them to vindicate one's self when the blame is being assigned, e.g. in a workplace. There is a reason why the old adage of "give it to me in writing" exists.

Also, even though people may say the wrong things when they are emotional, a repeated pattern of them expressing those sentiments can reveal their real disposition. In this way, a record of what is said, word for word, can be useful to uncover someone's real thoughts. I already do this with my brain, but it is easier when it's in searchable, dated text.