This is true, but it is a problem that currently exists.
For orthogonality's sake, I would suggest that if most of your posts come from 1:1 conversations, post by post, then perhaps there is a better medium to communicate with that person than through a board like HN.
The people that I have those conversations with are people I only know through HN, and the discussions come out of what happened here. Those discussions would not have naturally happened in another medium.
As an example I give you this discussion. In this thread I have 2 posts, both with 1 karma right now. I do not expect them to go up much if at all. Where would you suggest that I should have gone for this conversation with you?
EDIT: I changed the evaluation to not make the avg karma relative to total karma but instead to the median value of average karma of all users. The downside to this is that Median(AvgKarma) would need to be calculated quite often throughout the day because I doubt pg wants to pay for the resources to run that everytime someone votes.
Whether or not this is a good thing is a matter of preference. (My preference should be obvious.)
Though, truth be told, I've never worried much about karma.