This is very tricky to do. Let's say that it's the iPhone4 launch day and every story is about that phone. Some users might want all those duplicate posts, to get different perspectives on the story. Others might just be annoyed and want all those duplicates skipped.

I think filtering is probably the way to go in this case. Instead of detecting duplicates the reader should allow you to filter out all stories tagged "iPhone4".