Catan is merely ok from a serious gaming perspective, but for most people sitting around having fun, it's a go-to. There's value in it looking nice and being easy to understand, and I can see it staying popular 20 years from now. The top listed modern games have only ever come up in niche board game group settings for me, while even chess and Diplomacy have been much more popular. This is only my experience, but I'm probably not alone in thinking the top games are all very uncommon.
I've actually only ever played Diplomacy over Backstabbr.com with friends, so analogous to email/postal play (if you still count that as a board game). Completed several games now, and I've been wanting to do it in-person. That 1900 variant has more detailed notes than any other variant I've seen, and I'd like to give it a try!
I agree with your evaluation of the top listed games on BGG.
Catan can be fun, especially the first few dozen times you play it. But it wears thin even for a casual group. We have better casual games these days. And even back then, we did: I found that eg Bohnanza released about the same time as Catan holds up much better to repeated plays; even though it looks much less impressive at first. The big problem with Catan is that the initial settlement placement determines so much of the rest of the game, and there aren't that many meaningful decisions left in the rest of the game.
I did most of my in-person games of Diplomacy when I was working in Cambridge, Cambridgeshire. I was working for a startup spun off from the local university, and noticed that the university societies were actually (mostly) open to the general public. So I just joined the Diplomacy society.
That's very lucky of you. Diplomacy is my favorite game, and some of my friends like it too, but it can be hard to get 7 people committed to something that might last 2 months (or 4 hours if in-person). I'm sure I can find some club in California for this if I try, but I'm already in too many clubs.
In a 'frictionless vacuum', Diplomacy is one of my favourite games.
After a grueling day playing Diplomacy, I usually had second thoughts. (But that wouldn't keep me from doing it again a few months later.) But I did learn that I prefer play-by-email after all. The games are higher quality, and you can actually finish them.
I've actually only ever played Diplomacy over Backstabbr.com with friends, so analogous to email/postal play (if you still count that as a board game). Completed several games now, and I've been wanting to do it in-person. That 1900 variant has more detailed notes than any other variant I've seen, and I'd like to give it a try!