AP is push based (which actually causes the "every instance has its own set of comments" problem). You can run pullers on a small instance to get a better experience if the remote sides support listing posts, but the standard sync process is no more than receiving HTTPS calls and storing the JSON contents in the right place.

There's some additional overhead (doing HTTPS calls for verifying signatures, for instance) but that information can be cached pretty effectively.

Pushing contents is no more than posting HTTPS calls to every server in your follow lists, and possibly exposing said content in a GET API for pullers, though that's entirely optional.

Mastodon is heavy because of the way the backend is written (I blame Ruby on Rails for tha one), but there are fully featured ActivityPub servers out there that are orders of magnitude more efficient. Mastodon devs prefer the ease of development over performance but that's a choice, not an inherent problem of ActivityPub.

I thought AP was push-based though? Might be wrong of course, and I think stuff like media is its own thing