The trade off is the kernel decides what is paged into RAM and what stays on disk. Less control, but also less overhead. This is a similar approach to Squid.
I don’t think users are going to be happy with that approach in the long term. It’s well advised to control eviction behavior yourself. (Varnish learned this lesson ages ago, notwithstanding phk’s original paper.)
If you're running a Redis instance as the only thing on the VM instance it's on (very likely in my experience), is there a reason to reimplement all this policy logic in Redis, rather than simply tuning the caching policy of the existing OS kernel to make it perform well for Redis (maybe at the expense of performing well for any other process you might run under that same kernel)?
In theory that makes sense, but I'm not aware of any mechanism that allows you to tune a kernel's VMM to suit an arbitrary workload/access pattern. Are you?
Certainly we can do smarter things if we know an access is going to result in a page. But we're not in a position to take advantage of that until green threads are in place.
... which is great until a consumer or replica decides to replay a topic/partition from the beginning, and forces all the recently-written data to be evicted, adversely impacting performance for all the well-behaved consumers :(
Can you clarify what you mean by underpowered? Not every site wants to replace terabytes of disk with NVMe storage to make paging faster - especially sites that have a lot of throughput and long retention demands.
Second, even if quotas could be used - and there are plenty of sites where they cannot - there’s still a broker-replacement scenario that needs to be accounted for. Most folks don’t want to intentionally impede the recovery rate of a failed broker.
