well I think the biggest problem in authorization is definitv if you need object access control and implement lists, it makes things really really tricky and there is basically no good standard for it.
resource-based rbac is hard, extremly hard and it can easily become either a mess or a performance bottleneck.
I agree with your points. We have adopted a resource-based RBAC/ABAC and a policy language in simple YAML to address some of these challenges. In terms of performance, a side car architecture with SDKs give good response times. In such simple architecture the implementation of lists is the current challenge we are addressing. However, for majority of use cases abstracting the decision making logic to a centralized service and providing a simple API that addresses the question of can this principal, do this action on this resource with a true or false answer goes a long way.
If you're interested in resource-based rbac [1] with list endpoints [2], Oso supports both! You're right, it's a tough problem, we've invested a ton of time to make this work well without needing to rearchitect your app.