I like to keep a slack channel (and saved kibana search) for 500s for this exact reason. System-wide we should have no 500s, and when they happen I like to tackle them immediately. I also have daily reports for other various errors, like caught exceptions, invalid auths, etc just so I can see where things aren't going quite right in case it's indicative of something weird going on.