My advice from years of notarizing my apps is to make sure you do it at least once per day for each of your apps. If you only notarize once every release (say, every month or so), you are almost guaranteed to encounter some new cryptic error that you've never seen before, either due to some glitch in signing your app or frameworks, or else some server-side error such as new terms & conditions that you are being "encouraged" to agree to. It will take you hours to research and resolve them if they aren't spotted right away.
As others pointed out, https://github.com/mitchellh/gon is a great tool for doing this on your local machine (e.g., with a cron job). In addition, if you are building your app using a GitHub action (which I highly recommend if it is open-source), you can use my https://github.com/hubomatic/hubomat action to package, notarize, and staple a release build in one shot. The sample/template app does this automatically on every commit as well as once per day: https://github.com/hubomatic/MicroVector/actions.
So when this fails from a scheduled job, you at least know that something has changed on the Apple side and can investigate that right away. And if it fails as a result of a commit, then at least you can start looking at what changes you may have made to your entitlements or code signing settings or embedded frameworks or any of the other million things that can cause it to fail.
I auto notarize my app when I push to a release-candidate branch even if I don't deploy it. I also have a general code signing CI test that catches stuff before notarization would. I believe I've never, in thousands of pushes to this branch, hit a non-obvious notarization issue.
The main annoying thing so far for me using notarization long term is the terms and conditions signing step, which is silly because they're only updating the paid apps contract and we're notarizing explicitly so we can distribute outside the app store.
Smoke-testing your code signing is a good idea, and would probably catch most notarization issues. Aside from those, through, I've encountered numerous issues with embedded frameworks and app extensions whose error reporting wouldn't be described as obvious. Catching those right away rather than right before you are trying to deploy a release is critical.
I check that executables don't depend on libraries from outside the app, I check that I successfully shipped everything as universal2, and I check for stuff like .DS_Store and vim .swp files.
As others pointed out, https://github.com/mitchellh/gon is a great tool for doing this on your local machine (e.g., with a cron job). In addition, if you are building your app using a GitHub action (which I highly recommend if it is open-source), you can use my https://github.com/hubomatic/hubomat action to package, notarize, and staple a release build in one shot. The sample/template app does this automatically on every commit as well as once per day: https://github.com/hubomatic/MicroVector/actions.
So when this fails from a scheduled job, you at least know that something has changed on the Apple side and can investigate that right away. And if it fails as a result of a commit, then at least you can start looking at what changes you may have made to your entitlements or code signing settings or embedded frameworks or any of the other million things that can cause it to fail.