Ammonia is more comparable to natural gas or propane than oil.
Also crude oil is really toxic. It off gasses all sorts of questionable things, and touching it isn't recommended. I get what you mean by your comment, but oil isn't some super stable, safe material.
Oil burns and CH3 explodes easily, but we use it anyway because it makes economic sense.
The same will happen with ammonia. There are risks, but the energy that you could store is immense at a very low price.
Don't get me wrong, I was working in batteries and solar panels research in the early days. I heard the same arguments against it( why we should spend money when batteries and solar panels are so expensive and work so badly? They were expensive and worked badly)
Now we work on fuel cells. They are expensive and there is room for improvement.
Except oil is a liquid, while ammonia is a gas. You can put oil in an open container and it won't hurt anyone. Now try doing the same with ammonia...