Apparently it can be really difficult. I had a fall in my house onto an outstretched hand, my doc thought it was broken so she sent me for X-rays. She also told me that it is really easy to miss scaphoid fractures, so if it still hurts in two weeks to come back for more X-rays because sometimes the fracture can be easier to image as it heals. They also said that might not work and if I am still in pain without a diagnosed fracture at week 6 that I would need an MRI.

Thankfully it wasn't broken at all, it just hurt like hell for a week.