I mean, it's been identified as aircraft by pilots nearby. And it has the ability to disguise itself from radar. That's all you really need to conclude spy plane. I know that people had made sightings of previous US planes before they were declassified. A slightly scarier possibility is that its not American.
ATC primary radar is pretty weak. Not showing up (particularly after changing flight direction) doesn't mean much. It's very odd (and illegal) for something to be up in flight levels without a transponder though.
