I interned at a defense contractor, at a satellite office in a shared space. Everyone (including construction workers, movers, visiting officials, employees, etc) had to be badged and logged past the reception area. And truly restricted areas required swiping in with a proper level badge.
Obviously, it's just an anecdote, but I would be surprised if similar requirements weren't standard for the defense industry.
I had a rather long career in the military doing a huge variety of things, primarily with physical security. We were pretty good at our jobs (the US is generally militarily good at its job), but there were regular security lapses that would make regular people shudder. And this was involving activities with the State Department and TS facilities and materiel. You're right though: It's not like getting into a corporate satellite office. You're gonna need a badge of some kind (or correct-looking paperwork, et al) to BS your way past the guy at the door. But it's most definitely not the kind of thing people think of it.
Obviously, it's just an anecdote, but I would be surprised if similar requirements weren't standard for the defense industry.