I've read that clause and at first sight it would seem that it too is a restriction but clearly it was not interpreted that way historically or when it is being cited. How is there so much consensus in the interpretation?
The simple answer is that the Supreme Court of the United States said as much. Lower courts are bound to follow SCOTUS' holdings. Of course, precedent can always be overturned (i.e. if SCOTUS changes its mind), but the N&P Clause's interpretation is unlikely to be changed.
