One way to tackle the problem is to separate governance from the state, and say that we want a governance structure accountable to the workers, and populated by workers, but no associated state apparatus—no police, no army, no state lines, etc.

Many forms of anarchism turn into a kind of pastorial agrarianism because of this. It really is hard to come up with a governance structure beyond a certain size that meets the accountability requirements for the kind of society socialism wants, and so many socialist anarchists see the answer as the self-governance of communities small enough to manage some kind of general council.

The irony is that as best I gather, the founding fathers were shooting for much the same! (Federalism)

(Of course that was states rather than small towns or something, but states were far less populous then- and Rhode Island is still pretty small!)

This is way, somewhat famously, Noam Chomsky considers his anarchism a straightforward extension of the principles of enlightenment political philosophy to industrial society.

There's a layered irony, of course: most of the founding fathers were members of a political and economic elite which openly favored aristocracy and commerce over the 'regular folk.' Not all of them of course. The American Revolution was a strange tenuous cooperation between an economic elite that saw independence as the way out of a mercantile demand on the part of Britain, and a political class which was versed in Enlightenment political philosophy and saw an opportunity for using Locke and Montesquieu as a handbook for a new kind of government.