To a large extent that depends on your definition of empire. If an 'empire' is defined as a multi-billion dollar company then I agree with you, if it is anything over 10 million then there must be 10's of thousands of them.
I'm pretty sure my own company does not qualify as an 'empire' by any standard, but it keeps me free, off the streets and pays the bills, I couldn't wish for more. I wouldn't exchange it for a boring job in investment banking for whatever compensation you put on the table, there's more to life than money, as I've found out the hard way.
I am not after financial independence the easy way. I could never work for an investment bank. It's boring. And even if you're paid $200K a year at a bank, you're still a pawn. Nothing more than a mere cog in a big machine, unable to decide your own future. I don't advocate that. I am only stating that there are tradeoffs. You can trade money for freedom. You can trade excitement for safety. You can be a sheep and be bored, you can be a wolf and be lonely.
In any case, there's nothing particularly noble about empire per se. Sure, it pleases the ego. But there's always a bigger empire. And empires require HR departments and middle managers and other such distasteful things. You don't need to be the captain of an aircraft carrier. It's also fun to run a pirate ship.
I'm pretty sure my own company does not qualify as an 'empire' by any standard, but it keeps me free, off the streets and pays the bills, I couldn't wish for more. I wouldn't exchange it for a boring job in investment banking for whatever compensation you put on the table, there's more to life than money, as I've found out the hard way.