It varies from case to case, but you should keep doing it as long as it's the best option. Though it's hard to gauge, if the expected outcome is better than what you'd get for starting a new startup or working a desk job, then you should keep going at it.

1000 users is pretty good, in my opinion. They could help you morph your startup into something better. But don't give up on the startup, just change it around a little.