I think part of it is a marketing and documentation issue. I have been working to help get beanstalk into a better place specifically around ruby tooling and overall documentation. Check out https://github.com/nesquena/backburner for my solution for a 'resque' inspired DSL for using beanstalkd. We are using it in multiple projects. Also, I have been working to improve the ruby beanstalk client. Expect more news soon.
I used Beanstalkd for a few years and I can testify that it's an awesome piece of software.
That said, that was before Resque and Sidekiq existed. I now use Sidekiq for my new project since I'd rather not reinvent the wheel. I suspect most people think the same way.
The collaboration and navigation tools of Beanstalk are not up to par, last I checked. The file browsers on GitHub and Bitbucket are very responsive. The "pull request" feature is extremely useful for managing large teams where features are being developed concurrently. Beanstalk's best feature is its automatic deployment, but you can duplicate this in a more powerful way with post-commit hooks or continuous integration processes.
