> And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs.
Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
...shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
I've written loads of both, and I realize the thread had already started down this path, but how is a shell script really different from a python script? If you're targeting Bourne, your shell isn't a separate installation, but it isn't onerous to install your favorite python version on machines you control. Sure, python has an ecosystem of libraries that shell doesn't have, but you don't have to use those libraries if you don't want to. What terrible consequences does someone who prefers python to shell scripting face?
> What terrible consequences does someone who prefers python to shell scripting face?
Terrible consequences? Probably not. Shell script dependencies are easier to handle, because the commands you're relying on are built right into the OS. They're going to be there because they have to be. How often does sed, grep, awk, and cut change?
It's a matter of complexity. If you're munging files and using regexps, shell is fine, and quicker than Python. If you're starting to think "I need a set" or "I need real error handling", it's about the time I move to Python or Perl.
> called Rocks (built in Python and shell scripts)
Regardless of if it "worked", I think mixing languages in an application is a bad idea, and should be avoided whenever possible. There are a few reasons,
1) maintainability - you're requiring anyone to know more things to maintain the application
2) Attack surface - lots of security bugs come in between boundaries between tools
3) Correctness - it's harder to automate testing
Did you have any sort of automated testing?
Just because shell scripts can do a lot of heavy lifting, doesn't mean they should.
Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs.
Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.