Shell scripting gets a lot of hate, but after spending time to grok the language (and copious use of shellcheck), I've grown quite fond of it. This also has the nice side effect of emphasizing use of old unix tools.
I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays in POSIX shell for me. Stdin/stdout are your friend.
Certainly, shell scripting isn't the answer for everything, but with shellcheck (and bats!), I feel like it's a really reasonable default for systemy things. Heck, the status bar I use in my window manager is essentially just a 700 line bash script, with asyncronous update and everything. It's even quite readable and orthogonally organized, if I might boast. Have been pleasantly surprised over the years every time I go back and read it.
I think one of the biggest problems is the inconsistent runtime environment. I mean, yes, you can just say fuck POSIX and code for Bash only, but then again, not even Bash is consistent across its versions. Just to give an example: The infamous `set -e` behaves quite differently across different shells
And those are just POSIX compliant shells. I mean this doesn't matter if you just want to write a script for yourself but becomes frustrating when you want to level up you shell skills and write cross-platform scripts ;-)
That said, I have to add that I love shell scripting... I am not exactly sure why, but I guess it is because it lets you build powerful things with very few lines of code.
Yes. I have already been burned by it. Some scripts were developped without bashism in order to be portable on posix shells. Initially, they were used on ksh. Recently, they were tested only on bash. I had to use them on ubuntu (where /bin/sh is a link toward dash). Nothing was working and it was hard to identify the issues. I have changed the first line of all scripts -> problem fixed. Portability of scripts is an illusion: all you can do is test on all the target shells. If you do not have budget for testing on other shell than bash, do not waste your time trying to write portable code.
I agree, and I like having those ‘oh wow, I can do that in bash? That’s dangerous, but awesome’ moments just by researching how to get stuff done. Really people don’t know what they’re getting themselves into when they start down the shell scripting path.
When isn't this true, though? JavaScript behaves differently based on your browser and version. C runs differently based on your compiler version and system libs. This is just the nature of software.
I think the particular problem with shells is that POSIX (the standard) didn't really keep up with the development of new features over time. Instead, the different implementations introduced new features and nowadays there are some features which are quite common but not part of the standard (just from the top of my head: curl/wget; mktemp; bashisms).
With JS that is different. Yes, every browser has a few extra bits here and there, but there is very little that many browsers have that is not part of the standard. With compilers, it is different because the developer can decide which one he wants to use at build-time. After that, his program still might behave differently on different platforms (depending on APIs, etc.) but in general, the language that he uses behaves the same. With shell scripts that is not the case as the behavior depends on the run-time shell and the developer has very little influence on which shell will be used.
So while you are right that every programming language has this problem, some are better at managing it than others and shells are particularly bad at it in my opinion.
The problem is that "shell scripting" is "bash scripting", and the focus of that particular outgrowth of the original Bourne shell isn't really focused on that. It was mostly about interactive improvements, and while a few of the bits of added functionality help (like not needing to shell out for every bit of string manipulation), it doesn't contribute a lot to writing longer scripts. Zsh just doubled down on the interactive use. And let's never mention csh again.
Now there was one offshoot that tried to get better at longer scripts: The Korn Shell. In it's later edition (ksh93) it actually had hashes and a pretty good way of expanding its functionality. Heck, dtksh would've given Tcl/Tk a run for its money in the 90s if both ksh and Motif (the library dtksh used for its widgets) weren't proprietary at the time.
I worked on a "devops" system in the early '00s that was used to provision all kinds of Unix servers and workstations (including HP-UX, AIX, Solaris, Linux, BSD..), and the major parts of it was written in ksh93. It worked quite well.
I still think that all of this is inferior to both Perl5 and Tcl/Tk, but ksh93 made some decent strides way back when.
I have a special place in my heart for Perl5, but I would never use it for anything DevOps related... Perl’s motto is “there’s always another way to write it” — and that’s the exact opposite of what you want in a DevOps environment :)
Sorry to say, but I find the criticism in this form to be quite meaningless. What does a motto have to do with any kind of validity for a certain purpose? Python has the "Zen of Python", does this inherently make it a better solution?
"There are many ways to skin a cat" is older than computers, and it applies to basically any programming language. Some languages make it harder to do it in a different manner, some allow it, but don't offer it without including some additional package. C has no hash tables, but it's not that hard to get them, for example.
I'd go so far as to say, that plenty of current programming languages allow more ways to do a thing out of the box than Perl. What mythical language does offer a straight and completely obvious path from problem to solution? Haven't I ascended enough in the halls of programmer-dom to be told of this coding panacea?
And why would this be particularly bad for DevOps? Are DevOps programmers worse and/or more easily confused?
Just recently we had a thread here about different ways to remove duplicates from arrays, and that was done in Golang, devops Holy Grail du jour.
> And why would this be particularly bad for DevOps? Are DevOps programmers worse and/or more easily confused?
DevOps is all about consistency and readability... neither of which are strong points of Perl. Perl was the de-facto cross-platform DevOps language in the 90s (seriously quite a few Mac programs of the day installed Perl as a dependency). There’s a reason the industry moved away from it 15 years ago.
I’ve been building in Perl5 for over 20 years, and it’s really easy to paint yourself into an edge case where things don’t quite work the way you expect them to. Everyone who writes Perl learned it a slightly different way, which makes managing an active codebase of Perl “fun” — it has a tendency to devolve into spaghetti code over time once you get a lot of contributors.
Bats, shellcheck and jq has completely transformed the shell scripting experience for me. To the point where I’ll reach for bash much more often than other languages these days, for anything terminal based.
jq also has some really nice quick interactive features. My favorite super simple one is that jq beautifies JSON by very virtue of just piping it into jq with no additional arguments.
Haven't tried BATS before, but it sounds like the missing part of our bash codebase, which I am currently mostly skeptical of because it isn't unit tested.
I'd fear though that having BATS would just increase the tendency to let a bash script grow from 50 lines through to 200, when it should've just been ported to Python at 100.
Actually, my experience with bats is that it forces you to modularize and keep things small. Easier to test things that way. Bats is truly an indispensable tool when doing shell scripting – I even use it to test CLIs built in other languages too!
Pipeline thinking: This is the pipe-and-filter architectural style. Well documented in the industry.
For pipe-and-filter system management use case, I tend to think that Powershell is the peak of the shell evolution currently, but that is probably my limited insight. When I learnt shell scripting, zsh was the best available :) but I preferred the ksh.
Shell (Bash) scripting have a few huge problems, mostly with error handling and number arithmetic. The arithmetic one isn't a large problem for its niche, but the error handling is so large a bother that it's best avoided for anything complex, even if something else (AKA, Python) will lead to a much larger code.
That said, there are niches where bad error handling isn't as much problematic. Ironically, Bash is much more fit for web programming than for system scripting.
> I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables.
ever since I started dabbling in racket I've grown fond of not dealing with state and look at evreything like a function. (I'm not a functional programmer).
The old UNIX philosophy with pipes is very close to that as in every command in the pipeline is a function in similar way. I'm not saying hacking together a bash script is FP but it's still possible to apply the FP thinking in the UNIX pipeline better than building a monolith that handles the whole pipe. I can pass state around (env vars) but it's an extra effort and it forces me to think whether I can get it done without state first (via optimizing the input output formats).
> [...] the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays [...]
Could you elaborate please? I see that data - especially sequential, array-like data - is conveniently passed via pipes (and e.g. jq https://stedolan.github.io/jq/ uses this concept), but I feel you're saying something deeper or a different kind of use-case... [perhaps because you say "state" not "data")
The sibling comment by @glic3rinu pretty much says it, but I'll elaborate a little bit. Can't make any claims about being anything deep, but in a sense, my shift in perspective came from treating program state as "data in pipes" and doing that as extensively as possible.
A big part of this was figuring out I could the `read` builtin to parse stuff into local variables:
if read -r foo bar baz _; do
# do stuff with $foo, $bar, $baz
# $_ contains anything extra on the line
done
Also, once everything is on stdin, all the standard unix tools Just Work without having to echo and print everywhere, e.g. something like
echo $some_local variable | unix_tool
For me, this turned a lot ugly scripts into primarily a collection of functions that compose nicely via pipelines.
Anyway, using `read` works especially nicely with tabular data too. For example, a naive CSV parser might look like
while IFS=',' read -r field1 field2 field3 _; do
# do stuff
done
By "state" I think OP is referring to an style of programming heavily based on global variables (global state). Function calls set global variables that the caller can read after. Functions and imports (sourcing) are all riddle with "side-effects", making the execution of the program very difficult to follow (spaghetti code).
Because "return values" in bash are implemented by echoing (printing to stdout):
return=$(my_function arg1)
I believe OP has the point that echoing friendly-parsable output is a way of implementing "data-structures" in shells.
result=$(my_function arg1)
result_a=$(echo "$result" | grep A)
result_b=$(echo "$result" | grep B)
After writing a decent backup shell script in fish, I converted it to bash as IntelliJ didn't have support for fish and also non strict Posix compatibility wasn't too comfortable and then realized it's quite a pain to write long script in bash knowing modern languages are far easier to write with, I may just go back to using perl for shell scripting.
All of the later scripting languages aren't really meant for writing shell script as invoking commands is just tedious instead of just using back ticks in perl.
It is very inconvenient, ineffective and slow. For example lists are slow as hell. And the syntax is kind of workaround - look at '[' executable for example. It definitely has its place in CS history and it made things possible in the past but using it to anything more complex nowadays is a bad, bad, ba idea!
I am glad people prefer Python or Ruby for more complicated stuff and Go, Java or compiled langages like Rust, C/C++ for performance-intensive jobs.
Perl is fast but the syntax is terribly non-intuitive and I would mark it 'do not use for new designs'.
I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays in POSIX shell for me. Stdin/stdout are your friend.
Certainly, shell scripting isn't the answer for everything, but with shellcheck (and bats!), I feel like it's a really reasonable default for systemy things. Heck, the status bar I use in my window manager is essentially just a 700 line bash script, with asyncronous update and everything. It's even quite readable and orthogonally organized, if I might boast. Have been pleasantly surprised over the years every time I go back and read it.