Almost 10 years ago, when I just started to work, I had to do a repetitive task. Something in IIS about 30-50 subdomain bindings. I told to myself “I’m not going to do that manually, such a waste of time, let’s write a script!”. 5 hours later, I was still working on this script and I was stuck. I asked some help to a senior colleague and his reply was: “you know, it would have taken you only 10 min to do it manually”. :)
You are so right from a business standpoint. But what about personal development? This is one of the ways developers are born. Sure the first one or ten attempts to automate something might be a net loss to the business, but eventually that junior employee has leveled up their programming skills and developed an estimating sense of when to automate and when to suck it up. It's easy to sit at our place of knowledge and tut at the beginers but we were all beginers once.
I'm an "early" programmer and still poking my way around various toy side projects to learn concepts and such.
Ubuntu and Bash had been somewhat of a mystery to me previously, but I eventually got a virtual dev environment setup and was comfortable getting in with navigating to my Vagrant folder, then executing 'vagrant up', 'vagrant ssh', etc. and then navigating to my project.
That comfort turned to annoyance as I realized how unnecessary those few keystrokes were. And despite how little time they took, I found myself getting angry at the repetition of having to enter them.
So I decided to conquer my Bash scripting fears, and wrote what many would consider to be a trivial script to navigate to the Vagrant folder on my host machine, boot up the VM, and then navigate to the appropriate project folder in the VM. Along the way I learned more about aliases, and also how to pass commands from a host to a guest machine which was one of the roadblocks I encountered.
I now can boot up and get into my project folder with a simple three character alias from within any directory on my host machine.
Again--any experienced developer would be able to script this in minutes most likely. Back then it took hours. And now those are the most satisfying three characters I type, as every time I enter them, I get reminded of the progress I'm making and that even trivial things may be candidates for automation.
I know, right? I'm actually leaning in the direction of making it a little less spartan by having it either accept a parameter for a project name to go directly to that folder, or prompt me to enter a number to select from a list of projects gathered from the folder names in my parent project directory.
Doesn't every developer eventually alias building to 'b' or maybe 'm'? It's like a law of nature. Or you can just rebuild automatically every time you save a file, like js people do.
Just in most cases its not necessarily better to have written the script. You could learn the script in a situation in which the time payoff isnt so terrible.
How can you possibly estimate the time payoff if you've never done it before?
In any case, I think it's more than just the time payoff. Getting repetitive tasks automated allows you to reallocate that cognitive overhead to other details, and that's a payoff that's hard to quantify, but is critical for leaving your mind free to focus on more important complexity.
Quite often you have to manually do the task at least once to determine how to automate it. That tells you your upper bound on how much time can be saved.
If for example it only took 15 minutes of manual work it's highly unlikely that the payback of automation would occur any time soon. That's especially true because the 2nd or 3rd time there may be different special cases and you'd have to go back and change your automation.
Automation's payoffs aren't necessarily in time saved. Quality can be a more important consideration. "Did I forget a step?" "Did I forget to verify this aspect of the configuration?" "Oh ... am I sure I typed in the new domain name correctly each of the eight times I needed to type it in in the various contexts?"
Automating a task usually means each important parameter is entered only once, that all steps are completed, and that the results are checked.
Of course if we agree programming is not so easy, verifying that the script actually works as intended will take roughly the same time. Not to mention thinking about what it should do when faced with an unusual situation.
Automation can still be a benefit even if it initially takes more time. People doing repetitive tasks make mistakes.
Spend 10 hours writing a script to automate 50x10 minute jobs. You spend an extra hour, but you end up with 50 consistent results, rather than 40 successful jobs and 10 with minor mistakes.
Often the time invested in writing an automation script far outweighs the time spent doing it manually, even when you account for every time the script will ever be run. And it can still be worth it if the script never makes a mistake that human might.
even when it doesn't. I find that if you are careful about adhering to the unix philosophy of tiny scripts that do one thing well then you end up with lots of tools to solve problems specific to your problem domain that get reused in future tools.
This xkcd is really relevant, being a chart of the time invested vs time saved, not even including the accuracy and learning value: https://xkcd.com/1205/
My rationale is that every time I automate a task, I get better at automating tasks. Losing a battle here and there helps me win more battles later on, and by the end of my life I will have netted more time saved than lost.
Also, automation work is much more interesting than doing stupid, repetitive tasks.
I always look at it as decreasing a dependency on me personally. If I'm sick, the task can't be done at all, for example. So even if you spend an hour automating something that can be done manually in 10 minutes, it's a good use of time because the script doesn't get sick, doesn't go on vacation, doesn't leave for another company, etc.
The flip side of this is your company gradually becoming dependent on a sprawling and ever-growing tangle of likely undocumented scripts. This can be a time bomb (no matter how good your intentions) waiting to explode if you ever leave.
I disagree. If you make a mess, you have a mess... but at least your intentions were at one point clear enough to encode as a computer program. While painful to pick apart during an emergency, it's easier than finding the one person in the world with that knowledge and getting them to explain it to you. Having your business depend on a messy pile of scripts is better than having your business depend on someone who died a few months ago. Having your business depend on clean well-maintained code is even better... but great is the enemy of good.
Not necessarily. If the script is documented and written cleanly (e.g Fabric3 which has very little magic, and reads almost like Bash), then in can server as "runnable documentation". In this case automation could be really good for continuity and busfactor.
But yeah... got to put some effort into it. Like all docs, script-docs worn't write themselves.
Agree that it's not unavoidable, but the problem is deeper than just the scripts themselves being well documented and straightforward. The main risk (IME, anyway) is awareness / discovery of the scripts in a timely manner - the fact that everything 'just works' means that it can be months or years before something breaks, at which point it may be too late to ask the person who set it up. Hopefully there's good handover from the outgoing staff member to a technically proficient colleague. If so then everything might be OK but if not then a catastrophic failure may well be the first sign that something is wrong.
(A recent example of much simpler automation going horribly wrong was a company losing their top-level domain name because it was set to auto-renew using an ex-employee's personal credit card, and their email address as contact. This worked for years until that card was cancelled and the warning emails bounced off the closed account.)
I can think of a few benefits, even if it takes longer: you may need to do something similar again and now you have the script, you've gained experience and you might right the next script faster, correctness.
I think correctness is the most important one. Even if a manual task is quick, ensuring that you do it correctly and exactly the same way for all 30-50 things is extremely tedious. If one out of those 50 subdomains was behaving differently and you needed to investigate exactly why, that could quickly erode any time savings.
Correctness is huge. The other day a colleague and I were figuring out some things about scheduled work days over the next few years. After I had futzed about in a spreadsheet for five minutes, he said "this is ridiculous!" and started furiously paging through a paper calendar. We had our results nearly simultaneously, but he was off by over a month. After that we just used the spreadsheet.
Correctness also works the other way around. We needed to link to databases with non-matching customer IDs, the analyst on the team spent several hours automating fuzzy matching of customer names in R before we concluded it was easier and more correct to build a mapping table by hand for the few 100 that didn’t match.
From a short time perspective you wasted your time. But in the long run you learned valuable skills that later may have enabled you to automate other more important things.
I am a script junkie. I have to, just have to script every task out. Here's what I've realized.
First do it manually, then automate it.
Doing manual tasks is frustrating enough to motivate me to automate it. Plus I have a much better understanding of the inputs, the required outputs and constraints for the script to be more useful.
Let your scripts evolve.
Don't force everything in the first go. Simplicity is the key. Let them be rigid, readable and in a single file at first. Then, introduce flexibility as required. Abstract out core functions into individual scripts/functions/modules. This builds a certain sense of familiarity with the code that is just too endearing and eventually very productive.
Reuse your mini-scripts.
Each of my scripts does one thing. There are super-scripts that combine many of those for a task that I am performing at the moment. DRY principle. The more simple, atomic and mutually agreeable your scripts are the more powerful they get. This is just the unix way of doing things. Nothing new.
Write consistent code comments
You will write. You will forget. Make the code comments personal and consistent enough for you to immediately recognize them.
Use version control
Absolutely use it. There will be many versions of your core scripts/functions that dont agree with each other. My personal preference is a version number (semver) suffix on each file to identify which core script is used by which task script.
Reap the rewards
The rewards of putting in the effort to me have been akin to compound interest on an early investment. I have reused core scripts so heavily that prototyping any new idea is pretty easy. If there is something that a script does not do, I know what I need to read over the next weekend :)
Scripts/automation can be worth it even for one-shot tasks for which time to create a script > time to do the task manually:
a) They help in applying forethought to a process. Maybe you can even dry-run the thing.
b) They help documenting what has been done, and how, which can be invaluable in a team setting.
c) They train you into getting incrementally better at automating, and at sensing when to automate and when not to†. If it's painful, you're not doing enough of it.
† I find that for the ones that would take ages to properly script, more often than not a combination of manual (like preparing a normalised input for the script using vim) and automated subtasks can do wonders, leveraging the best of wetware and software worlds.
The only thing that I will do manually in any environment is putting sensitive data into configuration. With IAM roles, I can usually avoid that. I will create the roles manually.
If you go by the old mantra of treating servers like cattle and not pets, you have to automate everything. My VMs (EC2 instances) can be destroyed, replicated, auto scaled, etc. at any time. I have to automate no matter how long it takes to figure out.
I make these tradeoffs a lot. Automating is fun but time consuming. Doing things manually is tedious. However, automating has one benefit: you make things easier for the next time you do something similar because you learn something. Usually requests for tedious things are not isolated incidents. So I have a bunch of commandline stuff that I know how to glue together like csvkit, jq, xargs, and a simple bash script that adds functions to do rest against our api server. Using that I can do easy bulk operations against our db without investing in product features, ui, etc.
sure, but mess it up once and you spent time to figure the error and fix it.
or go on holidays and your colleagues just gotta run the script.
the big benefits in scripts that is often overlooked: Given the same conditions and input, the output will always meet your expectations. There's no chance for human error.
Upfront time investment to build automation script sometimes more than manual. But most of the time it's a one time effort to save more of your future time.
From a pure time savings perspective, this makes sense - naturally, links to XKCD are already popping up.
From a maintenance perspective, a well-written script can serve as a reference after the original task was long forgotten. A poor substitute for documentation, but when performing software archaeology on legacy systems you really need everything you can get.
I like what the guy did, but am I the only one who feels most of these "I did x with y" articles always end up trying to sell you something? What happened to just sharing our knowledge with each other?
Most content on the internet is created for marketing purposes, unfortunately. I don't have much room to talk, as I do it myself. That doesn't mean that we don't enjoy creating the content, or sharing our knowledge with others, but it can definitely get old being the consumer of these pieces.
I’m not sure about that. How do you measure? I would expect that the vast quantity of content created is not to sell, its by individual creators writing these comments, tweeting, blogging, etc.
Perhaps the majority of content read, but that’s because it is marketed.
I’m not sure how you would measure. Count all the bytes on .com vs .org?
Some of the meaning of the code is embedded in how commander is set up, which if one is unfamiliar with it means the code is less explicit. Explicit is good, and there is a sense in which explicitness leads to simplicity.
Exactly, it’s a lot clearer to your future self, and any other maintainers how the application works when you aren’t adhering to a “frameworks” way of doing something. Especially for a simple cli.
Abstractions can be good, but I feel as though something like commander is really just an unneccessary abstraction.
It’s extren explicit...? Each option is laid out clearly. Each option Maps to a key on the object that is passed to the function called at the end of a commander chain.
Tom L is a good writer - I have gotten a lot out of his books and articles over the years - excellent point here:
"Since February 2015, the SRE (site reliability engineering) team at Stack Overflow has switched from a mixture of Python and Bash to Go. Even though Go isn't a scripting language, for small programs it compiles and runs nearly as fast as Python takes to start. At Stack Overflow we tend to prefer compiled, type-checked languages for large programs, especially when multiple people are collaborating, and, therefore, no one person is familiar with every line of code. Our policy was that Bash scripts couldn't be larger than 100 lines and Python programs couldn't be larger than 1,000 lines. Those seemed like reasonable limits. Rewriting scripts when they grew beyond the limit, however, was a lot of work. It was better to start in Go and avoid the conversion."
Not Golang, per se, but over time you are limited by your tools. Big bash scripts get unwieldy. Nodejs changes underlying dependencies (coincidentally wrestling with 4 yr old nodejs this week).
What I noticed about this - I'm not thrilled right now about using node to write console helper scripts, but the CLI lib they're using here does have a pretty nice interface. I do like Python generally for doing scripting type things, but there doesn't seem to be any mindshare for any cli helper libs for it other than click, which IMO has a really clumsy, awkward interface. Ruby is generally my favorite go-to now, and it has a bunch of cli helper libs with a variety of types of interfaces, at least some of which have what I consider to be a really slick interface.
If I was better at Python, I might try my hand at writing a better Python cli lib myself. I'm surprised nobody else has yet.
I'm interested in finding a Python CLI lib for building command suite apps. I've written a few with Click, and its architecture seems weird and clumsy to me. I looked at that website, and it doesn't list any command suite libs other than Click. There's a few other CLI-related libs, but for doing stuff like making prompts work nicely.
My usual go-to for that kind of stuff is Escort for Ruby. Everything is defined at a place and level that makes sense IMO.
I have had some nice results with swift. It’s still predominately Apple tech (macOS,iOS) but the packages manager is nice and it’s has nice “script” feel for a compiled language.
In respect of some of the comments elsewhere saying that the person would probably have been better off just doing the damn job, XKCD has, of course, got there first.
I reference these regularly to keep me on track doing things that actually matter, rather than tinkering endlessly to save me having to do things that are less interesting, but would take less time.
Funny, I have those exact two comics sitting right outside my cubicle. #1319 was put there because my team lead thought I wrote too many scripts to automate silly stuff. I put #1205 there beside it so that I could rationalize all the automation I do.
On the subject of automation, has anyone been able to do or is there even any scope for script level automation for video post-production? I'm talking specifically Adobe Premiere. I have to constantly do a series of the same actions on different content assets much like re-skinning. If anyone has had any success - I'd be interested to learn from you.
I have been looking into that of late and have found that the what little scripting options Adobe provides are woefully inadequate for anything more than basic macros.
I have been looking at FFMPEG for automating video post-production, and recommend you do the same.
What I want to know is - how can I automate the Google Admin interface. Because holy crap, is it slow.
Yeah, I'm sure there is an API :-) I just haven't gotten around to looking into it yet, it's one of those annoying things that never gets to the point of seriousness, but gets a few choice words from me when I have to use Google Admin.
Fortunately, much of GSuite admin api functionality is nicely wrapped up in the free / open / well maintained GAM cli - https://github.com/jay0lee/GAM/wiki
