Because it’s fun, here’s an alternative spelling of `echo .dump | sqlite3`:
<<<.dump sqlite3
This avoids invoking echo unnecessarily. Every time you do it you might just save a picowatt-hour or two! (It shouldn’t be more expensive. But be sure to spend a few watt-hours benchmarking it rigorously.)
This uses two things that shell users are commonly unfamiliar with:
• <<<: as `> filename` redirects stdout to the named file and `< filename` pipes the named file into stdin, `<<< word` pipes the word into stdin. (The whitespace is optional, and word follows normal argument splitting so you can do things like `<<<"Hello, world!"`.)
• Ordering: although most commonly written at the end of commands, redirections can be placed at the start as well. So if you do things like `2>/dev/null >&2` to suppress all output¹, that bit can go at the start or the end. I’ve become increasingly partial to putting redirections at the start of the line, especially in cases where I’m appending to a file for a while, because a leading `>>filename` supports visual alignment better than a trailing.²
And since we’re talking about different ways: as given, this is feeding the SQL to sqlite3 via stdin, but you can also pass it on the command line, after the filename. But things like this in Git (aliases, diff textconv, probably more) work by taking the string and appending the filename to the end, so you need a way of reordering the arguments. The solution is an immediately-invoked function:
f() { sqlite3 "$@" .dump; }; f
So when you diff mydb.sqlite, it runs `sh -c 'f() { sqlite3 "$@" .dump }; f mydb.sqlite'` or equivalent, which winds up executing `sqlite3 mydb.sqlite .dump`, as desired.
I use this technique a number of times in my Git aliases, saving the bother of putting them in separate shell scripts somewhere where path management is a bother, at the cost of maintaining a one-liner with sometimes too many semicolons.³
—⁂—
¹ “Take stderr (2) and redirect it (>) to /dev/null, then take stdout (default/implicit, could also write 1 explicitly) and redirect it (>) to stderr (&2).” There are plenty of other ways of writing this!
² Lists can be a better solution for this specific case, allowing you to redirect to the file only once for a whole bunch of commands:
{
<<<"Line one"
some-command
<<<"End of $thing"
} > filename
³ My longest is thirteen lines, though half of them barely count as lines. The line from my ~/.config/git/config, within [alias]:
# Revise into the commit that last changed File
rf = "!f() { if [ $# -eq 0 ]; then REV=\"$(git status --porcelain --untracked-files=no | sed '/^ /d;s/^.. //' | xargs -n1 git rev-list -1 HEAD -- | uniq)\"; NUM_REVS=\"$(echo \"$REV\" | wc -l)\"; if [ $NUM_REVS -ne 1 ]; then >&2 echo Files in the index were not all last modified in the same commit; exit 1; fi; else REV=\"$(git rev-list -1 HEAD -- \"$1\")\"; shift; fi; git revise \"$REV\" \"$@\"; }; f"
Seems a bit over the top when you could also just provide it as a command line argument to sqlite3 which is much more obvious than <<< and works in every shell.
Review what’s going on. Git gives the file name as the last argument, so you can’t pass .dump as the last argument like that without the contortion of an immediately-invoked function.
Shh! Don’t spoil things! Picowatt-hours, I say. Picowatt-hours!
But seriously, although echo is typically a shell built-in and distinctly faster than /usr/bin/echo, it’s still much slower than <<<, presumably because it still has to set up a pipe and an extra… shall we say pseudoprocess.
Comparing behaviours for feeding text into `true` (typically a shell built-in, so that process spawn times doesn’t drown the signal):
try() {
echo -e "\e[32;1m$1\e[m"
for (( run = 0; run < 5; run++ )); do
time (for (( i = 0; i < 1000; i++ )); do
$2
done)
done
}
a() { /usr/bin/echo .dump | true; }; try /usr/bin/echo a
b() { echo .dump | true; }; try echo b
c() { <<<.dump true; }; try '<<<' c
d() { true; }; try "no piping" d
My best times of the five runs, under bash/zsh, expressed in time per iteration:
• /usr/bin/echo: 750μs/845μs
• echo: 469μs/371μs
• <<<: 11μs/31μs
• No piping: 3μs/10μs
So… yeah, on a very slightly older or slower machine than mine, using <<< may save you more than half a millisecond. That’s a much bigger difference than I expected—I was expecting it to be well under 200μs, maybe under 100μs, though the more I think about it the more I realise my expectation may have been unreasonable.
Improved benchmarking script (I just wasn’t thinking carefully at the time, just getting something out quick; but eval is obviously better than requiring a function):
try() {
echo -e "\e[32;1m$1\e[m"
for (( run = 0; run < 5; run++ )); do
time (for (( i = 0; i < 1000; i++ )); do
eval $1
done)
done
}
try "/usr/bin/echo .dump | true"
try "echo .dump | true"
try "<<<.dump true"
try "true"
I also want to note that this is thoroughly into the nanowatt-hour scale (even at the unrealistically low figure of 1W power consumption, 300μs makes it 83⅓ nWh). Tens or hundreds of thousands of picowatt-hours!
There is another difference that users may care about here. zsh will create a temp file for each here-string in the here-string version, and bash may do too¹. Whether those files hit a disk is a matter of system configuration, and whether such a disk is magically quick or swirling rust is a different issue too.
[Just noting that the implementation here isn't equivalent for nerdsnipe-ery, not arguing for real attempts at optimisation of simple pipelines.]
¹ Always with older versions, only for large strings with newer versions.
Running under zsh, link 0 -> '/tmp/zshqqjTS0 (deleted)'; under bash, link 0 -> 'pipe:[1509353]'.
(I know <(…) is substituted in zsh with /proc/self/fd/… for a fd corresponding to a pipe:[…], and regular | piping makes 0 be a pipe:[…].)
I know some common configurations use concrete /tmp. Mine is tmpfs.
Thanks for the info! I like sharing these kinds of things because I have found them interesting and expect a few others will too, and people often add to them details I hadn’t known and like to know!
You can see the cutover on buffer size from my install of bash 5.2.15. Anything under 64k of here-string will still use a pipe.
In case you weren't aware, you can also force a "real" file with zsh process substitution by using =(…)¹. It can be useful when the tool you're using doesn't behave correctly with <(…), if it wishes to seek across it for example. Sadly, =(…) isn't supported in bash [yet?].
¹ In your case it still wouldn't hit a disk as it is in /tmp, but it at least becomes seekable.
This uses two things that shell users are commonly unfamiliar with:
• <<<: as `> filename` redirects stdout to the named file and `< filename` pipes the named file into stdin, `<<< word` pipes the word into stdin. (The whitespace is optional, and word follows normal argument splitting so you can do things like `<<<"Hello, world!"`.)
• Ordering: although most commonly written at the end of commands, redirections can be placed at the start as well. So if you do things like `2>/dev/null >&2` to suppress all output¹, that bit can go at the start or the end. I’ve become increasingly partial to putting redirections at the start of the line, especially in cases where I’m appending to a file for a while, because a leading `>>filename` supports visual alignment better than a trailing.²
And since we’re talking about different ways: as given, this is feeding the SQL to sqlite3 via stdin, but you can also pass it on the command line, after the filename. But things like this in Git (aliases, diff textconv, probably more) work by taking the string and appending the filename to the end, so you need a way of reordering the arguments. The solution is an immediately-invoked function:
So when you diff mydb.sqlite, it runs `sh -c 'f() { sqlite3 "$@" .dump }; f mydb.sqlite'` or equivalent, which winds up executing `sqlite3 mydb.sqlite .dump`, as desired.I use this technique a number of times in my Git aliases, saving the bother of putting them in separate shell scripts somewhere where path management is a bother, at the cost of maintaining a one-liner with sometimes too many semicolons.³
—⁂—
¹ “Take stderr (2) and redirect it (>) to /dev/null, then take stdout (default/implicit, could also write 1 explicitly) and redirect it (>) to stderr (&2).” There are plenty of other ways of writing this!
² Lists can be a better solution for this specific case, allowing you to redirect to the file only once for a whole bunch of commands:
³ My longest is thirteen lines, though half of them barely count as lines. The line from my ~/.config/git/config, within [alias]: