If anyone is interested there were several great posts on Hacker News a while about about useful UNIX commands [1, 2, 3]. I have also created several screencasts about command commands like the following [4], and one about the Filesystem Hierarchy Standard [5].
ls man pwd cd top ps df du cp mv rm mkdir rmdir less cat vi
Great links, though I admit when I saw the OP I was expecting content like that and was pleasantly surprised that they really were "Unix commands I wish I'd discovered years earlier", and not the standard array of unix tips+tricks that you can get by without for awhile but as you become more advanced become second-nature.
"Man ascii." The number of times that I wound up generating my own ascii table from whatever language I was working with...
"xxd" I always either used search in emacs hex-mode (which I find cumbersome and annoying but haven't taken the time to customize), or rolled my own search using a programming language. This would have come in handy SO many times back when I used to muck around with videogame save files and the like. I never even knew it existed nor how to discover that it did.
'cal' I learned about early on so never missed it and the 'ssh' stuff kind of fits into the regular "unix tips and tricks" category.
You reminded me of something. There is actually a really cool and simple site called asciiflow [1], which I use all the time to draw diagrams for explaining things in email, etc. It's pretty cool, and was even submitted several times to HN [2, 3].
Wow. This is really cool!
Not only can you make the ascii drawings, but there is an integration with a service called ditaa (unaffiliated) that then turns your ascii drawings into actual images.
I am keeping these.
http://ditaa.sourceforge.net/
I can't browse www.asciiflow.com, the name does not resolve:
$ host www.asciiflow.com
Host www.asciiflow.com not found: 3(NXDOMAIN)
Caused by both authoritative nameservers for asciiflow.com which return the correct answer (a CNAME to ghs.google.com), but with status=NXDOMAIN (wtf?):
$ dig +short asciiflow.com ns
ns2.123-reg.co.uk.
ns.123-reg.co.uk.
$ ; <<>> DiG 9.8.1-P1 <<>> @ns2.123-reg.co.uk www.asciiflow.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 65092
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;www.asciiflow.com. IN A
;; ANSWER SECTION:
www.asciiflow.com. 86400 IN CNAME ghs.google.com.
;; AUTHORITY SECTION:
google.com. 14400 IN SOA ns.123-reg.co.uk. hostmaster.google.com. 2013052701 86400 0 604800 86400
;; Query time: 207 msec
;; SERVER: 92.51.159.40#53(92.51.159.40)
;; WHEN: Wed Sep 11 00:27:52 2013
;; MSG SIZE rcvd: 123
Thanks for the suggestion! Actually, I initially wanted to do one on more advanced commands, but I thought I'd better lay a foundation first. I also want to cover using pipe (|) to chain these commands together. I'll make sure to include these.
Say you have a set of tasks you need to do, and some of those depend on others. Now, you want to perform those tasks in order. The question then becomes: in which order? That's the problem that tsort (1) solves.
It's a way to do some of what make does without having to write a Makefile. 99.9% of the time, it never comes up, but when it does, it is very, very handy to be able to this in a shell pipeline. HTH.
i usually click those links, see the stuff and roll up my eyes. nothing learned, but upvote so more people learn about them anyway.
this time, the very first one `man ascii` got me. Genius. Never though about looking for that in man before. But in my defense, this was added in 1999 :)
Let's say my pwd is `/home/projects` and i want to edit `/home/projects/a_huge_sprawling_app/940j_394/lol/flight_controller.rb`
`vim **/flight_controller.rb`
opens our flight_controller.rb straight away. In terms of effort, this allows you to basically omit a `find -name` when you're in a rush to edit some damn file where the hell is it again damn we need to reevaluate this directory str...
----
Double bang to re-use your last terminal entry. One great use, taking all the pain out of forgetting your sudos
FWIW, the double-star recursive globbing was added in Bash 4.0. My reading of the manual[1] suggests it is disabled by default, and can be enabled by setting the 'globstar' shell option. I believe it's enabled by default in zsh.
I just tried it on bash 4.2.25, and it worked, even though I just learned about it.
Maybe not every OS enables the same default options for Bash (I'm using Lubuntu 12.10 at the moment)
Yep, that is a fair point. I should have qualified my statement about bash's defaults, since many OSs include their own custom start-up files that may modify the defaults.
Re: command line globbing with globstar -- is there some way to get it to cache filepaths that are frequently accessed? It's quite slow for me (I usually use aliases for that, which I was contemplating getting rid of).
Thanks a lot for wising me up to the new hotness offered by bash 4+ (zsh is also pretty slow for me for some reason, even though people keep telling me to switch to that. Maybe I'm more impatient than most folks).
Sorry for the late reply. No built in way to cache globbed filepaths as far as I know. I did a little googling but to no avail. I suspect it wouldn't be too complicated to hack something together though. Another potential option: buy an SSD.
My favorite is the parallel jobs feature of xargs. For example, say you want to run a script you wrote called process-video.sh to do some processing on all the video files in a directory (extracting audio to MP3, converting format, etc.). You want to use all 8 of your cores. You could write a Makefile and run it with -j9, or you can do this:
This immediately forks 9 instances of process-video.sh on the first 9 .flv files in the current directory, then starts a new instance whenever a running instance completes, so 9 instances are always in flight. (I usually set to number of cores plus one for CPU-bound tasks, hence 9 for my i7 with eight cores [1].)
If you add -print0 to the find command and -0 to the xargs command, it uses null-terminated filenames (which does the right thing when filenames contain whitespace).
[1] Logical cores. Most i7's have four physical cores which become eight logical cores through the magic of hyperthreading.
If you like xargs, but want more flexibility, I'd highly suggest GNU parallel. Such flexibility includes running jobs on multiple computers, running intensive command using all available CPU's (like xargs -P), and creating unique scripts to handle multiple parameters.
Parallel also lets you transparently run stuff on remote servers, automatically handling stuff like copying files back and forth. When I have some heavy ad-hoc data processing to do my new favorite trick is spinning up 50-100 ec2 spot instances, point GNU parallel at them and just fire and forget.
Even though CPU is usually the resource that limits throughput for video processing in my experience, each process will presumably do some amount of I/O as well.
If you have 8 cores running 8 jobs, then whenever one of those jobs needs to do I/O you have a core sitting idle while that job waits for the disk. If you have 8 cores running 9 jobs, then all your cores will still be fully utilized when a single job is doing I/O.
I think you actually want to do plus a small percentage, perhaps 5%, so ceil(1.05*N) jobs on N cores. That is, I have a gut feeling that 64 cores doing 65 jobs would still result in underutilization, 67-68 jobs would be better (provided the workload doesn't become I/O-limited with that much CPU power, and provided the number of videos to be converted is still much larger than the number of cores, so you don't run afoul of Amdahl's law).
These are really just rules of thumb based on my gut feelings and mental models of how the system works; it might be fun to actually benchmark it and see if my ideas correspond to reality. (You probably want to reboot first, or a bunch of unrelated I/O, to flush the disk cache.)
I definitely agree with the n+1, however when you're dealing with HyperThreading I don't know that I'd bother. The extra 'cores' provided by HyperThreading are not full cores.
The idea behind HyperThreading is basically the same as your n+1 idea. A HyperThreaded core has duplicated circuitry to handle state (registers, etc) but not the execution resources. When one thread is stalled/not using the execution resources (e.g., waiting on disk IO, waiting on a fetch from RAM, etc), the other thread is already in processor ready to go and will be executed instead.
Obviously this is a bit of a simplification, but your n+1 idea is essentially already implemented in the hardware.
That said, if enough of your cores are still stalled waiting on disk/memory access and are stalled long enough to make the context switch worth it, it may be beneficial. If not, however, you might actually end up seeing some minor slow down as the processor is forced to switch between the competing processes.
I'd like to get a better sense of how well hyperthreading works in practice on stuff like this. Is it really the case that my physical cores' pipelines are < 50% utilized by video processing, such that the hyperthreads are equivalent to having a whole other core? Because I'd have guessed that, IO aside, you really want slightly fewer processes than virtual cores.
it's quite a common pattern in commandline apps to check if the stdout is a tty or not, and do things differently if so. So when you pipe it through `cat', it does the no-fancy-included output, much like the colouring that `ls' will apply interactively, but not when piped/redirected.
It's important for grepping and sedding - the colors are escape codes and would mess with your regular expressions otherwise (you want to find foobar , but bar was blue so you got foo\e[1;34mbar\e[0m instead, and it doesn't match with foobar).
On the other hand, if you want colors displayed you can force grep/ack to display the color 'always' and then pipe the output into 'less -R'. The -R (or --RAW-CONTROL-CHARS) flag has less display output such that color escape sequences work as you might expect, coloring the output displayed by less.
Because, in many - if not most - situations, spawning another process is insanely lightweight compared to the cognitive/physical load of your alternative (which includes remembering the switch and its value ("never" is hardly intuitive), remembering whether or not you're using a system that supports it (surprise, surprise, OSX doesn't), and typing all those extra characters).
Admittedly, I never actually do this because I don't ever care to not see colour (what would be the motive?), but I can understand why the OP does what they do.
Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems.
for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza;
do
ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts";
done
This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a for loop and scp to push the files out and then run md5 to verify they are correct. It was a hack but useful for a small set of machines.
for host in ... ; do ssh -l root $host "..." & done; wait
and it's very parallel. All that key exchange takes a while, so it's not ridiculously fast, but it's pretty good. If you've got ControlMaster set up already to be per-host, this might very well be very fast the second time.
Also check out xargs with the -P 10 -n 1 flags (for example) -- to limit at most 10 parallel tasks running at the same time. Useful if you don't have GNU parallel installed where you happen to be working (everyone has xargs installed).
The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.:
% (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\)
Or:
% cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys
This is a bit of a pet peeve of mine, but cd/tar should likely be
joined with && - in other words, in what directory do you want tar
to (not) run if cd fails? So consider:
% (cd /foo && tar cpf - .) | ssh bar '(cd /baz && tar xpf -)'
It would be an interesting project to look at people's shell usage and try to determine when they started using Unix. I have loads of Irixisms that have only recently been crowded out of my head by Darwin BSDism.
Forgive my ignorance, but I don't know what what it means to connect "across the channel", but it sounds important. Do you know a link that explains the concept? I've failed to Google it.
It means that whatever you put into the ssh process's STDIN gets sent to the STDIN of the remote process. For example you can do this:
tar cj $bigdir | ssh host 'tar xj'
That creates a tar.bz2 archive on your machine, sends it over ssh to the remote host, where it's unpacked. Ad hoc compressed and encrypted file transfers made easy.
Sure, of course. But I've used that tar|ssh combination so much that it was the first example of using the rsh-like stdout/in pairing that came to mind. As much as I bitch about Unix and POSIX and horrific shell brain damage, I still reach for crazy shell pipelines when I have data manipulation tasks up to some pretty large scale n.
He's talking about forwarding stdin and stdout across the SSH connection to the program that SSH is running on the other end; such that the stdin of ssh is passed unmodified to the program and vice-versa.
Are you familiar with stdin/stdout/stderr on Unix and how they interact? What ssh does that I find so powerful is ensure that what you put into stdout on one side of a pipe flows out of stdin on the process that ssh starts on the remote host.
Of course, ssh also gives you the output of the remote process, so you can put an ssh command in the middle of a pipeline. My day to day life no longer involves interacting with lots of remote Unix systems, but when it did, it was really, really useful.
"typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome. "
Sounds awesome, but I really didn't understand what he means. Is he talking about using a certificate (-i option) to log in ?
Yeah, you have a cert set up and put the public cert in the identities file for each server you want to connect to. Then just SSH into each box with agent forwarding ('-a' I think, but you're best to check that).
So you'll need to create an SSH cert first:
ssh-keygen -b 4096 -t rsa -C "$USER created on $(hostname)" -f $USER.key
Keep the private key, that's essential you keep this safe as it's literally your key to access any servers you do the next step for.
On each server SSH, append your public key to the of /home/$USER/.ssh/authorized_keys (creating the file if it doesn't exist). It must be the public key!
There'll be better guides online for this stuff so I suggest you have a read through them before blindly pasting the stuff I've posted into your terminal. But it's all quite simple stuff once you've grasped the difference between a public and private key (if you weren't already aware - you may well be)
Scp doesn't get all file types correct (fifos, etc), while tar does.
Piped tar commands do, but are generally inferior to rsync (at least if you ever need to copy more than once). I believe I've heard (here) that piped tar commands can be useful for the first sync (possibly better compression, no overhead of file checksums), then rsync thereafter.
because so many programs behave differently when attached to a tty (like prompting for input) and I just want a plain text log that says when it succeeded or where it blew up.
Agreed, and my problem with even the usage above is that I often realize too late that "whatever" is not going to finish before I need to unplug. Then I have to kill "whatever" and restart with nohup and logging. tmux/screen always has me covered.
I recently used xxd to illustrate the concept of text encoding, and put together a little vimscript to make it easier to visualize:
"Toggle hex edit mode
nmap <Leader>h :call ToggleHex()<CR>
let g:hex_mode_on = 0
function! ToggleHex()
if g:hex_mode_on
execute "%!xxd -r"
let g:hex_mode_on = 0
else
execute "%!xxd"
let g:hex_mode_on = 1
endif
endfunction
Stick this in your .vimrc, type something, use \h to convert it to hex, change a value, then \h to convert it back and observe how the text has changed. Not super useful, but a neat party trick.
I first ran into clipboard copy/paste on Mac. Discovered "xclip" on Linux. Aliased copy to 'xc' and paste to 'xp', later added 'xpc' to paste from the secondary (application) clipboard.
In 2005 I wrote similar utilities for windows called ccopy, ccut & ppaste. I recently dug them up and put them on GitHub. https://github.com/FigBug/ccopyppaste
Rather than 'man 7 ascii', there's the 'ascii' command itself.
It will also provide encodings for single characters:
$ ascii a
ASCII 6/1 is decimal 097, hex 61, octal 141, bits 01100001: prints as `a'
Official name: Miniscule a
Other names: Small a, Lowercase a
ASCII 0/10 is decimal 010, hex 0a, octal 012, bits 00001010: called ^J, LF, NL
Official name: Line Feed
Other names: Newline, \n
I find it fascinating that someone could be using something as complex and slow as Hadoop and not know of sort(1). Not faulting any lack of awareness, but it is really interesting that Google map/reduce and Hadoop marketing (word of mouth?) are so effective while the contents of /usr/bin are like buried treasure.
The funny part is that one of the early uses of Google's map/reduce approach was probably to distribute the job of sorting search results among many servers. And here you are sorting Hadoop results with something as small and simple as sort(1).
One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this:
You can chain commands together to quickly get ball park figures. I do this all the time when reviewing log data to get simple counts and look for anomalies.
Let me break this down a little. I cat data.txt, which just prints the contents to the console, I use grep to remove the # header, then use awk to print the second data column, the sort the values for my next procedure, and lastly I use uniq -c, which takes the input and counts all the unique values. So you can see that "4 104" means that 4 instances of 104 were found. I use this often to look at httpd logs for IP addresses or to review sudo commands, etc. Very handy!
ps. there are optimizations that could be done here, but that what is great about unix and "one function" utilities. There are many ways to the same destination!
Yes, there are some ways to simplify. You can get rid of the "UUOC". Instead of
cat data.txt | grep -v '#'
you can do:
grep -v '#' data.txt
or you can simplify further by moving the grepping into awk, so that the full command becomes
awk '!/#/{print $2}' data.txt | sort -n | uniq -c
sort -n is good to use for sorting numbers. You could probably pull the sort | uniq -c into awk somehow, but I have already demonstrated the full extent of my awk knowledge. One of these days I will get around to actually learning it.
I recently found out that zsh lets you magically popd as well, as long as you're cding deeper into the tree. If you jump to a different subtree it will pop its internal stack until it finds a parent of the target dir.
I recommend Quicksilver on mac, free and powerful. It lets you to find a lot of stuff (application, contact, email, etc.) and manipulate them (send, forward, print, etc.)
Alread rocks. You can develop workflows for common groups of tasks and it integrates well with iTunes, 1Password, Terminal and other system actions/apps.
I second the Quicksilver recommendation. When you combine Quicksilver triggers (keyboard shortcut available regardless of what application you're in) with applescript, and fabric it's very powerful. I have a trigger that that restarts all of my local services using fabric and opens up new terminal windows running tails of the relevant logs.
The Applescript is this if anyone wants to do something similar:
tell application "Terminal"
do script "tail -F /somepath/some.log"
do script "tail -F /anotherpath/another.log"
do script "sudo fab -f '/pathto/fabfile.py' local deploy"
activate
end tell
Here's a detailed explanation of setting up Triggers:
Which works by creating two named pipes and feeding them to diff and pumping your processes data into them. Process substitution is built on named pipes, they are the enabling mechanism that makes it work. Try this in a terminal:
echo <(true)
And you'll see the anonymous named pipe created by the subshell and passed to echo.
GNU Parallel. My new favorite command. I have probably written at least two or three ad-hoc, buggy, feature poor versions of that command before discovering it.
Since I didn't install vim on that box (just use basic vi), xxd wouldn't have installed as part of it. Where as in a jail on the same box, xxd does exist:
root@primus:/usr/ports # which vim xxd
vim: Command not found.
xxd: Command not found.
root@primus:/usr/ports # jexec 1 which vim xxd
/usr/local/bin/vim
/usr/local/bin/xxd
I've known about 'od', but I've always used 'hexdump' instead. Amusingly, I just noticed that they are literally the same (hard-linked) binary on OS X 10.6, even though the arguments differ somewhat. The man pages states that 'od' is part of POSIX, which aligns with your comment about portability.
This is also a common UNIXism: programs that behave differently depending on what they're called as. Another example is gzip (gunzip = gzip -d, gzcat = gzip -dc).
`mdfind' in non-Mac UNIXs is spelled `locate' (and the corresponding `updatedb' updates the indexing db, obviously (although you don't have to do this often, as it runs as a cron job)).
`man 7 charsets` is a great read which recommends checking out the pages for console(4), console_codes(4), console_ioctl(4), ascii(7), iso_8859-1(7), unicode(7) and utf-8(7).
I have the bad habit of trying out everything in /usr/bin (linux, OX) and in c:\windows\system32 (and further on, digging Program Files / Applications / etc.).
This is how I discovered myself "fsutil" (Windows) :)
> Can't they just directly access inodes for high speed counting?
Don't know what you mean by that exactly. But, I'd be stunned if find is actually reading the actual files rather than just reading the directory entries in directory "files". Which is what I'd call accessing inodes "directly"
If you're dealing with a truly large number of files you could use "find . -type f -printf 1 | wc -c" which would just print a single character to the pipe and count the characters. I really doubt this is significantly slower than one program that does both would be. It'll do roughly the same amount of work.
It can be really slow, even for a few tens of thousands of tiles. I remember watching Nautilus's "properties" panel slowly count up the number of files and directories.
I'm not sure what this has to do with piping (why would nautilus pipe the output of find when doing a directory listing?), but I doubt it's the pipe that's causing performance issues. Uncached file system access has a tendency to be slow[0].
du just uses the fts API, so if you want something directly equivalent (omitting options and error handling) it was only barely too long to post: http://pastie.org/8315296
Compared to the equivalent find, code #1 is much faster in this completely scientific best of several runs:
$ time (find OSS/llvm -type f -or -type l | wc -l)
real 0m4.681s
$ time du -d0 OSS/llvm
real 0m4.863s
$ time ./fts OSS/llvm
real 0m0.559s
Ostensibly because of the lack of stat calls. On the other hand the second paste and find with no arguments are equivalent.
function lc() { if [[ "$#" -gt 1 ]]; then for DIR in "$@"; do echo -n "$DIR - " ; ls -AU1 $DIR | wc -l ; done ; else ls -AU1 "$@" | wc -l ; fi; }
Then "lc dir/" will print the number of files in the directory, and "lc dir/*" will print the number of files in each of the subdirectories. This exactly what you're looking for because it doesn't descend into the tree. If you're working with directories that may have tens or hundreds of thousands of files in them, it's useful to check first.
There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next pass. If you do this before that data gets removed from the cache it will be much faster.
imac:~
$ time find . | wc -l
419191
real 0m37.865s
user 0m0.385s
sys 0m2.611s
imac:~
$ time find . | wc -l
419195
real 0m10.503s
user 0m0.330s
sys 0m1.430s
The output from find is a list of filenames separated by newlines. The wc command is reading from stdin and counting those newlines[1] -- wc itself has just two arguments in its argv array here ("wc", "-l"). You will not exceed its argument list. Further, it is not at all a hack to count the number of files this way; rather using pipes to compose functionality is the Unix way[2].
But fwiw, if you want a count of the inodes in use in the entire filesystem, you can get that directly from "df -i".
In any case, as an exercise to the reader, I encourage you to grab the source to find and add the -count output option you desire.
[1] Technically a filename can contain newlines, so this would throw off the count. The fix for that would be to use find's -print0 output option and then pipe to something which can count on the nulls. In practice, you're unlikely to have such filenames.
I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l".
Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?
I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count.
All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.
There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.
You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself.
In this invocation find and wc both have two fixed arguments, you're never going to hit MAX_ARG_PAGES. The output from find is passed to wc over a pipe, and to my knowledge there's no upper limit on how much data you can transfer over a pipe. I've piped huge disk images.
It's not the pipe through wc, but statting the directory entries which slows you down.
If your locatedb is current, you can use locate for this:
$ locate $PWD | wc -l
That runs in just a hair under a second (0.908s real) for the first instance. 'find' takes a few seconds on first pass (it's got to read from disk), but is surprising fast (0.227s real) on subsequent runs.
So I guess it depends on whether you're doing it just once or multiple times.
Never knew about the "cal" command. I especially like that you can quickly look up a previous or future month/year.
>> cal 3 1973
quickly shows a calendar of March 1973
The "+n" syntax is not standards compliant, from the tail info page (on fedora 19):
On older systems, the leading '-' can be replaced by '+' in the obsolete option syntax with the same meaning as in counts, and obsolete usage overrides normal usage when the two conflict. This obsolete behavior can be enabled or disabled with the '_POSIX2_VERSION' environment variable (*note Standards conformance::).
alias d='[ ${#DIRSTACK[@]} -gt 0 ] && for i in `jot ${#DIRSTACK[@]} 1 ${#DIRSTACK[@]} 1`; do echo ${DIRSTACK[$i-1]}; done; echo $PWD'
alias popd='[ ${#DIRSTACK[@]} -gt 0 ] && { cd "${DIRSTACK[${#DIRSTACK[@]}-1]}"; unset DIRSTACK[${#DIRSTACK[@]}-1]; }; d'
alias pushd='DIRSTACK[${#DIRSTACK[@]}]="$PWD"; cd'
For others who don't know it: http://en.wikipedia.org/wiki/Pushd_and_popd. It essentially is a stack for the working directory, so you can get back to where you were previously quickly and easily.
My biggest improvement was about finding out about .inputrc and configuring the Bash to use VIM-keybindings which is pretty handy if you are used to the editor. Also the following mapping from ESC to pressing jf via
imap jf <ESC>
was very nice as I find it much more ergonomic. And to use it in Bash I have
set editing-mode vi
set keymap vi
$if mode=vi
set keymap vi-insert
"jf": vi-movement-mode
$endif
I have my Bash set to vi keybindings, but I had no idea it was possible to remap them. Now I can finally stop hitting escape in Bash too. Thank you so much!
Yeah it is great if you can use similar commands across applications. Btw. there is also a nice history inside VIM (implemented as a separate buffer) that can be accessed using q: so you can scroll through the history with jfkl. E.g. I use that for accessing files: If you are working on a project you will probably edit similar files. If you use one "programming" vim sessions where you always edit files via :tabe you will find all the recent edited files in your history and using q:/ you can search for them pretty fast. So over time you have automatically a working environment adapted to your current project.
Better yet use 'less +F'. It's as easy to remember as tail -f.
The advantage of using less here is that you can easily interrupt the tailing (ctrl-c). For example, to search. Then just hit 'F' again to restart tailing. But now your search pattern will be highlighted as the data streams by.
I have been using both "tail -f" and less for years, and never knew about this. Thank you. In three lines of text, you have made my logfile-centric life oh-so-much easier to deal with.
I believe xxd is actually part of vim and not a Unix command itself. It's used to support hex-editing binary files, but you can see why it's useful in other roles as well.
The origins of xxd seem to be a bit murky, as I've seen others say the same thing about xxd being available only if vim is installed, but I've never seen a Unix system that didn't have vi(m) included.
$ man xxd | grep vi
Use xxd as a filter within an editor such as vim(1) to hexdump a region
Use xxd as a filter within an editor such as vim(1) to recover a binary
Use xxd as a filter within an editor such as vim(1) to recover one line
From what I've been able to find, vim was released in 1991, and xxd has been around since at least 1990, whereas vi was written by Bill Joy in 1976. (according to Wikipedia)
Interesting bit of trivia, perhaps someone with a good memory can clear it up for us?
There are no hits for xxd on the Unix Tree (http://minnie.tuhs.org/cgi-bin/utree.pl) so it's definitely not a traditional Unix tool. I'm sitting in front of a BSD system which includes nvi. I don't have xxd.
There was a time when I, annoyed at being thrown into vi(m) whenever something spawned an editor, rather than setting the EDITOR environment variable on login, removed vi and replaced it with symlinks to emacs on the servers I managed.
I'm a bit more tolerant of vi now, but just because I "never" see it courtesy of suitable EDITOR entries.
At any password prompt, if you knowingly mis-keyed your password, rather than pressing return, getting the "your an idiot" message (often after an annoying 1 - 2 second delay), and having to retype the command, you can actually press ctrl-U, which is the "Clear the line" bash command, and start over typing your password again.
For the ssh section, I'd also include sshfs. It's super handy when you want to use local (or GUI) applications for a file on a server, or just to mount your home computer as a drive while you are away. I personally use it to mount my backup server's drive on my Chromebook, effectively giving myself a 500GB hard drive whenever I need it.
I'm feeling rather good about myself because although I'm a complete novice at linux, I knew the cal command. Yay me! I read about it here:
http://linuxcommand.org/tlcl.php
>I also use ctrl R <keyword> at the bash prompt but that only gives me the most recent match. //
Are you sure, I'm using BASH on Kubuntu and I get previous matches by repeated use of ctrl+R.
Actually inspired by a previous post here I made a script "hs" that searches archived history files and shows me all past match lines. Before that I did what you do now.
When do you need man ascii and xxd? I'm a front-end developer so I have no clue about these low-level things but I'm interested and would like to learn.
There are a lot of usecases for bot of them, mostly debugging I's say. So I'm gonna tell you what I used them for in the past months:
The ASCII table might be useful when you're dealing with encoding problems and want to know e.g. what exactly happened in a broken URL encoding. I used both of them for binary exploitation in the security class I took at university: When utilizing buffer overflows, it's important to get every byte at the right place of your payload. Sometimes, you also need to treat it as a string and that's where ASCII comes in.
These are only two specific usecases. As I said, there are a lot more. Development of binary file parsers could be another.
There are free unix shells out there, google will find them. Don't expect anything fancy, but you will be able to try all the commands talked about here.
To connect to the shell from Windows, you can use the Putty program (google for it), although the shell documentation will almost certainly tell you how to do it.
Wow, now I feel old. I thought everyone got started with SunOS on a Sparcstation 2? Seriously, don't most people encounter it either through installing a Linux distribution or firing up a terminal on OSX? Or is even my-first-Unix a virtual experience?
3. There is always nice command that does what you want but you somehow forget it. I can't believe that Common Lisp has just 900+ symbols but I routinely forget some of them when programming.
My favorite part is when you implement something you could've had running in a minute or so, but instead developed it for hours or days and it's still not as good as the existing solution. I often get that with C and libraries. There has to be a better way to search through existing stuff which is good and proven, not just for C of course.
Haskell has hoogle [0], that lets you search by type signature. If you already know roughly what kind of function you want, it's a lot easier to sift through a tiny handful of (or even a single) functions that match a given type signature.
ISTR Forth had something similar, or maybe I'm just thinking that in Forth, words are almost uniquely defined by their stack annotations.
[2] https://news.ycombinator.com/item?id=5022457
[3] https://news.ycombinator.com/item?id=4985393
[4] http://sysadmincasts.com/episodes/13-crash-course-on-common-...
[5] http://sysadmincasts.com/episodes/12-crash-course-on-the-fil...