- fd: A simple, fast and user-friendly alternative to 'find'
https://github.com/sharkdp/fd
- ripgrep: ripgrep recursively searches directories for a regex pattern
https://github.com/BurntSushi/ripgrep
- exa: Replacement for 'ls' written in Rust
https://github.com/ogham/exa
- bat: A 'cat' replacement. I recommend following the customizations.
https://github.com/sharkdp/bat
Written in Python:
- asciinema: Terminal session recorder
https://github.com/asciinema/asciinema
- httpie: Modern command line HTTP client
https://github.com/jakubroztocil/httpie
- visidata: A terminal spreadsheet multitool for discovering and arranging data
https://github.com/saulpw/visidata
- youtube-dl: Command-line program to download videos from YouTube.com and other video sites
https://github.com/rg3/youtube-dl
- pgcli: Postgres cli with autocomplete and syntax highlighting
https://github.com/dbcli/pgcli
Written in C
- jq: Command-line JSON processor
https://github.com/stedolan/jq
- tmux: a terminal multiplexer
https://github.com/tmux/tmux
- pspg: postgres pager (you can combine it with pgcli)
https://github.com/okbob/pspg
I wish HN handled long horizontal scrolling lines on mobile better. It’s aggravating that we are a site full of programmers and we can’t get the basics of usability in 2018.
One thing that I think most people don't consider often is that making things more usable isn't always good. Medium and message are interlinked. If everyone in the world switched to mobile, it would change the content of the articles we read and it would change the discussion of those articles. When you make something more usable on a different platform, you're encouraging the biases that the platform compels. In the case of mobile this means encouraging shallower content and briefer discussions, among other things.
Sure making things more usable may sometimes have trade-offs. But I'm struggling to think of the disadvantage of basic markdown support on a site where everyone knows markdown, and even assumes it is supported. People are constantly posting markdown lists that end up all on one line.
I assume the reason it hasnt been fixed is that nobody can be bothered, which is fine. But it's still a little embarrassing. Kind of like how Stackoverflow doesn't support triple backticks. I often see questions with formatting screwed up because people assume a site that heavily relies on markdown supports an extremely common markdown feature.
I know. I'm pointing out why this is a bad thing, because most people who advocate for better mobile support on Hacker News don't see themselves as advocating for a reduction in the quality of discussion and a reduction in the depth and complexity of the articles being shared.
- rq: A tool for doing record analysis and transformation (replacement for jq, supports yaml, json, avro, protocol buffers, etc.)
https://github.com/dflemstr/rq
As you are using tmux, also check out the awesome tmuxp written in Python. The author is very thorough with the package; testing and documentation. Tmuxp allows you to quickly launch pre-configured tmux sessions.
The benefit of tmuxp is pretty clear from its README[1]. It gives this example:
session_name: 4-pane-split
windows:
- window_name: dev window
layout: tiled
shell_command_before:
- cd ~/ # run as a first command in all panes
panes:
- shell_command: # pane no. 1
- cd /var/log # run multiple commands in this pane
- ls -al | grep \.log
- echo second pane # pane no. 2
- echo third pane # pane no. 3
- echo forth pane # pane no. 4
The example is very simple (I use tmuxp to heavily configure the session initialization), but it shows how you can have a config file to set a session's name, set the number and names of windows in there, set the layout of panes differently if you want in each window, run commands like "cd", set env vars differently if you want, in each pane.
You can find a lot more examples of tmuxp config on its doc site[2].
tmux doesn't allow one to have such quick, easy-to-read config files. You can of course do most of this in your .tmux.conf, but it will be too complicated to allow reuse and quick modifications.
I understand. But the point I am trying to make applies there too. A script to do what I am doing in the tmuxp config I last pasted would be too unmanageable, and not as readable and reusable.
Thanks for the links. In the future, please don’t use indentation to set off lists like this. It makes it difficult to read due to side scrolling, particularly on mobile. People often use - (hyphen) or * set of bullet points, and separate lines with new lines. You don’t get <li> elements, but it does work. Plus, your links will work, too :)
Given that we're members of HN (as opposed to someone running the site), I try to provide constructive advice that others can actually use to improve the situation. The admins are well aware of the situation; continuing to argue that it's a problem with the platform while refusing to do what you can to work with the given situation seems like a needless waste of a comment.
If by terminal programs you mean command-line interface (CLI) programs, then you'll never regret learning how core Unix commands work, in themselves and in pipelines, and their POSIX-defined interfaces. I use these often: make grep find sed xargs m4 cut tr cat tee sort uniq head tail pg/more/less ps pkill/pgrep. My favourite among them is make, once you learn how it's meant to be used, it's an incredibly useful and flexible tool.
Some other useful CLI programs that I use and are not part of POSIX are: strace, perl, webalizer, rcs. RCS might be a mostly-forgotten VCS tool, but it is very suitable for version-controlling projects that consist of single files. I mostly use it for plain-text or Org-mode documents.
If you meant terminal-UI (TUI) programs, like top/htop, vi, etc., I don't really use them. But I've used these in the past:
- mutt: a very nice mail client
- newsbeuter: RSS/ATOM feeds client
- vi: I do occasionally use vi in some situations
In the past years I've started to use Emacs tools instead of these because Emacs is a very rewarding, composable and comfortable working environment. Still, I use CLI programs through Bash/shell-mode all the time, because it's the most efficient way to do many tasks.
I like the composition of tools, it lets you extend your usage pretty easily. I personally run this 20 times a day: find -name '*.java' | xargs grep 'pattern'
But at least there you're searching for specific files. I was talking only about running a find to launch an instance of grep for every file found. That's more keystrokes to get a slower result.
For what it's worth, my biggest usage of find is similar to your java example (except a different file extension). I also do do '-exec sed...') a lot too.
I basically live in vim and tmux when I'm doing development work. Tmux IMO basically turns a terminal from a handy utility to a fully productive workspace.
I switched to ripgrep for my grepping needs, I find it fast and the syntax and output coloring handy.
Just found fastmod recently, https://github.com/facebookincubator/fastmod. Very handy for doing bulk substitutions of strings in project directories - it can prompt with a diff for every change and do full regexes, so you can try out tricky things and make sure it isn't doing anything crazy before you let it make 500 changes throughout your repo.
Also a fan of gitsh, https://github.com/thoughtbot/gitsh. Helps out with using Git on the command line just the right amount IMO - just avoiding the git prefix, tab completion, and a little syntax highlighting, instead of trying to redo the whole interface.
I don't use it that much, but jq, https://stedolan.github.io/jq/ is really handy for slicing and dicing json data without diving into the complexity of writing actual programs.
And on OS X, homebrew is indispensable for actually managing all of your CLI programs.
I also have found this Google Drive cli handy, https://github.com/odeke-em/drive. If you want to sync some but not all of your Google Drive files, and only push and pull on command instead of continuously, it's great.
git-standup seems cool, but misses the one thing I would want from it: show me the commits I've made from each repo that I work in, sorted by time. That way I can see that I made commit 1 in repo X, then commit 2 in repo Y, then commit 3 in repo X. Also instead of "-d 4" to see 4 days ago, "January 15th" or "Thursday" would be handy.
Yeah, I'm only interested in the actual flow of all my work, not the flow of my work in individual repos. I can use 'git log' to list my work in a single repo.
>This blog is intended as a quick look at terminal-based applications available in Linux, and in some similar operating systems. Not only are these ideal for modern, low-power and space-saving computers, but they go a long way toward invigorating out-of-date hardware.
locate may be the most-overlooked timesaver on the GNU distro. For searches of more than ~50000 directories, it's an order of magnitude faster than find:
$ time locate -0 /{opt,usr}/\*.gitignore |
xargs -0 stat --format=%s | wc
387 387 1225
real 0m0.087s
$ time find /{opt,usr} -name '.gitignore' -print0 2>/dev/null |
xargs -0 stat --format=%s | wc
387 387 1225
real 0m0.748s
The speedup comes from reading a single database file rather than every directory within the tree to be searched. If the files you want were last touched today, a manually triggered rebuild of the database may be needed:
$ sudo updatedb
The argument syntax for path specification of locate I find more convenient as well.
+1 for remind and wyrd. Remind is really, really powerful. You can express things like an appointment that fires "every 3 weeks starting May 4th, but if that day falls on a weekend or holiday then {skip it, move it to the following/preceding weekday that isn't a holiday}", and lots more. Wyrd is a really nice console frontend to remind (I just found that there are graphical frontends available too). Finally with rem2ics you can express really complicated things in Remind and then export the results as an iCalendar-compatible format and import that into whatever other calendar you want.
I used screen, remind+wyrd, mutt, and irssi (and even ledger) in school; I got a job at a bigco that pretty much required me to use gmail and gcal, and I stopped using console things as much. Lately I've been toying around with moving my mail flow back into console / console-ish inside emacs.
Emacs, tmux, tmuxp, ripgrep, eless[1], ncdu, exa (ls replacement. I often use 'exa --tree'), sed, awk, peco (I use this in aliases to select one or more items from a command outputting multiple values), hugo, dtrx, textract (npm), wget, shellcheck, rsync, htop.
I find myself using noti a lot. It's effectively just a message generator that can receive data over stdout or trigger manually. It uses whatever the most appropriate notification method for your system is, so it feels very seemless. I use it for long-running tasks where I don't really know how long it's going to take.
- googler: Google from the terminal
- Buku: Bookmark manager like a text-based mini-web
- nnn: The missing terminal file browser for X
- bcal: Storage expression calculator
- imgp: Multi-core batch image resizer and rotator
- keysniffer: Linux kernel mode debugfs keylogger
- ddgr: DuckDuckGo from the terminal
- pdd: Tiny date, time diff calculator
MPS-Youtube is extremely useful if you don't mind the CLI and want to quickly play a YouTube video.
translate-shell if you're a language person. Very complete interface to most popular translation services.
atools(aunpack, apack, als, etc) for quick archive manipulation if you're lazy.
Since all the good suggestions are taken, I want to mention a couple of diversions:
fortune: prints something silly or thought-provoking. You can get extra fortune cookie files or add your own.
cowsay: prints an ascii-art cow (or other creature) with a speech-bubble filled with the text of your choice. You can pipe fortune into cowsay for extra fun.
$ fortune | cowsay
______________________________________
/ Publishing a volume of verse is like \
| dropping a rose petal down the Grand |
\ Canyon and waiting for the echo. /
--------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
My latest find is to use a combination of weechat and wee-slack for most of my slack usage (it's note quite there when it comes to editing, or in-line reactions). But works well for the chat part:
Just ensure you're running a 256-bit color terminal emulator, and for most uses it doesn't even differ that much from the GUI version. Recently I've been doing a lot of work by SSH-ing to my desktop machine from a small netbook and using the terminal Emacs client. If you run Emacs in server mode, the transition is so smooth that at this point, I don't bother getting out of bed if I need to work on something late in the evening, I just take the netbook and SSH over.
Agreed. I also like to use TRAMP [1] for remote editing files in Emacs: it gets a local copy of a file you're editing so that it only requires a connection when you save/load (as opposed to constantly streaming Emacs over SSH). Not suitable for all tasks, but it means I'm rarely at the mercy of my lackluster internet connection.
If you don't have a reason to use Emacs from the terminal emulator, use the GUI. And for SSH, it's very hard to reject Tramp. One must is to make it less verbose though: (setq tramp-verbose 1). Then it becomes faster (at least in my perception) and way less noisy.
I find using emacs in the terminal less bothersome on osx because of osx's inane out of the box window management; I can replace it completely with managing emacs frames in screen and that experience doesn't change when I move between platforms.
Maybe I didn't word myself clearly enough - my reason for using Emacs from the terminal emulator is working from a small, portable machine on the go, and pick up stuff where I left it. It so happens that my travel machine is a Windows tablet/netbook, and Emacs has some performance quirks there. SSH-ing to a beefy desktop machine (that also runs an instance of the product I'm working on) saves me from fiddling with Tramp, and lets me almost seamlessly switch between travel mode and working on the desktop.
Piggybacking on OP's suggestion of youtube-dl, i have this alias for quickly grabbing the audio-only version of a youtube video. (I use it when I come across a song on youtube that I can't find for sale or streaming anywhere else)
alias dl-audio='youtube-dl --extract-audio --audio-format mp3'
If you have ffmpeg installed, just drop the video ID after dl-audio, and you'll get an mp3 of the highest quality audio stream available for the video.
There's so many programs and scripts I use it's hard to keep them all straight. So, instead of trying I just exported a list of every command in my history that I've used at least twice:
micro, mc, ag silver searcher, stat, find, aria2c, xev, w3m, wmctrl, nohup, xdotool, docker, php, ffmpeg, phantomjs, du, ls [From my history file, all others are homemade scripts and little functions in .bashrc]
which shows total disk usage for every file and folder in the current directory. I've actually aliased it as `dush` because I use it so often.
Bonus tip: if you're using btrfs, it has a built-in du that keeps track of cow statistics, so you can see which files/folders are deduped and which aren't. It's invoked with
btrfs filesystem du -s *
or (because the btrfs binary understands abbreviations)
Thanks. Will try it in details. Currently, it is taking like really forever to scan my bunchs of old school hard drives! du options are easy to remember, ergonomy first^ May I suggest a good one to remember `ls -ltrapR` <- It always kill anyone aside ;)
For me personally, I guess netcat, socat, tcpdump and SSH (for tunneling) qualify the most. I'm working in a Windows environment where I'm usually not allowed to install new software, and I miss those by far the most.
Often, I need to do a test run of something which involves slightly rerouting network traffic and I'll spend half a day hacking together some horrible "solution" that I could literally have done in 3 lines of bash code on a UNIX-like system. Maybe throw hexdump for looking at binary stuff in there too.
Edit: Oh, and of course GDB! If only I could have the accumulated days of log forensics back that were done just because it's still faster to do that instead of getting the visual studio debugger to work remotely. Whatever happened to KISS? -_-
It's a bit like a mix of (x)stow and apt for Windows. Or the good parts of arch aur with binary distribution.
Should probably also mention that I'm experimenting with asdf as a version manager - to avoid a long list of setup in bashrc to adjust paths to various interpreters. It follows a lot of best practices that various dedicated version managers seem to pick up eventually:
Some time ago I wrote just for fun this couple of lines:
> What else I'm missing out on
echo 'Currently there are ' $(ver=(/bin /sbin /usr/bin /usr/sbin) && ls ${ver[*]} | wc -l) 'commands installed and available in the system'
This prints the man page of 3 random commands. One by one. Is a way to discover a few new commands each day. To go to the next command and exit press q. I hope you enjoy it.
commandoftheday=(/bin /sbin /usr/bin /usr/sbin) && ls ${commandoftheday} |sort -R |head -3 |xargs man '{}'
As web applications are being split in micro services and independant entities, it becomes harder and harder to setup a local dev environment and you have to handle dozen of shell terminals.
GCE aims to take that pain away by offering you the ability to configure the setup of your projects once and run it in the blink of an eye.
- `ffmpeg` one utility for video manipulation to rule them all.
- `ping` to check whether I am connected to the internet or not.
- `curl -vvv maktabkhooneh.org` to check where is the connection problem happens.
- `howdoi` search StackOverflow for snippets to solve your problem from the command line.
tmux: A terminal multiplexer, aka turns your terminal to a minimalist workspace (the only sane way I found to copy terminal output without using a mouse)
REQUEST: jq is great for json - but is there a "jq for xml"?
Some webservices are still only available in xml (e.g. uv levels). I currently use XMLStarlet for xml in the terminal. There are extra annoyances in xml that you can't avoid (like namespaces), but it could be a lot closer to jq.
Thanks, seems the right idea, but browsing the code, decode just interprets a single xpath expression. While that is useful (and I've used xpath from the CL like before), I found I quickly needed to be able to chain them together etc.
Tmux and ripgrep are two that I use daily. I used to use cloc then switched to tokei, then to my own scc which I use frequently along with lc. Hyperfine is something I have been using a lot and despite having ripgrep I still use grep often with xargs.
I must confess based on the strength of the argument buruntsushi had with the Ripgrep comparison I switched from ag to rg and have not looked back. I was a daily ag user but I don’t need the advanced regex support and I do like the speed.
I was going to say xterm.. But I guess ssh, git, vim, gcc, make, node, tar, docker, elinks, mutt ls, ps, cp, mv, mkdir, timidity, minicom, tmux, diff, jq and some I can't recall right now
`cd` is not a command; it is built into your shell.
Some systems, like OS X and CentOS, map the cd man page to builtin which lists all the shell builtins and lets you know you should look at your shell's man page.
I've also been writing Python versions of some other Linux commands, and will post about them here some time later.
Also, for saving (and then viewing) man pages (about commands or other topics), without the formatting characters (such as ^H), on some Linux distros, I find this useful:
m, a Unix shell utility to save cleaned-up man pages as text:
> Also, for saving (and then viewing) man pages (about commands or other topics), without the formatting characters (such as ^H), on some Linux distros, I find this useful:
^H is one of the two backspace characters (the other being ^?) depending on which VT standard you're configured to use. So you shouldn't really need a dedicate script to do that as it's just a problem between your pseudo-TTY (Linux console) and your terminal emulator (PuTTY / iTerm / xterm / etc).
Thankfully both are easily configurable. However without knowing which terminal emulator you're using I couldn't walk you throw configuring that. In terms of the TTY; you can change the backspace character via the `stty` command:
stty erase [hit backspace on your keyboard]
Another thought is it might be a case that the TTY is largely configured correctly already but you're overwriting your environmental variables with non-standard values (eg the $TERM var with a value that differs from the actual terminal you're using) which is causing the pager (`less` / `more` etc. `man` will use this for the paging) to break the standard your terminal is expecting. But either way, this is definitely a configuration problem rather than something that should be fixed with additional parsing scripts.
> you shouldn't really need a dedicate script to do that
It isn't because the terminal is mis-configured and/or mis-interpreting those characters, it's because these characters have historically been used in a special way by troff's ascii output driver, as used by the man command.
Bold characters are "emulated" by outputting the character itself, followed by ^H, then by repeating the character, and underline is emulated by outputting _ (the underscore character) followed by ^H, followed by the character to be underlined. This is the same way that bold/underline was achieved on a manual typewriter or by old teletype terminals with paper output.
Pagers like "more" & "less" have in-built behaviour that knows how to interpret these sequences and render bold or underline appropriately, and if you "cat" the file your terminal would probably ignore them, but if you open a file with those sequences in a text editor, you're going to see a bunch of unnecessary ^H characters. The OP's script uses the "col" command to remove the unnecessary ^Bs (and the preceding character) that troff has output.
By default, GNU groff doesn't actually output those sequences anymore and uses ANSI escape codes instead. AFAIK many (most?) distributions actually compile out the ANSI behaviour in favour of the old way though (because "more" and "less" don't actually behave correctly with the ANSI characters by default), but some don't, which breaks the script (it's broken in Cygwin, for instance).
FWIW, if you need a fool-proof way to convert a man page to plain old ASCII with no escapes at all, it's easiest just to redirect the output of "man" to a file:
man ls > ls.txt
The long-winded way (with groff) is something like:
>Pagers like "more" & "less" have in-built behaviour that knows how to interpret these sequences and render bold or underline appropriately,
Right
>and if you "cat" the file your terminal would probably ignore them
I think the terminal does not ignore them. My guess is that it interprets the characters, but it has no noticeable visual effect on the screen. E.g. if the letter "c" in "cat" is output as "c^hc" (to make in bold in print), the terminal will just print "c", a backspace, then "c" again, which to the user will look the same as a single "c".
You described the issue and cause better than I did :)
I didn't know about the ANSI behavior, interesting.
>FWIW, if you need a fool-proof way to convert a man page to plain old ASCII with no escapes at all, it's easiest just to redirect the output of "man" to a file:
> man ls > ls.txt
IIRC, even with that way, the control chars still appeared in the file (at least on some Unix version), which is why I wrote the script in the first place.
> the control chars still appeared in the file (at least on some Unix version)
That's a good point. My whole reply is a bit GNU-centric: the "col -bx" solution should work with everything except groff in ANSI mode (looks like the flag to go back to the standard troff behaviour is GROFF_NO_SGR [1][2], in case anybody is interested).
Ah fair enough. The few times I have seen this issue was when jumping on old Solaris SPARC boxes which also mangled the the backspace key in the interactive terminal.
So it sounds like I've put 2 and 2 together and gotten 5.
Though coincidentally I do the redirect trick a lot as was as querying the gzipped raw files too (one of the projects I'm working on requires building a cutdown man page parser).
I first created that script called m years ago on some Unix boxes that I was using. Could have been HP-UX or other version. And I've used it over the years on many Unix and Linux versions that I worked on. It could be that in some cases, the script is needed due to a tty or TERM or other configuration issue. But I'm pretty sure that I've had the need for it (to remove those control characters) on at least some systems where such config was okay. I know this because, while I do tweak env. var., stty and other settings now and then, I do not always (need to) do so, and have still found control chars [1] in the man output, even when redirecting to file. That is why I created the script, because when working on a C, Python or any other project where I need to read man pages, I often like to redirect the man output to a file (in my ~/man dir) and then read them using vi/vim.
[1] See what I say in that post (about m) about nroff and troff.
Also, I do know about stty, have used it for years, although there is less need for it these days. Used to do a lot of tweaking and experimenting with the erase option (for which, BTW, instead of backspace key in your example, we can also write a literal ^h, i.e. caret, then letter h), intr option, onlcr, ocrnl and others. Used to be good fun and sometimes frustrating too, because docs for this areas were somewhat lacking then.
Yeah I'm familiar with groff et al. I used to get this issue on old UNIX boxes but not seen it in any of the Linux distros I use daily (not Freebase).
The other thing worth trying is seeing whether your locals are set correctly and/or set the GROFF_TYPESETTER environment variable. Though this might only by GNU specific. Or just redirect the output (as suggested by the other reply) which is how I used to get around the issue on SunOS at least.
I don't make a habit of commenting on shell scripts what I haven't first read. But thank you all the same.
I used to use 'col' alot to resolve the tabs Vs spaces problem with different actors imposing different preferences in their source code. It's a handy tool but often overlooked.
>I don't make a habit of commenting on shell scripts what I haven't first read. But thank you all the same.
Sorry, I should have added "if you haven't already read it", in my earlier comment.
>I used to use 'col' alot to resolve the tabs Vs spaces problem with different actors imposing different preferences in their source code. It's a handy tool but often overlooked.
Interesting, I may not have noticed or known about the tabs option of col. Cool. Seems it is like the entab program in an edition of the K&R C book.
