I do something very similar, but without the prompt settings. I have settings in .bashrc[0] to have the history file based on date. I then use fzf[1] (fzf-tmux is great) and a grep-like tool(sift[2]) to use for ctrl-r that fuzzy-searches history and orders by usage frequency[3]. This way I can easily search for the command I'm thinking of fairly quickly. Particularly useful for those times I want to run a command again that was quite long or had more than a couple options/flags.
For those who want a well thought out solution, I have a co-worker that created a very useful history shell script with lots of integrated tools for search. Just source in your bash profile.
Yes, the stars are missing, but not where you think. The --files option has a star on the inside of each quote. This makes the search only show history for the host you are on (I sync my Dotfiles between machines).
sift searches a path/folder so the star you have is not needed.
But yeah, HOSTNAME_SHORT is something I derive from HOSTNAME.
the difference is that the author's version backs up every command to a file as it happens. If you have multiple terminals open, you do not get the same output running "history".
That does make multiple terminals play nice with eachother, but it's still different, as OP's still flushes to HISTFILE after every command. But, then we can do that with
I used to do the same thing to figure out which commands I should create short aliases for. Sounded like a good idea at the time but then I realized that I'm creating a file with an awful lot of interesting information in it and I not getting a lot in return. So I set my HISTSIZE to 1000 which is more than enough for interactive shell use and I don't have to worry about having stuff like "youtube-dl fuckmesilly.com/${insane_porn_title}" lying around on servers I have access to. (Or which server you can connect to with my private key -- you might as well remove HashKnownHosts from your ssh config if you log it all yourself.)
not sure about "not getting a lot in return", there have been plenty of times where I have had to hand build something with strange defaults or additional commands, then six months later I am downloading the new version of the source and having the history of what I did back then is a lifesaver. Not to mention the times you remember you edited something somewhere and partially remember where/when but not exactly so you can just fuzzy search vim dirname and filter.
It does depend on what one does on their computer, obviously, but in my opinion for developers having a more or less permanent history with a fuzzy matcher (fzf is what I use) is extremely important.
I personally have bash set up so it saves a new history file each month, and a custom fzf alias to search on all of them, and I wouldn't want to have it any other way, I also treat these history files as confidential and make sure they are not in git etc. and if I have to input any passwords on the command line I just prepend the command with a space so it's not saved
Servers should have zero history saved on the disk. It gives any intruder an easy place to look for passwords, private keys, etc that may have been accidentally recorded and gives clues about related systems.
If you have administrative stuff you need to do more than once, write a little script or alias for it. Depending on history for this is just lazy.
OTOH what about audit trail? Are there any standard solutions for saving commands input at servers without giving person inputting those commands access to the logs?
Also, silly idea for a DOS attack vector: script-spam enough commands to have the audit history consume all available space on server.
We use rootsh[1] logging to syslog, which gets forwarded to a logging server, which in turn is periodically copied to a wholly separate AWS account, so that in case of breach of the main account the audit logs are intact.
Excellent point if anyone can get access to your home directory files.
I work around the security issues by not backing up history and having encrypted file systems on all of my Linux laptops. I don't save history on my servers.
Yep, also back in the day it was common protocol to have .history files chmod'd to 600 by default.
Having an unlimited history is great. Sometimes there'd be a tool or resource I'd vaguely remember skimming (e.g., "oh yeah, I was working on ___, so it must have been on (day +/- 24 hours)". I'd find a command that more or less is chronologically synced with the command, then go into Chrome's SQLite history db in my User Profile to find it.
That wasn't necessary. You could easily have made the very same point without calling someone's personal pastime/hobby, "garbage".
It's one thing to use porn as an example of "things that may be awkward if found in my bash history", it's something else to get judgemental about it. Unnecessary and off-topic.
The default bash setting is to not write history until the session terminates. In which case the whole session will be forgotten, not just the stuff after overwriting HISTFILE (I always set it to /dev/null FWIW, didn't know blank was sufficient).
If you find yourself doing a series of `cd` and `ls` commands frequently, I feel like you might be using `cd` and tab completion ineffectively.
Instead of doing:
ls
cd foo/
ls
cd bar/
ls
cd baz/
You can do:
ls
cd foo/<tab><tab>bar/<tab><tab>baz/
This isn't what it will look like, it's just the key sequence (if this doesn't make sense let me know and I'll try to explain better). <tab><tab> will list all the possible completions, which is equivalent to an ls, without having to type in an ls and then another cd.
You can frequently avoid even more `cd`/`ls` shenanigans by judiciously using `pushd`.
Tab autocomplete will show everything, and showing a screen full of name is not particularly useful. Most of the time I use ls it's something like "ls (asterisk)foo(asterisk)".
But if there are multiple candidate completions, you don't get a list of them, the way you do from <tab> <tab> with no glob - it just completes unconditionally to the first candidate.
So you're saying you just showed me the one feature bash has over zsh!? Bash has an env var named HISTIGNORE that will allow you to not put commonly used boring commands in your history. Mine looks like:
Just a point on security, many advocate that logging commands is a major security weakness. Similar to why SSH now hashes entries in ~/.ssh/known_hosts by default. The idea is you don't want to provide hints on which remote systems you connect to, as these can offer a springboard to the intruder.
And one has to be mindful of that time you ran:
export AWS_CREDENTIAL=xpXfLVsY/77Nr+m1mKmys719h0m2z2BCYSv9d5r
That is then an increased risk of breach because it is kept around for a long time. YMMV. Defense in depth, don't use production secrets in development, etc, etc.
Too bad the author didn't Grep "password" plus a few lines on either side. Even if you sudo <stuff> and type outside the prompt once every few hundred attempts it's still gonna turn up a lot after a year or more.
Even the most novice of adversaries would have a field day with the bash history of a lower level IT admin.
I do the opposite and maybe weird - I disable history, and keep the history file contents hand curated.
Whenever I come up/across/use a command I'd like to keep for future or use frequently, it gets appended to the history file.
I use `unset HISTFILE` to make sure when I exit the history file doesn't get all the cruft in.
Every now and then I might do some housecleaning, including going through the history file then `cp ~/.bash_history{,.2}` to keep a clean copy around.
As far as saving, heres a handy function:
`getlast() {
fc -ln "$1" "$1" | sed '1s/^[[:space:]]*//'
}`
1. Regarding commands with credentials or sensitive info, bash allows you to prefix the command with a space character to omit saving the command to the history. This requires remembering this fact, and of course you get nothing in your history to refer to later.
2. I remember a HN submission for a piece of software (or collection of softwares?) that provided a centralized log of your command history across multiple machines. In this way, if you are working on multiple machines in a cluster, you can later search back through your histories from those machines in a single location. My Google Fu is failing me now though.
I have a little bit of bash to save my history files and a custom search, hss, to search across them. This makes the exposure an order of degrees, if it's in bash history already then it's going to hang around longer. If I weren't doing this if still have the same problem with bash's built-in/default history.
I started this in response to a HN post some years ago.
For better pretty much everything, one may wish to try zsh. It's pretty much a strict superset of bash, but better.
There is one aspect in which it's significantly worse, though: it's fiendishly difficult to configure and understand. I just use configs provided by others, which is not ideal but works.
bash (with all features) is also a strict superset of bash (with default configuration that people think is all there is to it, especially if they use macs [1]), but better.
[1] If you are using a mac and haven't installed a newer copy of bash you're using a version that's nearing on 10 years old and does indeed mostly suck.
Isn't this going to be pretty slow? It starts 5 subshells for every prompt. You could at least change $(pwd) to $PWD and switch the square brackets to double square brackets.
I don't get it, isn't your bash history already logged to `.bash_history`? What's the point of using PROMPT_COMMAND to send it in addition to another file?
A warning: do NOT do what one engineers I used to work with did, and upload his "dot files" onto GitHub for portability. A lot of secrets can be stored in a few months' worth of bash history.
there's nothing wrong with uploading your dotfiles -- lots of people do that -- just as long as you do it sanely and don't upload things like .ssh/id_* and .{bash|zsh}_history
For the zsh users out there: I wrote a zsh plugin [0] a while ago which does a similar thing (writes time, directory and command to a file) + gives you a better (directory sensitive) history search.
I go one big step further than this and log everything that comes across the screen.
One time it saved me from a crontab -r that wiped out a 100+ line crontab. I had viewed it recently so I just copied it out of my history.
On a day to day basis it's more about looking up old queries I typed out, the results of those queries at that time, bash commands and their results, the state of a file I edited at a certain time, a stack trace, the output of an ls -l command, etc. Anything I ever do in a session.
Some of those have their own logs or ways to capture history, but a way to capture everything at once is much more comprehensive and less sensitive to forgetting to set the size of the bash history on a given machine, archiving the bash/psql history files when a machine goes away, etc.
And besides not having to worry about the availability of distributed history/log data, being able to grep everything at once is invaluable.
script gives you the whole console, including output and control characters. It's a useful tool but might be a little overkill for keeping track of what commands you have run.
This is useful if you are developing locally, but what if a lot of your command line usage is on remote servers that you have SSH'd into?
I believe iTerm2 has some support for logging remote commands, but I'm not sure it quite does the trick. Anybody know of another tool that will log both local and remote commands?
Another problem for me is the ephemeral nature of many of the boxes I work on - Vagrant boxes, spot instances on AWS. I suppose I could do this and then export the bash history as part of the teardown process.
At the moment, I get around this by storing as much as I can in searchable scratch files, but this relies on me knowing what to copy. I'm sure there's a better way.
I use the built-in script command sometimes when I'm using SSH, set to write a date+server named text file to my history folder. You could do that and awk out the output lines afterwards.
You would also need to unset HISTFILESIZE, export PROMPT_COMMAND="history -a", export HISTTIMEFORMAT="%d/%m/%y %T ", and set up log rotation to get the behavior listed in the article.
I have many shells open at once, sometimes dozens. Only one of them "wins" when saving history. If Bash has out-of-the-box support for merging multiple shell history, it's not obvious in the man page. And I have them open precisely because they're different contexts and I don't really want them sharing history. If you want a log of everything you run, you need to make it some other way.
So, basically, in combination with sp332's and raldi's points, the answer is that it is completely not the same.
I took the bash-history related stuff from https://github.com/mrzool/bash-sensible and have been VERY happy with it. Among other things, it makes sure all your histories from various open terminals get merged, not overwrite each other -- I think maybe just `shopt -s histappend` is enough for that? But I was messing around with my settings regarding history settings for a while tweaking and tweaking, until I found bash-sensible, tried it out cut and paste, and found it was perfect.
## SANE HISTORY DEFAULTS ##
# Append to the history file, don't overwrite it
shopt -s histappend
# Save multi-line commands as one command
shopt -s cmdhist
# Record each line as it gets issued
PROMPT_COMMAND='history -a'
# Huge history. Doesn't appear to slow things down, so why not?
Yours is a good additional idea, but it is in no way a substitute. You might want to look at some history you forgot to log. Having intentional logging on top of comprehensive logging is like markup metadata, very handy too.
it makes me sad when everybody doesn't see the same things the same way. I'm just waiting now for systemd to disable bash history in favor of logging in binary form all mouse rolls intermixed with key clicks till it wipes it all out when you log out in case yours is a student account on a shared university machine. Don't worry, a GUI editor for it all is on the wishlist.
6 of one, half dozen of another. It really depends on how you see things. For security reasons we've disabled logging. And users don't access servers at the shell anyway so no need for it there. We log app usage in app.
I find screen logging a bit more useful than shell logging. Shell only records the commands sent and not the result. Screen records everything including interactive like nano.
the most precious thing to a computer should be the human input, that's the thing that your brain worked on and that's what primarily should be recorded. (that's what source code is) Computers are deterministic, replaying your input should yield the same output again.
That is why it is soo important to save .bash_history (seriously, that's not a dot file you should be searching hard for)
Damnit, this is so much smarter and easier than what I did!
I've been trying to leverage bash's history, and wanted to avoid putting a file write into my prompt hook. But I have a big kludgy system for exporting bash history separately for each day, de-duping the overlapping history, and trying to make sure bash always logs the command I typed.
I'm still fighting with bash sometimes not saving my history commands. I've done everything recommended in all corners of the Internet, but certain keyboard combos will still cause history lines to not get saved. It's maddening, and the solution here avoids it completely.
As far as I can tell, bash deals with multiple shell sessions exactly how I'd expect (and want) it to: an individual terminal doesn't have access to the live session from other terminals, but when you close it it appends it's session to the shared history file, so each session ends up a continuous block in that file.
I don't know what it does exactly, but it always seems to lose history after a reboot. Either the last shell session to close overwrites the others, or they fail to save the history if not closed cleanly enough.
Wouldn't the ideal solution not be a component of a simple file writer, but rather, a buffered writer? A command gets executed, thrown into buffer. A wait process in bash checks buffer, and then writes to .bash_history?
On Mac OS X and iTerm2 I've noticed a thing where pasting in commands means they don't get added to the history and using up arrow or search to find them doesn't work. Anyone seen that? It's one of those things that annoys me every time it happens, but I'm always in the middle of a task when it occurs so I never get a chance to figure it out.
This is controlled by an environment variable, HISTCONTROL:
A colon-separated list of values controlling how commands are saved on the history list. If the list of values includes ignorespace, lines which begin with a space character are not
saved in the history list. A value of ignoredups causes lines matching the previous history entry to not be saved. A value of ignoreboth is shorthand for ignorespace and
ignoredups. A value of erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. Any value not in the above list
is ignored. If HISTCONTROL is unset, or does not include a valid value, all lines read by the shell parser are saved on the history list, subject to the value of HISTIGNORE. The
second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of HISTCONTROL.
This is probably it, thanks, I didn't know this could occur. It only happens when I copy the output of a particular command, not always, so I'm guessing it has a space.
I've just tried and I can't reproduce. It might be because you're copying a leading space: if a command is prefixed by white space it doesn't get saved in the history.
I implemented bash audit logging[0] by making use of Ryan Caloras's bash-prexec[1] project which provides a fairly robust and resilient way to implement ZSH's preexec and precmd functionality.
Some of the features of my solution are that it creates a sub folder in the user's home directory called ~/.bash_history and underneath this it will create sub folders for each month (YYYY-MM) and under each of those sub folders will reside a daily audit log file of all the bash command history (YYYY-MM-DD). The audit script logs both login and log outs, as well as each command executed in bash.
^ Here is my comment on a previous post for how I am logging all of my bash history in a logical manner (keeping track of terminals and timestamps, etc.) to an sqlite database.
I've been tempted to do something similar. The context information (e.g. environment, working directory) would be especially helpful, since that's what I'm currently missing. Have you tried https://github.com/umang1912/advanced-shell-history? That was where I planned to start.
Most of these are extremely complex for almost no good reason. Storing you bash history "in the cloud" seems like a dumb idea to me, requiring a Python script for this seems gratuitous... and essentially all of these are using PROMPT_COMMAND (except bash-history-sqlite, which seems more legit), which doesn't have the correct behavior. I remember all sorts of issues with the timing of what is logged, how pipes are handled, and interaction with expected history behavior. I'm going to counter-ask: have you tried my script? ;P I recommend starting with my script, but obviously I'm going to be biased. Almost all of my script is the actual SQL parts: you will note that I have almost no shell shenanigans. At least from the perspective of asking me why I haven't tried something else, I feel like you would have to provide some rather extreme motivation at this point to work with one of these much more complex scripts.
"For the last three and a half years, every single command I’ve run from the command line on my MacBook Pro has been logged... the return I’ve gotten on that small investment is immense."
I'm still not sure why it's helpful to have three and a half years of your bash shell logs.
The logger in me appended hostname and the tty to the file name and the log message. You could also add a user name to this as well[0].
I love this idea because you can easily analyse your commands then modify or automate your workflow where it makes sense. I've done this in the past to create a two letter aliases to frequently used bash commands.
I've written a bash history server in go and it has saved me more time than it took me to write it.
It stores commands in a sqlite database. For each command it also saves the time, the user and the host.
It permits global search with some useful features (grep A/B/C switches, regexp, etc).
I wrote it to scratch an itch, thus the documentation isn't user friendly.
If anyone is interested, shameless plug: https://github.com/andmarios/bashistdb
A long time ago I set my history file to save up to the previous 1000 commands, and I ended up mostly learning grep for the first time because I wanted to search quickly for various recent commands. Stuff like "history | grep _some ip_" or things like that.
Storing the entire history would no doubt be helpful, but I feel similarly to what other people have said so far, which is that it may not be the safest thing to do? Probably depends on usecase.
I have bash script in my bash_profile to support separate history files named with tty, month and year (so separate files for each tty). It looks for recent history files for the given tty and reloads old history on first startup.
Have cron job to signal bash session to flush history with command timestamps/dates to disk daily.
Handy for recalling commands/command switches you ran months ago.
I like this idea but I know that I have accidentally entered my password or my password with a typo in it on the terminal on at least one occasion.
The only way round this that I can see is to grep out the password - but then my password (or a substring of it) is now in plaintext in my .zshrc/.bashrc
I can't think of a solution to this. Does anyone have anything?
I wrote a command called 'lc', which list commands in a directory. I've found that what I really want is context-specific command history,and this provides it.
I frequently review my shell history to pickup past command invocations, but I think it has limits.
At some point, you have to clean things up and create a script for repetitive tasks, or write wrappers to tame complicated command line syntax. I think curl and groff are my favorite examples of commands that have too many options.
Doing the same. Also including the git branch (if applicable)
It's not obvious, but with this mechanism commands are only logged once they complete. So the log file does not include commands while they are still running. Run into that from time to time.
I recently had a bash hiccup causing the history to vanish. Not a problem for 90% of things but for a few long rsync commands (the one time crafts for you) it was a PITA. Or an opportunity in RTFM says the noob.
with El Capitan, my bash history isn't logged, with each fresh session letting me know Permission Denied on creating ~/.bash_history. The internet searching tells me to disable csrutil which I did, to no effect.
I use the reverse lookup and zsh history heavily and it has served me well. Whats the benefit of this, logs that go back for a longer time ? I am not sure what the default setting on zsh is tbh.
Don't go overloading `require()` just because you can - the modules API has been locked down for a couple of years now, messing with the semantics causes all sort of fallout. - https://nodejs.org/api/modules.html (I'm looking at you webpack, babel)
Instead better would've been to use a custom method; internally the `require.resolve()` API can still be used to get the same semantics as `require()`
I also use script to log both input and output of every terminal, and the timing option to record the timing of that input and output too. Naturally stored with caution, and with a shortcut to a non-logging terminal for dealing with anything sensitive.
---
[0] HISTFILE="${HOME}/Sync/Dotfiles/history/$(date -u +%Y-%m-%d.%H.%M.%S)_${HOSTNAME_SHORT}_$$"; export HISTFILE
[1] https://github.com/junegunn/fzf
[2] https://sift-tool.org/
[3] __fzf_history__() {
sift --no-color -e "^[^#]" --files "_${HOSTNAME_SHORT}_" -N --no-filename $HOME/Sync/Dotfiles/history | sort | uniq -c | sort | $(__fzfcmd) +s --tac +m -n1..,.. --tiebreak=index --toggle-sort=ctrl-r | sed "s/ [0-9] *//"
}