How and Why to Log Your Bash History

I do something very similar, but without the prompt settings. I have settings in .bashrc[0] to have the history file based on date. I then use fzf[1] (fzf-tmux is great) and a grep-like tool(sift[2]) to use for ctrl-r that fuzzy-searches history and orders by usage frequency[3]. This way I can easily search for the command I'm thinking of fairly quickly. Particularly useful for those times I want to run a command again that was quite long or had more than a couple options/flags.

---

[0] HISTFILE="${HOME}/Sync/Dotfiles/history/$(date -u +%Y-%m-%d.%H.%M.%S)_${HOSTNAME_SHORT}_$$"; export HISTFILE

[1] https://github.com/junegunn/fzf

[2] https://sift-tool.org/

[3] __fzf_history__() {

sift --no-color -e "^[^#]" --files "_${HOSTNAME_SHORT}_" -N --no-filename $HOME/Sync/Dotfiles/history | sort | uniq -c | sort | $(__fzfcmd) +s --tac +m -n1..,.. --tiebreak=index --toggle-sort=ctrl-r | sed "s/ [0-9] *//"

}

This entire blog post could be reduced to use the HISTFORMAT variable, which the author doesn't apparently know about. Put this in your ~/.bashrc:

    export HISTTIMEFORMAT='%Y-%m-%d %H:%M.%S | '

    $ history | tail
    50198  2016-05-31 10:15.57 | cd docker
    50199  2016-05-31 10:16.03 | cd rpms/
    50200  2016-05-31 10:16.11 | scp docker* omniscience:/tmp/
    50201  2016-05-31 10:14.06 | screen -ls
    50202  2016-05-31 08:33.34 | screen -x
    50203  2016-05-31 19:06.53 | grep HIST .bashrc
    50204  2016-05-31 19:07.46 | history | tail
    50205  2016-05-31 19:08.10 | task ls
    50206  2016-05-31 19:08.23 | docker ps -qa
    50207  2016-05-31 19:08.30 | history | tail

    $ grep HIST .bashrc
    HISTCONTROL=ignoredups:ignorespace
    # for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
    export HISTFILESIZE=99999
    export HISTSIZE=99999
    export HISTTIMEFORMAT='%Y-%m-%d %H:%M.%S | '
    export HISTIGNORE="ls:exit:history:[bf]g:jobs"

I used to do the same thing to figure out which commands I should create short aliases for. Sounded like a good idea at the time but then I realized that I'm creating a file with an awful lot of interesting information in it and I not getting a lot in return. So I set my HISTSIZE to 1000 which is more than enough for interactive shell use and I don't have to worry about having stuff like "youtube-dl fuckmesilly.com/${insane_porn_title}" lying around on servers I have access to. (Or which server you can connect to with my private key -- you might as well remove HashKnownHosts from your ssh config if you log it all yourself.)

TLDR: Overwriting history is a feature.

I'm a big fan of saving history - trying to remember all the shell based commands we use is a nightmare!

If you're using zsh, a tip I picked up from [0] was to alias all your common commands like

    'cd', 'ls' 'fg'...

    ' cd' ' ls' ' fg' ... 

    setopt HIST_IGNORE_SPACE

[0] http://chneukirchen.org/blog/archive/2012/02/10-new-zsh-tric...

EDIT: formatting

Just a point on security, many advocate that logging commands is a major security weakness. Similar to why SSH now hashes entries in ~/.ssh/known_hosts by default. The idea is you don't want to provide hints on which remote systems you connect to, as these can offer a springboard to the intruder.

Just for those out there using zsh or fish I used to following in my .zshrc to get this working:

    precmd() {
        eval 'if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history | tail -n 1)" >>! ~/Dropbox/Logs/Bash/Macbook/bash-history-$(date "+%Y-%m-%d").log; fi'
    }

I do the opposite and maybe weird - I disable history, and keep the history file contents hand curated. Whenever I come up/across/use a command I'd like to keep for future or use frequently, it gets appended to the history file.

Well given that we copy and paste a lot, by mistake there can be credentials and sensitive data in the bash history...

Moreover, bash history doesn't work nicely across different terminal tabs or computers.

Does anybody know a tool providing better command line history?

I would just do this:

export HISTFILESIZE=5000000 export HISTSIZE=5000000 export HISTTIMEFORMAT="%F %T " shopt -s histverify shopt -s histappend

Isn't this going to be pretty slow? It starts 5 subshells for every prompt. You could at least change $(pwd) to $PWD and switch the square brackets to double square brackets.

I don't get it, isn't your bash history already logged to `.bash_history`? What's the point of using PROMPT_COMMAND to send it in addition to another file?

A warning: do NOT do what one engineers I used to work with did, and upload his "dot files" onto GitHub for portability. A lot of secrets can be stored in a few months' worth of bash history.

For the zsh users out there: I wrote a zsh plugin [0] a while ago which does a similar thing (writes time, directory and command to a file) + gives you a better (directory sensitive) history search.

[0] https://github.com/tymm/zsh-directory-history

I go one big step further than this and log everything that comes across the screen.

One time it saved me from a crontab -r that wiped out a 100+ line crontab. I had viewed it recently so I just copied it out of my history.

On a day to day basis it's more about looking up old queries I typed out, the results of those queries at that time, bash commands and their results, the state of a file I edited at a certain time, a stack trace, the output of an ls -l command, etc. Anything I ever do in a session.

Some of those have their own logs or ways to capture history, but a way to capture everything at once is much more comprehensive and less sensitive to forgetting to set the size of the bash history on a given machine, archiving the bash/psql history files when a machine goes away, etc.

And besides not having to worry about the availability of distributed history/log data, being able to grep everything at once is invaluable.

This is useful if you are developing locally, but what if a lot of your command line usage is on remote servers that you have SSH'd into?

I believe iTerm2 has some support for logging remote commands, but I'm not sure it quite does the trick. Anybody know of another tool that will log both local and remote commands?

Why logging to a file when you could just set the HITSIZE variable in your .bashrc ? (plus this will give you ctrl+r search which is a must)

Use screen instead. Ctrl-a and "H" to enable and the same to disable. Log is saved in user directory as screen.<session>

That way I have control of when I log and what gets logged. And way easier to find than hunting for dot files.

Damnit, this is so much smarter and easier than what I did!

I've been trying to leverage bash's history, and wanted to avoid putting a file write into my prompt hook. But I have a big kludgy system for exporting bash history separately for each day, de-duping the overlapping history, and trying to make sure bash always logs the command I typed.

I'm still fighting with bash sometimes not saving my history commands. I've done everything recommended in all corners of the Internet, but certain keyboard combos will still cause history lines to not get saved. It's maddening, and the solution here avoids it completely.

This looks like quite a dirty hack, but I'll try it.

I wish bash history wasn't so broken. I suppose once upon a time it was ok to assume people only had one shell session at a time.

Possibly a good opportunity to ask:

On Mac OS X and iTerm2 I've noticed a thing where pasting in commands means they don't get added to the history and using up arrow or search to find them doesn't work. Anyone seen that? It's one of those things that annoys me every time it happens, but I'm always in the middle of a task when it occurs so I never get a chance to figure it out.

I implemented bash audit logging[0] by making use of Ryan Caloras's bash-prexec[1] project which provides a fairly robust and resilient way to implement ZSH's preexec and precmd functionality.

Some of the features of my solution are that it creates a sub folder in the user's home directory called ~/.bash_history and underneath this it will create sub folders for each month (YYYY-MM) and under each of those sub folders will reside a daily audit log file of all the bash command history (YYYY-MM-DD). The audit script logs both login and log outs, as well as each command executed in bash.

---

[0] https://github.com/onelittlehope/bash-prompt [1] https://github.com/rcaloras/bash-preexec

https://news.ycombinator.com/item?id=10695305

^ Here is my comment on a previous post for how I am logging all of my bash history in a logical manner (keeping track of terminals and timestamps, etc.) to an sqlite database.

"For the last three and a half years, every single command I’ve run from the command line on my MacBook Pro has been logged... the return I’ve gotten on that small investment is immense."

I'm still not sure why it's helpful to have three and a half years of your bash shell logs.

The logger in me appended hostname and the tty to the file name and the log message. You could also add a user name to this as well[0].

I love this idea because you can easily analyse your commands then modify or automate your workflow where it makes sense. I've done this in the past to create a two letter aliases to frequently used bash commands.

[0] export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(hostname) $(tty) $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log.`hostname`; fi'

I've written a bash history server in go and it has saved me more time than it took me to write it.

It stores commands in a sqlite database. For each command it also saves the time, the user and the host. It permits global search with some useful features (grep A/B/C switches, regexp, etc).

I wrote it to scratch an itch, thus the documentation isn't user friendly. If anyone is interested, shameless plug: https://github.com/andmarios/bashistdb

ICYMI: Sensible Bash addresses this well:

https://github.com/mrzool/bash-sensible

A long time ago I set my history file to save up to the previous 1000 commands, and I ended up mostly learning grep for the first time because I wanted to search quickly for various recent commands. Stuff like "history | grep _some ip_" or things like that.

Storing the entire history would no doubt be helpful, but I feel similarly to what other people have said so far, which is that it may not be the safest thing to do? Probably depends on usecase.

I have bash script in my bash_profile to support separate history files named with tty, month and year (so separate files for each tty). It looks for recent history files for the given tty and reloads old history on first startup.

Have cron job to signal bash session to flush history with command timestamps/dates to disk daily.

Handy for recalling commands/command switches you ran months ago.

I like this idea but I know that I have accidentally entered my password or my password with a typo in it on the terminal on at least one occasion.

The only way round this that I can see is to grep out the password - but then my password (or a substring of it) is now in plaintext in my .zshrc/.bashrc

I can't think of a solution to this. Does anyone have anything?

I wrote a command called 'lc', which list commands in a directory. I've found that what I really want is context-specific command history,and this provides it.

Linky: https://github.com/pconerly/lc-listcommands-bash

At work unfortunately the sys admins disable customisation. Especially HIST_SIZE etc :(

At regular intervals, especially when I have important commands to save I do a:

  cp ~/.bash_history ~/bash_history_bak_2016-05-31.txt

  cat ~/bash_history* | grep <important-command> 

I frequently review my shell history to pickup past command invocations, but I think it has limits.

At some point, you have to clean things up and create a script for repetitive tasks, or write wrappers to tame complicated command line syntax. I think curl and groff are my favorite examples of commands that have too many options.

In powershell there is a built in cmdlet for logging your history: start-transcript. Details here: http://til.secretgeek.net/powershell/Transcript.html

Doing the same. Also including the git branch (if applicable)

It's not obvious, but with this mechanism commands are only logged once they complete. So the log file does not include commands while they are still running. Run into that from time to time.

I recently had a bash hiccup causing the history to vanish. Not a problem for 90% of things but for a few long rsync commands (the one time crafts for you) it was a PITA. Or an opportunity in RTFM says the noob.

with El Capitan, my bash history isn't logged, with each fresh session letting me know Permission Denied on creating ~/.bash_history. The internet searching tells me to disable csrutil which I did, to no effect.

:-[

I use the reverse lookup and zsh history heavily and it has served me well. Whats the benefit of this, logs that go back for a longer time ? I am not sure what the default setting on zsh is tbh.

My job would be substantially harder if I couldn't ask bash history the question "how did someone solve this problem last time?"

ctrl-r does a fuzzy search in the history and provides most of what I want out of the box. I wonder why this hadn't been mentioned yet.

Don't go overloading `require()` just because you can - the modules API has been locked down for a couple of years now, messing with the semantics causes all sort of fallout. - https://nodejs.org/api/modules.html (I'm looking at you webpack, babel)

Instead better would've been to use a custom method; internally the `require.resolve()` API can still be used to get the same semantics as `require()`

I also use script to log both input and output of every terminal, and the timing option to record the timing of that input and output too. Naturally stored with caution, and with a shortcut to a non-logging terminal for dealing with anything sensitive.

export HISTFILESIZE=50000

export HISTSIZE=50000