Jay Taylor's notes
back to listing indexBash eternal history
[web search]
zshtcsh
csh
dash (for lazy masochistic sysadmins)
sh
Other (please comment)
( 3383 votes ~ 28 comments )
Recent Weblogs
Bash eternal history
Posted by ateijelo on Wed 15 Aug 2007 at 10:08
Many times I've found myself using Ctrl-R in Bash to get a old command four times the terminal width, just to find out that too many days have passed and it's no longer in the .bash_history file. Here are two lines that will keep track of every command line you type at the bash prompt and use no external processes at all, just plain bash.
My first approach to this problem was increasing the maximum number of lines in the history to a very large quantity. But no matter how large it was, there was always a moment when I needed a long command I typed many months ago and it had already left the history. The current solution came to my mind when I learned about the PROMPT_COMMAND variable, a command that bash executes before showing each prompt. Here are the two lines:
export HISTTIMEFORMAT="%s "
PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo $$ $USER \
"$(history 1)" >> ~/.bash_eternal_history'
One goal I set to myself was to achieve it without using any external process, so bash wouldn't have to fork a new process after every ENTER pressed at its prompt. The first line sets the format of history lines to include the date as a Unix timestamp, so you can now when you typed every command. The second line, which is the core of the solution, first ensures that, if a previous PROMPT_COMMAND was set, it gets executed before our stuff and then appends a line of the format:
PID USER INDEX TIMESTAMP COMMAND
to a file called .bash_eternal_history in the current user home.
Adding the username, which at first seemed unnecesary, became useful later to distiguish between "sudo -s" sessions and normal sessions which retain the same value for "~/", and so append lines to the same .bash_eternal_history file.
I hope some of you find these two lines as useful as I do. :-)
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Just as a sidenote: the "-p" parameter of mysql is not followed by the password, it's just there to ask the password to be prompted interactively.
Your remark is still valid for other programs, though (like ncftp -p password or ldapsearch -w password).
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
in general, not a good idea to write a password on the command-line like this.
[ Parent | Reply to this comment ]
Just one problem to be aware of... when you first log in, there will be a random entry written to the file based on the the previous command you wrote that was saved to history in your last session.
thanks!
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
On startup, the history is initialized from the file named by the varia- able HISTFILE (default ~/.bash_history). The file named by the value of HISTFILE is truncated, if necessary, to contain no more than the number of lines specified by the value of HISTFILESIZE. When an interactive shell exits, the last $HISTSIZE lines are copied from the history list to $HISTFILE. If the histappend shell option is enabled (see the descrip- tion of shopt under SHELL BUILTIN COMMANDS below), the lines are appended to the history file, otherwise the history file is overwritten. If HIST- FILE is unset, or if the history file is unwritable, the history is not saved. After saving the history, the history file is truncated to con- tain no more than HISTFILESIZE lines. If HISTFILESIZE is not set, no truncation is performed.I haven't actually started using it yet (although I plan to), but I think you enable it by just adding
shopt -s histappendto your .bashrc.
[ Parent | Reply to this comment ]
If you unset HISTSIZE, every Bash session keeps in memory all the commands entered into it (instead of the last $HISTSIZE only.) If you unset HISTFILESIZE, at the end of each session Bash inconditionally appends all the commands it has in memory to the history file (instead of truncating the file to be at most $HISTFILESIZE lines.)
I've been using the following in my .bashrc for a long time:
unset HISTSIZE
unset HISTFILESIZE
Granted, it does'n have that fancy "PID USER INDEX TIMESTAMP" thing, but it's simple, builtin and fast.
[ Parent | Reply to this comment ]
And that would be a problem for me
$> wc .bash_history.archive
1871262 5533327 45226246 .bash_history.archive
I wonder how bash would perform... lets see...
koshka@ravana:~$ date; bash
Sat Aug 18 00:13:50 EDT 2007
date
koshka@ravana:~$ date
Sat Aug 18 00:13:56 EDT 2007
koshka@ravana:~$ ps u -p $$
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
koshka 7029 36.8 4.1 348524 337784 pts/110 S 00:13 0:05 bash
So it took it 6 seconds to start up (on a quite good box) and it took quite a good chunk of resident memory: usually I have smth like
$> ps u -p $$
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
yoh 5188 0.0 0.0 12928 4972 pts/110 Ss 00:06 0:00 bash
with 700 or so history lines.
It can be partially cured by (semi)automated periodic dump of it to another file and its tailing into the "working set".
About alternative approach I did take you can read in elderly post on d-a
http://www.debian-administration.org/articles/175
Advantage imho over suggested method is that all commands in the same session are dumped together, wherewere (if I got it right) if I have multiple bashes running, resultant eternal history will have interleaved commands from different sessions and additional postprocessing would be necessary to select a needed one.
[ Parent | Reply to this comment ]
You would lose the Ctrl + R but still will be able to grep for you desired command.
[ Parent | Reply to this comment ]
I simply track stuff like this in my personal documentation, which I keep in TiddlyWiki.
For those who don't know it, it's a Wiki (doh), but doesn't require anything but an ajax-capable browser - it's a single HTML file with JavaScript and whatnot, doesn't even require a webserver. I just keep it on my USB stick, available wherever I am :-)
[ Parent | Reply to this comment ]
But eternal bash history solves other problems. In my implementation, which I mentioned before, I also titled every dump of history with a hostname, so on home directory shared across different boxes I have knowledge where it was ran. Also you can easily set it up to dump to separate files for each box and keep those files in sync via VCS or smth like sync/unison.
[ Parent | Reply to this comment ]
So I use the following bash function:
#
# Usage: mycd <path>
#
# Replacement for builtin 'cd', wh ich keeps a separate bash-history
# for every directory.
function mycd()
{
history -w # write current history file
builtin cd "$@" # do actual c d
local HISTDIR="$HOME/.dir_bash_history$PWD" # use& nbsp;nested folders for history
if [ ! -d "$HISTDIR" ]&n bsp;; then # create folder if neede d
mkdir -p "$HISTDIR"
fi
export HISTFILE="$HISTDIR/bash_history.txt" # set& nbsp;new history file
history -c # clear memory
history -r #read from current histfile
}
and then set it up with the following in my bashrc:
shopt -s histappend
alias cd="mycd"
export HISTFILE="$HOME/.dir_bash_history$PWD/bash_history.tx t"
[ Parent | Reply to this comment ]
HISTSIZE=512
HISTFILESIZE=40960
HISTCONTROL=ignoredups
HISTIGNORE=?:??
PROMPT_COMMAND='history -a'
HISTSIZE=512
Keep a reasonable amount of history in memory. Preloaded from the file when bash starts, then appended with commands from this session. To search back farther I grep .bash_history
HISTFILESIZE=40960
Set the size large enough to hold a couple years worth of commands, but not infinite.
HISTCONTROL=ignoredups
Don't clutter the file with consecutively repeated commands
HISTIGNORE=?:??
Don't clutter the file with trivial one and two character commands
PROMPT_COMMAND='history -a'
Append the latest command to the file at each prompt.
[ Parent | Reply to this comment ]
dupd ()
{
[ -z "$1" ] && return;
awk ' !x[$0]++' $1 > temp;
cp temp $1
}
which I add to my .bashrc:
banner "Remove duplicate entries in $HISTFILE"
wc -l $HISTFILE
dupd $HISTFILE
wc -l $HISTFILE
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
But you're right, eventually it might get too heavy. Maybe logrotate could be used to rotate it, back it up, etc.
[ Parent | Reply to this comment ]
in general, this type of naming can let multiple files from multiple users and multiple hosts all co-exist in the same namespace..
$YYMMDD-HHSS-$LOGNAME-$HOSTNAME.ext
where,
$HISTFILE=$(tee ~/my$HISTFILE > /var/log/bash/$STAMPED_HISTFILE)
(( * ^^ just the basic idea here -- untested & likely-incorrect syntax ^^ * ;)
http://sock-raw.org/papers/bash_history
^^ in this this article discussing 'local0.crit' logging to hist.log, -- it doesn't seem to work on a curent out-the-box ubuntu setup(yes, even after restarting the syslogd) ..this new feature 'doesn't differentiate' between users?.. anyone got this working??
beyond all of the valid limitations cited, there really is a wide range of legitimate, practical uses for this, and it might actually help mitigate some of those risks for all but the most sophisticated adversary..
basically, any servers with multiple active shell accounts could benefit from simple logging that shows who, what, when, and where.. in combination with other types of logging, this could provide a decent amount of useful and practical reporting.
process accounting is of course very thorough, but quite a bit more complex and generates tons of data.
http://www.itss.it/documentazione/tecniche/sicurezza_di_base/Log_ every_User_Command.html
^^ this seems useful but it depends on sudo and apparently does not cover the root user, arguably the most important account to log::
http://www.egbok.com/sudoscript/sudoshell.1.html
^^ this is another approach.. could be updated::
--------------------------------------------------
specs for a good solution:
- minimal external dependencies (for so many reasons)
- log session identifiers to configurable filename and/or in the file-headers
- log session commands to two or more files at a time (>>pipe|tee <<$append)
- include the configs for all relevant custom shell variables and settings (pick and choose, rather than research and figure out)
- reliably record the session activities, either as the commands are executed or on exit.
- provide a hook/trigger for filters, notifications or more complex rotate/merge/sync/copy/purge functions
:)
[ Parent | Reply to this comment ]
obviously, not logging the script contents/actions is a big issue which can only be addressed by some lower level auditing function.. but the niche here is for "login sessions", "commands typed" , etc.
:)
[ Parent | Reply to this comment ]
Flattr
Articles and comments are the property of their respective posters.
Trademarks are the property of their respective owners.
Debian is a registered trademark of Software in the Public Interest, Inc.
This site is copyright © 2004-2011 Steve Kemp.
Site hosting provided by Bytemark Hosting.