Logging the output of long commands run multiple times

T. Kurt Bond

2021-07-07 12:47

I often run commands that produce a lot of output that needs to saved for debugging, and often the commands have to be repeated multiple times to get things to work. For example, building software from source, often using the familiar ./configure; make; make install paradigm.

So, the first thing is to try is to use the venerable tee command.

./configure 2>&1 | tee Log.configure
make 2>&1 | tee Log.make
make install 2>&1 | tee Log.make-install

To make the log files easy to find I use a Log. prefix.

But I often need to run the commands multiple times, and want to save each run under a new filename, so if the filename already exists I want to add a number to the end and then increment the number until I find one that hasn't been used. And I'd like the filename to have the date in YYYY-MM-DD format, so the resulting names look like Log.make-install-2021-07-07_2.

So I wrote a bash function incf (increment filename) to put in .bashrc that generates such a name:

incf () {
    # Construct a filename from PREFIX, "_YYYY-MM-DD",  optionally _N (where N
    # is 1 or greater) if the filename already exists, and optionally SUFFIX.
    # Example: "incf file .tar.gz" results in "file_2021-07-07.tar.gz", or
    # "file_2021-07-07_N.tar.gz" if "file_2021-07-07.tar.gz" already exists,
    # where N is 1 or greater.
    local prefix suffix fileprefix i testname sep1 sep2
    prefix="$1"
    suffix="$2"
    sep1="_"
    sep2="_"
    fileprefix="${prefix}${sep1}$(date +%F)"
    let i=0
    # The zeroth filename doesn't have the number.
    testname="${fileprefix}${suffix}"
    while true
    do
      [ ! -e "$testname" ] && break
      ((i++))
      testname="${fileprefix}${sep2}${i}${suffix}"
    done
    echo "$testname"
}

And then I wrote a bash function that uses incf to generate the Log. filename, potentially in a different directory:

logf () {
    # Construct a filename, possibly in another directory, that starts with
    # "Log." and ends with "YYYY-MM-DD" and optionally "_N", where N is 1 or
    # greater, if the filename already exists.
    local dn bn fn
    dn="$(dirname "$1")"
    bn="Log.$(basename "$1")"
    fn="$(incf "$dn/$bn")"
    echo $fn
}

And then I wrote a log command that uses logf and tees its input into that file:

log () {
    # tee the input into a log file.
    tee $(logf "$1")
}

So running ./configure 2>&1 | log ~/tmp/configure generates a file Log.configure_2021-07-07 in the ~/tmp directory.

But what if I specify a lot of options to the command, and would like record if it in the log file, so if I get interrupted and then come back some time later I can use the same command?

First I wrote a base function, cleanname, that takes a string and converts it to something that should be safe to use as a filename.

cleanname () {
    # Clean up a string so it is (relatively) safe to use as a filename.
    local cmd="$*" name
    name=$(echo "$cmd" | sed 's/[ =";?*&^%$#@!~`|()<>]/-/g' | \
               sed "s#[/']#-#g" | sed -E 's/--+/-/g' | \
               sed -E 's/(^[-.]+|-+$)//g' | \
               sed -E 's/\.\.\.*/./g')
    echo "$name"
}

Then I wrote a bash function, exlog, to use the whole command with its options as part of the filename (constructed with cleanname, and also include the whole command in the log output:

exlog () {
    # Execute a shell command and log it to "Log.<cmd-as-safe-filename>"
    local cmd="$*" name="$(cleanname "$@")"
    name="$(logf $name)"
    printf 'Logging to %s\n' "$name"
    (echo "cmd was: $cmd"; time "$@") 2>&1 | tee $name
}

So running the command

exlog ../configure --prefix=/Users/tkb/sw/versions/groff/git

produces the file

Log.configure-prefix-Users-tkb-sw-versions-groff-git_2021-07-07

and it contains the line

cmd was: ../configure --prefix=/Users/tkb/sw/versions/groff/git

and running it again produces the file

Log.configure-prefix-Users-tkb-sw-versions-groff-git_2021-07-07_1

This code is available in a gist.

Last edited: 2021-07-09 15:30:53 EDT

Lacking Natural Simplicity

Logging the output of long commands run multiple times

About

Links

Lacking Natural Simplicity

Comments

About

Links