Piping STDERR to STDOUT for filtering, done right

People ask how to pipe STDERR under Bash (the most commonly used shell for Linux users) and get frustrating or bad answers. This is because doing it right is very ugly.

When you want to pipe only STDERR, you probably want to filter STDERR somehow, perhaps using grep or awk or sed, but you don't want to damage the output on STDOUT.

Suppose, for example, that you have a program which produces nice output, but also generates errors on STDERR. Of those errors, there are a few you're interested in, but most simply clutter the output. So you want to filter the STDERR output, but preserve the STDOUT output.

We all know we can redirect STDERR to STDOUT with the syntax

# some_program 2>&1

but this would simply merge STDOUT and STDERR, so downstream you wouldn't know which what was an error and what was plain old output. In bash, you cannot pipe STDERR directly like you can in csh. Instead, you need to flip STDERR and STDOUT, do your filtering, then flip them back.

Before I show you the command, keep in mind two things:

1. The syntax "2>&1" seems to mean "send the output of STDERR to STDOUT" (STDERR is #2 and STDOUT is #1) and it certainly does, but the way it accomplishes this makes it a little weird. STDERR has a location in memory containing a file descriptor. So does STDOUT. "2>&1" actually copies the file descriptor in #1's momory location to #2's memory location. So while the affect is to send them both to the same place, what you've really done is made a copy of #1's file descriptor and destroyed the reference to the original #2 file descriptor.

2. You can only pipe or redirect for the command you're in. So

# echo BOB ; echo DAN > /dev/null

doesn't make BOB disappear. It just makes DAN disappear.

OK. So how do we do this? Well, to flip #2 and #1, we use the trick

3>&1 1>&2 2>&3

This creates an entirely new output memory location called #3, and copies the file descriptor of #1 into it. It then puts the file descriptor of #2 in #1. Lastly, it puts the file descriptor of #3 in #2. The order of operations is important. We need to save #1 before we redirect it, and we need make #1 a copy of #2 before we make #2 a copy of #3. Any difference in the order would send the same thing to two places.

The exact same sequence flips them back.

A real world example:

Let's use this to filter tar. The archiving program tar produces a lot of error messages on STDERR, some of which are important and some of which can be ignored. We'd like to see only the ones that show that there's really something wrong.

# tar -cf /tmp/junk.tar /tmp/real /tmp/fake
tar: Removing leading `/' from member names
tar: /tmp/fake: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

The directory /tmp/real exists, but the directory /tmp/fake doesn't. The other messages are things we always see. We'd rather only see the message about /tmp/fake. We do this using egrep.

# tar -cf /tmp/junk.tar /tmp/real /tmp/fake 3>&1 1>&2 2>&3 | egrep -v "tar: Removing leading|tar: Error exit delayed from previous errors"
tar: /tmp/fake: Cannot stat: No such file or directory

Better. But STDERR is now on STDOUT and vice versa. We should flip them back. And here's where my second point from above comes in. You can't just add the flipping syntax at the end of the egrep command. The egrep command has it's own STDERR location, distinct from the one tar is using. You need to bunch them together before flipping them back. You do this by using parenthesis.

# (tar -cf /tmp/junk.tar /tmp/real /tmp/fake 3>&1 1>&2 2>&3 | egrep -v "tar: Removing leading|tar: Error exit delayed from previous errors") 3>&1 1>&2 2>&3
tar: /tmp/fake: Cannot stat: No such file or directory

Yes, it's a mess. But now that you have the pattern, you can use it when you really need it. For example, when you only want to see certain types of STDERR output in cron jobs. You can put the above syntax in lines of your crontab, or you can wrap your commands in scripts, so as not to junk up your crontab file.

In a nutshell, it's:

( COMMAND 3>&1 1>&2 2>&3 | grep -v ANNOYING_ERRORS ) 3>&1 1>&2 2>&3 | grep TARGET_STRINGS

---

Paul Pomerleau is the author of Networking for English Majors, which teaches people with liberal arts and social science degrees how to get jobs in computer networking. You don't need a computer science degree to get a well-paying job in computing.