Stuff I Learned Scripting - Fun with STDERR

Published: 2012-01-12. Last Updated: 2012-01-12 14:30:00 UTC
by Rob VandenBrink (Version: 1)
5 comment(s)

Say you’re writing a long Windows CMD script, something like an audit script that’ll take a good 20-30 minutes to complete.

Now say the whole script is being redirected to a report file - as scripts get more complicated, I'm finding that almost everything I write ends up doing this.  Something like below (just to pick a random SEC579 example):

audit-esx.cmd servername userid password > reportfile.html

If all goes well, you see *nothing* on your screen for the next 20+ minutes (unless you’ve got a good port of tee available) – but if it gets stuck, it's going to be 20+ minutes, or likely longer, before you realize that your script is borked …

What to do? What to do? - - Use STDERR !

As the script goes through, insert an echo for each test (or meaningful phase) in your script to STDERR:

echo Audit Check SomeMeaningfulName >&2

or, if you’ve parameterized your script enough:

echo Check %CHK% >&2

"&2" means "send this to STDERR".

So, instead of a blank screen as the audit runs, the screen will be a show you useful info on it's progress:

C:\sans\sec579\audit>audit-vms esx01.sec579.com root Passw0rd > esx01-audit-vms.html
Audit Check VMX01
Audit Check VMX02
Audit Check VMX10
Audit Check VMX11
Audit Check VMX12
Audit Check VMX20
Audit Check VMX21
Audit Check VMX22
… and so on, until it's done

Another neat trick will allow you to echo to a file AND to STDERR in windows.  The example below will take the output of "somecommand", echo it to STDERR (which you'll see on the screen), and also echo it to the file "outputfile.txt"

somecommand > &2 > outputfile.txt

In linux, I'd normally do this using "tee" as mentioned, mostly because I'm lazy.  The problem in this case with using tee is that it goes to STDOUT, rather than to STDERR, so if you're using it in combination with other redirection, you may not get what you expect:

somecommand | tee outputfile.txt

To fix this, you might string your command serially with cat, but that means that you won't see the command output on STDERR until the command is completely finished, rather than in (more or less) real time.

somecommand > outputfile.txt ; cat outputfile.txt >&2

To see everything at the same time, I'll still use tee, but we'll also use a temp file descriptor (3) and dump the STDOUT output of tee to STDERR, as shown below

(somecommand | tee outfile.txt) 3>&2

I hope this was useful - if you've got a neat take on using STDERR, or STDIN or STDOUT for that matter, in Windows (or *nix) scripts, by all means pass them along in our comment form !

===============
Rob VandenBrink
Metafore

Keywords: Scripting stderr
5 comment(s)

Comments

And to complete the set... sometimes you want to capture the whole script output, not use STDERR as your monitoring channel.

To redirect both STDOUT and STDERR to the same file, do this:

somecommand > outputfile.txt 2> outputfile.txt

Or following Rob's handle names, redirect STDERR to STDOUT and then redirect STDOUT to the destination file:

somecommand 2>&1 > outputfile.txt

And because there's more than one way to do it, use TEE and TAIL from the GnuWin32 ports of classic *nix command line tools to Windows, available here: http://gnuwin32.sourceforge.net/

Then you can tee in the middle of your redirection, or if you realize later that your script is still running, run tail on the output file:

tail outputfile.txt

Or use tail to continually update the progress by following the updates to the file:

tail -F outputfile.txt

(presumably, you'd open a new command line window, because the first one is still busy waiting for your script to finish)

Andrew from Vancouver

Jan 12th 2012
1 decade ago

Great call Andrew!
Even with Microsoft pushing their *nix toolset (which is pretty good) as part of Windows, I still find myself using the gnuwin32 set more often than not, only because I can pull a single executable and use it in a client environment, without installing anything on their servers or workstations.

Rob VandenBrink

Jan 12th 2012
1 decade ago

I like to use the transcript feature of Powershell for this type of scenario as well.

Justin

Jan 12th 2012
1 decade ago

Merging STDOUT and STDERR can get you in to trouble sometimes if the process in question interleaves output to them. The C STDIO library attempts to determine if output is going to a terminal or to a file/pipe. If the latter, it generally turns on buffering for the output handles. If the program in question doesn't autoflush buffers, then the output from STDOUT and STDERR can end up interleaved on buffer flushes rather than lines, which can make a mess of the saved output. I once had to deal with a Windows server process that directed all of its logging and debugging information to STDOUT and STDERR, and it took me a long time to find a workaround that would enable us to save the output to a file (and thus make it available over the network). The interleaving problem was a pain because the buffers were all internal to the EXE in question. I ended up writing a Perl script that ran in the same terminal window in parallel with the actual EXE. The terminal window was fairly wide and had massive scroll back buffers. The Perl script used Win32::API to make Win32 API calls to constantly screen-scrape the entire terminal window and diff it against the previous screen-scrape, then output the changes. Line wrapping was unavoidably treated as a line break since I had no access to the underlying output, just how it got dumped to the screen, but this approach solved the interleaving problem and ended up being pretty reliable. Looks like I used GetConsoleScreenBufferInfo and ReadConsoleOutputCharacter and SetConsoleScreenBufferSize and GetStdHandle. The buffer height was set so that the buffer size was just under 64K (i.e. int(64K / width)). Looks like I had a tenth of a second interval on screen scrapes.

Anonymous

Jan 12th 2012
1 decade ago

This version is more efficient:
yourcommand | tee /dev/stderr

Anon

Jan 16th 2012
1 decade ago

Login here to join the discussion.


Top of page
Diary Archives