The Surly Admin

Monitoring the Progress of a PowerShell Job

Working in a big Enterprise is a whole different animal than small business (which has been my space for a long time).  I’m finding myself doing a lot more multi-threading because of the pure scale of things that need to be done and running a single threaded sequence just takes too long.  But that doesn’t mean I don’t want to give my users feedback about the progress of my scripts, in fact, the larger the work load the more important feedback becomes.  I recently discovered a technique that allows me to do just that–but not before putting my own spin on it.

As always, I find what others have done before me and add to it.  To monitor the progress of my background jobs I found this post from Boe Prox, the man I’ve probably taken the most code from!

The Setup

In this case I have 3 background jobs I’m submitting, each processing potentially hundreds of servers.  Since this is liable to take awhile, I want the primary script to give the user feedback about the progress through the list of servers.  My initial testing used the verbose stream to simply return a number, so the primary script would query each background job, get that number and calculate the percentage from there.  This worked great but then I ran into a major problem (major for me, anyway).

When running with -Verbose on, which I often do, as soon as I did a Receive-Job on the background job it would list all of the numbers from my progress.  Not a show stopper by any means, but the overall output looked incredibly messy.  I attempted to redirect the verbose output but pretty much failed on that.  So reading down Boe’s page I saw we could use the Progress stream.  By simply putting a number in PercentComplete I could do the same thing and when I receive the job, no extra output!  Success!

Almost.

Problem here is this technique works great as long as your progress never goes above 100, but in my case I had over 400 servers I was processing and as soon as I hit 101 Write-Progress throws an error.  The fix turned out to be incredibly simple.  Instead of using PercentComplete, I simply used the Status field to place my number.  This allows me to have far more flexibility than the PercentComplete.  Of course, you could use PercentComplete, even if you had multiple jobs, by getting the percent for each job, adding those numbers up and then dividing by the number of background jobs to get an average and then post that in your primary script.

But in this case, I wanted my primary Write-Progress to not only have the percent complete, but show how many servers have been processed.  That means I needed the raw number and–as usual–nothing is ever easy.  Consider the following test script:

This is a simple script that create 5 background jobs (as defined by $Num).  After submitting we’ll begin monitoring the progress of the job.  We define $TotProg as 0 to start, then query the StatusDescription–and since this returns an array we only want the last element, hence the -1 element reference–and add it to our $TotProg.  After that we check if $TotProg is greater than 0 (otherwise you’ll get a divide by zero error), display our progress bar, wait a few seconds and loop again.  Go ahead and run the code (don’t step through it in an ISE, to really see what I mean you have to just run it).

It doesn’t happen every time, so you might have to run it a couple of times, but notice the error.  784%?  What the heck?  Where did that come from?  I tried various “Start-Sleep” scenario’s to hide this, but I either ended up hiding most of the progress–because it would complete before I finally got around to getting good progress–or it would still give me the errors!  So what’s going on?  As you submit jobs it takes PowerShell a few seconds to get everything ready for the jobs.  This causes two problems, the first one I get around with the Try/Catch block.  If the sub-job hasn’t been spawned yet, checking on the sub-job with this code ($Job | Get-Job).ChildJobs[0].Progress.StatusDescription[-1] will trigger a terminating error.  The Try/Catch block catches that, waits half a second and loops back around.

The second problem is the mysterious uber-percentage.  This one was really difficult to troubleshoot because whenever I stepped through the script it would invariably work just fine.  Frustrating, right?  So running it at full speed was the only way to trigger the problem.  But what was going on?  Well, it looks like as the sub-job has spawned, but before it’s fully begun running the progress stream will have random [char]’s in them.  So I have to get the value, test if it’s a [char] and if so set the value to 0.  If not accept the actual value–which will be a string–add it to $TotProg and calculate the proper percentage.

Here’s the functioning Do loop:

Now you have a way to easily communicate progress of your background jobs to the “mother” script, which can really improve the user experience.  Especially for scripts that will be taking quite some time to complete.  I’m curious, how are you using multi-threading?

June 23, 2014 - Posted by Martin9700 | PowerShell | Background Jobs, Multi-thread, multithread, Write-Progress