The Surly Admin

Father, husband, IT Pro, cancer survivor

Dynamic Properties in Objects…. and performance!

So last post I talked about dynamic properties in objects, and it was pretty cool.  Then a crazy thing happened, that same colleague who was playing around with a faster Get-ChildItem project?  He decided to get some real work done and began working on a Office365 script but the Get-MailboxStatistics cmdlet is a little bit different with O365 in the TotalItemSize property is deserialized and pretty much only has a lousy string output.  So how to get the raw number without the ToMB() method?  A little Googling and we found an Exchange blog post using RegEx to strip out all the extra crud, and a dynamic property!  In the years I’ve been using PowerShell I’ve never once seen a dynamic property and the very day I decided to learn it we actually found a real life usage!  What are the odds?  Anyway, the twist was the Exchange team was applying the Add-Member to an array of objects.  I didn’t realize you could do that.  And it worked!

But what about performance?

Typically we’d use a calculated field to strip out the crud and calculate the number, right?  It’s the PowerShell way.  This method seemed really cool and we wanted to use it, but the fear is it’d be slower than a Select-Object with calculated field.  So I tried it with some sample data and ran them both through Measure-Command.  And my jaw dropped.  The dynamic property and Add-Member technique was about 15x faster than the Select-Object with calculated field.  What?!  A few more test proved it out.

Prove it

Remember this last “wish list’ item I posted yesterday?

Get-ChildItem $Path Recurse | Select Class MyDynamicFileObject Property BaseName,FullName,SizeGB,SizeMB

Let’s actually do it in real life, this time using both the Select-Object and Add-Member technique’s and see who wins.  Here’s the test code:

cls
Write-Verbose "$(Get-Date): Getting DIR information…" Verbose
$Dir = "C:\dropbox\test"
$Files = Get-ChildItem $Dir File Recurse | Select FullName,BaseName,Length
Write-Verbose "$(Get-Date): $($Files.Count) files found" Verbose
Write-Verbose "$(Get-Date): Using calculated fields in Select…" Verbose
Measure-Command {
$Files | Select Fullname,BaseName,Length,@{Name="SizeMB";Expression={ [math]::Round($_.Length / 1MB,2) }},@{Name="SizeGB";Expression={ [math]::Round($_.Length / 1GB,2) }}
}
Write-Verbose "$(Get-Date): Now calculating with dynamic properties…" Verbose
Measure-Command {
$Files |
Add-Member MemberType ScriptProperty Name SizeMB Value { [math]::Round($this.Length / 1MB,2) } PassThru |
Add-Member MemberType ScriptProperty Name SizeGB Value { [math]::Round($this.Length / 1GB,2) }
$Files
}
Write-Verbose "$(Get-Date): Done!" Verbose

This is against my test folder where I keep a bunch of files (mostly scraps from other scripts!).  There are about 234 files in these folders.  The results?

VERBOSE: 01/24/2015 06:37:54: Getting DIR information…
VERBOSE: 01/24/2015 06:37:54: Using calculated fields in Select…
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 24
Ticks : 241311
TotalDays : 2.79295138888889E-07
TotalHours : 6.70308333333333E-06
TotalMinutes : 0.000402185
TotalSeconds : 0.0241311
TotalMilliseconds : 24.1311
VERBOSE: 01/24/2015 06:37:54: Now calculating with dynamic properties…
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 7
Ticks : 78253
TotalDays : 9.05706018518518E-08
TotalHours : 2.17369444444444E-06
TotalMinutes : 0.000130421666666667
TotalSeconds : 0.0078253
TotalMilliseconds : 7.8253
VERBOSE: 01/24/2015 06:37:54: Done!

view raw
TestResult1.txt
hosted with ❤ by GitHub

Not the 15x speed difference we saw when doing string manipulation, but still 3x faster.  So next I decided to run it against my entire C: drive.  There are 216,177 files on my PC!

VERBOSE: 01/24/2015 06:40:58: Getting DIR information…
VERBOSE: 01/24/2015 06:42:52: 216177 files found
VERBOSE: 01/24/2015 06:42:52: Using calculated fields in Select…
Days : 0
Hours : 0
Minutes : 1
Seconds : 32
Milliseconds : 50
Ticks : 920506180
TotalDays : 0.0010654006712963
TotalHours : 0.0255696161111111
TotalMinutes : 1.53417696666667
TotalSeconds : 92.050618
TotalMilliseconds : 92050.618
VERBOSE: 01/24/2015 06:44:24: Now calculating with dynamic properties…
Days : 0
Hours : 0
Minutes : 0
Seconds : 20
Milliseconds : 259
Ticks : 202592197
TotalDays : 0.000234481709490741
TotalHours : 0.00562756102777778
TotalMinutes : 0.337653661666667
TotalSeconds : 20.2592197
TotalMilliseconds : 20259.2197
VERBOSE: 01/24/2015 06:44:44: Done!

view raw
TestResult2.txt
hosted with ❤ by GitHub

This time we got a little over FOUR time faster.

Limitations

If you find yourself working with really big data sets and you need a calculated field, there’s no question this is a great technique and much faster than Select-Object with calculated fields.  But there are some limitations.  With Select you can actually draw information from other sources.  On Friday I needed to produce a report of all of our VM’s that were not on hardware version 10 and had a 2TB virtual disk (we actually have a few).  Turns out VMware does not support VMDK’s over 2TB on hardware version 9.  So I setup a loop on our vCenter servers, then a loop on our VM’s and used Get-HardDisk on each VM.  But the problem is the pipeline from Get-HardDisk doesn’t include the vCenter information or the VM name, so I used calculated fields in a Select statement to refer to that information from my loop.  Since the object that can be created from Get-HardDisk does not contain that information I wouldn’t be able to use the Add-Member method to get information from the loop.

OK, technically that’s not true.  There is some reference information in Get-HardDisk, so in the value parameter I could use Get-VM and from that data use Get-Host to get that information, but it would be evaluated every time you display the object which would be painfully slow!

Conclusion

I love this.  I love its speed, and I love its simplicity.  It’s a narrow use case, admittedly, but another powerful weapon in my arsenal.  But Holy Crap!  I’ve been trying to get people off of Add-Member (at least for normal object creation) for years and now suddenly I’m going to embrace like a champ!  Fun stuff!

January 27, 2015 - Posted by | Powershell - Performance | ,

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: