The Surly Admin

Building Modules

If you’re looking to read about Modules and what they are and how to make them, this isn’t the blog post for you.  No, the problem I have is living with Modules.  I’ve only recently begun using them with the new job, but they have quickly made a place in my heart because being able to just call a module over and reuse code saves so much time!  But, as always happens, I want to make a change or fix a bug and that’s where modules fall on their face (a little).  This is my attempt to address that fact.

The Problem

We have a couple of modules now at work, one is what I call the “uber” module and it pretty much has everything in it.  This is great for your PowerShell session since you often just want to be able to run something and walk away.  But when you’re running a script, I often prefer using smaller, more focused modules, mostly from a–probably misguided–notion of better performance.  I don’t just mean speed performance, but memory usage and being a good neighbor on our script servers.

But fixing a problem in a module is kind of a pain.  Not going to sugar coat this one.  You have to make the change, import the module back in with the Force switch (so it overwrites what’s already in memory) and then run your test.  Rinse and repeat 50 times and you’re getting annoyed pretty heavily.  So the other option is to copy the function out of the module, paste it into a PS1 file, save it and then troubleshoot from there.  When done you copy the function back out and paste it into the module.  Much better but still a pain.

So my co-worker suggested we just have the module dot source the functions.  So each function has it’s own PS1 which the module is dot sourcing into memory.  Works pretty well but there are two issues.  First you go over the network to get the module, then you repeated go back to the network to load each module (our module has about 10 calls right now) and while this is fast there is a perceptible pause when starting PowerShell.  The second problem came with our Start-Exchange function.  This function simply does some implicit remoting to our Exchange servers so we can run the Exchange cmdlets without actually installing the Exchange tools on the workstation.  And it plain doesn’t work when dot sourcing from a module.  The script runs fine, and the remote session is created and commands imported perfectly, except you just can’t use them.  PowerShell is completely unaware of the cmdlet’s.  I suspect this is a scope issue.  I finally ended up creating another PS1 that didn’t have all this as a function and then in the module simply creating an alias to this new file.  (Pre-publishing edit: Turns out the the technique I’m describing below didn’t solve this problem either, something else is going on!  I’ve seen some interesting solutions that I haven’t tested yet, so I should post a follow-up article later).

The Solution

Wouldn’t it be nice to have these modules have the functions right in them, but still be able to edit and troubleshoot from individual PS1 files?  You could keep your testing data and code in there, but still have the function in the module.  Well, PS1 files are just text files and PowerShell is pretty good at manipulating text files, why not read that file in, extract the Function and copy them all into a module?

The Update (February 8th, 2015)

Of course, no project is ever really finished and I just added a nice update to the Publish-Module script.  There is now a #Publish and #EndPublish region which allows you to include code in your module.  Want to put a cool alias in your module?  Put it in between #publish and #endpublish.  Turn on implicit remoting to your Exchange 2013 server?  You guessed it.  Here are some examples of how it works:

Specifications

So let’s outline what this script needs to do:

• Be directed at a folder, where the PS1 files are located
• PS1’s are placed in sub-folders, with the name of the folder being the name of the module
• Sub-folders of the module level would be modules of their own, but the parent folder would also contain those functions, while the subfolder would also be a standalone module with just the PS1 files under it.
• Extract functions only, any other code in the PS1 (assumed it is being used for testing) is to be ignored.
• Create a module manifest file — not strictly needed but you automate to tie up all these loose ends so let’s do that.

But I wanted some stretch goals too:

• Customize manifest file (implemented)
• Detect when changes to the underlying PS1 files are made and increment the ModuleVersion accordingly (implemented)
• Automated creation of help files (not implemented – yet!)

Testing

The first step was trying to pull the function out of the text file.  My first thought was to use RegEx for this, and under very tight controls this did work.  But as soon as I placed testing code in the file along with the function is quickly broke down.  It might well be possible to do with RegEx but it’s beyond my humble capabilities.  Looking at the pattern, it was obvious I needed to find the word “Function” and the opening curly bracket, and then grab all the code until the closing curly bracket.  But how do you program that?  I decided to try a line by line approach, look for “Function” and turn a flag “On” telling the script to save every line from now on.  Then I’d have to count all of the opening curly brackets, and subtract all the closing curly brackets until my counter got down to zero, indicating the end of the function.  Here’s some sample code:

I decided to use a System.Collections.ArrayList because adding to this array type using the Add method is blindingly fast and much more efficient then adding to a standard array.  I define two arrays, one called $Functions which will hold all the functions found, not only in 1 PS1 but all of the PS1’s (the code above is not the full code, but meant as a sample of how I extracted the Functions).  $Function is the current function being pulled, once finished it will added to $Functions and the loop will continue.

The real fun was the two $Count lines.  One does a Split on {, subtracts 1 just because that makes more sense to me, and then the second line splits on the closing curly brakets and subtracts it back down.  Then just check for when $Count is zero and boom, you have the full function.  Then add each line to $Function, and eventually add the whole function to $Functions and Out-File them (the -joins just flatten things out to a string and make the module look nice).

The Problems

No script escapes without a few problems and this one actually had quite a few.  First up was the Function line itself.  You see, sometimes I code Function like this:

And then sometimes I code it like this:

The difference is subtle, but had a profound (and bad) effect on the function extraction code.  Had to test if the opening curly bracket was on the same line as Function, and deal with it if it isn’t.

Second problem was the #requires comment.  This is a great ability within PowerShell to allow you to specify which version of PowerShell must be running for the script to work.  So many people are still on version 2.0 of PowerShell and I write most of my stuff using 3.0 syntax so it’s important.  The problem is you can’t have multiple #requires in a module!  Well, technically you can but they all have to be the same.  If you have a requires statement that has 2.0 in it, and then another that has 3.0 in it the module won’t load!  So I had to put some code in that read the #requires line, converted the number to a [version] type and then compare it to the next version and keep the highest.  I then put that into the manifest file and removed all of the #requires from the functions.  This allows you to troubleshoot things properly and not worry about the module build.

Module Version

I really wanted an automated way of to increment the module version if there are changes.  But how?  You’d have to detect there were changes to the underlying PS1 files and then increment the module version.  But there are 4 separate fields to a version type: Major, Minor, Build and Revision.  What to do?

I decide to get the LastWriteTime of the manifest file since that should be the last time the build script was run (give or take).  From this I can simply filter out the files that have changed since the last time we did this.  Simple percent calculation after that an then some If/ElseIf’s to determine which of the 4 version properties I convert.  Interesting note about the [version] type in PowerShell (.Net really) is that all of them are read-only.  So I have to use Select to pull them into a custom object which I can change easily enough (and the properties are nicely named too, so that makes it easier).  Then simply put it back together as a string.

Interesting fact about a PowerShell module manifest is that it’s just a hashtable, so by using Invoke-Expression on the whole text file I can convert it into a hashtable in memory and manipulate it as needed.  This also makes for a very convenient splat for the New-Manifest cmdlet.  Now you can manually go in and edit the manifest with whatever information you want and that will never be lost by the build process.

Real Life

Sometimes you have what you think is a great idea and it just hits people with a resounding meh.  That was certainly the case at work, much to my disappointment.  But I’ve been using this script for a couple of weeks now and I have to say it performs exactly as intended.  I’ve added new functions to my modules, and modified/fixed a couple of others and I just run the script and bang I have all new modules.

You can find the source code, as usual, at Spiceworks:

Publish-Module

January 20, 2015 - Posted by Martin9700 | PowerShell | Modules