Pester showed me a bug in our existing build process. Can you find it?

Words: 729

Time to read: ~ 4 minutes

Continuous Improvement

Working on the goal of continuous improvement of our processes, I got given access to the PowerShell scripts for our Build Process.

Credit where credit is due, these PowerShell scripts were created by people unfamiliar with the language.

They have done a great job with their research to build scripts that do what they want so I’m not going to nit-pick or critique.

I’m just going to improve and show/teach my colleagues why it’s an improvement.

Original State

The current state of the script is monolithic.

We have a single script that defines functions and then calls them later on. All mixed in with different foreach () and Write-Hosts.

Here’s a rough outline of the script.

$param01 = $args[0]
$param02 = $args[1]
$param03 = $args[2] -replace 'randomstring'

... Generic PowerShell commands ...

function 01 {
    function 01 definition
}

function 02 {
    function 02 definition
}

function GetPreviousTag {
    function GetPreviousTag definition
}

... More generic PowerShell commands ...
... that call our GetPreviousTag function ...
... etc ...

That was it.

1 giant script.
0 tests.

Extracting Functions for Tests

Now scripts are notoriously hard to test, I’ve written about how I’ve done that before but, honestly, if you really want to know then you need to check out Jakub Jares ( blog | twitter ).

Knowing how difficult testing scripts are, the first thing I decided to do was take the functions in the script and split them out. This way they can be abstracted away and tested safely.

I also didn’t want to take on too much at one time so I choose a random function, GetPreviousTag, and only actioned that one first.

Taking a look at GetPreviousTag

The simplest test that I can think of is a pass/fail test.

What do we expect to happen when it passes and what do we expect to happen when it fails.

To figure that out we’ll need to see the GetPreviousTag function. So I copied and pasted the code over to its own file GetPreviousTag.ps1. (sanitised, of course)

function GetPreviousTag {
    # Run the "git describe" command to return the latest tag
    $lastTag = git describe
    # If no tag is present then return false
    if ([string]::IsNullOrEmpty($lastTag)) {
        return $false
    }
    else {
        # If a tag is returned then we need to ensure that its in our expected format:
        # If a commit has taken place but the tag hasn't been bumped then the git describe command will return 
        # refs/tags/1.1.0.a.1-33-gcfsxxxxx, we only want the 1.1.0.a.1 part of the tag so we split off everything after
        # the "-" and trim the "refs/tags/" text.   
        $lastTagTrimmed = $lastTag.Split("-") | Select-Object -First 1
        $lastTagTrimmed = $lastTagTrimmed -replace 'refs/tags/',''
        # Verify that last tag is now in the expected format
        if ([regex]::Match($lastTagTrimmed,'\d+\.\d+\.\d+\.\c\.\d+')) {
            return $lastTagTrimmed
        }
        else {
            return $false
        }
    }
}

It’s nicely commented and glancing through it, we can see what it does.

  • Gets the output of git describe
    • If there’s no output:
      • return $false
    • If there is output:
      • Split on a dash, and get the first split
      • Remove the string ‘refs/tags/’
        • If the remainder matches the regex:
          • Return the remainder
        • If the remainder does not match the regex:
          • return $false

So we have our pass outcome, the remainder, and fail outcome, $false.

More importantly, can you see the bug?

The Pester Test

Here is the Pester test I created for the above function.

It’s relatively simple but even this simple test highlighted the bug that had gone hidden for so long.

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path) -replace '\.Tests\.', '.'
. "$here\$sut"

Describe "GetPreviousTag" {
    Context -Name 'Pass' -Fixture {
        Mock -CommandName git -MockWith {
            'refs/tags/1.1.0.a.1-33-gcfsxxxxx'
        }

        It -Name 'returns expected previous tag' -Test {
            GetPreviousTag | Should -BeExactly '1.1.0.a.1'
        }
    }

    Context -Name 'Fail : empty git describe' -Fixture {
        Mock -CommandName git -MockWith {}

        It -Name 'returns false' -Test {
            GetPreviousTag | Should -BeFalse
        }
    }

    Context -Name 'Fail : regex does not match' -Fixture {
        Mock -CommandName git -MockWith {
            'refs/tags/NothingToSeeHere-33-gcfsxxxxx'
        }

        It -Name 'returns false' -Test {
            GetPreviousTag | Should -BeFalse
        }
    }
}

Thanks to the above Pester test, I was able to find the bug, fix it, and also be in a position to improve the function in the future.

If you can’t find the bug, run the above test and it should show you.

Finally

If there’s one thing to take away from this post, it is to test your scripts

I’ve found Pester so useful that I decided to put my money where my mouth is…literally.

It’s more than deserved. Now back to continuous improvement…

When does a Failing Pester Test return Green?

Words: 716
Time to read: ~ 3.5 minutes
TL;DR: Don't nest It blocks

I’ve been working more with different people in different departments more and more lately at work.

“Great” you might say and I would agree with you.

This has meant that any and all scripts that I write to give to these departments have to be tested so that they don’t fail when they are run.

“Great” you might say and, again, I would agree with you.

New-Fixture

For PowerShell scripts, I’ve been using Pester. When I want to write a new function, I run a Pester command to create a .ps1 and a .Tests.ps1 file.

It’s a simple enough setup – I checkout a new branch for the command and switch to it.

git checkout -b NewFunction

I know that Pester has a command that I can use to create a framework of a new function and a framwork of a Pester test file for that function.

Get-Command -Verb New -Module Pester |
    Where-Object Definition -like '*create*scripts*tests*'

This returns the command New-Fixture and a quick scan of the help shows that this is exactly what we are looking for.

(Get-Help New-Fixture).Synopsis

This function generates two scripts, one that defines a function
and another one that contains its tests.

(Get-Help New-Fixture).Synopsis

TDD

An effect of working with more and more departments is that I have less and less time to spend on these functions. That’s meant that I’ve had to take a hard look at what would be a “nice to have” and a “must have”.

Because of this I’ve spent more time on planning and writing the tests first. Then I write my scripts to pass these tests and only these tests.

If it is a “must have” then it gets a test. If it has a test then the script gets that functionality, or property, or parameter, etc.

If it’s a “nice to have” then it doesn’t get a test. If it doesn’t get a test then it’s not in the script. When I have time later then I’ll go back and see if I’ll add it

I have yet to have time to go back and add stuff but then I’ve yet to have a need to add any of the “nice to have”s.

You may think that thinking about what’s needed and writing the tests takes time and you’d be correct.

But anytime spent on the planning part is saved by not spending time writing code that isn’t needed and getting it to work. Overall, it’s a major time saving technique.

Back on Track

Here is what is in the .Tests.ps1 file created from the New-Fixture command.

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path) -replace '.Tests.', '.'
. "$here\$sut"
Describe "<function name>" {
It "does something useful" {
$true | Should -Be $false
}
}

Contents of .Tests.ps1 file

It’s a nice template and all you need to do is modify that It block to have your test instead.

Here is where my Pester test failed and returned a green.

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path) -replace '\.Tests\.', '.'
. "$here\$sut"

Describe "Test-FakeFunction" {
    It "does something useful" {
        It 'should pass by deafult' {
            $true | Should -Be $false
        }
    }
}

It has a helpful message for you…

Do you see it?

The Reason

In modifying the template, I forgot to take out the original It block and just put my It block inside it.

This lead to my block “pass by deafult” failed
(as it should cause of that typo) but the original, parent block it was in “does something useful” passed!

It surprised me that I had a passing and failing tests since I thought I had only written 1 test but the maintainers of Pester included a helpful little error message for us.

You are already in a test case.

In the error message…3 times

Further Action

I’m an advocate of “if you see an issue, raise it on Github”. Even if you don’t fix the issue yourself, somebody else more than likely will come along and fix it for everyone.

Am I going to raise an issue for this on Github though?

  • An issue where it was due to my bad typing?
  • An issue where they already have it raised in the error message?
  • An issue where I accidentally put an It block inside an It block when I shouldn’t have?

For this instance, I think we’re okay…

Pester Testing Self Contained Scripts

Words: 1009

Time to read: ~ 5 minutes

Update:
    2019-02-22: Added a test for Jakub’s (don’t freak out, don’t freak out) comment.
    2019-06-13: Check out Jakub’s post on the topic – Such elegance http://jakubjares.com/2019/06/09/2019-07-testing-whole-scripts/

Continue reading “Pester Testing Self Contained Scripts”