Summary: How to Implement Automatic Testing using Selenium and PowerShell in Azure DevOps Server.
Date: Around 2020
Refactor: 29 April 2025: Checked links and formatting. Rebranded Team Foundation Server (TFS) to Azure Devops Server.

In this article I will use the Selenium webdriver for Firefox and Chrome in PowerShell to implement automatic testing in TFS. I will this combined with the earlier create build and release pipeline in which we deployed a website. Here we will take the necessary steps to test the website.

Used technologies:

• Selenium webdriver for Firefox and Chrome
• TFS build and release tasks
• PowerShell

Note that because we use PowerShell to do the test, the results are not considered an automatic test from the point of view from TFS. This requires a testplan, in an appropriate format. To create that you'll need Visual Studio (Code). Using PowerShell to do the same thing should be considered as a very fast and easy to deploy lightweight version of automatic testing. Possibilities are limited but should be sufficient in most cases.

To succesfully run Selenium tests we need the following files:

• Selenium:
  • WebDriver.dll
  • WebDriver.Support.dll
• Browser driver chrome:
  • chromedriver.exe
• Browser driver firefox:
  • geckodriver.exe

• Go to the Selenium download page: https://www.seleniumhq.org/download/
• Download the C# package under Selenium Client & WebDriver Language Bindings
• Extract the downloaded file: selenium-dotnet-3.14.0.zip
• Under the dist directory, locate the Selenium.WebDriver.3.14.0.nupkg file and add the .zip extension, so you can extract it
• Under the lib directory there are three directories, net35, net40 and net45 for the corresponding dotnet versions.

Repeat the last two steps for the Selenium.Support.3.14.0.nupkg file.

• Go to the Selenium download page: https://www.seleniumhq.org/download/
• Locate the Mozilla Geckodriver download link and the Google Chrome Driver download link.

Download the latest versions.

First thing is to add the PowerShell script to the TFS Code Repostory. Go to Code → PowerShell folder (create it if it doesn't exist yet). Create a new file as seleniumtest.ps1 and add the code (see below the code for necessary explanation:

# Check for.net version of minimum 4.5:
if (!(Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\' |  Get-ItemPropertyValue -Name Release | Foreach-Object { $_ -ge 394802 })){
	exit
}else{
	write-host "mimimum .net 4.5 version found. Continue"
}
 
# Load the Selenium .Net library and paths to the various test libraries
# webdriver & support dll
Add-Type -Path "D:\selenium\WebDriver.dll"
Add-Type -Path "D:\selenium\WebDriver.Support.dll"
# Add path for chromedriver.exe and firefox geckodriver.exe
$env:PATH += ";D:\selenium"
 
# Testplan variables
$testurl = "http://tfs.build.getshifting.com/"
$testname = "dtap"
$resultsdir = "D:\selenium\testresults"
[OpenQA.Selenium.ScreenshotImageFormat]$ImageFormat = [OpenQA.Selenium.ScreenshotImageFormat]::Png
 
### Firefox testplan #########################################################################################################
# Start firefox headless
Write-Host "Start firefox testplan, starting firefox headless"
[OpenQA.Selenium.Firefox.FirefoxOptions]$ffoptions = new-object OpenQA.Selenium.Firefox.FirefoxOptions
$ffoptions.addArguments('-headless')
$ffdriver = New-Object OpenQA.Selenium.Firefox.FirefoxDriver ($ffoptions)
# Sometimes headless doesn't work through arguments. Setting the headless environment variable to make sure we're headless
$env:MOZ_HEADLESS = 1
 
# Start firefox testplan
Write-Host "Go to url $testurl"
$ffdriver.Navigate().GoToURL($testurl)
 
# Get Evidence that the website works
$ffcurrenturl = $ffdriver.url
$ffcurrenturltitle = $ffdriver.title
Write-Host "Current FF url  $ffcurrenturl and title $ffcurrenturltitle"
# Get Pagesource
Write-Host "Create pagesource $ffcurrenturl"
$ffdriver.PageSource | Out-File "$resultsdir\ff$testname.html" -Force
# Get Screenshots
Write-Host "Create screenshot $ffcurrenturl"
$Screenshot = [OpenQA.Selenium.Support.Extensions.WebDriverExtensions]::TakeScreenshot($ffdriver)
$Screenshot.SaveAsFile("$resultsdir\ff$testname.png", $ImageFormat)
 
# Close firefox
Write-Host "Close and quit firefox browser and selenium webdriver. "
$ffdriver.Close()
$ffdriver.Quit()
 
### Chrome testplan #########################################################################################################
# Start Chrome headless
Write-Host "Start chrome testplan, starting chrome headless"
$chromeOptions = New-Object OpenQA.Selenium.Chrome.ChromeOptions
$chromeOptions.addArguments('headless')
$chromeDriver = New-Object OpenQA.Selenium.Chrome.ChromeDriver($chromeOptions)
 
# Start chrome testplan
Write-Host "Go to url $testurl"
$chromedriver.Navigate().GoToURL($testurl)
 
# Get Evidence that the website works
$chromecurrenturl = $chromeDriver.url
$chromecurrenturltitle = $chromeDriver.title
Write-Host "Current Chrome url  $chromecurrenturl and title $chromecurrenturltitle"
# Get Pagesource
Write-Host "Create pagesource $chromecurrenturl"
$chromeDriver.PageSource | Out-File "$resultsdir\chrome$testname.html" -Force
# Get Screenshots
Write-Host "Create screenshot $chromecurrenturl"
$Screenshot = [OpenQA.Selenium.Support.Extensions.WebDriverExtensions]::TakeScreenshot($chromeDriver)
$Screenshot.SaveAsFile("$resultsdir\chrome$testname.png", $ImageFormat)
 
# Close Chrome
Write-Host "Close and quit chrome browser and selenium webdriver. "
$chromeDriver.Close()
$chromeDriver.Quit()

The script performs the following steps:

• Check for the required .Net version
  • You can do that manually using the following PowerShell command: Get-WindowsFeature -name NET* | where installed
• Add the required path variables for selenium. I put the required files in D:\selenium and created the directory D:\selenium\results
• Start the testplan per browser:
  • Go to url
  • Output some information like the url and the pagetitle to the logfile
  • Download the pagesource and make screenshots of the page

Note that this quite a simple testplan. We only go to the url and create some screenshots. In real life you should do some logging in and test more of the actual site.

We need to set the script on the build server so it can be run during the release. We need to do so now because the powershell folder is not included in the artifact. In the build definition, click “+ Add Task” and search for “Copy Files”. Click the “Copy Files” task and click Add. Configure the task like this:

• Display Name: Copy Files to: D:\selenium
• Source Folder: Powershell (use the icon with the three dots to navigate to the correct folder)
• Contents: seleniumtest.ps1
• Target Folder: D:\selenium
• Advanced settings:
  • Overwrite: Enabled

In the release we currently have one phase which runs on the deployment group and which actually deploys the website. We now need to add a second phase that runs on the build agent so the selenium script can run and the results can be collected and emailed.

In the TFS portal go to Build and Release → Release and select the release we need to edit, and click Edit. Select the correct environment and click on the Phases and tasks link. Once you're in the Release definition click the three dots next to “Build environment” and click “Add Agent Phase”. Configure the phase like this:

• Display Name: PowerShell Selenium Test
• Agent queue: Select the queue that holds the agent with the selenium files on it.
• Additional Options:
  • Skip download of artifacts: Enabled (by default the artifact is downloaded but we don't need the artifact for this phase)

Now we can add tasks in the phase.

In the release definition, in the newly created Agent Phase, click “+” and search for “PowerShell”. Click the “PowerShell” task and click Add. Configure the task like this:

• Display name: PS Start Selenium Test
• Type: File Path
• Script Path: d:\selenium\seleniumtest.ps1
• Advanced settings:
  • Working Folder: D:\Selenium
• Control Options:
  • Continue on error: Enabled (The firefox browser produces output in headless mode that is being diagnosed as an error in TFS. By enabling this option these errors are ignored)

In the release definition, in the newly created Agent Phase, click “+” and search for “Archive Files”. Click the “Archive Files” task and click Add. Configure the task like this:

• Display name: Zip PS Selenium Testresults
• Root folder (or file) to archive: D:\Selenium\Testresults
• Archive Type: zip
• Archive file to create: $(system.artifactsdirectory)/$(Build.BuildId).zip
• Replace existing archive: enabled

In the release definition, in the newly created Agent Phase, click “+” and search for “PowerShell”. Click the “PowerShell” task and click Add. Configure the task like this:

• Version: 2.*
• Display name: PS Email Selenium Testresults
• Type: Inline
• Script:

$mailserver = "smtp.getshifting.com"
$toemail = "$(release.requestedForEmail)"
#$toemail = "$(release.requestedFor)"
Write-host "email $toemail"
# Check for valid getshifting address
if ($toemail -notlike "*getshifting.com"){$toemail = "postmaster@getshifting.com"}
Write-host "email $toemail"
$fromemail = "$(agent.machineName)@getshifting.com"
$subject = "$(release.definitionName) Selenium Testresults"
$info = "See the attachment for the testresults. <br>See $(release.releaseWebUrl) for the full results. "
 
Send-MailMessage -To $toemail -From $fromemail -SmtpServer $mailserver -Subject $subject -BodyAsHtml -Body $info -Attachments "$(System.ArtifactsDirectory)/$(Build.BuildId).zip"
 
#Remove zip file
remove-item "$(System.ArtifactsDirectory)/$(Build.BuildId).zip"

This sends the email to a valid getshifting email address. This is also the case if the $(release.requestedForEmail) variable is not set, or if it is set to a non-getshifting email address.

The next step is to add some more test steps: Implement Automatic Testing using Selenium and PowerShell in Azure DevOps Server - More TestPlans The next step is to implement another testplan, but with authentication: Implement Automatic Testing With Authentication using Selenium and PowerShell in Azure Devops Server

• https://newspaint.wordpress.com/2017/03/23/using-powershell-2-0-with-selenium-to-automate-internet-explorer-firefox-and-chrome/
• https://tech.mavericksevmont.com/blog/powershell-selenium-automate-web-browser-interactions-part-i/
• https://tech.mavericksevmont.com/blog/powershell-selenium-automate-web-browser-interactions-part-ii/
• https://tech.mavericksevmont.com/blog/powershell-selenium-automate-web-browser-interactions-part-iii/