Summary: How to Implement Automatic Testing using Selenium and PowerShell in Azure DevOps Server.
Date: Around 2020
Refactor: 29 April 2025: Checked links and formatting. Rebranded Team Foundation Server (TFS) to Azure Devops Server.
In this article I will use the Selenium webdriver for Firefox and Chrome in PowerShell to implement automatic testing in TFS. I will this combined with the earlier create build and release pipeline in which we deployed a website. Here we will take the necessary steps to test the website.
Used technologies:
Note that because we use PowerShell to do the test, the results are not considered an automatic test from the point of view from TFS. This requires a testplan, in an appropriate format. To create that you'll need Visual Studio (Code). Using PowerShell to do the same thing should be considered as a very fast and easy to deploy lightweight version of automatic testing. Possibilities are limited but should be sufficient in most cases.
To succesfully run Selenium tests we need the following files:
Repeat the last two steps for the Selenium.Support.3.14.0.nupkg file.
Download the latest versions.
First thing is to add the PowerShell script to the TFS Code Repostory. Go to Code → PowerShell folder (create it if it doesn't exist yet). Create a new file as seleniumtest.ps1 and add the code (see below the code for necessary explanation:
# Check for.net version of minimum 4.5: if (!(Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\' | Get-ItemPropertyValue -Name Release | Foreach-Object { $_ -ge 394802 })){ exit }else{ write-host "mimimum .net 4.5 version found. Continue" } # Load the Selenium .Net library and paths to the various test libraries # webdriver & support dll Add-Type -Path "D:\selenium\WebDriver.dll" Add-Type -Path "D:\selenium\WebDriver.Support.dll" # Add path for chromedriver.exe and firefox geckodriver.exe $env:PATH += ";D:\selenium" # Testplan variables $testurl = "http://tfs.build.getshifting.com/" $testname = "dtap" $resultsdir = "D:\selenium\testresults" [OpenQA.Selenium.ScreenshotImageFormat]$ImageFormat = [OpenQA.Selenium.ScreenshotImageFormat]::Png ### Firefox testplan ######################################################################################################### # Start firefox headless Write-Host "Start firefox testplan, starting firefox headless" [OpenQA.Selenium.Firefox.FirefoxOptions]$ffoptions = new-object OpenQA.Selenium.Firefox.FirefoxOptions $ffoptions.addArguments('-headless') $ffdriver = New-Object OpenQA.Selenium.Firefox.FirefoxDriver ($ffoptions) # Sometimes headless doesn't work through arguments. Setting the headless environment variable to make sure we're headless $env:MOZ_HEADLESS = 1 # Start firefox testplan Write-Host "Go to url $testurl" $ffdriver.Navigate().GoToURL($testurl) # Get Evidence that the website works $ffcurrenturl = $ffdriver.url $ffcurrenturltitle = $ffdriver.title Write-Host "Current FF url $ffcurrenturl and title $ffcurrenturltitle" # Get Pagesource Write-Host "Create pagesource $ffcurrenturl" $ffdriver.PageSource | Out-File "$resultsdir\ff$testname.html" -Force # Get Screenshots Write-Host "Create screenshot $ffcurrenturl" $Screenshot = [OpenQA.Selenium.Support.Extensions.WebDriverExtensions]::TakeScreenshot($ffdriver) $Screenshot.SaveAsFile("$resultsdir\ff$testname.png", $ImageFormat) # Close firefox Write-Host "Close and quit firefox browser and selenium webdriver. " $ffdriver.Close() $ffdriver.Quit() ### Chrome testplan ######################################################################################################### # Start Chrome headless Write-Host "Start chrome testplan, starting chrome headless" $chromeOptions = New-Object OpenQA.Selenium.Chrome.ChromeOptions $chromeOptions.addArguments('headless') $chromeDriver = New-Object OpenQA.Selenium.Chrome.ChromeDriver($chromeOptions) # Start chrome testplan Write-Host "Go to url $testurl" $chromedriver.Navigate().GoToURL($testurl) # Get Evidence that the website works $chromecurrenturl = $chromeDriver.url $chromecurrenturltitle = $chromeDriver.title Write-Host "Current Chrome url $chromecurrenturl and title $chromecurrenturltitle" # Get Pagesource Write-Host "Create pagesource $chromecurrenturl" $chromeDriver.PageSource | Out-File "$resultsdir\chrome$testname.html" -Force # Get Screenshots Write-Host "Create screenshot $chromecurrenturl" $Screenshot = [OpenQA.Selenium.Support.Extensions.WebDriverExtensions]::TakeScreenshot($chromeDriver) $Screenshot.SaveAsFile("$resultsdir\chrome$testname.png", $ImageFormat) # Close Chrome Write-Host "Close and quit chrome browser and selenium webdriver. " $chromeDriver.Close() $chromeDriver.Quit()
The script performs the following steps:
Get-WindowsFeature -name NET* | where installedNote that this quite a simple testplan. We only go to the url and create some screenshots. In real life you should do some logging in and test more of the actual site.
We need to set the script on the build server so it can be run during the release. We need to do so now because the powershell folder is not included in the artifact. In the build definition, click “+ Add Task” and search for “Copy Files”. Click the “Copy Files” task and click Add. Configure the task like this:
In the release we currently have one phase which runs on the deployment group and which actually deploys the website. We now need to add a second phase that runs on the build agent so the selenium script can run and the results can be collected and emailed.
In the TFS portal go to Build and Release → Release and select the release we need to edit, and click Edit. Select the correct environment and click on the Phases and tasks link. Once you're in the Release definition click the three dots next to “Build environment” and click “Add Agent Phase”. Configure the phase like this:
Now we can add tasks in the phase.
In the release definition, in the newly created Agent Phase, click “+” and search for “PowerShell”. Click the “PowerShell” task and click Add. Configure the task like this:
In the release definition, in the newly created Agent Phase, click “+” and search for “Archive Files”. Click the “Archive Files” task and click Add. Configure the task like this:
In the release definition, in the newly created Agent Phase, click “+” and search for “PowerShell”. Click the “PowerShell” task and click Add. Configure the task like this:
$mailserver = "smtp.getshifting.com" $toemail = "$(release.requestedForEmail)" #$toemail = "$(release.requestedFor)" Write-host "email $toemail" # Check for valid getshifting address if ($toemail -notlike "*getshifting.com"){$toemail = "postmaster@getshifting.com"} Write-host "email $toemail" $fromemail = "$(agent.machineName)@getshifting.com" $subject = "$(release.definitionName) Selenium Testresults" $info = "See the attachment for the testresults. <br>See $(release.releaseWebUrl) for the full results. " Send-MailMessage -To $toemail -From $fromemail -SmtpServer $mailserver -Subject $subject -BodyAsHtml -Body $info -Attachments "$(System.ArtifactsDirectory)/$(Build.BuildId).zip" #Remove zip file remove-item "$(System.ArtifactsDirectory)/$(Build.BuildId).zip"
This sends the email to a valid getshifting email address. This is also the case if the $(release.requestedForEmail) variable is not set, or if it is set to a non-getshifting email address.
The next step is to add some more test steps: Implement Automatic Testing using Selenium and PowerShell in Azure DevOps Server - More TestPlans The next step is to implement another testplan, but with authentication: Implement Automatic Testing With Authentication using Selenium and PowerShell in Azure Devops Server