= Azure Devops Server Maintenance Build =

**Summary**: How to check an isolated environment with Azure DevOps Server 2018. \\
**Date**: 27 May 2020 \\
**Refactor**: 29 April 2025: Checked links and formatting. Rebranded Team Foundation Server (TFS) to Azure Devops Server. \\
{{tag>azuredevops powershell iis}}

> Note that this is done on [[tfs2018|TFS Server 2018]]

When using TFS for deploying an isolated test environment you might have the need to maintain that environment or get some monitoring working. We have the specific case that the environment is cloned from another environment so that means that names and ip addresses are also the same. So this article shows you how to create the infrastructure and the build definition to monitor such environment.

= Infrastructure =

First you need a server to run a build agent on. Note that this server needs to be able to talk to the TFS server by hostname. So you might have to open up some firewall ports, provide NATting and add hostnames to DNS or the hostfile, it all depends on your specific setup. Once you can access the build server from the server, you need to install the build agent. I already explained on how to do that in [[tfs2018#build_agent|this article]].

= Build Definition =

There are several ways to do this, but I opted to create a PowerShell script that will run on the Build Agent and create some html files that can be published as an artifact so you can deploy them to a webserver.

== Create the Build Definition ==

=== New Repository ===

First create a new repository to store the scripts we'll make later on. Go to Code -> The current repository dropdown on the left and click "+ New Repository" . Use GIT and give it a name, her I'll use DevOps-Maintenance.

=== New Build Definition ===

Go to Build and Release -> Builds and click "+ New".

Make sure that when you create the Build Definition to select the Agent queue in which you installed the Build Agent in the step above. Also, make note of the name as you'll need it later on. The name I use in this article is: DevOps-DEV-Maintenance

== Add PowerShell Script ==

Before you can run a powershell script you first need to create one. In the repository we created above create a new folder called powershell and then create a new file within the new folder. You can name it how you like but here I'm using basicmonitoring.ps1. Note that the ps1 extension is required.

This is the script I'm using to monitor state, diskusage and uptime and stopped automatic services. It also provides the server description and adds a custom object to the servers to know on what date they were added. It also sends out monitoring using Microsoft Graph.
<code powershell>

# Author: Sjoerd Hooft / https://www.linkedin.com/in/sjoerdhooft/

### Versioning and Functionality ##################################################################
### Information on Script or Application ###
# Name: Basic Monitoring for DTA Environments.
### Script & Application key users ###
# Script maintained by Sjoerd Hooft
### Versioning ###
### 2020 05 27 - Sjoerd Hooft - First Version ###
# Get all Windows AD Servers
# Check for availability
# Check for uptime
# Check for disk usage
# Check for stopped automatic services
# Output to a html page, using color coding for warnings and error
# Email an overview of all down server to Infra using Graph
# Email individual server problems to Infra and Applications using Graph
### 2020 08 25 - Sjoerd Hooft ###
# Adding Appearance Date to server object and html output
###################################################################################################

### Requirements ##################################################################################
### Powershell
# The script can run on both powershell 5 and powershell core (version 6 and 7). Where needed if versions are behaving different default values are assigned.
# The script is designed to run in TFS and has the following requirements:
### Pipeline name: DevOps-TST-Maintenance (or equivalence for other DTA environments)
### Named parameters: -admin : the name of an account that has domain admin privileges
###                   -adminpass : the matching password
###                   -graphsecret : The secret for the AD Enterprise Application for outgoing email
###################################################################################################

### Bugs ##########################################################################################
### No script bugs known yet
#
###################################################################################################

### How-To ########################################################################################
# Please read the comments on specific sectors for explanation
# Not all build servers are created equally. Check for valid directories. --------- Change to artifact directory, is always available.
### When changing and testing ###
# Set test-mode to "on", instead of "off"
# Verify the output directory exists on the server and is writable
###################################################################################################

### Script Overview ###############################################################################
### Fase 1 ########################################################################################
# Set all Variables
# Define Functions
### Fase 2 ########################################################################################
# Set Graph variables and verify graph connection
# Define Graph Functions
### Fase 3 ########################################################################################
# Run script
# Create output
###################################################################################################

########################################## Start Fase 1 ###########################################

### Script Variables ###
# If used, the param statement MUST be the first thing in your script or function
Param(
   [string]$admin,
   [string]$adminpass,
   [string]$graphsecret
)

### Test Mode ###
# Set to on or off
### This enables:
# more logging
# outputs to a different location
# redirects all outgoing emails to the requestor
# limits the amount of server Email to 10.
### This depends on
# the way the build was started
# an override option in the queue build variables for manual starts
### Set email defaults
$global:to = "Infra@getshifting_com,postmaster@getshifting_com"
$global:cc = ""
$global:bcc = ""
$requesteremail = $env:BUILD_REQUESTEDFOREMAIL
$fallbackemail  = "postmaster@getshifting_com"
# Determine testmode
if ($env:BUILD_REASON -eq "Manual"){
    if ([string]::IsNullOrEmpty($env:EnableTestMode)){
        Write-Host "##[section] Manually triggered build without test mode preference. Test Mode is on. "
        $global:testmode = "on"
        if (![string]::IsNullOrEmpty($requesteremail) -and ($requesteremail -match '@getshifting_com')){
            $global:to = $requesteremail
        }else{
            $global:to = $fallbackemail
        }
    }else {
        Write-Host "##[section] Manually triggered build with test mode preference. "
        if ($env:EnableTestMode -eq "on"){
            $global:testmode = "on"
            if (![string]::IsNullOrEmpty($requesteremail) -and ($requesteremail -match '@getshifting_com')){
                $global:to = $requesteremail
            }else{
                $global:to = $fallbackemail
            }
        } elseif ($env:EnableTestMode -eq "off") {
            $global:testmode = "off"
        } else {
            Write-Host "##[section] Manually triggered build with invalid test mode preference. Exiting now"
            exit 1
        }
    }
} elseif ($env:BUILD_REASON -eq "Schedule"){
    Write-Host "##[section] A schedule triggered the build. Test Mode is off. "
    $global:testmode = "off"
} else {
    Write-Host "##[section] The build was triggered by an event like CI, pullrequest or otherwise. Test Mode is on. "
    $global:testmode = "on"
    if (![string]::IsNullOrEmpty($requesteremail) -and ($requesteremail -match '@getshifting_com')){
        $global:to = $requesteremail
    }else{
        $global:to = $fallbackemail
    }
}

### Determine D-T-A environment depending on builddefinition name ###
### Note: The preferred way to do this would be to include the build agent name, however right now, some environments have the same name for the build agent.
if ($env:BUILD_DEFINITIONNAME -eq "DevOps01-Maintenance"){$global:dtaenvironment = "DTA-01"}
if ($env:BUILD_DEFINITIONNAME -eq "DevOps02-Maintenance"){$global:dtaenvironment = "DTA-02"}
if ($env:BUILD_DEFINITIONNAME -eq "DevOps-Dev-Maintenance"){$global:dtaenvironment = "DTA-DEV"}
if ($env:BUILD_DEFINITIONNAME -eq "DevOps-Test-Maintenance"){$global:dtaenvironment = "DTA-TEST"}
if ($global:testmode -eq "on"){
    Write-Host "##[section] DTA BUILD definition: $($env:BUILD_DEFINITIONNAME)"
    Write-Host "##[section] DTA environment: $global:dtaenvironment"
}

### Create password object ###
# adminpass is now a string so must be converted
$adminpasssec = ConvertTo-SecureString -String $adminpass -AsPlainText -Force
# Checking credentials
if ($global:testmode -eq "on"){Write-Host "Credential variables user is $admin and password is $adminpasssec"}
# Create the Credentials object
$admincreds = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $admin,$adminpasssec

### Set Output Files ###
$filename = "serverstatusv3.html"
$outputfile = "$($env:BUILD_ARTIFACTSTAGINGDIRECTORY)\$($global:dtaenvironment)$($filename)"
# if ($global:testmode -eq "on"){
#     if (Test-path -Path "E:\AplOps\Output"){
#         $outputfile = "E:\AplOps\Output\$($global:dtaenvironment)$($filename)"
#     }elseif (Test-path -Path "D:\AplOps\Output"){
#         $outputfile = "D:\AplOps\Output\$($global:dtaenvironment)$($filename)"
#     }else{
#         $outputfile = "C:\Temp\$($global:dtaenvironment)$($filename)"
#     }
# } else{
#     $outputfile = "$($env:BUILD_ARTIFACTSTAGINGDIRECTORY)\$($global:dtaenvironment)$($filename)"
# }

# Disks to check for freespace - if changed you also need to change the html tables in the functions writeHTMLHeader and writeHTMLData
$diskLetters = "C","D","E","F","G","H"

### Functions ###
### writeHTMLHeader
### Write HTML page head and the table header
function writeHTMLHeader() {
$rundate = (Get-Date -UFormat "%A, %d %B %Y %R")
# Do not use whitespace with a multi-line string -  lines cannot be indented or the tabs/spaces will be embedded in the string
$htmlheader = @"
<!DOCTYPE html>
<html lang='en'>
<head>
<title>$global:dtaenvironment Monitoring</title>
<meta charset='utf-8'>
<meta name='viewport' content='width=device-width, initial-scale=1'>
<style>
    table {
        border-collapse: collapse;
        width:100%;
    }
    table, th, td {
        border: 1px solid #dddddd;
        text-align: left;
    }
</style>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.16.0/umd/popper.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.0/js/bootstrap.min.js"></script>
</head>
<body>
<div class="container-fluid">
<h1>Status for $global:dtaenvironment from $rundate</h1>
<input class="form-control" id="searchBar" type="text" placeholder="Search... ">
<table>
<tr>
<th style="width: 200px;">ServerName</th>
<th style="width: 200px;">Description</th>
<th style="width: 100px;">ServerStatus</th>
<th style="width: 100px;">Uptime</th>
<th style="width: 100px;">Refresh Date</th>
<th style="width: 100px;">C Free in GB/%</th>
<th style="width: 100px;">D Free in GB/%</th>
<th style="width: 100px;">E Free in GB/%</th>
<th style="width: 100px;">F Free in GB/%</th>
<th style="width: 100px;">G Free in GB/%</th>
<th style="width: 100px;">H Free in GB/%</th>
<th>Services - ServiceName(exitcode)</th>
</tr>
<tbody id="dtaTable">
"@
$htmlheader | Out-File $outputfile
}

### Functions: writeHTMLData
### Write table data rows
# Note: The variable name when the function is called and the variable name, as well as the variable itself all must be the same name
function writeHTMLData{
param(
    $computername,
    $serverstate,
    $hruptime,
    $cdisk,
    $ddisk,
    $edisk,
    $fdisk,
    $gdisk,
    $hdisk,
    $allservices,
    $state,
    $description,
    $refreshdate
)
if ($global:testmode -eq "on"){Write-Host "-computername $computername -serverstate $serverstate -hruptime $hruptime -cdisk $cdisk -ddisk $ddisk -edisk $edisk -fdisk $fdisk -gdisk $gdisk -hdisk $hdisk -allservices $allservices -state $state -description $description -refreshdate $refreshdate"}
# Do not use whitespace with a multi-line string - lines cannot be indented or the tabs/spaces will be embedded in the string
# Change the color of the table row depending on the state
if ($state -eq "yellow"){
$htmldata1 = @"
<tr style="background-color: yellow;">
"@
}elseif ($state -eq "orange"){
$htmldata1 = @"
<tr style="background-color: orange;">
"@
}elseif ($state -eq "red"){
$htmldata1 = @"
<tr style="background-color: red;">
"@
}else{
$htmldata1 = @"
<tr>
"@
}
$htmldata2 = @"
<td style="width: 200px;">$computername</td>
<td style="width: 200px;">$description</td>
<td style="width: 100px;">$serverstate</td>
<td style="width: 100px;">$hruptime</td>
<td style="width: 100px;">$refreshdate</td>
<td style="width: 100px;">$cdisk</td>
<td style="width: 100px;">$ddisk</td>
<td style="width: 100px;">$edisk</td>
<td style="width: 100px;">$fdisk</td>
<td style="width: 100px;">$gdisk</td>
<td style="width: 100px;">$hdisk</td>
<td>$allservices</td>
</tr>
"@
$htmldata1 | Out-File $outputfile -Append
$htmldata2 | Out-File $outputfile -Append
}


### Functions: writeHTMLFooter
### Write html page and table closing tags
function writeHTMLFooter(){
# Do not use whitespace with a multi-line string -  lines cannot be indented or the tabs/spaces will be embedded in the string
$htmlfooter = @'
</tbody>
</table>
<script>
$(document).ready(function(){
$("#searchBar").on("keyup", function() {
var value = $(this).val().toLowerCase();
$("#dtaTable tr").filter(function() {
$(this).toggle($(this).text().toLowerCase().indexOf(value) > -1)
});
});
});
</script>
</div>
</body>
</html>
'@
$htmlfooter | Out-File $outputfile -Append
}

### Functions: serverMail
### Send email for individual servers if the state is not green
# Note: The variable name when the function is called and the variable name, as well as the variable itself all must be the same name
function serverMail{
    param(
        $computername,
        $serverstate,
        $hruptime,
        $cdisk,
        $ddisk,
        $edisk,
        $fdisk,
        $gdisk,
        $hdisk,
        $allservices,
        $state,
        $description,
        $refreshdate
    )

    if ($global:testmode -eq "on"){
        Write-Host "Testmode is set to on, limiting the amount of individual server emails to 10, and sending all emails to $global:to."
        $global:servermailteller ++
        if ($global:servermailteller -gt 10){
            Write-Host "We've already send 10 mails, return to the main script"
            return
        }
    }
    if ($global:testmode -eq "on"){Write-Host "Send Server Mail for server $computername and state $serverstate"}
    if ($global:testmode -eq "on"){Write-Host "-computername $computername -serverstate $serverstate -hruptime $hruptime -cdisk $cdisk -ddisk $ddisk -edisk $edisk -fdisk $fdisk -gdisk $gdisk -hdisk $hdisk -allservices $allservices -state $state -description $description -refreshdate $refreshdate"}

    # Change email addresses, depending on servername. Keep Servername in alphabetical order. Don't do this in testmode.
    if ($global:testmode -ne "on"){
        if ($computername -eq "server1"){$serverto = "sjoerd@getshifting_com"}
        if ($computername -eq "server2"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server3"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server4"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server5"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server6"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server7"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server8"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server9"){$serverto = "user2@getshifting_com"}
        if ($computername -eq "server10"){$serverto = "user3@getshifting_com"}

    }
    # is serverto is not set yet, set it to the global default
    if ([string]::IsNullOrEmpty($serverto)){$serverto = $global:to}

    # Set custom subject
    $subjectserver = $global:Subject + " - $computername"

    # Set custom body
    $bodyserver += "<br>There is a problem with server $computername in $global:dtaenvironment <br>"
    if ($state -eq "red"){$bodyserver += "<p style=color:red>State = $serverstate </p><br>"}
    if (![string]::IsNullOrEmpty($description)){$bodyserver += "Server description is $description <br> "}
    if ($state -eq "orange"){$bodyserver += "<p style=color:orange>State = $serverstate </p><br>"}
    if ($state -eq "yellow"){$bodyserver += "<p style=color:yellow>State = $serverstate </p><br>"}
    if (![string]::IsNullOrEmpty($hruptime)){$bodyserver += "Uptime is $hruptime <br> "}
    if (![string]::IsNullOrEmpty($refreshdate)){$bodyserver += "Refresh Date is $refreshdate <br> "}
    if (![string]::IsNullOrEmpty($cdisk)){$bodyserver += "C drive = $cdisk <br>"}
    if (![string]::IsNullOrEmpty($ddisk)){$bodyserver += "D drive = $ddisk <br>"}
    if (![string]::IsNullOrEmpty($edisk)){$bodyserver += "E drive = $edisk <br>"}
    if (![string]::IsNullOrEmpty($fdisk)){$bodyserver += "F drive = $fdisk <br>"}
    if (![string]::IsNullOrEmpty($gdisk)){$bodyserver += "G drive = $gdisk <br>"}
    if (![string]::IsNullOrEmpty($hdisk)){$bodyserver += "H drive = $hdisk <br>"}
    if (![string]::IsNullOrEmpty($allservices)){$bodyserver += "The following services should start automatically but are not running, including their exitcode = $allservices <br><br>"}

    $body = $bodystart + $bodyserver
    if ($global:testmode -eq "on"){Write-Host "Body: $body"}
    # Send email
    SendGraphEmail -To $serverto -CC $CC -BCC $global:BCC -subject $subjectserver -emailtype $emailtype -body $body -from $From -oauthgraph $oauthgraph

    # Reset values
    $bodyserver = $null
    $serverto = $null

}

### Functions: Set warning level
### Determine the new warning level depending on the current state and the severity of the just found warning
Function SetWarningLevel(){
    param (
        $state,
        $severity
        )
    # If state is already highest
    if (($state -eq "red") -or ($severity -eq "red")){
        return "red"
    }elseif (($state -eq "orange") -or ($severity -eq "orange")){
        return "orange"
    }elseif (($state -eq "yellow") -or ($severity -eq "yellow")){
        return "yellow"
    }else{
        return "green"
    }
}

### Functions: ToHumanReadable
### Display the uptime in a Human Readable Format
Function ToHumanReadable(){
    param($timespan)

    $sb = New-Object System.Text.StringBuilder

    If ($timespan.TotalHours -lt 1) {
      [void]$sb.Append($timespan.Minutes)
      [void]$sb.Append("m")
      return $timespan.Minutes + "minutes"
    } else{
      If ($timespan.Days -gt 0) {
          [void]$sb.Append($timespan.Days)
          [void]$sb.Append("d")
          [void]$sb.Append(", ")
        }
        If ($timespan.Hours -gt 0) {
          [void]$sb.Append($timespan.Hours)
          [void]$sb.Append("h")
          [void]$sb.Append(", ")
        }
        If ($timespan.Minutes -gt 0) {
          [void]$sb.Append($timespan.Minutes)
          [void]$sb.Append("m")
        }
        return $sb.ToString()
    }
}

### Functions: Check Availability
### Check if a server responds to ping, and if so, responds to wmi
Function CheckAvailability(){
    param ($computername)
    if (Test-Connection $computername -quiet -count 1){
		# Server responds to ping
		# Powershell 6 does not support wmi like this anymore: https://docs.microsoft.com/en-us/powershell/scripting/whats-new/breaking-changes-ps6?view=powershell-6
		if ($PSVersionTable.PSVersion.Major -eq 6){
			Return "OK"}
		elseif (get-wmiobject -ErrorAction SilentlyContinue -computername $computername "win32_process" -Credential $admincreds){
			# Server responds to wmi
			Return "OK"}
		else{Return "WMIError"}}
	else{Return "PingError"}
}

### Functions: Check Hard Disk Usage
### Function to check hard disk free space and percentage
Function CheckHardDiskUsage() {
	param ($hostname, $deviceID)
    Try
	{
    	$HardDisk = $null
		$HardDisk = Get-WmiObject Win32_LogicalDisk -ComputerName $hostname -Filter "DeviceID='$deviceID' and Drivetype='3'" -ErrorAction Stop -Credential $admincreds| Select-Object Size,FreeSpace
        if (!(([string]::IsNullOrEmpty($HardDisk)))){
            $DiskTotalSize = $HardDisk.Size
            $DiskFreeSpace = $HardDisk.FreeSpace
            $frSpace=[Math]::Round(($DiskFreeSpace/1073741824),2)
            $PercentageDS = (($DiskFreeSpace / $DiskTotalSize ) * 100); $PercentageDS = [math]::round($PercentageDS, 2)

            Add-Member -InputObject $HardDisk -MemberType NoteProperty -Name PercentageDS -Value $PercentageDS
            Add-Member -InputObject $HardDisk -MemberType NoteProperty -Name frSpace -Value $frSpace
		}
		return $HardDisk
    }Catch{
        write-host "Error returned while checking the Hard Disk usage. Perfmon Counters may be fault"
    }
}


########################################## Start Fase 2 ###########################################

### Set Graph Variables ###
[string]$ClientID = $env:AzureDevOpsClientID #Require
[string]$ClientSecret = $graphsecret #Require
[string]$TenantDomain = $env:Tenant #Require

if ($global:testmode -eq "on"){
    Write-Output "Check Azure Enterprise Application details"
    Write-Output "ClientID: $ClientID"
    Write-Output "Tenant Domain: $TenantDomain"
}

### Start Graph Connection ###
try{
    $bodyGraph = @{grant_type="client_credentials";scope="https://graph.microsoft.com/.default";client_id=$ClientID;client_secret=$ClientSecret}
    $oauthGraph = Invoke-RestMethod -Method Post -Uri https://login.microsoftonline.com/$tenantdomain/oauth2/v2.0/token -Body $bodygraph
    if ($global:testmode -eq "on"){Write-Output "OauthGraph: $oauthGraph"}
}catch{
    Write-Host "##vso[task.LogIssue type=error;]$env:TASK_DISPLAYNAME - Something went wrong with the Graph connection";
    Write-Host "##vso[task.LogIssue type=error;]$env:TASK_DISPLAYNAME - Error message: $($_.Exception.Message)";
    #exit 1 # Enable this if you don't want the monitoring to run without email functionality
}

### Set Email defaults ###
$global:From = "postmaster@getshifting_com" #Require # Set to a shared mailbox which will hold the send emails
$global:Subject = "DTA Alert - $global:dtaenvironment" #Require
$global:Bodystart = "This email is sent from: <strong>Agent:</strong> $($env:AGENT_NAME); <strong>Build Definition:</strong> $($env:BUILD_DEFINITIONNAME); <strong>Requestor:</strong>  $($env:BUILD_QUEUEDBY); <strong>Build ID:</strong> $($env:BUILD_BUILDID)" #Require
[bool]$BodyAsHtml = $true
if ($BodyAsHtml -eq "True"){$global:emailtype = "html"}else{$global:emailtype = "text"}

if ($global:testmode -eq "on"){
    Write-Output "Check input parameters email defaults"
    Write-Output "To: $global:To"
    Write-Output "Cc: $global:Cc"
    Write-Output "Bcc: $global:Bcc"
    Write-Output "From: $global:From"
    Write-Output "Subject: $global:Subject"
    Write-Output "Body: $Bodystart"
    Write-Output "HTML: $BodyAsHtml"
}

### Send Email ###
Function SendGraphEmail(){
    param(
        $To,
        $Cc,
        $Bcc,
        $subject,
        $emailtype,
        $body,
        $From,
        $oauthgraph
    )

    ### Don't send emails in DTA-Test and DTA-DEV
    if (($global:dtaenvironment -eq "DTA-DEV") -OR ($global:dtaenvironment -eq "DTA-XXX")){ #replace by dta-test to disable email for dta-test
        if ($global:testmode -eq "on") {Write-Output "Not sending email for $global:dtaenvironment, subject: $subject"}
        return
    }

    ### Convert email addresses to JSON format

    # Convert To Address
    $tosplit = @($to.Split(','))
    $ToinJSON = $tosplit | %{'{"EmailAddress": {"Address": "'+$_+'"}},'}
    $ToinJSON = ([string]$ToinJSON).Substring(0, ([string]$ToinJSON).Length - 1)

    # Convert CC Address
    if (![string]::IsNullOrEmpty($cc)){
        if ($global:testmode -eq "on") {Write-Output "cc not empty, start json format"}
        $ccsplit = @($cc.Split(','))
        $CCinJSON = $ccsplit | %{'{"EmailAddress": {"Address": "'+$_+'"}},'}
        $CCinJSON = ([string]$CCinJSON).Substring(0, ([string]$CCinJSON).Length - 1)
    }else{
        $CCinJSON = ""
    }

    # Comvert BCC address
    if (![string]::IsNullOrEmpty($bcc)){
        if ($global:testmode -eq "on") {Write-Output "bcc not empty, start json format"}
        $bccsplit = @($bcc.Split(','))
        $BCCinJSON = $bccsplit | %{'{"EmailAddress": {"Address": "'+$_+'"}},'}
        $BCCinJSON = ([string]$BCCinJSON).Substring(0, ([string]$BCCinJSON).Length - 1)
    }else{
        $BCCinJSON = ""
    }

    if ($global:testmode -eq "on"){
        write-host "Display all email address in JSON"
        Write-Output "To: $to"
        Write-Output "ToSplit: $tosplit"
        Write-Output "ToJson: $ToinJSON"
        Write-Output "CC: $cc"
        Write-Output "CCSplit: $ccsplit"
        Write-Output "CCJson: $CCinJSON"
        Write-Output "BCC: $bcc"
        Write-Output "BCCSplit: $bccsplit"
        Write-Output "BCCJson: $BCCinJSON"
    }


    # Create email in JSON
    $reqBody='{
        "message": {
        "subject": "",
        "body": {
            "contentType": "",
            "content": ""
        },
        "toRecipients": [
            PlaceHolderTo
        ],
        "ccRecipients": [
            PlaceHolderCC
        ],
        "bccRecipients":[
            PlaceHolderBCC
        ],
        "replyTo":[

        ]
        }
    }'
    $reqBody = $reqBody -replace "PlaceholderTo",$ToinJSON
    $reqBody = $reqBody -replace "PlaceholderCC",$CCinJSON
    $reqBody = $reqBody -replace "PlaceholderBCC",$BCCinJSON

    if ($global:testmode -eq "on"){Write-Output "Final Check reqBody Before JSON: $reqBody"}
    $reqBody = $reqBody | ConvertFrom-Json
    $reqBody.message.subject = $Subject
    $reqBody.message.body.contentType = $emailtype
    $reqBody.message.body.content = $Body
    if ($global:testmode -eq "on"){Write-Output "Final Check reqBody After JSON: $reqBody"}

    ### Send email ###
    try{
        Invoke-RestMethod -Method Post -Uri "https://graph.microsoft.com/v1.0/users/$($From)/sendMail" -Headers @{'Authorization'="$($oauthgraph.token_type) $($oauthgraph.access_token)"; 'Content-type'="application/json"} -Body ($reqBody | ConvertTo-Json -Depth 4 | Out-String)
    }catch{
        Write-Host "##vso[task.LogIssue type=error;]$env:TASK_DISPLAYNAME - Something went wrong with sending email";
        Write-Host "##vso[task.LogIssue type=error;]$env:TASK_DISPLAYNAME - Error message: $($_.Exception.Message)";
        #exit 1 # Enable this if you don't want the monitoring to run without email functionality
    }
}

########################################## Start Fase 3 ###########################################

# Get a list of AD servers and other AD computer objects that are restored to a test environment
$computers = Get-ADComputer -Filter {(operatingSystem -like "*windows*Server*" -or Name -like "w10-mdm*")} -Properties Description,extensionAttribute1 | Sort-Object Name # Add more patterns by adding -or Name like "xxx*"
$computercount = $computers.count
if ($global:testmode -eq "on"){write-host "Found $computercount servers to check in ActiveDirectory. "}

# Start Report
writeHTMLHeader

# Creating array to hold servernames that are down.
$allserversdown = @()
$serverdowncount = 0

# Define yesterday for the refreshdate
$date = (get-date).addDays(-1)
[string]$yesterday = (Get-Date $date -UFormat "%d %B %Y")

# Start checking all servers in this environment
$teller = 1
ForEach ($computer in $computers){
    $state = "green"
    $computername = [string]$computer.Name
    $description = [string]$computer.Description
    $serverstate = CheckAvailability $computername
	write-host "Working on $teller of $computercount : $computername = $serverstate"

    if ($serverstate -eq "PingError"){
        $serverdowncount ++
        $state = SetWarningLevel $state "red"
        $serverinfo = "" | Select-Object name,status
        $serverinfo.name = $computername
        $serverinfo.status = $serverstate
        $allserversdown += $serverinfo
    }

    if ($serverstate -eq "WMIError"){
        $state = SetWarningLevel $state "yellow"
    }

    # Checking and Setting Refresh Date
    # As the script always runs in the morning, the refresh date (or so to say, the date the restore is made from) is always yesterday
    $refreshdate = [string]$computer.extensionAttribute1
    if ([string]::IsNullOrEmpty($refreshdate)){
        Set-ADComputer -Identity $computer -add @{"extensionAttribute1"=$yesterday} -Credential $admincreds
        if ($global:testmode -eq "on"){write-host "$computer has updated extensionAttribute1 into refreshdate $yesterday. "}
        $refreshdate = $yesterday
    }else{
        if ($global:testmode -eq "on"){write-host "$computer already has extensionAttribute1 updated to the refreshdate $refreshdate. "}
    }


    # Checking diskspace
    if ($serverstate -eq "OK"){
        foreach ($disk in $diskLetters){
            $HardDisk = CheckHardDiskUsage -hostname $computername -deviceID "$($disk):"
            if (!(([string]::IsNullOrEmpty($HardDisk)))){
                $PercentageDS = $HardDisk.PercentageDS
                if ($PercentageDS -le 5){$state = SetWarningLevel $state "red"}
                elseif (($PercentageDS -gt 5) -and ($PercentageDS -le 10)){$state = SetWarningLevel $state "orange"}
                elseif (($PercentageDS -gt 10) -and ($PercentageDS -le 15)){$state = SetWarningLevel $state "yellow"}
                $frSpace = $HardDisk.frSpace
                if ($global:testmode -eq "on"){write-host "Disk is $disk, Free disk space is $frSpace GB, Percentage is $PercentageDS %"}
                if ($disk -eq "C"){$cdisk = "$($frSpace)G / $($PercentageDS)%"}
                if ($disk -eq "D"){$ddisk = "$($frSpace)G / $($PercentageDS)%"}
                if ($disk -eq "E"){$edisk = "$($frSpace)G / $($PercentageDS)%"}
                if ($disk -eq "F"){$fdisk = "$($frSpace)G / $($PercentageDS)%"}
                if ($disk -eq "G"){$gdisk = "$($frSpace)G / $($PercentageDS)%"}
                if ($disk -eq "H"){$hdisk = "$($frSpace)G / $($PercentageDS)%"}
                # Resetting values
                $PercentageDS = 0
                $frSpace = 0
                $HardDisk = $null
            }
        }
    }

    # Checking uptime
    if ($serverstate -eq "OK"){
        try { $wmi=Get-WmiObject -class Win32_OperatingSystem -computer $computername -Credential $admincreds}
        catch { $wmi = $null }
        if (!(([string]::IsNullOrEmpty($wmi)))){
            $LBTime=$wmi.ConvertToDateTime($wmi.Lastbootuptime)
            [TimeSpan]$uptime = New-TimeSpan $LBTime $(get-date)
            $hruptime = ToHumanReadable($uptime)
            if ($global:testmode -eq "on"){Write-Host "last boot time = $LBTime , uptime is now $hruptime"}
        }
        else {
            write-host "WMI connection failed - check WMI for corruption"
        }
    }

    # Check automated services
    if ($serverstate -eq "OK"){
        if ($global:testmode -eq "on"){write-host "$computername Start Checking on Services."}
        try { $wmi=Get-WmiObject -Class Win32_Service -ComputerName $computername -Filter "startmode = 'auto' AND state != 'running' AND name != 'RemoteRegistry' AND name != 'WbioSrvc' AND name != 'clr_optimization_v4.0.30319_32' AND name != 'clr_optimization_v4.0.30319_64' AND name != 'sppsvc' AND name != 'CDPSvc' AND name != 'MapsBroker' AND name != 'ShellHWDetection' AND name != 'NetPipeActivator' AND name != 'NetTcpActivator' AND name != 'ATAGateway'" -Credential $admincreds | Select-Object name,startname,exitcode,displayname}
        catch { $wmi = $null }
        if (!(([string]::IsNullOrEmpty($wmi)))){
            if ($global:testmode -eq "on"){write-host "Stopped Automatic Services: Name - DisplayName - StartName - ExitCode"}
            foreach ($stoppedservice in $wmi){
                if ($global:testmode -eq "on"){write-host "$($stoppedservice.Name) - $($stoppedservice.displayname) - $($stoppedservice.startname) - $($stoppedservice.exitcode)"}
                # Create output and warninglevel
                if ($stoppedservice.exitcode -ne 0){$state = SetWarningLevel $state "orange"}
                $service = "$($stoppedservice.displayname)($($stoppedservice.exitcode)); "
                $allservices = $allservices + $service -join '; '
            }

        }
    }

    # Add all the found data to the html table
    if ($global:testmode -eq "on"){Write-Host "-computername $computername -serverstate $serverstate -hruptime $hruptime -cdisk $cdisk -ddisk $ddisk -edisk $edisk -fdisk $fdisk -gdisk $gdisk -hdisk $hdisk -allservices $allservices -state $state -description $description -refreshdate $refreshdate"}
    writeHTMLData -computername $computername -serverstate $serverstate -hruptime $hruptime -cdisk $cdisk -ddisk $ddisk -edisk $edisk -fdisk $fdisk -gdisk $gdisk -hdisk $hdisk -allservices $allservices -state $state -description $description -refreshdate $refreshdate

    # Send individual server emails
    if (($state -eq "red") -or ($state -eq "orange")){
        serverMail -computername $computername -serverstate $serverstate -hruptime $hruptime -cdisk $cdisk -ddisk $ddisk -edisk $edisk -fdisk $fdisk -gdisk $gdisk -hdisk $hdisk -allservices $allservices -state $state -description $description -refreshdate $refreshdate
    }

    # Reset values
    $description = $null
    $hruptime = $null
    $cdisk = $null
    $ddisk = $null
    $edisk = $null
    $fdisk = $null
    $gdisk = $null
    $hdisk = $null
    $allservices = $null
    $state = "green"
    $refreshdate = $null

    # Add one to the counter
    $teller ++
}

# Finish up the output file
writeHTMLFooter

# Send email to infra with all servers that are down
if ($serverdowncount -gt 0){
    # Convert the list of all servers that are down to a list that can be re-used from the email
    $alldownasstring = ($allserversdown | select name -ExpandProperty Name) -join '";"'
    $alldownasstringforarrayinput = '@("' + $alldownasstring + '")'
    # Setup the new body
    $bodyinfra += "<br><br>See all servers that cannot be pinged. If these servers should be removed from Active Directoy please use the commands below the list. <br><br>Server Name - Server Status<br>"
    foreach ($server in $allserversdown){$bodyinfra += "$($server.name) - $($server.status)<br>"}
    $bodyinfra += "<br>Use the following command to remove all these servers from Active Directory<br>"
    $bodyinfra += "<p style=color:red>Please don't do this in production<br>"
    $bodyinfra += "No really. Don't do this in production</p>"
    $bodyinfra += "<p style=font-family:monospace>" + '$removeservers' +" = $alldownasstringforarrayinput <br><br>"
    #$bodyinfra += 'ForEach ($server in $removeservers){write-host "Remove server: $server"; Remove-ADComputer $server -Confirm:$false}' + "</p>"
    $bodyinfra += 'ForEach ($server in $removeservers){write-host "Remove server: $server"; Get-ADObject -LDAPFilter "(objectClass=Computer)" | ? Name -eq $server | Remove-ADObject -Recursive' + "</p>"
    $body = $bodystart + $bodyinfra
    if ($global:testmode -eq "on"){Write-Host "Body: $body"}
    # Send email
    SendGraphEmail -To $To -CC $CC -BCC $global:BCC -subject $subject -emailtype $emailtype -body $body -from $From -oauthgraph $oauthgraph
}else {
    $bodyinfra += "<br><br>There are no servers that cannot be pinged in $global:dtaenvironment <br>"
    $body = $bodystart + $bodyinfra
    if ($global:testmode -eq "on"){Write-Host "Body: $body"}
    # Send email
    SendGraphEmail -To $To -CC $CC -BCC $global:BCC -subject $subject -emailtype $emailtype -body $body -from $From -oauthgraph $oauthgraph
}


</code>

Note:
* The scripts expects the user and the user password as input. You need to add these to the variables.
* The script is ready for multiple environments and different types of build agent servers. Modify the script to your need accordingly.

== Task: Run PowerShell Script ==

Follow these steps to add the script to the build definition:
* Add a task of the type PowerShell, version 1
* Modify these settings:
  * Type: File Path
  * Script Path: powershell/basicmonitoring.ps1
  * Arguments: -admin $(devadmin) -adminpass $(devadminpass)
  * Control Option: Continue on Error should be enabled

=== Add Variables ===

Go to the variables tab and add the variables devadmin and devadminpass as required by input. Mark the devadminpass as a secret.

== Task: Copy Files ==

Follow these steps to copy the output files to the build artifact directory:
* Add a taks of the type Copy Files, version 2
* Modify these settings:
  * Source Folder: E:\DevOps\Output (but depends on the exact output. Check the script above)
  * Contents: {{{**}}}
  * Target Folder: $(build.artifactstagingdirectory)

== Task: Publish Build Artifacts ==

Follow these steps to publish the artifact so it can be released:
* Add a task of the type Publish Build Artifacts
* Modify these settings:
  * Path to Publish: $(build.artifactstagingdirectory)
  * Artifact Name: Maintenance
  * Artifact Type: Server

= Release =

As shown before in [[tfsbuildrelease#create_release]] you can now create a new release pipeline or add the artifact to a current release. As [[tfsbuildrelease]] already shows you how to create a release pipeline follow these steps to add this artifact to a existing one:

* Go to Releases and edit the Release Pipeline you want to edit
* In the Artifact section you can click +Add to select the Build Definition as a source. Note thtat you must have run the build in order to be able to select it. Click done to save, and you'll see that the artifact will have been added to the artifact section.
* Now in the environment section click the link to the phases and tasks
* As I am releasing to a website click the plus sign to add a task: Deploy IIS website
* Modify these settings:
  * Package or Folder: $(System.DefaultWorkingDirectory)/DevOps-DEV-Maintenance/maintenance

That's all, you can now configure CI/CD if needed.