wiki.getshifting.com

--- Sjoerd Hooft's InFormation Technology ---

User Tools

Site Tools


start

This is an old revision of the document!


SHIFT-WIKI - Sjoerd Hooft's InFormation Technology

This WIKI is my personal documentation blog. Please enjoy it and feel free to reach out through blue sky if you have a question, remark, improvement or observation.


Thoughts on Upgrading Terraform Providers

Summary: On this wiki page I will cover my experience in upgrading the AzureRM Terraform provider from version 3.112 to 4.51.
Date: 8 November 2025

Recently I had to upgrade the AzureRM terraform provider. We hadn't upgraded it in a while due to other priorities, but once these were addressed, I set about the task of upgrading the provider. On this page I will try to give some background on the process and the provider itself, as well as some of the issues I encountered.

Background

The AzureRM provider is the official Terraform provider for managing Microsoft Azure resources. It is maintained by HashiCorp and Microsoft, and it allows users to define and manage Azure infrastructure using Terraform's declarative configuration language. The provider is regularly updated to add new features, fix bugs, and improve performance.

The latest version can be seen on the terraform registry. It allows you to check available resources and the available options for these resources.

Upgrade Process

Hashicorp, the company behind terraform, provides a general guide on upgrading providers. The process basically comes down to:

  • Get your current deployment stable and working
    • Make sure you can do a `terraform init`, `terraform plan` and `terraform apply` without any issues
    • Create a backup of your terraform state file
    • Also install the latest version of the terraform CLI. Note that if you're running terraform from a pipeline on which you cannot control the terraform CLI version, you should install the same version as installed on the pipeline to avoid any discrepancies.
  • Update provider configuration
    • Update the version in your `provider` block to the desired version
  • Create a plan
    • Run `terraform init` to download the new provider version
      • Use the `-upgrade` flag to ensure that the latest version is downloaded
    • Run `terraform plan` to see what changes will be made
  • Identify and document errors, warnings, or actions
    • Tackle one issue at a time. Start with errors, and only proceed to warnings once all errors are resolved. I had several occurences that after removing certain errors, also warnings dissapeared.
    • Refactor your configuration as needed
  • Apply your plan
    • Run `terraform apply` to implement the changes

And an additional note, if you're working with a pipeline to deploy your terraform code, try to be able to perform `terraform plan` locally first. This will save you a lot of time.

Update the Provider Version

This was our old provider block:

terraform {
  required_version = ">= 0.14.9"
 
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.112"
    }
  }
}

And this is our new provider block. As you can see, only the version number has changed:

terraform {
  required_version = ">= 0.14.9"
 
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "4.51.0"
    }
  }
}

Run the Plan Locally

The following commands were run in a PowerShell terminal on my local machine:

az login
az account set --subscription 30b3c71d-a123-a123-a123-abcd12345678
cd "C:\Repos\tf\applications\main"
terraform version
# Do a normal terraform init. It will tell you to use -upgrade to get the latest provider version
terraform init `
  -backend-config=storage_account_name="saeuwdevtf" `
  -backend-config=container_name="terraform" `
  -backend-config=key="dev.tfstate" `
  -backend-config=resource_group_name="rg-euw-dev-tf" `
  -backend-config=subscription_id="30b3c71d-a123-a123-a123-abcd12345678"
# So do a terraform init with -upgrade
terraform init `
  -backend-config=storage_account_name="saeuwdevtf" `
  -backend-config=container_name="terraform" `
  -backend-config=key="dev.tfstate" `
  -backend-config=resource_group_name="rg-euw-dev-tf" `
  -backend-config=subscription_id="30b3c71d-a123-a123-a123-abcd12345678" `
  -upgrade
# Then do a plan, and make sure to provide your tfvars file
terraform plan `
  -var-file="env/dev.tfvars" `
  -input="false"

Issues

When running the `terraform plan` command, several issues are to be expected. I'll address some of the issues I encountered below.

Deprecated Attributes

We had some resources with deprecated attributes. Especially the Kubernetes cluster resource had many attributes that were no longer supported. The way to tackle them is like this:

  • Lookup the resource in the terraform registry
  • Try to find the deprecated attribute. Sometimes it's only renamed, but then the description stayed the same. Other times it is completely removed. If that's the case it's isually mentioned in the old version of the documentation.
    • To find the old version of the documentation, you can use the version selector (it's almost at the top) on the terraform registry page of the resource.
  • Rename or remove the attribute in your terraform code

New Default Values

Both on the kubernetes resource as well as on storage account we had some attributes that had new default values. This can happen because the new provider suddenly has support for an attribute and sets a default value different from the Azure default value. These settings are usually new, so you need to check them thoroughly. For example, on the storage account the options to allow for cross tenant copy got disabled. And for the kubernetes cluster the node upgrade channel got changed. This actually caused an node image upgrade during the night because I missed that during the plan.

Provider Changes

We also had some errors and warning for the provider itself. It suddenly required a subscription id and also some of the attributes of the provider got renamed.

This was our old provider configuration:

provider "azurerm" {
  skip_provider_registration = true
  features {
    virtual_machine {
      skip_shutdown_and_force_delete = true
      delete_os_disk_on_deletion     = true
    }
  }
}

And this is our new provider configuration:

provider "azurerm" {
  subscription_id                 = var.env_subscription_id
  resource_provider_registrations = "none"
  features {
    virtual_machine {
      skip_shutdown_and_force_delete = true
      delete_os_disk_on_deletion     = true
    }
  }
}
Note that adding the subscription id solved a lot of decoding errors:
│ Warning: Failed to decode resource from state
│
│ Error decoding "module.restore_storage_account.module.diagnostic_settings.azurerm_monitor_diagnostic_setting.diagnostic_setting[0]"
│ from prior state: unsupported attribute "log"

Conclusion

Upgrading terraform providers can be a tedious task, especially when there are many breaking changes. However, by following a systematic approach and addressing issues one at a time, the process can be managed effectively. Always ensure to back up your state file before making any changes, and test thoroughly after the upgrade to ensure everything is functioning as expected. The whole process took me two working days with time for other issues and tasks as well in between.

2025/11/08 16:23

How to Add Text to Each File in a Directory in a Repository

Summary: This wiki page shows how I added google ads to each wiki page in a Repository using powershell, and use a VS Code extension to make sure new files get the same treatment.
Date: 25 October 2025

Background

When using Google Auto Adsense on my wiki, I get ads like everywhere, which makes the pages look cluttered and hard to read. My solution is to add an ad at the bottom of each page. However, I don not want to do that manually.

Using a VS Code extension to run a script on file save

I originally wanted to use git hooks to run a script. However, git hooks have a notourious bad reputation when running them on Windows. So instead I opted to use a VS Code extension that runs a script when saving a file.

I previously used this Run on Save extension for work, which worked very well, so I decided to use it here again.

After installing the extension, you need to add some configuration to your settings.json file. I've used the vscode settings extensively before, so I just needed to add the following settings:

    "emeraldwalk.runonsave": {
        // Messages to show before & after all commands
        "message": "*** All Start ***",
        "messageAfter": "*** All Complete ***",
        // Show elappsed time for all commands
        "showElapsed": true,
        "commands": [
            {
                "match": "\\.txt$",
                "notMatch": "\\drafts/.*$",
                "cmd": "pwsh.exe -ExecutionPolicy Bypass -File ${workspaceFolder}\\ads\\Add-AdsToWikiPages.ps1"
            }
        ]
    }

This will run the powershell script located in the ads folder every time a .txt file is saved in the dokuwiki/pages folder.

The Powershell script

This is the powershell script me and copilot created together and I tested very thoroughly before using it on my actual wiki pages:

  • It starts with defining the function that can later be called with parameters.
  • It has an option to exclude certain files from being modified.
  • It checks if the string to be added is already present in the file, and only adds it if it's not there yet.
  • It appends the string at the end of the file, ensuring proper formatting with new lines.
function Add-StringToFiles {
  param(
    [Parameter(Mandatory)]
    [string]$FolderPath,
    [Parameter(Mandatory)]
    [string]$StringToAdd,
    [string[]]$ExcludeList = @()
  )
 
  Get-ChildItem -Path $FolderPath -File -Filter "*.txt" | ForEach-Object {
    $filePath = $_.FullName
    $fileName = $_.Name
 
    # Skip if file is in exclude list
    if ($ExcludeList -contains $fileName) {
      Write-Host "Skipping excluded file: $fileName"
      return
    }
 
    $content = Get-Content -Path $filePath -Raw
 
    if ($content -notmatch [regex]::Escape("//This wiki has been made possible by://")) {
      $newContent = "$content`r`n$StringToAdd`r`n"
      Set-Content -Path $filePath -Value $newContent
    }
  }
}
 
$folder = "C:\Repos\GetShifting\knowledge\dokuwiki\pages"
$excludeFiles = @("overview.txt", "alltags.txt", "sidebar.txt")
$adString = '//This wiki has been made possible by://
 
<HTML>
<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-8613096447910897"
     crossorigin="anonymous"></script>
<!-- Wiki End of Page -->
<ins class="adsbygoogle"
     style="display:block"
     data-ad-client="ca-pub-8613096447910897"
     data-ad-slot="6221699236"
     data-ad-format="auto"
     data-full-width-responsive="true"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script>
</HTML>
'
 
Add-StringToFiles -FolderPath $folder -StringToAdd $adString -ExcludeList $excludeFiles

This wiki has been made possible by:

2025/10/25 18:06

Using VS Code and Lens to connect to a private AKS cluster

Summary: This wiki page explains how to use VS Code and Lens to connect to a private AKS cluster.
Date: 15 September 2025

There are many ways to connect to a private AKS cluster. This page assumes you have a jumpbox VM in the same virtual network as the AKS cluster, and that you can connect to the jumpbox using Azure Bastion. Using VS Code with the extensions explained below is the easiest way to connect to a private AKS cluster, but also has it's limitations. For that reason, I'll also explain how to use Lens to connect to the private AKS cluster.

Design

See here the design of the network setup to connect to a private AKS cluster using a jumpbox through Azure Bastion. For simplicity, setup and requirements that are not relevant for this page, like private DNS, are left out:

Network overview of an private AKS cluster


VS Code

To use VS Code to connect to a private AKS cluster, make sure you have the 'Remote Explorer' and 'Kubernetes' extensions installed. Then, follow these steps:

  • Optional: If applicable, activate the required Privileged Identity Management (PIM) groups to assign permissions for your account. You need at least 'Reader' permissions on the resource group where the bastion is located, and, depending on what you'll need to do, 'Reader' or 'Contributor' permissions on the resource group where the jumpbox and AKS cluster are located.
  • Use the following Azure CLI commands to connect to Azure and set the subscribtion to the one with the bastion:
    # We need to use the azure cli as creating the tunnel to the jumpbox will only work with the azure cli
    az logout
    # Disable the subscription selector feature as well as sign in with WAM
    az config set core.login_experience_v2=off
    az config set core.enable_broker_on_windows=false
    az login --tenant 7e4an71d-a123-a123-a123-abcd12345678
    # Set the subscription to the bastion subscription
    az account set --subscription aa123456-a123-a123-a123-abcd12345678
  • Use the following Azure CLI command to create a ssh tunnel to the jumpbox through the bastion:
    az network bastion tunnel --name bas-001 --resource-group rg-bastion --target-resource-id /subscriptions/aa123456-a123-a123-a123-abcd12345678/resourceGroups/rg-cluster/providers/Microsoft.Compute/virtualMachines/vm-jumpbox --resource-port 22 --port 50022
  • To use the 'Remote Explorer' extension, you need to add an entry to your ssh config file (usually located at `C:\Users\<your-username>\.ssh\config`):
    # Read more about SSH config files: https://linux.die.net/man/5/ssh_config
    Host vm-jumpbox
        HostName 127.0.0.1
        User azadmin
        Port 50022
        IdentityFile C:/Users/sjoer/.ssh/private_cluster.pem
    • Note that the `IdentityFile` must point to the private key that matches the public key configured on the jumpbox VM.
  • In VS Code, open the Remote Explorer extension from the activity bar → SSH Targets → Open 127.0.0.1 (linux) in a new window
    • This will open a new VS Code window connected to the jumpbox VM. You can now open a terminal in this window and run commands on the jumpbox VM. To be able to connect to the AKS cluster, you need to install various tools on the jumpbox first. Note that all of the command below can be obtained by going to your cluster in the Azure portal and then to 'Connect':
# Install the Azure CLI: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
# Download and install the Microsoft signing key
sudo mkdir -p /etc/apt/keyrings
curl -sLS https://packages.microsoft.com/keys/microsoft.asc |
    gpg --dearmor |
    sudo tee /etc/apt/keyrings/microsoft.gpg > /dev/null
sudo chmod go+r /etc/apt/keyrings/microsoft.gpg
 
# Add the Azure CLI software repository
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=`dpkg --print-architecture` signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" |
    sudo tee /etc/apt/sources.list.d/azure-cli.list
 
# Update repository information and install the azure-cli package
sudo apt-get update
sudo apt-get install azure-cli
 
# Install kubectl and kubelogin: https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-install-cli
sudo az aks install-cli
 
# Login to Azure
az login
 
# Set subscription to the one with the AKS cluster and login to the AKS cluster
az account set --subscription aa123456-a123-a123-a123-abcd12345679
az aks get-credentials --admin --name aks-privatecluster --resource-group rg-privatecluster --overwrite-existing
 
# Use kubelogin plugin for authentication
kubelogin convert-kubeconfig -l azurecli
 
# Verify your aks connection
kubectl cluster-info
  • You can now use the 'Kubernetes' extension to view and manage the AKS cluster. Open the 'Kubernetes' extension from the activity bar → Click on the refresh button in the 'Clusters' section to load the clusters from your kubeconfig file. You should see your AKS cluster listed there. You can now expand the cluster to view its resources and perform various actions like viewing logs, executing commands in pods and more.
Note that some of the above steps can be automated using VS Code Tasks.

Lens

Lens is a popular Kubernetes IDE that makes it easy to manage and monitor Kubernetes clusters. You can download it from here. To connect to a private AKS cluster using Lens, follow these steps:

  • Lens automatically uses your local kube config to connect to clusters. To use Lens to connect to a privatecluster first connect to the privatecluster using the jumpbox as explained above. Then, copy the kubeconfig file from the jumpbox to your local machine. From your home directory on the jumpbox, use cat .kube/config to display the content of the kubeconfig file. Copy the content and save it to a file on your local machine, for example C:\Users\sjoer\.kube\privatecluster-config
  • Open the file in your VS Code and make the following changes:
    • Change server: https://aks-privatecluster-m8k6low2.privatelink.westeurope.azmk8s.io:443 to server: https://localhost:6443
    • Add the line insecure-skip-tls-verify: true under the cluster section:
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: LS1QLSJBXXX
          server: https://localhost:6443
        insecure-skip-tls-verify: true
        name: aks-privatecluster
      contexts:
      - context:
          cluster: aks-privatecluster
          namespace: appops
          user: clusterAdmin_rg-privatecluster_aks-privatecluster
        name: aks-privatecluster-admin
      current-context: aks-privatecluster-admin
      kind: Config
      preferences: {}
      users:
      - name: clusterAdmin_rg-privatecluster_aks-privatecluster
        user:
          client-certificate-data: LS1QLSJBNMXXXLS0tCg==
          client-key-data: LS1QLSJBNMXXX
          token: xxx
  • In Lens, go to File → Add Cluster → Select kubeconfig file and select the file you just created. You should now see the AKS cluster in Lens.
  • Before you can manage it, you first need to create a ssh tunnel to the jumpbox through the bastion:
    az network bastion ssh --name bas-001 --resource-group "rg-bastion" --target-resource-id /subscriptions/aa123456-a123-a123-a123-abcd12345678/resourceGroups/rg-cluster/providers/Microsoft.Compute/virtualMachines/vm-jumpbox --auth-type AAD -- -L 6443:aks-privatecluster-m8k6low2.privatelink.westeurope.azmk8s.io:443
Note: The command above is different from the previous command as it used a ssh tunnel. Note that it also uses AAD authentication which is optional. A privatekey as setup before in the ssh config will work just as well.

This wiki has been made possible by:

2025/09/21 10:35

How to Clone a PVC in Kubernetes

Summary: This wiki page shows how to clone a Persistent Volume Claim (PVC) in Kubernetes.
Date: 14 September 2025

When doing an Argo CD sync I got an error on one of our PVCs. For one of our applications we upgraded the storage class but hadn't had the time yet to convert the actual PVC:

one or more objects failed to apply, reason: PersistentVolumeClaim "seq" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims   core.PersistentVolumeClaimSpec{    ... // 2 identical fields    Resources: {Requests: {s"storage": {i: {...}, Format: "BinarySI"}}},    VolumeName: "pvc-2bc75dce-01e6-4f8b-ae06-3fc6c6657dac", -  StorageClassName: &"default", +  StorageClassName: &"managed-premium",    VolumeMode: &"Filesystem",    DataSource: nil,    ... // 2 identical fields   }

As you can see, the storage class 'default' is changed to 'managed-premium', but unfortunately, this cannot be done online in Kubernetes, as that setting is immutable. Follow the procedure below to use 'korb' to quickly clone the PVC to a new one with the correct storage class.

Current Situation

This is the current pvc manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seq
  namespace: appops
  uid: 33dd11a4-e97e-4ce2-85e5-94efbad9e087
  resourceVersion: '737538059'
  creationTimestamp: '2024-08-07T14:57:02Z'
  labels:
    app: seq
    chart: seq-2024.3.1
    heritage: Helm
    k8slens-edit-resource-version: v1
    release: seq
  annotations:
    argocd.argoproj.io/tracking-id: appops:/PersistentVolumeClaim:appops/seq
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/selected-node: aks-system-12344567-vmss000000
    volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/appops/persistentvolumeclaims/seq
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Ti
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  volumeName: pvc-2bc75dce-01e6-4f8b-ae06-3fc6c6657dac
  storageClassName: default
  volumeMode: Filesystem

Note that the disk is 1TB in size. From the monitoring we see that it is 82% full.

Approach

There are several ways to clone a PVC, but I decided to create a quick clone using korb. Korb is a command-line tool to clone Kubernetes Persistent Volume Claims (PVCs) and migrate data between different storage classes. It supports various strategies for copying data, including snapshot-based and copy-twice methods.

Follow the steps below:

  • Check the latest release from: the korb release page
  • Use the following commands to install the korb binary on your linux system, and make sure to replace the version number if a newer one is available:
    # Download the latest release and make it executable
    curl -LO https://github.com/BeryJu/korb/releases/download/v2.3.4/korb_2.3.4_linux_amd64.tar.gz
    tar -xvzf korb_2.3.4_linux_amd64.tar.gz
    sudo mv korb /usr/local/bin/korb
    # Scale the application with the pvc to 0 replicas
    kubectl scale deployment seq --replicas=0 -n appops
    # Start the clone using korb. Note that the container image parameter is only necessary when working on a private cluster
    korb --new-pvc-storage-class=managed-premium --strategy=copy-twice-name --new-pvc-namespace=appops --source-namespace=appops --container-image=acreuwprd.azurecr.io/docker/beryju/korb-mover:v2 seq
    # Once the clone is ready, scale the application back to 1 replica
    kubectl scale deployment seq --replicas=1 -n appops

Note the following:

  • The korb command above was done on a private (Azure Kubernetes) cluster, so I had to add the image to the container registry before the command worked. You can do that with the following commands (replace the acr name with your own):
    az login
    docker pull ghcr.io/beryju/korb-mover:v2
    docker tag ghcr.io/beryju/korb-mover:v2 acreuwprd.azurecr.io/docker/beryju/korb-mover:v2
    docker push acreuwprd.azurecr.io/docker/beryju/korb-mover:v2
  • The strategy 'copy-twice-name' means that the pvc will first be cloned to a temporary pvc with a temporary name, and then right away to the final pvc with the original name. This works best in an environment with Argo CD which tracks the pvc by name.
  • The clone process from the standard storage class to the managed-premium storage class took 3 hours and 20 minutes for a 1 TB pvc that was 82% full. The second clone (to the final pvc with the original name) was some faster, but still took a long time. Keep that in mind when planning a maintenance window.

Result

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seq
  namespace: appops
  uid: 1946a79c-37aa-4f01-8ce0-ed090d2b9b67
  resourceVersion: '743857155'
  creationTimestamp: '2025-09-10T22:24:19Z'
  labels:
    app: seq
    chart: seq-2024.3.1
    heritage: Helm
    k8slens-edit-resource-version: v1
    release: seq
  annotations:
    argocd.argoproj.io/tracking-id: appops:/PersistentVolumeClaim:appops/seq
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/selected-node: aks-system-12344567-vmss000000
    volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/appops/persistentvolumeclaims/seq
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Ti
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  volumeName: pvc-1946a79c-37aa-4f01-8ce0-ed090d2b9b67
  storageClassName: managed-premium
  volumeMode: Filesystem

This wiki has been made possible by:

2025/09/14 14:29
start.1748779162.txt.gz · Last modified: by 127.0.0.1