Summary: Usually you'd set your kubernetes cluster to automatically decide on the number of nodes in the cluster, but what if you know you'll be deploying a new application which will require a large amount of resources? Sometimes you just want to be ahead of things and change things manually. On this page I'll show you how to do that on an Azure Kubernetes cluster.
Date: 20 December 2024

Use the following command to check the current load of the cluster:

azadmin@vm-euw-dev-jumpbox:~$ kubectl top node
NAME                             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-system-67293637-vmss000000   529m         13%    7108Mi          56%
aks-system-67293637-vmss000001   186m         4%     4534Mi          36%
aks-system-67293637-vmss000002   214m         5%     4057Mi          32%
aks-user-16792673-vmss000001     466m         12%    10724Mi         85%
aks-user-16792673-vmss000002     312m         8%     11114Mi         88%
aks-user-16792673-vmss00001q     270m         6%     9307Mi          73%
aks-user-16792673-vmss00001r     386m         10%    9648Mi          76%
aks-user-16792673-vmss00001t     490m         12%    9138Mi          72%
aks-user-16792673-vmss00001u     374m         9%     10972Mi         87%
aks-user-16792673-vmss00001v     473m         12%    11594Mi         92%
aks-user-16792673-vmss00001x     304m         7%     9701Mi          77%

As you can see, there is still plenty of CPU resources available on the user pool, but memory is getting tight. We'll need to expand the cluster to make sure we have enough resources available for the new deployment.

We will use the Azure CLI to manually expand the cluster. First we'll disable the autoscaler, then we'll manually scale the cluster out, and after the deployment we'll enable the autoscaler again.

# First, we set some variables so we don't have to type them out every time
subId="bc624dd8-a267-8dda-3665-33dc762456d2"
rg="rg-euw-dev-akscluster"
aksClusterName="aks-euw-dev-akscluster"
userNodePool="user"
 
# Then, we set the correct subscription. This assumes you're already logged in to azure, if not, do so with 'az login'
az account set --subscription $subId
az account show
 
# Get the current number of nodes and the cluster autoscaler settings
az aks show --resource-group $rg \
  --name $aksClusterName \
  --query agentPoolProfiles
### From the output check for the agentpool with the name user and check the following settings:
#### "count": 8,
#### "enableAutoScaling": true,
#### "maxCount": 8,
#### "minCount": 3,
#### "name": "user",
### This shows that autoscaler is on and that the cluster is at it's max capacity of 8 nodes.
 
# Disable the autoscaler on the cluster so we can manually scale the cluster out for a deployment
az aks nodepool update \
  --resource-group $rg \
  --cluster-name $aksClusterName \
  --name $userNodePool \
  --disable-cluster-autoscaler
 
# Change the number of nodes for the nodepool manually
az aks scale \
  --resource-group $rg \
  --name $aksClusterName \
  --node-count 10 \
  --nodepool-name $userNodePool

We have now changed the number of nodes in the userpool to 10 instead of 8. It could take some time for the nodes to be fully available in the cluster.

Once the deployment has been done, it's best to revert the cluster to use the autoscaler again. This way you don't have to worry about scaling the cluster manually:

# After the deployment has been done, set the autoscaler back on
az aks nodepool update \
  --resource-group $rg \
  --cluster-name $aksClusterName \
  --name $userNodePool \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10

It could also be that you do not want to scale the cluster manually but would like to allow the autoscaler to scale the cluster out. Use the command below to change the max count of the autoscaler so it can scale out to more nodes:

# To change the setting for a running nodepool with autoscaler already enabled
az aks nodepool update \
  --resource-group $rg \
  --cluster-name $aksClusterName \
  --name $userNodePool \
  --update-cluster-autoscaler \
  --min-count 3 \
  --max-count 10