Ubuntu OS Upgrade on Kubernetes Environment

Note

Kubernetes is not upgraded as part of the OS upgrade process. If necessary, after completing the OS upgrade, user's can follow the standard Kubernetes upgrade documentation to upgrade Kubernetes.

1. Prerequisites

Python version: 3.12.X

Rdaf CLI version: 1.4.1

RDAF Infra Tags: 1.0.4

RDAF Platform/Worker/App tags: 8.1.0.1

Important

Internet should be available on all nodes.

2. Preparation Steps

• Always take VM-level snapshots in vCenter for all the VMs.

• Always choose default options when prompted during upgrade.

• Each VM upgrade is approximately 1.5 hours.

3. Upgrade Order

• APP/Platform VMs

• Worker VMs

• Infra VMs

• Registry VM

• Gateway VMs

4. Upgrade Process

4.1 Steps to Execute on Ubuntu OS Upgraded VM

Step 1. create the directory for storing the Kubernetes GPG key by running. This command creates the /etc/apt/keyrings directory if it does not already exist.

sudo mkdir -p /etc/apt/keyrings

Step 2. Download the Kubernetes Release key and save it as a GPG keyring file. This command fetches the Kubernetes release key, converts it to a GPG keyring format, and stores it in the specified directory.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

Step 3. Add the Kubernetes package repository to your system's sources lis. This command writes the repository information into a new sources list file named kubernetes.list.

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Example Output

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /

Step 4. Put the kubelet, kubeadm, and kubectl packages on hold to prevent them from being automatically upgraded. This command marks these packages to be held at their current versions during system updates.

sudo apt-mark hold kubelet kubeadm kubectl

Example Output

sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.

Step 5. This command displays the current OS version details, such as the distributor ID, description, release number, and codename

lsb_release -a

Example Output

rdauser@kubsv10820:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Important

We should first cordon the node, and once all pods are in a pending state, we can proceed with the OS upgrade on the respective VM.

4.2 Steps to Execute on CLI or Master Node

Step 1. Cordon the node to prevent new pods from being scheduled on it.

kubectl cordon <NodeName>

Step 2. Drain the node by evicting all the pods, ignoring daemonsets, and deleting emptyDir data.

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force

Step 3. Ensure that the pods on the node move to the 'Pending' state. If they do not, you may need to force delete remaining pods.

kubectl -n rda-fabric delete pod --force <pod-name>

4.3 Steps to Execute on Ubuntu OS Upgrade VM

• Before upgrading, temporarily disable the Kubernetes APT repository, This command comments out the Kubernetes repository entry, preventing it from interfering during OS upgrade procedures.

if [ -f /etc/apt/sources.list.d/kubernetes.list ]; then
    sudo sed -i 's/^[[:space:]]*deb /# deb /' /etc/apt/sources.list.d/kubernetes.list
fi

Example Output

rdauser@kubsv10820:~$ if [ -f /etc/apt/sources.list.d/kubernetes.list ]; then
>   sudo sed -i 's/^[[:space:]]*deb /# deb /' /etc/apt/sources.list.d/kubernetes.list
> fi
rdauser@kubsv10820:~$ 

4.4 Steps to Upgrade Ubuntu from 20.04 to 22.04

Step 1. Back up important configuration files and check your current OS version.

cp /etc/sysctl.conf /home/rdauser
sudo cp /etc/sudoers /home/rdauser
cp /etc/docker/daemon.json /home/rdauser
lsb_release -a

Example Output

rdauser@kubsv10820:~$ cp /etc/sysctl.conf /home/rdauser
rdauser@kubsv10820:~$ sudo cp /etc/sudoers /home/rdauser
rdauser@kubsv10820:~$ cp /etc/docker/daemon.json /home/rdauser
rdauser@kubsv10820:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Step 2. Perform the system update and upgrade.

sudo apt update -y && sudo apt full-upgrade -y

Step 3. Update the Docker daemon configuration.

sudo vi /etc/docker/daemon.json

Step 4. Replace the content with the following configuration.

{
  "tls": true,
  "tlscacert": "/etc/tlscerts/ca/ca.pem",
  "fixed-cidr-v6": "fdc6:ef49:83db::/64",
  "ip6tables": true,
  "exec-opts": [
    "native.cgroupdriver=cgroupfs"
  ],
  "tlsverify": true,
  "default-cgroupns-mode": "host",
  "storage-driver": "overlay2",
  "hosts": [
    "unix:///var/run/docker.sock",
    "tcp://0.0.0.0:2376"
  ],
  "tlskey": "/etc/tlscerts/server/server.key",
  "ipv6": true,
  "debug": false,
  "tlscert": "/etc/tlscerts/server/server.pem",
  "userland-proxy": false,
  "experimental": true,
  "live-restore": true
}

Step 5. Save the file, then reboot the VM.

sudo init 6

Step 6. After reboot, update the Docker service configuration. Comment out the existing ExecStart line and add the new line as highlighted below.

sudo vi /usr/lib/systemd/system/docker.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Step 7. Save the file, then reload systemd and restart Docker

sudo systemctl daemon-reload && sudo systemctl restart docker

Example Output

sudo systemctl daemon-reload && sudo systemctl restart docker

Note

These steps apply exclusively to the REGISTRY VM.

• Before upgrading the OS on the Registry VM, ensure the Python symlink is correctly set to avoid Python-related issues during the upgrade. These commands remove any existing /usr/bin/python3 link and create a new symbolic link pointing to Python 3.8.

sudo rm -f /usr/bin/python3
sudo ln -s /usr/bin/python3.8 /usr/bin/python3

For remaining VMs (excluding the Registry VM), continue with the following steps

Step 8. Check which packages are on hold.

apt-mark showhold

Example Output

rdauser@kubsv10820:~$ apt-mark showhold
kubeadm
kubectl
kubelet

Step 9. Update and fully upgrade the system, This will fetch the latest updates and upgrade all packages to their latest versions

sudo apt update -y && sudo apt full-upgrade -y

Example Output

rdauser@kubsv10820:~$ sudo apt update -y && sudo apt full-upgrade -y
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libevent-core-2.1-7 libevent-pthreads-2.1-7 libfwupdplugin1 libopts25 libxmlb1 linux-headers-5.4.0-109 linux-headers-5.4.0-109-generic linux-headers-5.4.0-110
  linux-headers-5.4.0-110-generic linux-image-5.4.0-109-generic linux-image-5.4.0-110-generic linux-modules-5.4.0-109-generic linux-modules-5.4.0-110-generic
  linux-modules-extra-5.4.0-109-generic linux-modules-extra-5.4.0-110-generic sntp
Use 'sudo apt autoremove' to remove them.
The following security updates require Ubuntu Pro with 'esm-infra' enabled:
  cloud-init linux-headers-generic jq linux-libc-dev openssl libblockdev-swap2
  sntp libssh-4 libpython3.8-minimal git-man libsystemd0 gcc-10-base
  linux-image-generic python2.7-minimal libsqlite3-0 python3-urllib3
  libsnmp-base libitm1 libjq1 sudo python2.7 libpython3.8 python3.8
  open-vm-tools git libblockdev-crypto2 udev libblockdev-loop2 libquadmath0
  libblockdev-fs2 libblockdev-part2 python3-requests libudev1 libsoup2.4-1
  udisks2 libsnmp35 libtsan0 libubsan1 mysql-client-core-8.0 python3.8-minimal
  systemd-sysv libblockdev2 libpam-systemd liblsan0 systemd libgomp1
  libssl-dev libblockdev-utils2 libmysqlclient21 libnss-systemd
  libblockdev-part-err2 libgcc-s1 libxml2 libpython2.7-minimal
  libpython3.8-stdlib libgnutls30 libudisks2-0 libatomic1 libssl1.1 libcc1-0
  libpython2.7-stdlib libstdc++6 linux-generic libxslt1.1
Learn more about Ubuntu Pro at https://ubuntu.com/pro
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
rdauser@kubsv10820:~$

Step 10. To start the OS upgrade, run the following command.

sudo do-release-upgrade -f DistUpgradeViewNonInteractive

Step 11. Once the upgrade completes, reboot the system.

sudo init 6

Step 12. After reboot, update the Docker service configuration. Comment out the existing ExecStart line and add the new line as highlighted below.

sudo vi /usr/lib/systemd/system/docker.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Step 13. Save the file, then reload systemd and restart Docker.

sudo systemctl daemon-reload && sudo systemctl restart docker

Step 14. Verify the OS version to ensure the upgrade was successful

lsb_release -a

Example Output

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

Step 15. Update packages

sudo apt update -y && sudo apt full-upgrade -y

Example Output

rdauser@kubsv10820:~$ sudo apt update -y && sudo apt full-upgrade -y
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libffi7
Use 'sudo apt autoremove' to remove it.
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  python2.7-minimal python2.7 libpython2.7-minimal libpython2.7-stdlib
Learn more about Ubuntu Pro at https://ubuntu.com/pro
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

4.5 Steps to Upgrade Ubuntu from 22.04 to 24.04

Step 1. To start the OS upgrade, run the following command.

sudo do-release-upgrade -f DistUpgradeViewNonInteractive

Step 2. Once the upgrade completes, reboot the system.

sudo init 6

Step 3. After reboot, update the Docker service configuration. Comment out the existing ExecStart line and add the new line as highlighted below.

sudo vi /usr/lib/systemd/system/docker.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Step 4. Save the file, then reload systemd and restart Docker.

sudo systemctl daemon-reload && sudo systemctl restart docker

Step 5. Verify the OS version to ensure the upgrade was successful

lsb_release -a

Example Output

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.3 LTS
Release:        24.04
Codename:       noble

5. Post-Upgrade Steps on Registry VM Only

Step 1. Remove the deadsnakes PPA repository.

sudo add-apt-repository --remove ppa:deadsnakes/ppa

Step 2. clean up the package cache and update the package list.

sudo apt clean && sudo apt update

Step 3. Remove the current Python 3 installation

sudo apt remove python3

Step 4. Install the specific Python 3.12 standard library version.

sudo apt install libpython3.12-stdlib=3.12.3-1ubuntu0.5

Step 5. Install necessary tools like vim and ufw.

sudo apt install vim ufw

Step 6. Configure pip to break system packages to avoid conflicts.

python3 -m pip config set global.break-system-packages true

6. Cleanup and Confirm No Pending Upgrades

Step 1. Perform cleanup to ensure no pending configurations or upgrades remain using the below given commands.

sudo dpkg --configure -a
sudo apt-get -y -f install
sudo apt-get

Example Output

rdauser@kubsv10820:~$ sudo dpkg --configure -a
rdauser@kubsv10820:~$ sudo apt-get -y -f install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libnsl2 mailcap
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
rdauser@kubsv10820:~$ sudo apt-get
apt 2.8.3 (amd64)
Usage: apt-get [options] command
       apt-get [options] install|remove pkg1 [pkg2 ...]
       apt-get [options] source pkg1 [pkg2 ...]

apt-get is a command line interface for retrieval of packages
and information about them from authenticated sources and
for installation, upgrade and removal of packages together
with their dependencies.

Most used commands:
  update - Retrieve new lists of packages
  upgrade - Perform an upgrade
  install - Install new packages (pkg is libc6 not libc6.deb)
  reinstall - Reinstall packages (pkg is libc6 not libc6.deb)
  remove - Remove packages
  purge - Remove packages and config files
  autoremove - Remove automatically all unused packages
  dist-upgrade - Distribution upgrade, see apt-get(8)
  dselect-upgrade - Follow dselect selections
  build-dep - Configure build-dependencies for source packages
  satisfy - Satisfy dependency strings
  clean - Erase downloaded archive files
  autoclean - Erase old downloaded archive files
  check - Verify that there are no broken dependencies
  source - Download source archives
  download - Download the binary package into the current directory
  changelog - Download and display the changelog for the given package

See apt-get(8) for more information about the available commands.
Configuration options and syntax is detailed in apt.conf(5).
Information about how to configure sources can be found in sources.list(5).
Package and version choices can be expressed via apt_preferences(5).
Security details are available in apt-secure(8).
                                        This APT has Super Cow Powers.

Step 2. Reboot the VM. This completes the post-upgrade setup on the Registry VM.

sudo init 6

Example Output

rdauser@kubsv10820:~$ sudo init 6

Broadcast message from root@kubsv10820 on pts/1 (Tue 2025-10-14 07:03:01 UTC):

The system will reboot now!

rdauser@kubsv10820:~$

7. Validate Ubuntu Version and Enable Kubernetes Repository

Step 1. Verify the current Ubuntu OS version.

lsb_release -a

Note

The expected output should show version 24.04

Example Output

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04 LTS
Release:        24.04
Codename:       noble

Step 2. Uncomment the Kubernetes repo in the sources list.

sudo sed -i 's/^# deb /deb /' /etc/apt/sources.list.d/kubernetes.list 2>/dev/null || true

Step 3. Update your package list to include the new repository.

sudo apt-get update

Example Output

rdauser@kubsv10820:~$ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu noble-backports InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu noble/main amd64 Components [464 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.6 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu noble/universe amd64 Components [3,871 kB]
Get:8 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [212 B]
Get:9 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [52.2 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu noble/multiverse amd64 Components [35.0 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:13 http://us.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [378 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:15 http://us.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,140 B]
Get:16 http://us.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:17 http://us.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [11.0 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:19 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [212 B]
Fetched 5,017 kB in 2s (2,365 kB/s)
Reading package lists... Done

Step 4. Remove any residual update manager and distribution upgrade logs

sudo rm -rf /var/lib/update-manager/*
sudo rm -rf /var/log/dist-upgrade/*

Step 5. Remove cached package files to free up space.

sudo apt clean

Step 6. Re-execute the systemd manager configuration to recognize any changes.

sudo systemctl daemon-reexec

Step 7. Reload systemd units to ensure all service configurations are reloaded.

sudo systemctl daemon-reload

Step 8. Restart the Kubernetes kubelet service to apply any changes.

sudo systemctl restart kubelet

Note

These commands are part of the maintenance procedure. Use them carefully, especially in production environments.

Step 9. Check the status of the kubelet service to ensure it is running correctly.

systemctl status kubelet

Example Output

rdauser@kubsv10820:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2025-10-14 07:11:03 UTC; 2s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 7522 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 7522 (code=exited, status=1/FAILURE)
        CPU: 179ms

8. Kubernetes Node Management and Troubleshooting

Step 1. On the CLI VM or Master VM, execute the following command to mark the node as schedulable.

kubectl uncordon <node-name>

Note

Replace <node-name> with the actual name of your node.

Step 2. Check the node status using the following command

kubectl get nodes

Step 3. Verify pods in specific namespaces and labels using the below commands.

kubectl get pods -n rda-fabric -l app_category=rdaf-infra -o wide
kubectl get pods -n rda-fabric -l app_category=rdaf-platform -o wide
kubectl get pods -n rda-fabric -l app_component=rda-worker -o wide
kubectl get pods -n rda-fabric -l app_name=oia -o wide

Note

Troubleshooting lsb_release -a Command Error

• If executing lsb_release -a results in an error similar to

 File "/usr/bin/lsb_release", line 25, in <module>
    import lsb_release
ModuleNotFoundError: No module named 'lsb_release'

• You can resolve this by creating a symbolic link to the lsb_release.py script

sudo mkdir -p /usr/local/lib/python3.12/site-packages
sudo ln -s /usr/share/pyshared/lsb_release.py /usr/local/lib/python3.12/site-packages/lsb_release.py