SDB:K3s cluster deployment on MicroOS

Jump to: navigation, search
openSUSE Support Database
Portal - SDB Categories - How to write an article


Tested on openSUSE

Recommended articles

Related articles


What is K3s?

K3s is a CNCF sandbox project that delivers a lightweight certified Kubernetes distribution created by Rancher Labs. K3s is highly available and production-ready. It has a very small binary size and very low resource requirements. In simple terms: it is Kubernetes without the bloat.

Why K3s?

As openSUSE Kubic comes to an end it is important that we offer alternatives to users who are used to the immutable base operating system under Kubic - openSUSE MicroOS, with transacitonal-updates, along with the easy deployment of a Kubernetes cluster. K3s is fully supporting the transactional nature of MicroOS, supports SELinux, and requires less resources than a stock Kubernetes deployment making it a great candidate.

Example K3s cluster deployment

With Combustion

In the following examples you will learn how to use Combustion - since it is the easiest and fastest way to get it done; to deploy a single-node and a highly available multi-node.

Preparations

A USB flashdrive will be required, or if you’re using virtual machine (VM) the supported method to attach a drive - physical or virtual - under a VM, in the hypervisor of your choice where MicroOS can read the Combustion script from.

Example GUI tools are Gnome Disks, Gparted. Example CLI tools are parted, fdisk. In general, use the tool that works for you.

  1. Create a new GPT partition table on the flashdrive,

  2. Create a new primary partition,

  3. Format the newly created partition to ext4,

    mkfs.ext4 /dev/sdY

  4. Label the partition as combustion,

    e2label /dev/sdY combustion

  5. Mount it,

  6. Create a folder and name it, combustion.

Cluster deployment

Prepare the combustion script

The following script can be used to create a single master node to start the cluster with, install the applications you require - such as cockpit, configure the root password (it is not recommended to add one if you’re deploying in to production), create a new user with password, and add authorized_keys for easy login with ssh. You can configure these by setting the variables at the beginning of the script. Feel free to remove parts you don’t need or add things you may need: 

#!/bin/bash
 
set -e
 
## Note:
## To check if the k3s installation has been finished
## issue the "systemctl status k3sinstall.service" command.
## To finish the installation you must reboot!
## Once booted you can check the node with:
## "kubectl get nodes"
## For more check out:
## "https://documentation.suse.com/trd/kubernetes/pdf/kubernetes_ri_k3s-slemicro_color_en.pdf"
 
## Enable network
# combustion: network
## Post output on stdout
exec > >(exec tee -a /dev/tty0) 2>&1
## Add password for root user
## SUSE documentation recommends openssl passwd -6, mkpasswd --method=sha-512 works as well
## The default password that is set here is: linux
ROOT_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
SSH_ROOT_PUBLIC_KEY=ssh_key.pub
SSH_USER_PUBLIC_KEY=ssh_key.pub
USER_REQUIRED_PACKAGES='patterns-microos-cockpit cockpit bash-completion'
CREATE_NORMAL_USER=user  ## Add the username here to create a user, leave empty to skip creating one
NORMAL_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
NODE_HOSTNAME="tanis00"  ## If you want to add additional nodes to a cluster you must set the hostname or nodes will not be able to join
## K3s configuration
INSTALL_K3S_UPSTREAM=true  ## Set to false if you want to use the openSUSE rpm, also add the package name to USER_REQUIRED_PACKAGES
INSTALL_K3S_EXEC='server --cluster-init --write-kubeconfig-mode=644'
 
## Set hostname
echo $NODE_HOSTNAME > /etc/hostname
 
## Mount /var and /home so user can be created smoothly
if [ "$CREATE_NORMAL_USER" ]
then
    mount /var && mount /home
fi
 
## Set root password
echo root:$ROOT_USER_PASSWORD | chpasswd -e
## Add ssh public key as authorized key for the root user
mkdir -pm700 /root/.ssh/
cat $SSH_ROOT_PUBLIC_KEY >> /root/.ssh/authorized_keys
 
## User creation
if [ "$CREATE_NORMAL_USER" ]
then
    echo "User creation is requested, creating user."
    useradd -m $CREATE_NORMAL_USER -s /bin/bash -g users
    echo $CREATE_NORMAL_USER:$NORMAL_USER_PASSWORD | chpasswd -e
    echo $CREATE_NORMAL_USER "ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/adminusers
    mkdir -pm700 /home/$CREATE_NORMAL_USER/.ssh/
    chown -R $CREATE_NORMAL_USER:users /home/$CREATE_NORMAL_USER/.ssh/
    cat $SSH_USER_PUBLIC_KEY >> /home/$CREATE_NORMAL_USER/.ssh/authorized_keys
    echo "Requested user has been created, requested password has been set."
  else
    echo "No user will be created"
fi
 
## Install required packages
if [ "$USER_REQUIRED_PACKAGES" ]
then
    zypper ref && zypper --non-interactive install $USER_REQUIRED_PACKAGES
fi
 
if $INSTALL_K3S_UPSTREAM; then
    ## Download and install the latest k3s installer
    curl -L --output k3s_installer.sh https://get.k3s.io && install -m755 k3s_installer.sh /usr/bin/
    ## Create a systemd unit that installs k3s if not installed yet
    cat <<- EOF > /etc/systemd/system/install-rancher-k3s.service
    [Unit]
    Description=Run K3s installer
    Wants=network-online.target
    After=network.target network-online.target
    ConditionPathExists=/usr/bin/k3s_installer.sh
    ConditionPathExists=!/usr/local/bin/k3s
    [Service]
    Type=forking
    TimeoutStartSec=120
    Environment="INSTALL_K3S_EXEC=$INSTALL_K3S_EXEC"
    ExecStart=/usr/bin/k3s_installer.sh
    RemainAfterExit=yes
    KillMode=process
    [Install]
    WantedBy=multi-user.target
    EOF
fi
 
## Enable services
systemctl enable cockpit.socket
systemctl enable sshd
systemctl enable install-rancher-k3s.service
 
## Unmount var and home
if [ "$CREATE_NORMAL_USER" ]
then
    umount /var && umount /home
fi
 
echo "Configured with Combustion" > /etc/issue.d/combustion

The part that deploys K3s is actually a systemd unit /etc/systemd/system/install-rancher-k3s.service that starts right after the system is booted. The whole installation takes about a minute. Once it is finished it is advised to reboot your instance so the changes can take effect.

Once rebooted you can check the node(s) availability with kubectl get nodes.

Do note that this unit will deploy the latest stable version of K3s, if you require a specific version first check the availability of the version you require at: https://github.com/k3s-io/k3s/releases, then define INSTALL_K3S_VERSION as an environmental variable in the systemd unit that specifies the version. For example: Environment="INSTALL_K3S_VERSION=v1.22.9+k3s1"

Highly-available master node(s)

For a production workload it is recommended to deploy additional master nodes to achieve a highly-available cluster. It’s important to note that you need to change the hostname of your additional hosts, if there is a node with a matching name you will not be able to add that to the cluster!

You will need to extract the node-token from your first master node you initialize the cluster with. You can get this token from /var/lib/rancher/k3s/server/node-token.

You will also need to define the ip address or FQDN of the first master node to be able to connect to it. You can do this by setting the MASTER_NODE_ADDR variable.

You can use the following script (modify it accordingly to your needs, define variables at the beginning): 

#!/bin/bash
 
set -e
 
## Note:
## To check if the k3s installation has been finished
## issue the "systemctl status k3sinstall.service" command.
## To finish the installation you must reboot!
## Once booted you can check the node with:
## "kubectl get nodes"
## For more check out:
## "https://documentation.suse.com/trd/kubernetes/pdf/kubernetes_ri_k3s-slemicro_color_en.pdf"
 
## Enable network
# combustion: network
## Post output on stdout
exec > >(exec tee -a /dev/tty0) 2>&1
## Add password for root user
## SUSE documentation recommends openssl passwd -6, mkpasswd --method=sha-512 works as well
## The default password that is set here is: linux
ROOT_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
SSH_ROOT_PUBLIC_KEY=ssh_key.pub
SSH_USER_PUBLIC_KEY=ssh_key.pub
USER_REQUIRED_PACKAGES='patterns-microos-cockpit cockpit bash-completion'
CREATE_NORMAL_USER=user  ## Add the username here to create a user, leave empty to skip creating one
NORMAL_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
## K3s configuration
INSTALL_K3S_UPSTREAM=true  ## Set to false if you want to use the openSUSE rpm, also add the package name to USER_REQUIRED_PACKAGES
MASTER_NODE_ADDR='172.168.255.104'  ## The ip or FQDN of the first node
MASTER_NODE_K3S_TOKEN='K106bcc041130fd90b367680868839a458aae9b6f1b2deee6e2308a3ff330bd4b51::server:fda85366efe11d4c31b4bfdccedc2994'  ## You can find this on the master/CP node: /var/lib/rancher/k3s/server/node-token
INSTALL_K3S_EXEC='server'
NODE_HOSTNAME="tanis01"  ## If you want to add additional nodes to a cluster you must set the hostname or nodes will not be able to join
 
## Set hostname
echo $NODE_HOSTNAME > /etc/hostname
 
## Mount /var and /home so user can be created smoothly
if [ "$CREATE_NORMAL_USER" ]
then
    mount /var && mount /home
fi
 
## Set root password
echo root:$ROOT_USER_PASSWORD | chpasswd -e
## Add ssh public key as authorized key for the root user
mkdir -pm700 /root/.ssh/
cat $SSH_ROOT_PUBLIC_KEY >> /root/.ssh/authorized_keys
 
## User creation
if [ "$CREATE_NORMAL_USER" ]
then
    echo "User creation is requested, creating user."
    useradd -m $CREATE_NORMAL_USER -s /bin/bash -g users
    echo $CREATE_NORMAL_USER:$NORMAL_USER_PASSWORD | chpasswd -e
    echo $CREATE_NORMAL_USER "ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/adminusers
    mkdir -pm700 /home/$CREATE_NORMAL_USER/.ssh/
    chown -R $CREATE_NORMAL_USER:users /home/$CREATE_NORMAL_USER/.ssh/
    cat $SSH_USER_PUBLIC_KEY >> /home/$CREATE_NORMAL_USER/.ssh/authorized_keys
    echo "Requested user has been created, requested password has been set."
  else
    echo "No user will be created"
fi
 
## Install required packages
if [ "$USER_REQUIRED_PACKAGES" ]
then
    zypper ref && zypper --non-interactive install $USER_REQUIRED_PACKAGES
fi
 
if $INSTALL_K3S_UPSTREAM; then
    ## Download and install the latest k3s installer
    curl -L --output k3s_installer.sh https://get.k3s.io && install -m755 k3s_installer.sh /usr/bin/
    ## Create a systemd unit that installs k3s if not installed yet
    cat <<- EOF > /etc/systemd/system/install-rancher-k3s-master.service
    [Unit]
    Description=Run K3s installer
    Wants=network-online.target
    After=network.target network-online.target
    ConditionPathExists=/usr/bin/k3s_installer.sh
    ConditionPathExists=!/usr/local/bin/k3s
    [Service]
    Type=forking
    TimeoutStartSec=120
    Environment="K3S_URL=https://$MASTER_NODE_ADDR:6443"
    Environment="K3S_TOKEN=$MASTER_NODE_K3S_TOKEN"
    Environment="K3S_KUBECONFIG_MODE=644"
    Environment="INSTALL_K3S_EXEC=$INSTALL_K3S_EXEC"
    ExecStart=/usr/bin/k3s_installer.sh
    RemainAfterExit=yes
    KillMode=process
    [Install]
    WantedBy=multi-user.target
    EOF
fi
 
## Enable services
systemctl enable cockpit.socket
systemctl enable sshd
systemctl enable install-rancher-k3s-master.service
 
## Unmount var and home
if [ "$CREATE_NORMAL_USER" ]
then
    umount /var && umount /home
fi
 
echo "Configured with Combustion" > /etc/issue.d/combustion

The installation process is the same as with the first master node, once the installation finished reboot your instance and verify that the node has been added to your cluster with kubectl get nodes.

Deploy worker/agent node(s)

To add a worker or - as Rancher calls it - agent nodes to your K3s cluster you can use the following Combustion script.

Do note that you will need the node-token from your first master node /var/lib/rancher/k3s/server/node-token. Also, don’t forget to change the hostname. Like in case of adding additional master nodes, you will also need to define the ip address or FQDN of the first master node to be able to connect to it. You can do this by setting the MASTER_NODE_ADDR variable. 

#!/bin/bash
 
set -e
 
## Note:
## To check if the k3s installation has been finished
## issue the "systemctl status k3sinstall.service" command.
## To finish the installation you must reboot!
## Once booted you can check the node with:
## "kubectl get nodes"
## For more check out:
## "https://documentation.suse.com/trd/kubernetes/pdf/kubernetes_ri_k3s-slemicro_color_en.pdf"
 
## Enable network
# combustion: network
## Post output on stdout
exec > >(exec tee -a /dev/tty0) 2>&1
## Add password for root user
## SUSE documentation recommends openssl passwd -6, mkpasswd --method=sha-512 works as well
## The default password that is set here is: linux
ROOT_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
SSH_ROOT_PUBLIC_KEY=ssh_key.pub
SSH_USER_PUBLIC_KEY=ssh_key.pub
USER_REQUIRED_PACKAGES='patterns-microos-cockpit cockpit bash-completion'
CREATE_NORMAL_USER=user  ## Add the username here to create a user, leave empty to skip creating one
NORMAL_USER_PASSWORD='$6$NWOL.CUVsuDtWZhb$rs8JeaNfx/ZL.OvQPYugAbXAUA67va.geqLa96zNo8no/PqxvFDICB.Gb2BFlIZoTx095qHSkRevKIyy1/AKj.'
## K3s configuration
## You can set the Kubernetes version to be installed by setting INSTALL_K3S_VERSION in the [service] below
## the following way: Environment="INSTALL_K3S_VERSION=$[Version of K3s to download from Github]"
## Will attempt to download from the stable channel if not specified.
INSTALL_K3S_UPSTREAM=true  ## Set to false if you want to use the openSUSE rpm, also add the package name to USER_REQUIRED_PACKAGES
MASTER_NODE_ADDR='172.168.255.104'  ## The ip or FQDN of the first node
MASTER_NODE_K3S_TOKEN='K106bcc041130fd90b367680868839a458aae9b6f1b2deee6e2308a3ff330bd4b51::server:fda85366efe11d4c31b4bfdccedc2994'  ## You can find this on the master/CP node: /var/lib/rancher/k3s/server/node-token
NODE_HOSTNAME="tanis02"  ## If you want to add additional nodes to a cluster you must set the hostname or nodes will not be able to join
 
## Set hostname
echo $NODE_HOSTNAME > /etc/hostname
 
## Mount /var and /home so user can be created smoothly
if [ "$CREATE_NORMAL_USER" ]
then
    mount /var && mount /home
fi
 
## Set root password
echo root:$ROOT_USER_PASSWORD | chpasswd -e
## Add ssh public key as authorized key for the root user
mkdir -pm700 /root/.ssh/
cat $SSH_ROOT_PUBLIC_KEY >> /root/.ssh/authorized_keys
 
## User creation
if [ "$CREATE_NORMAL_USER" ]
then
    echo "User creation is requested, creating user."
    useradd -m $CREATE_NORMAL_USER -s /bin/bash -g users
    echo $CREATE_NORMAL_USER:$NORMAL_USER_PASSWORD | chpasswd -e
    echo $CREATE_NORMAL_USER "ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/adminusers
    mkdir -pm700 /home/$CREATE_NORMAL_USER/.ssh/
    chown -R $CREATE_NORMAL_USER:users /home/$CREATE_NORMAL_USER/.ssh/
    cat $SSH_USER_PUBLIC_KEY >> /home/$CREATE_NORMAL_USER/.ssh/authorized_keys
    echo "Requested user has been created, requested password has been set."
  else
    echo "No user will be created"
fi
 
## Install required packages
if [ "$USER_REQUIRED_PACKAGES" ]
then
    zypper ref && zypper --non-interactive install $USER_REQUIRED_PACKAGES
fi
 
if $INSTALL_K3S_UPSTREAM; then
    ## Download and install the latest k3s installer
    curl -L --output k3s_installer.sh https://get.k3s.io && install -m755 k3s_installer.sh /usr/bin/
    ## Create a systemd unit that installs k3s if not installed yet
    cat <<-EOF > /etc/systemd/system/install-rancher-k3s-worker.service
    [Unit]
    Description=Run K3s installer
    Wants=network-online.target
    After=network.target network-online.target
    ConditionPathExists=/usr/bin/k3s_installer.sh
    ConditionPathExists=!/usr/local/bin/k3s
    [Service]
    Type=forking
    TimeoutStartSec=120
    Environment="K3S_URL=https://$MASTER_NODE_ADDR:6443"
    Environment="K3S_TOKEN=$MASTER_NODE_K3S_TOKEN"
    Environment="K3S_KUBECONFIG_MODE=644"
    ExecStart=/usr/bin/k3s_installer.sh
    RemainAfterExit=yes
    KillMode=process
    [Install]
    WantedBy=multi-user.target
    EOF
fi
 
## Enable services
systemctl enable cockpit.socket
systemctl enable sshd
systemctl enable install-rancher-k3s-worker.service
 
## Unmount var and home
if [ "$CREATE_NORMAL_USER" ]
then
    umount /var && umount /home
fi
 
echo "Configured with Combustion" > /etc/issue.d/combustion

The installation process is the same as with the first master node, once the installation finished reboot your instance and verify that the node has been added to your cluster with kubectl get nodes.

Upgrading K3s

To update the k3s cluster you can utilize the same systemd unit by renaming the k3s binary and restarting the unit, for example to update the worker nodes: 

# mv /usr/local/bin/k3s /usr/local/bin/k3s_OLD

# systemctl restart install-rancher-k3s-worker.service

# systemctl reboot

Another option is to edit the the systemd unit, and remove the condition that checks for an existing k3s binary in /usr/local/bin (ConditionPathExists=!/usr/local/bin/k3s). For both scenarios you will need to reload systemd for the changes to take effect systemctl daemon-reload

Not the most elegant way perhaps, but it allows for easy automation with Ansible, including for nodes to be updated one-by-one.

If the above options seem unsuitable for your deployment, check out the documented upstream options for manual upgrades at: https://rancher.com/docs/k3s/latest/en/upgrades/basic/ or automated upgrades at: https://www.suse.com/c/rancher_blog/upgrade-a-k3s-kubernetes-cluster-with-system-upgrade-controller/.

Deploying K3s manually

Master node

# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --cluster-init --write-kubeconfig-mode=644" sh -

HA Master

Just like in the above examples you will need the node-token from the first master node (/var/lib/rancher/k3s/server/node-token) and need to define the master node ip address or FQDN. 

# curl -sfL https://get.k3s.io | K3S_URL="https://$MASTER_NODE_ADDR:6443" K3S_TOKEN="$MASTER_NODE_K3S_TOKEN" K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="server" sh -

Worker/agent node

Just like in the above examples you will need the node-token from the first master node (/var/lib/rancher/k3s/server/node-token), and need to define the master node ip address or FQDN. Since the INSTALL_K3S_EXEC variable is not defined it will be deployed as a worker or agent node. 

# curl -sfL https://get.k3s.io | K3S_URL="https://$MASTER_NODE_ADDR:6443" K3S_TOKEN="$MASTER_NODE_K3S_TOKEN" K3S_KUBECONFIG_MODE="644" sh -
Retrieved from "https://en.opensuse.org/index.php?title=SDB:K3s_cluster_deployment_on_MicroOS&oldid=165679"