Two-node K3S (LVM operator)

Overview

Kolab is deployed in a two-node K3S cluster using application-level replication for disaster recovery. Unlike the Replicated K3s deployment, this setup does not use DRBD for disk-level replication. Instead, a single LVM volume group per node is managed by the OpenShift LVM Operator (LVMS / TopoLVM), which provisions logical volumes on demand for Kubernetes workloads.

Two physical (or virtual) nodes, each running both a K3S server (control plane) and a K3S agent (workload).
K3S control-plane state is stored in an external MariaDB Galera cluster running on the infrastructure nodes, providing HA without DRBD.
External connections go through HAProxy with IP failover via keepalived (virtual IP), making failover transparent to clients.
Galera Arbitrator (garbd) runs on both infrastructure nodes; whichever survives provides the third vote to prevent split-brain.

This deployment avoids the operational complexity of DRBD (no kernel modules, no split-brain fencing, no manual mount/unmount procedures during failover) while still providing disaster recovery across two nodes.

Storage

Storage for Kubernetes workloads is managed internally by the LVM operator on the worker nodes. A volume group is created per node, and the LVM operator provisions logical volumes on demand for Kubernetes PersistentVolumeClaims using the topolvm-provisioner storage class.

On the infrastructure nodes (which run the control plane and are tainted NoSchedule for workload pods), dedicated logical volumes are pre-created by Ansible for the host services: MariaDB (mounted at /data/mysql) and the K3s server state (mounted at /var/lib/rancher/k3s/server).

Failure modes

One out of two nodes can fail without data loss for the surviving node's workloads.
Service continues automatically on the surviving node via the virtual IP — no manual failover is needed for external connections.
A single node must be able to handle the full production load (this is a disaster recovery setup, not a load-balancing setup).
K3S server startup depends on MariaDB being available — see the MariaDB Galera recovery section below.

Deployment method

The deployment is managed by:

Ansible playbooks (configured via inventory.yaml) to manage the virtualized nodes, which run K3S and other dependencies.
A Helm chart (configured via values.yaml) to manage the Kolab deployment running on K3S.

The Ansible playbooks assume that one or more KVM hypervisors are available to run the virtualized nodes, and the following topology is assumed:

Control host: A workstation with access to the hypervisors is used to manage the environment.
Hypervisor A (or a single hypervisor):
- Infrastructure VM 1 (infra1)
- Worker VM 1 (worker1)
Hypervisor B (optional, for true HA across failure domains):
- Infrastructure VM 2 (infra2)
- Worker VM 2 (worker2)

The Ansible playbooks are executed from the control host. It is required that passwordless SSH logins are configured (for the user executing the Ansible scripts) from the control host to the hypervisors for virtual machine setup.

First, an inventory.yaml file must be created to describe the deployment.
An initial run of the Ansible playbook will provision the VMs on the hypervisors, set up HAProxy, the MariaDB Galera cluster, and a K3S cluster, then deploy Kolab into the cluster.
An initial values.yaml will be generated from a template next to the inventory file.

Going forward, the inventory.yaml and values.yaml files should be kept in version control. Changes to the environment can be applied by either executing the full Ansible playbook again, or by only redeploying the Helm chart (via values.yaml).

Setup Instructions

Executed from the control host.

Create a directory where you intend to manage the Kolab installation from.

Download the latest release:

https://mirror.apheleia-it.ch/pub/kolab-kubernetes-latest.tar.gz

Extract the tarball:
```
tar xf kolab-kubernetes-latest.tar.gz
```

Copy the example inventory to create your own:

cp deployments/k3s-two-node/inventory.example.yaml inventory.yaml

Adjust inventory.yaml to match your environment:
Set virtual_ip to the floating IP that will be shared by HAProxy/keepalived.
Set all passwords (vm_root_password, db_*_password, k3s_db_password, etc.) — consider using Ansible Vault for secrets.
Set domain to your domain name.
Set chart_url if you are using a custom registry.
Populate the infrastructure_nodes and worker_nodes host groups with your hostnames.
Assign hosts to hypervisor_primary_group (configured as keepalived MASTER) and hypervisor_secondary_group (configured as keepalived BACKUP), and set qemu_uri per group or host if VMs are distributed across different hypervisors.
Set mariadb.bootstrap: true on exactly one infrastructure node (the one that will bootstrap the Galera cluster on first run).
Set the LoadBalancer IPs (proxy_lb_ip, imap_lb_ip, postfix_lb_ip, etc.) to match your MetalLB address pool.
Copy deployments/k3s-two-node/values.yaml to the same directory and adjust it:
Set domainName, adminPassword, image references, TLS certificate, Passport keys, and all other deployment-specific values.
Storage capacities reference the topolvm-provisioner storage class — ensure the values match the disk sizes allocated per node.

Run the Ansible provisioning process:

ansible-playbook -v -i inventory.yaml -D deployments/k3s-two-node/ansible/playbook.yaml

Navigate to the configured URL (add an /etc/hosts entry if DNS is not yet configured) and log in with admin@YOURDOMAIN.
Future changes can be applied via:
```
./kolabctl apply
```

Installing on a pre-existing VM (bypass VM provisioning)

It is possible to connect to an existing pair of systems instead of letting Ansible provision new VMs via KVM.

As the base image for manual installation, the following image is recommended: https://mirror.apheleia-it.ch/pub/kolab-minimal-0.1.qcow2 (default login: root/kolab)

Once the VMs are available, follow the setup instructions above with the following adjustments in inventory.yaml:

Disable VM provisioning by setting provision_vm: false in each host entry or in all.vars.
Set ansible_host for each host to the IP or hostname reachable from the control host, if the inventory hostname is not already resolvable.

Updating

To update the K3S nodes, run the full Ansible playbook again:

ansible-playbook -v -i inventory.yaml -D deployments/k3s-two-node/ansible/playbook.yaml

Note that this respects pinned image versions in values.yaml, so update those manually before running the playbook if you intend to upgrade Kolab component images.

To reinstall K3S without reprovisioning the VMs (e.g. after a failed K3S upgrade):

ansible-playbook -v -i inventory.yaml -D deployments/k3s-two-node/ansible/reinstall-k3s.yaml

MariaDB Galera Recovery

K3S uses MariaDB as its datastore backend. This creates a startup dependency: K3S cannot start if MariaDB is not running. If both nodes lose MariaDB simultaneously (e.g. after a full cluster power-off), neither K3S nor MariaDB will come up automatically.

One node still active

If one node is still up, it continues operating. Once the failed node is recovered, restart its MariaDB service:

systemctl start mariadb

MariaDB will automatically rejoin the Galera cluster, and K3S will follow.

All nodes down

When all nodes have been shut down, it is necessary to bootstrap a new Galera cluster from the node with the most recent data.

Determine which node is ahead by comparing WSREP sequence numbers on both nodes:
```
mysqld --wsrep-recover
```
Look for a line such as:
```
WSREP: Recovered position: 90823e7a-cf9e-11ef-a4e1-d69a8635a591:1873796
```
Bootstrap from the node with the highest sequence number. Set mariadb.bootstrap: true on that node in inventory.yaml.
Run the recovery playbook, specifying the bootstrap host:
```
ansible-playbook -i inventory.yaml deployments/k3s-two-node/ansible/recover-mariadb.yaml \
  -e 'mariadb_bootstrap_host=<hostname>'
```
This stops any stuck MariaDB/garbd processes, forces safe_to_bootstrap: 1 on the chosen node, and runs galera_new_cluster. The other node will then rejoin automatically.
Once MariaDB is healthy, K3S will start automatically (it polls the datastore on startup). If K3S does not start on its own, start it manually:
```
systemctl start k3s
```