Kubernetes HA Cluster using LXC

From campisano.org
Jump to navigation Jump to search

Architecture

Architecture used in this hands-on guide

  • name.com - for domain names
  • Oracle network load balancer - for load balancing request to servers
  • 2 Oracle ampere instances - as provisioned servers
  • HAProxy systemd daemon - in each server, to balancing requests from the load balancer to forward to the internal LXC containers
  • 2 LXC containers per server - to accommodate the Kubernetes cluster
  • LXC network - to provide connections between containers of all servers
  • kubeadm - to setup the Kubernetes cluster
  • Flannel - to setup the Kubernetes network
  • Traefik - in global deploy mode to be in each LXC container, to receive requests from HAProxy to forward to the Kubernetes services


static IP network definition

internal network CIDR

from https://www.davidc.net/sites/default/subnets/subnets.html?network=10.10.0.0&mask=16&division=1.0 

10.10.0.0/16	10.10.0.0 - 10.10.255.255	10.10.0.1 - 10.10.255.254	65534	Divide

note about the nomenclature scheme for the 10.10.0.0 ip address

  • the first two numbers (10.10.) are the prefix under that will be contained all the internal LXC resources
  • the third number will be used to identify provisioned servers (0-255, 255 max)
  • and forth number will be used to identify LXC containers accommodated by servers (1-254, 253 max theoretically, however we just want a handful of them per server, we can use 10.10.x.100 for the provisioned server, so the range will be 1-99, 98 max that is more than enough)

In this way, we can chose container names that remember both the parent server as well as the static IP used by the container

  • ip table:
resource name external network addr internal bridge addr
* server1 (unknown at this point) 10.10.1.100
c101 10.10.1.1
c102 10.10.1.2
* server2 (unknown at this point) 10.10.2.100
c201 10.10.2.1
c202 10.10.2.2



System cleanup

on the hosting machines, cleanup usefulness stuffs to avoid disk waste and provider's agents resource waste

on Debian, you can use apt-mark and apt-get autoremove --purge


Example useful for Jammy cleanup:

turn off phases updates, Ubuntu 24 specific stuff, see https://ubuntu.com/server/docs/explanation/software/about-apt-upgrade-and-phased-updates/#how-do-i-turn-off-phased-updates 

cat > /etc/apt/apt.conf.d/99-Phased-Updates << EOF
Update-Manager::Always-Include-Phased-Updates true;
APT::Get::Always-Include-Phased-Updates true;
EOF

then 

apt-get update
apt-mark auto `apt-mark showmanual`
apt-get install apt apt-utils base-files base-passwd coreutils dash debconf dpkg grep gzip hostname libc6 libc-bin nano passwd procps psmisc readline-common sed sysvinit-utils tar
apt-get install bash binutils bsdutils bzip2 ca-certificates curl dialog dnsutils findutils iptables iputils-ping kmod less locales lsof man mount mutt ncal netcat-traditional nmap telnet tzdata util-linux
apt-get install busybox-static command-not-found conntrack cron dmidecode friendly-recovery hwdata hdparm info iputils-tracepath irqbalance jq linux-oracle linux-headers-oracle mtr-tiny net-tools netcat-openbsd netfilter-persistent rsync rsyslog socat sudo tcl tcpdump time ubuntu-minimal ubuntu-standard unzip
apt-get autoremove --purge
apt-get dist-upgrade
apt-get clean
shutdown -r now

shutdown just to grant avoid surprises in the next future reboot


LXC install

Note, we will use the vanilla LXC, not the Ubuntu LXD one


Follow Lxc instructions


HAProxy install

HAProxy will be installed in any server to forward requests incoming from the external network load balancer to the internal LXC containers services

In the follow example, HAProxy will forward external incoming requests on 6443 port to the Kubernetes api listening at 6443 port inside each of the LXC containers of this example, roundrobin balancing the request between them: 

apt-get install haproxy
apt-get clean
cat > /etc/haproxy/haproxy.cfg << 'EOF'
global
    log stdout format raw local0
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    timeout connect 10s
    timeout client 300s
    timeout server 300s
    timeout tunnel 300s
    option tcp-check
    tcp-check connect
    balance roundrobin
    default-server downinter 5s fastinter 1s inter 3s fall 2 rise 3

frontend ft_lxc_kubeapi
    bind *:6443
    default_backend bk_lxc_kubeapi

backend bk_lxc_kubeapi
    server c101 10.10.1.1:6443 check
    server c102 10.10.1.2:6443 check
    server c201 10.10.2.1:6443 check
    server c202 10.10.2.2:6443 check

frontend ft_lxc_deploy
    bind *:30666
    default_backend bk_lxc_deploy

backend bk_lxc_deploy
    server c101 10.10.1.1:30666 check
    server c102 10.10.1.2:30666 check
    server c201 10.10.2.1:30666 check
    server c202 10.10.2.2:30666 check

frontend ft_lxc_gitolite
    bind *:30022
    default_backend bk_lxc_gitolite
 
backend bk_lxc_gitolite
    server c101 10.10.1.1:30022 check
    server c102 10.10.1.2:30022 check
    server c201 10.10.2.1:30022 check
    server c202 10.10.2.2:30022 check

frontend ft_lxc_traefik_http
    bind *:30080
    default_backend bk_lxc_traefik_http

backend bk_lxc_traefik_http
    server c101 10.10.1.1:30080 check
    server c102 10.10.1.2:30080 check
    server c201 10.10.2.1:30080 check
    server c202 10.10.2.2:30080 check

frontend ft_lxc_traefik_https
    bind *:30443
    default_backend bk_lxc_traefik_https

backend bk_lxc_traefik_https
    server c101 10.10.1.1:30443 check
    server c102 10.10.1.2:30443 check
    server c201 10.10.2.1:30443 check
    server c202 10.10.2.2:30443 check
EOF
systemctl restart haproxy
systemctl status haproxy

Kubernetes install under LXC

Follow the Install section of Kubernetes


In one of the LXC machines with Kubernetes binaries installed, do 

LB_ENDPOINT=<INTERNAL_LOADBALANCER_IP>:6443
kubeadm init --v=5 --ignore-preflight-errors=NumCPU,Mem --cri-socket unix:/run/containerd/containerd.sock --pod-network-cidr=192.168.0.0/16 --control-plane-endpoint "${LB_ENDPOINT}" --upload-certs
mkdir -p $HOME/.kube; cp -a /etc/kubernetes/admin.conf $HOME/.kube/config
kubectl get nodes

Fix kube proxy error inside LXD trying to change /proc/sys/stuffs 

kubectl get all -A
kubectl logs -n kube-system -l k8s-app=kube-proxy
kubectl -n kube-system get configmap kube-proxy -o yaml | sed 's/maxPerCore: null/maxPerCore: 0/' | sed 's/min: null/min: 0/' | kubectl apply -f -
kubectl -n kube-system delete pods -l k8s-app=kube-proxy
## wait pending state
kubectl get all -A

Configure Kubernetes network 

mkdir flannel
VERSION="0.28.7"
curl -fsSL "https://github.com/flannel-io/flannel/releases/download/v${VERSION}/kube-flannel.yml" | sed 's:10.244.0.0/16:192.168.0.0/16:g' > flannel/flannel.yml
kubectl apply -f flannel/flannel.yml
## wait ready state
kubectl get nodes

Obtain Kubernetes join command, all the nodes will be control-plane 

echo $(kubeadm token create --print-join-command) --v=5 --ignore-preflight-errors=NumCPU,Mem --control-plane --certificate-key $(kubeadm init phase upload-certs --upload-certs 2>/dev/null | sed -n '/Using certificate key/{n;p;}')

On each other nodes to configure 

kubeadm join ...
mkdir -p $HOME/.kube; cp -a /etc/kubernetes/admin.conf $HOME/.kube/config
kubectl get nodes

Untaint manager nodes to be able to accommodate services 

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
Retrieved from "https://www.campisano.org/mediawiki/index.php?title=Kubernetes_HA_Cluster_using_LXC&oldid=2909"