Lxc

From campisano.org
Jump to navigation Jump to search

Install

# note: avoid running docker daemon, or lxc containers will not have internet access
apt-get install lxc
apt-get clean

Modules

Required modules for privileged containers 

cat > /etc/modules-load.d/lxc_privileged_containers.conf << EOF
bridge
br_netfilter
ip_tables
iptable_filter
iptable_nat
netlink_diag
nf_conntrack
overlay
tun
udp_tunnel
vxlan
xfrm_user
xt_statistic
EOF
while IFS= read -r M; do modprobe -v "$M"; done < /etc/modules-load.d/lxc_privileged_containers.conf

Config

Prepare a customized common config with privileged and network settings

References: 

cat > /etc/lxc/custom.conf << EOF


### Basic settings, from /usr/share/lxc/config/common.conf

# Default configuration shared by all containers

# Setup the LXC devices in /dev/lxc/
lxc.tty.dir = lxc

# Allow for 1024 pseudo terminals
lxc.pty.max = 1024

# Setup 4 tty devices
lxc.tty.max = 4

# Drop some harmful capabilities
###lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio

# Ensure hostname is changed on clone
lxc.hook.clone = /usr/share/lxc/hooks/clonehostname

# Setup the default mounts
###lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0

# Block some syscalls which are not safe in privileged
# containers
###lxc.seccomp.profile = /usr/share/lxc/config/common.seccomp

# Lastly, include all the configs from /usr/share/lxc/config/common.conf.d/
lxc.include = /usr/share/lxc/config/common.conf.d/

### End basic settings, from /usr/share/lxc/config/common.conf



### Additional mounts
lxc.mount.auto=proc:rw sys:rw cgroup:rw
lxc.mount.entry = /dev/kmsg dev/kmsg none bind,create=file 0 0
lxc.mount.entry = /boot boot none bind,create=dir 0 0

# mount /lib/modules directly on the container destination will lead to an error due to recurring links
# so we mount it to an another folder now and at first boot we will manual link it to the destination
lxc.mount.entry = /lib/modules mnt/host/modules none bind,create=dir 0 0



### Auto start this container at boot
lxc.start.auto = 1



### Network
lxc.net =
lxc.net.0.type = veth
lxc.net.0.link = brvxlan0
lxc.net.0.name = eth0
lxc.net.0.flags = up



### Privileged
lxc.apparmor.profile=unconfined
lxc.cap.drop=
lxc.cgroup.devices.allow=a
lxc.cgroup.devices.deny=
lxc.cgroup2.devices.allow=a
lxc.cgroup2.devices.deny=

EOF

Service

The start/stop lxc service scripts will be in charge of create the host network that allow communication between the host and its containers as well as between the containers of all the hosts.

In the follow example there are two hosts, already connected in a same network, with arbitrary IPs 10.1.101.1, 10.1.101.2 and 10.1.101.3, both with a common network device named enp0s6

Create a env file for the lxc service: 

# three hosts will be used, with following networks, IPs and devices
# note that hosts need to know each other to add a rule on bridge forwarding database (FDB) 
cat > /etc/default/lxc-custom-net.env << 'EOF'
BRIDGE_CIDR=10.10.0.0/16
HOSTS='enp0s6 10.1.101.1 10.10.1.100
       enp0s6 10.1.101.2 10.10.2.100'
EOF

Create the start script for the lxc service: 

cat > /etc/default/lxc-custom-net-start.sh << 'EOF'
#!/bin/sh

set -o errexit -o nounset

# identify current host ip by device
while read -r TEST_DEVICE TEST_DEVICE_IP TEST_BRIDGE_IP
do
    HOST_DEVICE_IP=$( { ip -4 -o addr show dev "$TEST_DEVICE" 2>/dev/null || true; } | sed -n -E 's|.*inet ([0-9.]+)/.*|\1|p' )

    if test "$HOST_DEVICE_IP" = "$TEST_DEVICE_IP"
    then
        HOST_DEVICE=$TEST_DEVICE
        HOST_BRIDGE_IP=$TEST_BRIDGE_IP
        break
    fi
done <<EOVARS
$HOSTS
EOVARS

if test -z "${HOST_DEVICE:-}" || test -z "${HOST_DEVICE_IP:-}" || test -z "${HOST_BRIDGE_IP:-}"
then
   echo "Cannot find device ip. exit" >&2
   exit 1
fi

echo "found device ${HOST_DEVICE}, device ip ${HOST_DEVICE_IP}, bridge ip ${HOST_BRIDGE_IP}"

# create a bridge and assign a new ip to the current host
ip link add brvxlan0 type bridge
ip addr add ${HOST_BRIDGE_IP}/16 dev brvxlan0
ip link set brvxlan0 up

# create a vxlan (id 42 is arbitrary, nolearning is to disable mac learning to manually specify where to send traffic destined for a specific mac)
ip link add vxlan0 type vxlan id 42 dev $HOST_DEVICE dstport 4789 local $HOST_DEVICE_IP nolearning
ip link set vxlan0 up
ip link set vxlan0 master brvxlan0

# for each other host, add a bridge rule on current host
# for each other host, add a bridge rule on current host
while read -r OTHER_DEVICE OTHER_DEVICE_IP OTHER_BRIDGE_IP
do
    if test "$HOST_DEVICE_IP" != "$OTHER_DEVICE_IP"
    then
        bridge fdb append 00:00:00:00:00:00 dst $OTHER_DEVICE_IP dev vxlan0
    fi
done <<EOVARS
$HOSTS
EOVARS

# add a masquerade postrouting rule for current network
iptables -t nat -A POSTROUTING -s $BRIDGE_CIDR ! -d $BRIDGE_CIDR -j MASQUERADE
EOF

chmod 755 /etc/default/lxc-custom-net-start.sh

Create the stop script for the lxc service: 

cat > /etc/default/lxc-custom-net-stop.sh << 'EOF'
#!/bin/sh

set -o errexit -o nounset

ip link del vxlan0
ip link del brvxlan0
EOF

chmod 755 /etc/default/lxc-custom-net-stop.sh

Create the lxc service definition for systemd: 

cat > /etc/systemd/system/lxc-custom-net.service << 'EOF'
[Unit]
Description=LXC custom network
After=network.target

[Service]
Type=oneshot
EnvironmentFile=/etc/default/lxc-custom-net.env
ExecStart=/etc/default/lxc-custom-net-start.sh
ExecStop=/etc/default/lxc-custom-net-stop.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

Enable and start lxc service: 

systemctl daemon-reload
systemctl enable lxc-custom-net
systemctl start lxc-custom-net
systemctl status lxc-custom-net

Create the LXC containers

Create the LXC containers, we will use Debian trixie arm64 image


Example for c101: 

export CNT=c101
export CONTAINER_CIDR=10.10.1.1/16
export HOST_BRIDGE_IP=10.10.1.100

Create the container 

lxc-create "${CNT}" --template download -- --dist debian --release trixie --arch arm64

Reset network config and configure the container 

# create a storage folder in the host
mkdir -p /srv/lxc/storage/${CNT}
# replace common configs with the custom one
sed -i 's|/usr/share/lxc/config/common.conf|/etc/lxc/custom.conf|g' /var/lib/lxc/${CNT}/config
# add specific configs for this container
cat >> /var/lib/lxc/${CNT}/config << EOF

#################
# Custom config #
#################

## Host persistent mount point
lxc.mount.entry = /srv/lxc/storage/${CNT} mnt/host/storage none bind,create=dir 0 0

## Memory
lxc.cgroup2.memory.max = 2560M

## Custom
lxc.include = /etc/lxc/custom.conf
EOF

Start the container and enter 

lxc-start ${CNT} -l DEBUG -o ${CNT}_start.log
cat ${CNT}_start.log
lxc-ls --fancy
lxc-attach --keep-env ${CNT}

Configure shell history 

cat >> ~/.bashrc << 'EOF'

####
# bash history
####
export HISTTIMEFORMAT='%F %T '
export HISTSIZE=50000
export HISTFILESIZE=50000
export HISTCONTROL=ignorespace
EOF

Fix modules and dmesg wrong behaviors inside containers 

# fix /lib/modules
rm -rf /lib/modules
ln -s /mnt/host/modules /lib/modules
#
# fix dmesg
cat > /etc/sysctl.d/dmesg.conf << 'EOF'
kernel.dmesg_restrict = 0
EOF
sysctl -w kernel.dmesg_restrict=0

Configure systemd-networkd to use our defined static ip 

cat > /etc/systemd/network/eth0.network << EOF
[Match]
Name=eth0

[Network]
DHCP=no
Address=${CONTAINER_CIDR}
Gateway=${HOST_BRIDGE_IP}
DNS=8.8.8.8
DNS=1.1.1.1
Domains=~.
EOF
systemctl restart systemd-networkd

After that, you can see that the ip is correct and exit 

ip addr show dev eth0
ping 10.10.1.100
ping 8.8.8.8
exit
Retrieved from "https://www.campisano.org/mediawiki/index.php?title=Lxc&oldid=2908"