NetBSD as a Kubernetes Pod

by Emile `iMil' Heitor - 2023-10-23

I had to do it.
So here’s how to run a NetBSD micro-vm as… a Kubernetes pod.

First thing is to modify the start script from the previous article in order to add Docker-style networking, i.e. port forwarding from the host to the micro-vm. This is done using the hostfwd flag in qemu’s -netdev parameter

#!/bin/sh

kernel=$1

img=${2:-"root.img"}

[ -n "$3" ] && drive2="-drive file=${3},if=virtio"

qemu-system-x86_64 -enable-kvm -m 256 \
        -kernel $kernel -append "console=com root=ld0a" \
        -serial mon:stdio -display none \
        -drive file=${img},if=virtio $drive2 \
        -netdev user,id=net0,hostfwd=tcp::8080-:80 -device virtio-net,netdev=net0

In the previous experience we mapped the kernel and the root image from the host using Docker’s -v parameter, and while it’s possible to map files from the host using a Kubernetes volume, we will bundle NetBSD these files into the Docker image to make things easier.
Please refer to mksmolnb documentation to learn how to produce a minimal nginx micro-vm.

FROM alpine:latest

RUN apk add --quiet --no-cache qemu-system-x86_64 iproute2 bridge-utils

COPY netbsd-SMOL nginx.img startnb.sh ./
COPY qemu/qemu-ifup qemu/qemu-ifdown /etc/

CMD /startnb.sh /netbsd-SMOL nginx.img

Once the image is created, export it to a tar archive in order to add it to the “cluster”; I won’t upload smolBSD images to public repo just yet, I have a bigger plan, so for now, I’ll just make the images available to containerd as cached ones.

Warning, it took me way too long to understand that an image tagged with latest won’t be cached and then will be fetched no matter what, so tag your image accordingly.

builder$ docker save imil/smolbsd:0.1 > smolbsd.tar

There’s a lot of homelab Kubernetes all-in-one clusters out there, I picked one that works out of the box on the GNU/Linux Mint system I’m using, microk8s.
Let’s import the smolbsd image

cluster$ microk8s ctr image import smolbsd.tar

To spare some time I did the classic alias

cluster$ alias k='microk8s kubectl'

And now the real deal, here’s the pod manifest, basically this does the same as the Docker command line. Note the imagePullPolicy: Never key / value as we don’t want the container runtime to try and fetch the image “outside”

apiVersion: v1
kind: Pod
metadata:
  name: smolbsd-nginx
  labels:
    app.kubernetes.io/name: smolbsd-nginx
spec:
  containers:
  - name: nginx
    image: imil/smolbsd:0.1
    imagePullPolicy: Never
    ports:
    - containerPort: 8080
    volumeMounts:
    - mountPath: /dev/kvm
      name: dev-kvm
    securityContext:
      privileged: true
      capabilities:
        add:
          - NET_ADMIN
  volumes:
  - name: dev-kvm
    hostPath:
      path: /dev/kvm

We could have written a deployment, and I actually tried it, it works, but for the sake of simplicity of this blog post, we’ll keep it as simple as a single pod.
We expose the port 8080 from the container, which in turn is forwarded by qemu with the hostfwd parameter.

In order to make this pod service available, here the associated service, here again we expose the port 8080

apiVersion: v1
kind: Service
metadata:
  name: smolbsd-svc
spec:
  selector:
    app.kubernetes.io/name: smolbsd-nginx
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

We can join the two as a single yaml file, by separating both definitions with a --- and they invoke k apply

cluster$ k apply -f smolbsd.yaml

And here we go

cluster$ k get pods
NAME            READY   STATUS    RESTARTS   AGE
smolbsd-nginx   1/1     Running   0          13h
cluster$ k get services
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes    ClusterIP   10.152.183.1    <none>        443/TCP    22h
smolbsd-svc   ClusterIP   10.152.183.18   <none>        8080/TCP   19h
cluster$ k logs smolbsd-nginx 2>&1|tail -20
vioif1: leased 10.0.2.15 for 86400 seconds
vioif1: adding route to 10.0.2.0/24
vioif1: adding default route via 10.0.2.2
script_runreason: /libexec/dhcpcd-run-hooks: No such file or directory
forked to background, child pid 122

starting nginx.. done

Testing web server:
HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Mon, 23 Oct 2023 15:02:01 GMT
Content-Type: text/html
Content-Length: 38
Last-Modified: Mon, 23 Oct 2023 04:13:29 GMT
Connection: close
ETag: "6535f2e9-26"
Accept-Ranges: bytes

The moment of truth

$ curl http://10.152.183.18:8080
Welcome to nginx on NetBSD on Docker on Kubernetes!

NetBSD pod on Kubernetes [✔]