NetBSD

As you might be aware of, I am working on a minimal BSD/UNIX system called smolBSD. It is based on the NetBSD operating system, and its main target is virtual machines, more specifically microvms. This system is capable of fully booting the OS in less than one second using a specially trimmed kernel along with small, specialized root systems.
I was stunned to learn (but am I wrong?) that this work does not seem to have an equivalent, not even in the Linux world. FreeBSD Firecracker support sure permits to boot a kernel is about the same time my NetBSD hacked kernel does, but there’s no work around s slim and task dedicated root filesystem.

2025/01 Update https://github.com/NetBSDfr/smolBSD/tree/main/k8s

I had to do it.
So here’s how to run a NetBSD micro-vm as… a Kubernetes pod.

First thing is to modify the start script from the previous article in order to add Docker-style networking, i.e. port forwarding from the host to the micro-vm. This is done using the hostfwd flag in qemu’s -netdev parameter

#!/bin/sh

kernel=$1

img=${2:-"root.img"}

[ -n "$3" ] && drive2="-drive file=${3},if=virtio"

qemu-system-x86_64 -enable-kvm -m 256 \
        -kernel $kernel -append "console=com root=ld0a" \
        -serial mon:stdio -display none \
        -drive file=${img},if=virtio $drive2 \
        -netdev user,id=net0,hostfwd=tcp::8080-:80 -device virtio-net,netdev=net0

In the previous experience we mapped the kernel and the root image from the host using Docker’s -v parameter, and while it’s possible to map files from the host using a Kubernetes volume, we will bundle NetBSD these files into the Docker image to make things easier.
Please refer to mksmolnb documentation to learn how to produce a minimal nginx micro-vm.

I have this little toy project for quite a while now, and I have this idea of handling a fleet of NetBSD micro-vms with Kubernetes since I started my new job in which I am caring a k8s cluster.

I came to realize that starting a smolBSD micro-vm with Docker was not so difficult after all. Using mksmolnb’s startnb.sh I came up with this very simple Dockerfile:

FROM alpine:latest

RUN apk add --quiet --no-cache qemu-system-x86_64 iproute2 bridge-utils

COPY startnb.sh ./
COPY qemu/qemu-ifup qemu/qemu-ifdown /etc/

CMD /startnb.sh /netbsd-SMOL ${IMG} ${DISK}

qemu-ifup being a simple copy of Debian’s /etc/qemu-ifup.

In November 2018 AWS published an Open Source tool called Firecracker, mostly a virtual machine monitor relying on KVM, a small sized Linux kernel, and a stripped down version of Qemu. What baffled me was the speed at which the virtual machine would fire up and run the service. The whole process is to be compared to a container, but safer, as it does not share the kernel nor any resource, it is a separate and dedicated virtual machine.
If you want to learn more on Firecracker’s internals, here’s a very well put article.

pkgsrc’s collectd does not support the thermal plugin, so in order to publish thermal information I had to use the exec plugin:

LoadPlugin exec
# more plugins

<Plugin exec>
        Exec "nobody:nogroup" "/home/imil/bin/temp.sh"
</Plugin>

And write this simple script that reads CPUs temperature from NetBSD’s envstat command:

$ cat bin/temp.sh 
#!/bin/sh

hostname=$(hostname)
interval=10

while :
do
        envstat|awk '/cpu[0-9]/ {printf "%s %s\n",$1,$3}'|while read c t
        do
                echo "PUTVAL ${hostname}/temperature/temperature-zone${c#cpu} interval=${interval} N:${t%%.*}"
        done
        sleep ${interval}
done

I then send those values to an influxdb server:

I recently added a fantastic graphing tool named facette to pkgsrc. Facette knows how to pull data sources from various backends, and among them, the famous collectd.

In this article, we will see how to setup both on NetBSD but keep in mind it should also work for any platform supported by pkgsrc.

First up, collectd installation. It can be done either with pkgin (binary installation) or pkgsrc (source installation):

• with pkgin

$ sudo pkgin in collectd collectd-rrdtool

• with pkgsrc

 $ cd /usr/pkgsrc/sysutils/collectd
 $ sudo make install clean
 $ cd ../collectd-rrdtool
 $ sudo make install clean

Tune up a minimal collectd configuration

I have read a shitload of overcomplicated setups to bring up a postfix / dspam SMTP + antispam server, and finally came to a much lighter and simpler configuration by basically reading documentation and real life examples. Note this is suitable for a personnal and basic environment, no database, no virtual setup. Basic stuff.

The target system is NetBSD but this short doc should apply to pretty much any UNIX / Linux.

If you recently tried to install NetBSD 7.0 using Linux KVM you might have encountered the following failure:

This bug have been recently fixed on the 7-branch but the official ISO images are not yet updated, so you’ll have to use NetBSD daily builds mini-ISO which includes Christos fix to bus_dma.c

For the record, here’s the virt-install command I use:

sudo virt-install
    --virt-type kvm
    --name korriban
    --ram 4096 --disk path=/dev/vms/korriban,bus=virtio
    --vcpus 2
    --network bridge:br0,model=virtio
    --graphics vnc
    --accelerate
    --noautoconsole
    --cdrom /home/imil/iso/boot.iso
    --cpu host

Performances are really good, the host is a Debian GNU/Linux 8.0 amd64 running on Online’s Dedibox Classic 2015.

Proceeding with my Christmas presents, I have refactored my ${HOME} lab. One of the goals was to migrate my public gateway to a diskless Soekris Net6501 my beloved wife offered me :)

The overall PXE/NFS process is explained a billion times over the Internet, only particular point here is that I used dnsmasq instead of ISC DHCP.

Nevertheless I came across an issue that took me way too long to understand; while the boot process seemed perfectly fine, at some point, after Setting up ttys, init hung and the kernel showed the following message:

Snoopy is a pretty cool piece of software that can log every exec(3) call to syslog. When it comes to security, that feature can be really handy.

Yesterday (Dec. 5), I commited security/snoopy to pkgsrc. The package comes with GNU/Linux related scripts in order to modify /etc/ld.so.preload so libsnoopy is loaded before libc and achieve its role. NetBSD doesn’t have a ld.so.preload file, instead, we use a flexible /etc/ld.so.conf configuration file which has the following syntax: