foundation/runners/README.md

foundation-runners — the fenced Actions runner fleet (isolated stack)

Step-0 after the foundation stands. A separate Pulumi project/stack that provisions runner VM(s) on a libvirt host (crunchy01) and registers Forgejo Actions runners with a distinct label (fenced), so ecosystem/untrusted jobs (runs-on: fenced) execute off the forge VM — the R5 fence.

Why a separate stack (decoupling)

A @pulumi/libvirt provider dials the runner host on every up/refresh/preview of the stack it lives in. If the runner VM lived in bootstrap, then crunchy01 being down — or you not having access to it — would break pulumi refresh/up of the foundation itself (the classic Terraform coupling trap). Pulumi isolates this at the stack boundary: a provider only initializes when its own stack runs. So the fleet is its own project; bootstrap never imports it. Consequences:

• Foundation deploy/refresh never touches crunchy01.
• crunchy01 down ⇒ only this stack's refresh is affected, and only when you run it.
• One-way dependency: this stack mints a runner token from the forge, so it runs after the foundation is up.

Host prep (one-time, kept OUT of this stack)

The libvirt provider needs something to connect to, so install libvirt on the host out-of-band (not via this stack), and ensure a LAN bridge exists:

sudo apt-get update
sudo apt-get install -y qemu-kvm libvirt-daemon-system libvirt-clients \
  bridge-utils dnsmasq qemu-utils virtinst cloud-image-utils
sudo systemctl enable --now libvirtd
# a LAN bridge (br0) enslaving the physical NIC must already exist (crunchy01 had it).

Also required on the host, one-time:

• root SSH via key — the @pulumi/libvirt provider and the host firewall command connect as root (add the operator pubkey to /root/.ssh/authorized_keys).
• a libvirt storage pool — crunchy01 already had one named images (at /var/lib/libvirt/images), so the stack is configured with host.pool images. On a host with the conventional default pool, leave host.pool at its default.

Deploy

export RUNNER_SSH_KEY_PATH=~/.ssh/foundation-test_ed25519   # reaches host + VM (root)
cd runners
pulumi stack init crunchy         # isolated file backend, like bootstrap/provision
pulumi config set host.address 192.168.1.2
pulumi config set host.pool images          # crunchy01's pool (see host prep)
pulumi config set forge.address 204.168.234.72
pulumi config set vm.name foundation-runner-02
pulumi config set vm.ipCidr 192.168.1.16/24
pulumi up

pulumi up will: apply the kube-router-proof FORWARD timer on the host, create an Ubuntu VM on br0 (docker + qemu-guest-agent via cloud-init), mint a runner token from the forge, and register + run the fenced runner in the VM. Verify with a runs-on: fenced job on any repo.

Cutover: DONE. pulumi up on the crunchy stack created foundation-runner-02 (static .16, 8c/32G), registered the fenced runner, and a runs-on: fenced job ran on it green. The hand-built foundation-runner-01 was then retired (virsh destroy/undefine + disk removed), so the Pulumi-managed runner-02 is the sole fenced runner. (A now-offline crunchy-runner registration from the hand-built VM may still be listed on the forge — harmless; deregister at leisure.)

Gotchas baked into the code (learned the hard way)

• k3s host firewall. crunchy01 is a k3s node; kube-router sets FORWARD policy DROP + br_netfilter=1, dropping bridged VM↔LAN traffic. Fix = iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT, re-asserted by a 60s systemd timer (kube-router flushes iptables on resync, so a boot-only rule isn't enough).
• Ubuntu, not Debian genericcloud. Debian's cloud-init wrote netplan the image never applied → no IPv4 (static or DHCP). Ubuntu 24.04 renders + applies cleanly.
• NIC name-agnostic network-config. The cloud-init network-config matches the NIC by glob (match: {name: "e*"}), not a hardcoded enp1s0 — the libvirt.Domain may enumerate it as ens3/etc., which left the VM with no IP until matched generically.
• No qemuAgent: true. It makes the provider block on the guest agent (not up on a fresh boot) during create. We register over the VM's static IP, so it's not needed.
• Register dial window. The runner-register command uses dialErrorLimit: 30 so it waits ~5 min for the VM to boot + apply its IP, landing the runner in a single up.
• PTY console. The domain declares a pty serial console so virsh console <vm> works. (Don't back serial with a file — you lose interactive console.)
• Docker socket gid. act_runner runs as uid 1000; the daemon container gets --group-add <docker gid> so it can reach /var/run/docker.sock.
• IP is optional. The runner polls the forge outbound, so a fixed LAN IP isn't required — set vm.ipCidr empty for DHCP. Default is a static .15 for predictability.