foundation/documentation/sessions/SESSION_2026-07-01_003.md

Session 2026-07-01 #003 — the fenced runner fleet (R5): build → harden → codify → cutover

What was done

Continued from #002 (T14 + ecosystem CI done). Built the R5 fence the operator had deferred: a Forgejo Actions runner on a separate VM on separate hardware (crunchy01), so ecosystem/untrusted jobs (runs-on: fenced) run OFF the forge VM. Then hardened it, formalized it as a decoupled Pulumi stack, and did the live cutover.

1. Fenced runner — built by hand, then proven

• crunchy01 (192.168.1.2, 16c/128G, Debian 13, k3s node running the GitLab runners + nominatim) had /dev/kvm + passwordless sudo but no libvirt. Installed qemu-kvm/libvirt/virtinst/cloud-image-utils.
• Created an Ubuntu 24.04 VM on the LAN bridge br0, docker inside, registered a Forgejo runner (label fenced) against https://forge.olsitec.net.
• Proven: a runs-on: fenced job ran with kernel 6.8.0 (Ubuntu VM) + egress 62.176.248.112 (site IP), vs the Hetzner forge VM's 6.1.0 / 204.168.234.72 — i.e. it executed on crunchy, isolated from the forge.

2. Hardening

• kube-router-proof firewall. crunchy01's k3s/kube-router sets FORWARD policy DROP
  • br_netfilter=1, which drops bridged VM↔LAN traffic (incl. the runner→forge poll). Fix = iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT, re-asserted by a 60s systemd timer (libvirt-bridge-forward.timer) because kube-router flushes iptables on resync (a boot-only unit isn't enough).
• VM autostart; rotated the throwaway root console password; PTY console so virsh console works.

3. Formalized as a DECOUPLED Pulumi stack — runners/

New isolated project (peer to bootstrap/, provision/, offsite-backup/). Why decoupled (operator's explicit concern, and the answer to "is this a Pulumi problem like it was with Terraform"): a @pulumi/libvirt provider dials the runner host on every up/refresh, so putting it in bootstrap would make the foundation undeployable/unrefreshable whenever crunchy01 is down/unreachable. Pulumi isolates this at the stack boundary — bootstrap never imports runners/. One-way dependency: runners mints a token FROM the forge, so it's "step-0 after the foundation stands".

4. Live pulumi up cutover — DONE

Ran the crunchy stack live: created foundation-runner-02 (static 192.168.1.16, 8c/32G), registered the fenced runner, and a runs-on: fenced job ran GREEN on it. Then retired the hand-built VM (foundation-runner-01), so the Pulumi-managed runner-02 is the sole fenced runner. Bugs the live run surfaced (all fixed): NIC name isn't enp1s0 → match e*; drop qemuAgent:true (blocks on the agent at create); dialErrorLimit:30 for boot; fix the register token passing; host prep = root SSH + the images pool (crunchy has no default pool).

Current state

• foundation-runner-02 live on crunchy01 (192.168.1.16, 8c/32G), Pulumi-managed, label fenced, executing jobs. Its runner is a docker container in the VM.
• runners/ project committed to the foundation repo (index.ts, README, Pulumi.yaml, package.json). runners/Pulumi.crunchy.yaml + runners/state/ are gitignored (local to the operator workstation only — see Open threads: not backed up).
• Access: crunchy01 root@192.168.1.2 (operator key ~/.ssh/foundation-test_ed25519 now in root's authorized_keys; also andiolsi + sudo). libvirt installed; images pool; libvirt-bridge-forward.timer active. Forge VM root@204.168.234.72:222.
• Foundation repo master clean, all pushed. Forge admin platform-admin / Vault foundation/forgejo/service-credentials:forgejoAdminPassword.

NEW requirements from the operator (this session) — for the next agent

1. brix02 (192.168.1.3) runner with failover from crunchy01. Only when crunchy01 is unavailable should brix02 pick up jobs. Forgejo has no native standby: same-label runners load-balance, offline ones just get nothing. Two paths:
   • HA-on-outage (simple): register brix02 with the SAME fenced label — when crunchy is down, brix02 covers; when both up, they share load.
   • Strict standby (custom): keep brix02's runner STOPPED + a watchdog (systemd timer polling the Forgejo runners API) that starts it only when crunchy's runner is offline. The runners/ stack is already multi-host-capable via config — target brix02 with a second stack (pulumi stack init brix02, config set host.address 192.168.1.3, vm.name foundation-runner-03). First verify brix02 has KVM + libvirt + a bridge (same host prep as crunchy; brix02 is also the Graylog target — see memory).
2. k8s runner for heavy jobs. The seaspots GitLab pipelines (~/work/seaspots/gitlab/pipelines, .gitlab-ci.yml) run seaspots-s57-utils (registry.gitlab.com/seaspots/tools/seaspots-s57-utils:1.11.0: GDAL/ogr2ogr + tippecanoe) — tags: [heavy-compute], needs 16+ CPU / 64+ GB RAM / 100+ GB disk. These would crush the 8c/32G VM runner. The operator wants such heavy/containerized jobs to run on a Forgejo runner inside crunchy's k3s cluster (k8s-scheduled resources), with a distinct label (e.g. heavy/k8s). Design note: Forgejo act_runner executes jobs via docker or host mode — it has no mature native k8s executor like GitLab's. The next agent must evaluate: act_runner as a k8s Deployment (big resource requests) using host-mode or a DinD sidecar, vs. another approach. This is a design task, not yet started.

Open threads / backlog (from #002 + this session)

• runners stack state not backed up — only on the operator workstation (runners/state/, gitignored). A DR gap; consider backing it up like bootstrap's.
• Stale crunchy-runner registration on the forge (from the retired hand-built VM) — offline, harmless; deregister at leisure (Forgejo runners admin API/UI).
• Package registry (Stage-2) — publish @olsitec packages (olsicrypto, svelte-common) so the two registry-blocked 999_testing candidates (seaspots-homepage, token-service) build via reusable-docker-build.
• T15 — index.ts phase marker still T10-runner; Gate A/B comments; DAY-ZERO-TIMELINE.
• Hardening — pin floating image refs; pre-bake pulumi plugins into foundation-ci; MCP registration (D6); Forgejo v15 upgrade drops the reusable-workflow runs-on quirk.

Operating mode for next session: HIGH-RISK / INFRA (remote VMs, k3s, Docker, secrets).