foundation/documentation/sessions/HANDOVER.md

HANDOVER — next-session prompt (paste into a fresh context)

Living doc: overwritten each handover. Durable record = the dated SESSION_* files. Latest state = SESSION_2026-07-01_003.md (read it first, then #002 + #001).

---

Continue the olsitec-foundation build. You are the Lead Agent, HIGH-RISK / INFRA mode (remote VMs, k3s, Docker, secrets).

Required reads (in ~/work/olsitec-foundation/foundation/)

1. documentation/sessions/SESSION_2026-07-01_003.md ← runner fleet + the NEW asks below
2. documentation/sessions/SESSION_2026-07-01_002.md ← T14 + ecosystem CI · _001.md ← the egg
3. documentation/contracts/CONTRACT_001–004 + decisions/ADR_004,005,006,007 (ADR-007 first)
4. runners/README.md ← the decoupled runner-fleet stack (host prep, config, gotchas)
5. .forgejo/workflows/README.md ← the ecosystem-CI reusable-workflow contract (Forgejo-11 quirk)

Where things stand (all green / live)

• The egg is LIVE (6 containers on the Hetzner forge VM); T11/T13/T14 done; ecosystem CI (reusable workflows + selftest) green; https://forge.olsitec.net=200.
• The R5 fence is LIVE + codified. foundation-runner-02 (crunchy01 VM, 192.168.1.16, 8c/32G, label fenced) runs ecosystem/untrusted jobs OFF the forge VM. It's managed by the runners/ Pulumi stack — an isolated project (bootstrap never imports it), so foundation deploy/refresh never touches crunchy01. Stack = crunchy; config + state are gitignored (operator workstation only).
• Foundation repo master clean, all pushed.

THIS session's work (operator asks, in priority order)

1. brix02 runner with failover from crunchy01

Add a runner on brix02 (192.168.1.3) that picks up jobs only when crunchy01 is unavailable. Forgejo has no native standby — same-label runners load-balance; offline ones get nothing. Choose with the operator:

• HA-on-outage (simple, recommended): register brix02 with the SAME fenced label → when crunchy is down brix02 covers; when both up they share load.
• Strict standby (custom): brix02 runner kept STOPPED + a watchdog (systemd timer polling the Forgejo runners API) that starts it only when crunchy's runner is offline. The runners/ stack is multi-host-capable: cd runners && pulumi stack init brix02 && pulumi config set host.address 192.168.1.3 && ... vm.name foundation-runner-03. FIRST verify brix02 has KVM + libvirt + a LAN bridge (same host prep as crunchy — see runners/README §Host prep; brix02 is also the Graylog target, so it's a real box). Then pulumi up and prove a fenced job runs on it (and, for standby, that it idles while crunchy is up).

2. k8s runner for heavy (16CPU/64GB) jobs on crunchy's k3s

The seaspots GitLab pipelines (~/work/seaspots/gitlab/pipelines/.gitlab-ci.yml) run seaspots-s57-utils (ogr2ogr + tippecanoe; registry.gitlab.com/seaspots/tools/ seaspots-s57-utils:1.11.0), tags: [heavy-compute], needing 16+ CPU / 64+ GB RAM / 100+ GB disk — they'd crush the 8c/32G VM runner. Stand up a Forgejo runner inside crunchy's k3s cluster (k8s-scheduled resources) with a distinct label (e.g. heavy), so runs-on: heavy jobs run there. DESIGN TASK (not started): Forgejo act_runner executes via docker or host mode — no mature native k8s executor (unlike GitLab). Evaluate act_runner as a k8s Deployment with big resource requests (host-mode or a DinD sidecar) vs. alternatives. Note crunchy's k3s already runs the GitLab runners (namespace gitlab) — do not disturb them.

Operating essentials

• Forge VM: 204.168.234.72, SSH :222, key ~/.ssh/foundation-test_ed25519. Deploy: cd bootstrap && ./run.sh up. Passphrase: pass olsitec-foundation/PULUMI_CONFIG_PASSPHRASE. Forge admin: platform-admin / Vault foundation/forgejo/service-credentials:forgejoAdminPassword.
• crunchy01: root@192.168.1.2 (operator key in root's authorized_keys) OR andiolsi+sudo. libvirt installed; pool images; libvirt-bridge-forward.timer active (kube-router-proof). Runner fleet: cd runners; export RUNNER_SSH_KEY_PATH=~/.ssh/foundation-test_ed25519; export PULUMI_BACKEND_URL=file://$(pwd)/state; export PULUMI_CONFIG_PASSPHRASE=$(pass olsitec-foundation/PULUMI_CONFIG_PASSPHRASE); pulumi stack select crunchy.
• CI image: foundation-ci:latest, built on the forge VM (/tmp/ci-image); rebuild on toolchain change.
• Reuse mechanism: Forgejo 11 reusable workflows work but the CALLING job needs runs-on
  • SHORT cross-repo ref (.forgejo/workflows/README.md). Composite actions need FULL-URL.

Watchouts (HIGH-RISK)

• crunchy01 is a k3s node — the physdev-is-bridged FORWARD accept is what lets VMs reach the LAN; if a runner goes dark, check that rule / the timer first. Don't disturb k3s (gitlab namespace runners, nominatim, flannel/cni0).
• Never commit the passphrase / Vault root token / unseal keys. runners stack state lives only on the workstation (not backed up — a DR gap to address).
• Stale offline crunchy-runner registration on the forge (from the retired hand-built VM) — harmless; deregister at leisure. Don't pulumi up the prod olsicloud4-* stacks.

Standing backlog (after the two asks above)

• Package registry (Stage-2) — publish @olsitec pkgs so the C1/C5 docker candidates build.
• T15 — index.ts phase marker + Gate A/B comments + DAY-ZERO-TIMELINE.
• Hardening — pin floating image refs; pre-bake pulumi plugins; MCP (D6); Forgejo v15 upgrade; back up the runners stack state.

Validate each task live and commit atomically per task.