# Session 2026-06-30 #002 — Wave 2: data plane → forge → CI → backup (egg is LIVE)

## What was done
Built and deployed **all of Wave 2** live to the Helsinki VM (cx33, `204.168.234.72`,
SSH :222). The egg now runs **6 containers** and **`git clone git@git.olsitec.net:olsitec/foundation.git`
works**. Each task is a reviewable commit (atomic, conventional).

- **ADR-007** — control-plane ops via `@pulumi/command` `remote.Command` (docker-exec over SSH).
  Internal ports (PG 5432, Vault 8200, RustFS 9000) aren't published, so init/role/bucket/admin/token
  steps run inside the VM over the existing SSH path. Idempotent, readiness-gated, **secrets on stdin**
  (the command provider echoes the command on error → never inline; `environment` needs sshd AcceptEnv
  which the VM rejects). This is the cross-cutting mechanism for T03/T05/T06/T09/T10.
- **T03 postgres** — `foundation-postgres` (pg17), forgejo role+DB via remote.Command. `lib/remote.ts`
  (`vmConnection`) + `credentials.ts` generator half (CONTRACT_002).
- **T04 rustfs** — `foundation-rustfs` + 4 buckets + scoped service account (mc `svcacct add` works on
  RustFS; `mc ready` doesn't → gate on `mc ls`; mc busybox lacks grep → shell `case`). `IMAGE_MC` pinned.
- **T05 vault** — `foundation-vault` raft (`/vault/file`, IPC_LOCK). Init/unseal over docker-exec; keys
  emitted on stdout (secret, `logging:Stderr` so never streamed) → `run.sh` captures to
  `vaultCredentials:*`. `vault-unseal.sh` = passphrase-gated reboot helper (ADR-004). run.sh also pins
  the backend per-process (`PULUMI_BACKEND_URL`, no global `pulumi login`).
- **T06 credentials** — `writeCredentialsToVault` writes postgres+rustfs+forgejo service-credentials to
  the `foundation` kv-v2 mount via `vault kv put - ` (JSON on stdin). GATE A = `dependsOn vault.init`.
- **T07 caddy** — `foundation-caddy` public ingress (80/443), **DNS-01 TLS via Cloudflare** on a custom
  xcaddy image (`containers/caddy-cloudflare/Dockerfile`, `caddy-dns/cloudflare@v0.2.4`, built on the VM,
  image-id is the container image). Routes forge→Forgejo, s3→RustFS. Vault NOT proxied publicly.
- **T08 forgejo** — `foundation-forgejo` (fj11): external PG, RustFS blobs (default storage + LFS),
  config via `FORGEJO__` env. The image's openssh sshd owns container :22 (`START_SSH_SERVER=false`
  explicitly — a stale app.ini value crash-loops it on :22). HTTP 3000 via Caddy (200).
- **T09 forge bootstrap** — headless admin + org `olsitec` + auto-init repo `olsitec/foundation` +
  operator SSH key, all via docker-exec (`forgejo admin` CLI + the image's curl). **Opened firewall :22**
  (provision stack) so the scp-form clone works (VM admin sshd is on :222).
- **T10 runner** — `foundation-runner` (forgejo/runner:6). Idempotent register (token via
  `generate-runner-token`, never leaves the VM); daemon runs uid 1000 + host docker group (gid 996) for
  socket access. A hello-world `runs-on: docker` workflow ran to **success**.
- **T12 backup** — `backup/{backup,restore}.sh` + `*-remote.sh`. Bundle (pg_dumpall, forgejo repos
  tar.zst, vault raft snap, pulumi state, rustfs blobs, MANIFEST.json) → RustFS + **offsite** Synology
  bucket. `restore.sh` = non-destructive scratch-restore verifier — **PASSES from both rfs and offsite**.

## Current state
- Repo `~/work/olsitec-foundation/foundation`, branch `master`, latest commit = T12. Working tree clean.
- `cd bootstrap && ./run.sh up` is **idempotent — 41 unchanged**. Live containers: postgres, rustfs,
  vault, caddy, forgejo, runner (all healthy/up).
- Master passphrase: `pass olsitec-foundation/PULUMI_CONFIG_PASSPHRASE`. VM key
  `~/.ssh/foundation-test_ed25519` (also the registered Forgejo operator key).
- Verified: `https://forge.olsitec.net` = 200 (LE cert), `git clone git@git.olsitec.net:olsitec/foundation.git`
  (:22) **and** `ssh://…:2222/…` both clone; Vault paths populated; CI green; backup restorable offsite.

## Known gaps / next steps
- **age at-rest encryption** of backups (CONTRACT_004 §4.3) not yet applied — both backup destinations
  are private/access-controlled; generate the age key + encrypt-before-upload is the next hardening.
- **Determinism**: a `pulumi up --refresh` surfaces a spurious `foundation-net` `ipamConfigs` diff — do
  NOT apply it (recreating the network disconnects everything); plain `up` ignores it. Investigate before
  enabling refresh in CI.
- **Forgejo crypto secrets** (SECRET_KEY/INTERNAL_TOKEN/JWT) auto-generate in app.ini but aren't mirrored
  to Vault (`foundation/forgejo/service-credentials` has only admin user/pw). Capture them later.
- Runner is co-located + root-equivalent (host docker socket) — fence to a separate VM for untrusted CI
  (PLAN-002 R5). The docker gid (996) is host-specific — re-check on DR.
- Remaining PLAN-002 tasks: **T11** handover (push repo→Forgejo, switch origin), **T13** DR-to-fresh-VM,
  **T14** `.forgejo/workflows/`, **T15** index orchestration polish + DAY-ZERO checklist.

## Operating mode for next session: HIGH-RISK / INFRA (remote VM, Docker, secrets).