foundation/documentation/planning/PLAN-002-foundation-implementation.md

# PLAN-002 — `olsitec-foundation` Implementation Strategy (Master Roadmap)

> **Companion to** [PLAN-001-forgejo.md](PLAN-001-forgejo.md) (the vision) and
> [002_platform_architecture.md](002_platform_architecture.md) (the existing olsicloud4 K8s platform).
> **Status:** Draft for human ratification. **Mode at authoring:** EXPLORE (design only, no code).
> **Author role:** Lead platform architect. **Date:** 2026-06-30.
>
> This document is **not** an implementation. It is the strategy that AI agents execute.
> Confidence markers (High/Medium/Low) follow baseline PD-5.

---

## 0. The Pivotal Decision (read this first)

**PLAN-001 deploys Forgejo *onto Kubernetes* via ArgoCD + Helm. The foundation must NOT.**

The foundation is the **egg**: the thing every other platform is hatched from. Kubernetes,
ArgoCD, Helm, cert-manager and ESO are themselves *hatched* by the platform — so the foundation
cannot depend on them without creating an unrecoverable circular dependency
(DR-from-nothing would require rebuilding K8s, which needs git+registry+secrets, which *are* the
foundation).

### Recommendation — a layered platform (High confidence)

| Layer | What | Substrate | Lifecycle |
| ----- | ---- | --------- | --------- |
| **Layer 0 — `olsitec-foundation` (the egg)** | Forgejo (+ Actions + OCI/npm registry), PostgreSQL, Vault, RustFS, reverse proxy, 1 runner | **Plain OCI containers on ONE VM**, orchestrated by Pulumi `@pulumi/docker` over SSH. **No K8s/ArgoCD/Helm.** | `pulumi up` (manual day-zero → CI later) |
| **Layer 1+ — the olsicloud4 K8s platform & everything else** | K8s, ArgoCD, cert-manager, ESO, Authentik, Grafana/Prometheus, Longhorn, Renovate, additional registries | Kubernetes | **Consumes** Layer 0: repos in foundation-Forgejo, CI in foundation-Actions, images/charts in foundation-registry, secrets in foundation-Vault |

**Why this is correct and not a downgrade:**
- The existing repo *already* contains `pulumi/modules/docker/` (a `@pulumi/docker` SSH-to-host wrapper) and `pulumi/olsitec-core/run.sh` (Pulumi-initializes-Vault-then-captures-unseal-keys-back-into-passphrase-encrypted-config). The tooling is already pointed at this model. (High confidence — verified in source.)
- PLAN-001's K8s topology remains valid as a **future, optional HA path** for Forgejo (its "True HA is a step change" note). It is not thrown away — it is deferred to §8.

**Consequence:** Everywhere PLAN-001 says "StatefulSet / Helm / ArgoCD Application," Layer 0 reads "container + named volume / Pulumi `docker.Container` / Pulumi resource." The *data & state model* of PLAN-001 (git repos on a POSIX FS, Postgres, S3 for blobs) is unchanged and fully reused.

---

## 1. Architecture Review

### 1.1 Validated strengths of the vision
- **Forgejo as one binary** (forge + CI + OCI + npm + 20 registries) genuinely collapses GitLab's 4–5 services into one. (High) — confirmed in PLAN-001.
- **Single master passphrase as the only external secret** is achievable and already proven by `olsitec-core` (`PULUMI_CONFIG_PASSPHRASE` passphrase provider). (High)
- **Pulumi-owns-credentials / Vault-distributes** (ADR-002) is the right steady-state. (High)
- **Boring tech**: Postgres, Vault, S3, a reverse proxy, Docker containers. All well-understood. (High)

### 1.2 Weaknesses / risks identified

| # | Risk | Severity | Mitigation (see section) |
|---|------|----------|--------------------------|
| R1 | **Single VM = single point of failure.** Forgejo is irreducibly stateful (git repos on FS). | High | Frequent backups to RustFS + **offsite**; DR rebuild ≤ 1h, tested (§6). HA is Layer-1 future (§8). |
| R2 | **Vault auto-unseal paradox** — unattended reboot leaves Vault sealed; auto-unseal normally needs an external KMS (a SaaS or a second Vault). | High | Shamir unseal; keys held in passphrase-encrypted Pulumi config; passphrase-gated unseal helper (§4, §9). |
| R3 | **RustFS maturity.** RustFS is a young MinIO-compatible S3. Foundation backups depend on it. | Medium | Keep S3 usage to the documented S3 API surface; **never** make RustFS the *only* copy of backups (offsite replica is non-S3-only). Treat RustFS as replaceable behind the S3 boundary. (Medium confidence on RustFS stability — flag for second-opinion.) |
| R4 | **Pulumi state location before infra exists** (chicken-egg). | Medium | Local file backend during bootstrap → migrate to RustFS S3 backend after; state backed up offsite (§5, §9). |
| R5 | **Privileged runner.** Forgejo Actions docker backend needs a privileged daemon. | Medium | Runner on a **throwaway sidecar VM** (or same VM, contained), never sharing the forge's trust boundary (§4a of PLAN-001 reused). |
| R6 | **DinD/runner pulls from Docker Hub** → rate limits + SaaS dependency for CI base images. | Medium | Pull-through cache → mirror critical images into Forgejo's own OCI registry; pin by digest (§7, §8). |
| R7 | **TLS day-zero**: ACME needs DNS resolving + reachability before the service is public. | Medium | DNS-01 via existing Cloudflare token (already in platform) OR reverse-proxy internal CA for day-zero, swap to real certs once DNS resolves (§4 certs). |
| R8 | **Backup encryption keys / offsite creds** become a *second* must-survive secret. | Medium | Fold offsite + backup credentials into the same passphrase-encrypted config / Vault; never a bare file (§4, §6). |
| R9 | **Forgejo Actions feature-completeness vs GitLab CI** for existing pipelines (Kaniko, semantic-release, helm push). | Low | PLAN-001 already mapped every job → `runs-on: docker`. Reuse that mapping. (High) |

### 1.3 Hidden dependencies to make explicit
- **DNS** must resolve `forge.olsitec.de` (and friends) to the VM **before** TLS and **before** self-hosting handover. Who owns the zone? (Cloudflare, per existing platform.) → §9 Networking.
- **An SSH key** trusted by the VM is needed for Pulumi's Docker-over-SSH provider. That key's trust is a day-zero identity question (§9 Identity).
- **Container images** are an external dependency until mirrored. Pin by **digest** for determinism (§Determinism).
- **The operator workstation** is an implicit trusted host for the very first `pulumi up`. Its toolchain must be validated (preflight, §2).

### 1.4 Suggested additions / changes to the component list
- **Add a reverse proxy with automatic TLS** → recommend **Caddy** (auto-ACME, ~10-line Caddyfile, internal-CA fallback). Alternative: Traefik. nginx if maximum-boring is required but loses auto-TLS ergonomics. (Medium — Caddy recommended.)
- **Add a Docker Hub pull-through cache** (`registry:2`) at Layer 0 from day one (PLAN-001 component #6) — removes a SaaS rate-limit dependency for CI. (Medium)
- **Defer Valkey/Redis** — single-replica Forgejo needs no external queue/cache (PLAN-001 confirms). Add only with HA. (High)
- **Defer Meilisearch** — search is not foundational. (High)
- **Keep `@pulumi/random` for all credential generation** (reuse existing pattern). (High)
- **Vault PKI engine** becomes the internal CA in §8 (replacing Caddy's bootstrap internal CA).

---

## 2. Bootstrapping Strategy (empty VM → operational)

Phases are deployed by **one Pulumi project** with explicit ordering (component dependencies + a small number of phase gates). See §5 for the dependency graph and §9 for the full timeline.

```
Phase 0  PROVISION   Bare VM (Hetzner) + cloud-init: docker engine, ssh key, firewall.
Phase 1  PREFLIGHT   Cloned repo validates host+toolchain (pulumi, node/bun, docker, ssh, dns, age/gpg).
Phase 2  STATE+TRUST Pulumi local file backend; master passphrase set (PULUMI_CONFIG_PASSPHRASE via `pass`).
Phase 3  DATA PLANE  Docker network + PostgreSQL + RustFS (sealed/empty Vault container also started).
Phase 4  VAULT INIT  `vault operator init` → capture root token + unseal keys → write back to passphrase-
                     encrypted Pulumi config (PROVEN pattern, olsitec-core/run.sh) → unseal.
Phase 5  CREDENTIALS @pulumi/random generates all service creds → written to Vault KV v2 → RustFS buckets
                     created → Postgres roles/DBs created.
Phase 6  FORGE       Reverse proxy + Forgejo (app.ini rendered with secrets from Vault/Pulumi) come up;
                     Forgejo install-lock + first admin created deterministically.
Phase 7  HANDOVER    Push the foundation repo INTO Forgejo; switch git origin; create org + mirror infra
                     repos; register first Actions runner (token from Vault).
Phase 8  CI HANDOFF  A `.forgejo/workflows/` pipeline runs `pulumi preview` (then `up` on approval).
Phase 9  BACKUP+DR   First backup taken (forgejo dump + pg_dump + vault snapshot + pulumi state) → RustFS
                     → offsite. DR rebuild rehearsed on a fresh VM.
```

**Phase gates (only where strictly required):**
- Gate A after Phase 4: Vault must be initialized+unsealed before Phase 5 writes secrets.
- Gate B after Phase 6: Forgejo must be healthy before Phase 7 handover.
Everything else flows through ordinary Pulumi resource dependencies — no extra gates.

---

## 3. Repository Structure

A **single repo** = the DR unit. `git clone` + master passphrase ⇒ rebuild.

```
olsitec-foundation/
├── README.md                      # 5-line quickstart + DR pointer
├── VERSIONS                       # pinned versions+digests for every image & tool (determinism)
├── preflight/
│   ├── preflight.sh               # validates tools, versions, ssh, dns, docker reachability
│   └── checks/                    # individual check scripts (composable, testable)
├── pulumi/
│   ├── Pulumi.yaml                # single project
│   ├── Pulumi.foundation.yaml     # stack: passphrase-encrypted config + secrets (committable)
│   ├── index.ts                   # phase orchestration entrypoint
│   ├── config.ts                  # typed config schema (CONTRACT_001)
│   ├── components/                # one ComponentResource per concern
│   │   ├── network.ts             # docker network, firewall expectations
│   │   ├── postgres.ts
│   │   ├── rustfs.ts              # + bucket provisioning
│   │   ├── vault.ts               # container + init/unseal capture lib
│   │   ├── credentials.ts         # @pulumi/random → Vault writer (CONTRACT_002 paths)
│   │   ├── proxy.ts               # Caddy + TLS strategy
│   │   ├── forgejo.ts             # app.ini render, install-lock, first admin
│   │   └── runner.ts              # act_runner + registration-token flow
│   ├── phases/                    # thin orchestrators: dataPlane(), vaultInit(), forge(), handover()
│   └── lib/                       # vaultInitCapture(), renderTemplate(), digest pinning helpers
├── containers/                    # Dockerfiles for anything we build/mirror ourselves
├── config/                        # rendered template SOURCES: app.ini.tmpl, Caddyfile.tmpl, pg-init.sql
├── backup/
│   ├── backup.sh                  # forgejo dump + pg_dump + vault snapshot + pulumi state → RustFS → offsite
│   └── restore.sh                 # inverse, parametrized by target host
├── dr/
│   ├── RUNBOOK.md                 # human-readable DR procedure
│   └── restore-to-fresh-vm.sh     # automated rebuild used by the DR rehearsal test
├── docs/
│   ├── decisions/                 # ADRs (ADR_F001 layered-platform, etc.)
│   ├── DAY-ZERO-TIMELINE.md       # §9 timeline as an executable checklist
│   └── contracts/                 # CONTRACT_001..004 (§10)
├── .forgejo/workflows/            # CI: preflight.yml, pulumi-preview.yml, pulumi-up.yml, backup-verify.yml
└── .gitignore                     # state/ (local backend), node_modules, *.local
```

**Why this layout (High confidence):**
- **One repo = one DR unit.** Vision requirement: "freshly cloned repo capable of pre-flight validation."
- **`components/` mirror the deployment order** so an agent can own one file with a clear contract.
- **`config/` holds template *sources*, never rendered secrets** — rendered output carries secrets and stays in container/Vault only (PD-2: don't version secrets).
- **`VERSIONS` centralizes determinism** — preflight and CI both read it; upgrades are a one-line diff.
- **`.forgejo/workflows/` co-located** so the repo that defines CI is the repo CI deploys (self-hosting).

---

## 4. Secret Management

### 4.1 Root of trust
**The master passphrase** (`PULUMI_CONFIG_PASSPHRASE`) is the single root. It selects Pulumi's
`passphrase` secrets provider (already in use: `encryptionsalt` in `Pulumi.olsitec-core.yaml`).
Chain of trust:

```
Master passphrase
  └─ decrypts Pulumi stack config secrets (committable, `secure: v1:…`)
       └─ which hold Vault unseal keys + root token (captured at init)
            └─ Vault becomes the runtime distribution layer for ALL other secrets (ADR-002)
```

The passphrase is the **only** thing a human must carry out-of-band. Store it in `pass`
(operator side), and/or split it among operators with Shamir, and/or a hardware token. It is
never written to the platform.

### 4.2 Credential generation — deterministic vs random

| Class | Examples | Source | Rationale |
|-------|----------|--------|-----------|
| **Random / high-entropy** | all service passwords, Postgres pw, RustFS access/secret keys, Forgejo `SECRET_KEY` + `INTERNAL_TOKEN` + JWT secrets, OCI/npm registry tokens, runner registration token, Forgejo admin password | `@pulumi/random` → Vault KV v2 | secrets must be unguessable; rotation = `--replace` |
| **Derived / deterministic** | usernames, DB names, bucket names, container/DNS names, Vault mount layout, hostnames | computed from typed config | reproducible, non-secret, no entropy needed |
| **External (the ONLY one)** | master passphrase | human | root of trust |

This satisfies the vision: *"everything else should derive from that."*

### 4.3 Vault initialization & unseal (the hard part — High attention)
- **Init:** Pulumi runs `vault operator init` (Shamir, e.g. 5 keys / threshold 3) inside the Vault
  container, captures `unsealKeys` + `rootToken` as **stack outputs**, then `run.sh` (or a Pulumi
  `local.Command`) writes them back as passphrase-encrypted Pulumi config secrets. **This exact
  pattern already exists in `olsitec-core/run.sh`** — reuse it verbatim. (High confidence.)
- **Unseal on reboot (R2):** Vault seals on every restart. Options:
  1. **Passphrase-gated unseal helper** *(recommended)* — a small script reads the unseal keys from
     Pulumi config (decrypted by the passphrase the operator provides) and unseals. Deterministic,
     reproducible, **no external KMS, no SaaS**. Cost: VM reboots need an operator (or a
     passphrase made available to a boot service — a trade-off to decide).
  2. **Transit auto-unseal** — rejected at Layer 0 (needs a *second* Vault → circular).
  3. **KMS auto-unseal** — rejected (SaaS dependency, violates design goal).
  → Recommend (1) for Layer 0; revisit auto-unseal when a second trust anchor exists at Layer 1.
  (Medium confidence — this is the main open operational question; flag for second-opinion.)

### 4.4 Rotation
Per ADR-002: `pulumi up --replace` on the `RandomPassword` → new value in Vault → consumers reload.
At Layer 0, consumers are containers, so rotation triggers a container recreate (Pulumi handles the
dependency). Vault root token: rotate via `vault operator generate-root` after bootstrap; store new
token in config. Unseal-key rotation: `vault operator rekey`.

### 4.5 Recovery & backup of secrets
- **Vault data** backed up via `vault operator raft snapshot` → RustFS → offsite.
- **Unseal keys + root token** survive inside the passphrase-encrypted Pulumi stack config, which is
  in the repo (and the repo is backed up). So {repo + passphrase} reconstitutes Vault access.
- **Backup/offsite credentials (R8)** live in Vault *and* are mirrored into the passphrase-encrypted
  config, so they survive even total Vault loss.

---

## 5. Deployment Order & Dependency Graph

```
                       ┌─────────────────────────┐
                       │  master passphrase (ext) │
                       └───────────┬─────────────┘
                                   │ selects secrets provider
                            ┌──────▼───────┐
                            │ Pulumi state │  (local file backend → later RustFS S3)
                            └──────┬───────┘
                                   │
        ┌──────────────┬──────────┼───────────┬──────────────┐
        ▼              ▼          ▼            ▼              ▼
   docker network  PostgreSQL  RustFS    Vault(sealed)   Caddy(proxy)
        │              │          │            │              │
        │              │          │     [Gate A: init+unseal] │
        │              │          │            │              │
        │              │          │      Vault(unsealed)      │
        │              │          │            │              │
        │              └──────────┴─────┐ credentials.ts      │
        │                               ▼ (@pulumi/random→Vault)
        │                          Postgres roles/DBs, RustFS buckets created
        │                               │
        └───────────────────────────────┼──────────────────────┐
                                         ▼                       │
                                    Forgejo (app.ini ← Vault) ◄──┘ (proxy routes to it)
                                         │  [Gate B: healthy]
                                         ▼
                                  first admin + org + repos
                                         │
                                         ▼
                                  act_runner (token ← Vault)
                                         │
                                         ▼
                                  CI assumes deploy duty
```

### 5.1 What depends on what
- **Everything** depends on Pulumi state + passphrase.
- **credentials.ts** depends on Vault being unsealed (Gate A) and on Postgres/RustFS existing (to create roles/buckets with the generated creds).
- **Forgejo** depends on Postgres (DB), RustFS (blob storage), Vault (secrets), proxy (TLS/ingress).
- **Runner** depends on Forgejo (registration token) and on the proxy (to reach Forgejo).
- **CI** depends on the runner.

### 5.2 Circular dependencies & resolutions (summary; full list §9)
| Cycle | Resolution |
|-------|-----------|
| Pulumi needs a secret store; Vault is that store; Vault is deployed by Pulumi | Passphrase-encrypted config holds unseal keys at bootstrap; Vault holds the rest in steady state. |
| Forgejo hosts the repo that deploys Forgejo | Deploy Forgejo from the **local clone** first; then push repo in + switch origin (handover). |
| CI deploys the platform; CI runs on the platform | First `pulumi up` is **manual**; CI takes over only after the runner exists and a self-rebuild is proven. |
| Registry hosts CI base images; CI fills the registry | Pull from upstream via pull-through cache day-zero; mirror into Forgejo registry afterward. |
| TLS needs DNS+ACME; ACME account must be created | DNS-01 via existing Cloudflare token, or internal CA day-zero; real certs once DNS resolves. |

---

## 6. Disaster Recovery (total VM loss)

**Premise:** survive on {a VM, the repo, the master passphrase}.

### 6.1 What must exist to recover
1. **The repo** (git clone — mirrored offsite, see below).
2. **The master passphrase** (operator's head / `pass` / Shamir split).
3. **The latest backup bundle** in the **offsite** location: `forgejo-dump.zip`, `pg_dump.sql`,
   `vault-raft.snap`, `pulumi-state` (if not reconstructible), `rustfs-data` (or it is the offsite).

### 6.2 Procedure (target ≤ 1 hour; `dr/restore-to-fresh-vm.sh` automates most)
1. Provision a fresh VM (Phase 0 cloud-init).
2. `git clone` foundation repo; run `preflight/`.
3. Set `PULUMI_CONFIG_PASSPHRASE`; `pulumi login` (local backend) or restore state from offsite.
4. `pulumi up` Phases 3–4: data plane + Vault container. **Restore Vault** from raft snapshot
   (`vault operator raft snapshot restore`); unseal with keys from config.
5. **Restore Postgres** (`pg_restore`) and **RustFS data** (sync from offsite) before starting Forgejo.
6. `pulumi up` Phase 6: Forgejo against restored DB + restored data dir (git repos).
7. Re-register the runner (new token) — runners are stateless, never restored.
8. Validate: clone a repo, run a pipeline, push an image, read a Vault secret.

### 6.3 What is recreatable and **not** backed up
- Container images (re-pullable / rebuildable from pinned digests).
- Search indexes (Forgejo rebuilds).
- Caches, runner ephemeral state, pull-through cache contents.
- Pulumi state *if* the local backend is reconstructible — but back it up anyway (cheap insurance).

### 6.4 Offsite requirement (critical)
RustFS lives on the same VM → it cannot be the only backup copy (R3). Replicate the backup bundle to
a **second location with a different failure domain** that is **not SaaS by hard dependency**:
recommend a second small Hetzner VM / Storage Box in another DC, or a second self-hosted RustFS.
(If a SaaS S3 is used, it must be *additive*, never the sole copy — preserving the no-SaaS guarantee.)

---

## 7. Operational Lifecycle

### 7.1 Upgrades
- Bump the pinned **digest** in `VERSIONS` → PR → CI `pulumi preview` posts the plan → human approves
  → CI (or manual for Vault/Postgres major versions) `pulumi up`.
- **Snapshot before** every Forgejo/Postgres/Vault upgrade (PD-4): take a backup bundle first.
- Sequence: never upgrade Postgres and Forgejo in the same change; Vault upgrades are isolated.

### 7.2 Backups
- **`backup/backup.sh` on a timer** (systemd timer or Forgejo Actions scheduled workflow):
  `forgejo dump` (repos+metadata) + `pg_dump` + `vault raft snapshot` + `pulumi state export` →
  RustFS bucket `foundation-backups` → replicate offsite.
- **Verify** restorability weekly (`.forgejo/workflows/backup-verify.yml` restores into a scratch
  container and asserts row counts / repo presence). A backup that has never been restored is a guess.
- First backup is part of bootstrap (Phase 9) — **before** declaring the platform operational.

### 7.3 Monitoring & alerting
- **Bootstrap → minimal:** container healthchecks + an external uptime probe (offsite).
- **Layer 1:** Prometheus/Grafana (on K8s) scrape the foundation node-exporter + Forgejo `/metrics`.
- **Alerting trust rule:** the alerter must **not** run on the only host it watches. Put uptime/alert
  offsite so a dead VM can still page. (High confidence — common self-hosting footgun.)

### 7.4 Maintenance & the self-hosting milestone
- **Self-hosting is reached when** (all true): foundation repo lives in Forgejo; CI can `pulumi up`
  the foundation; a DR rebuild has succeeded end-to-end from offsite backup.
- After that, **all changes flow through Git + CI**, with manual `pulumi up` reserved as the documented
  break-glass for Layer-0-breaking changes (e.g., Vault/Postgres major upgrades).

---

## 8. Future Expansion (how Layer 1+ integrates)

Every future service integrates through the **same four foundation interfaces**, never bypassing them:
**(1) source repo in Forgejo, (2) images/charts in Forgejo's OCI registry, (3) secrets in Vault,
(4) CI in Forgejo Actions.** This keeps the egg the single root for everything.

| Service | Integration path | Notes |
|---------|------------------|-------|
| **Kubernetes** | Provisioned by a *new* Pulumi project whose repo lives in foundation-Forgejo; pulls images from foundation-registry; secrets from foundation-Vault. | This is where the **existing olsicloud4 K8s platform** reconnects — as a Layer-1 consumer. |
| **ArgoCD** | Deployed on K8s; its app repos are Forgejo repos; bootstrap secret (git token) from Vault. | Replaces gitlab.com source in `002_platform_architecture.md` with Forgejo. |
| **Internal PKI** | **Vault PKI secrets engine** becomes the org CA, replacing Caddy's bootstrap internal CA. cert-manager (Layer 1) uses the Vault issuer. | Promotes day-zero self-signed → real internal trust. |
| **Authentik (SSO/OIDC)** | Deployed at Layer 1; Forgejo, Grafana, ArgoCD become OIDC clients. Introduce SSO **after** the platform is stable — not day-zero (avoid an identity dependency in the egg). | Forgejo can also *be* a temporary OIDC provider before Authentik exists. |
| **Grafana / Prometheus** | Layer 1; scrape foundation + cluster; dashboards-as-code in Forgejo. | §7.3. |
| **Longhorn** | Layer-1 storage for stateful K8s workloads — **not** used by Layer 0 (Layer 0 uses host volumes). | Keeps the egg storage-simple. |
| **Renovate** | Self-hosted runner job in Forgejo Actions; opens PRs against `VERSIONS` and chart repos. | Automates §7.1 digest bumps. |
| **Additional registries** | Forgejo's bundled registries cover OCI/npm/Helm/+20; add Harbor only if policy/scanning demands it. | Prefer not adding parts. |

**Migration note:** the existing platform's gitlab.com dependency (git + OCI registry at
`registry.gitlab.com/olsitec-nci/charts`, ADR-002 paths under `olsicloud4/...`) is **retired** by
pointing those repos/registries at foundation-Forgejo. That migration is its own plan, gated on the
foundation being proven.

---

## 9. Bootstrap Paradoxes & Day-Zero Analysis

For each: *why it exists · what depends on what · automatable? · solution · deterministic?*

### 9.1 Infrastructure
- **First VM provisioning.** Paradox: Pulumi provisions infra, but the VM hosts Pulumi's target.
  → A **thin separate Hetzner Pulumi project** (already exists: `pulumi/hetzner-cloud`) or one
  cloud-init creates the VM + installs Docker + plants the operator SSH key. Automatable: **yes**.
  Deterministic: yes (image + cloud-init pinned). The VM is the one piece provisioned *before* the
  foundation Pulumi runs.
- **Pulumi's first credentials.** It needs (a) SSH to the VM, (b) the master passphrase. SSH key is
  the day-zero identity (§9.3); passphrase is the root of trust (§4). No other credential needed —
  everything else is generated. Deterministic: yes.
- **Pulumi state before infra exists (R4).** → **Local file backend** on the operator machine during
  bootstrap; migrate to RustFS S3 backend after Phase 5; back up state offsite. Automatable: yes.
  Deterministic: yes (state is data, not derived, so it is *backed up*, not regenerated).
- **First clone of the repo.** Before Forgejo exists the repo lives… somewhere external (operator
  workstation + an offsite git mirror — e.g. a bare repo on the backup host, or temporarily
  gitlab.com during migration). After handover, Forgejo is canonical. Automatable: partially (the
  very first clone is operator action). Deterministic: yes (content-addressed git).
- **Binary installation.** `preflight/` checks; a pinned installer script fetches exact versions from
  `VERSIONS`. Automatable: yes. Deterministic: yes (pinned).
- **Host validation.** `preflight/preflight.sh` asserts tool versions, docker reachability, ssh, dns,
  disk, clock. Fails closed before any deploy. Automatable: yes.

### 9.2 Secrets & Trust
- **Root of trust:** master passphrase (§4.1). **Minimal external secret:** that passphrase, nothing
  else.
- **Vault init / unseal keys / initial creds:** §4.3 — proven `olsitec-core` capture pattern.
- **Deterministic vs random creds:** §4.2.
- **Rotation / recovery after total loss:** §4.4 / §4.5 + §6.

### 9.3 Identity
- **First administrator:** created **non-interactively** by Pulumi via `forgejo admin user create`
  (container exec) or `FORGEJO__security__INSTALL_LOCK=true` + env, with an admin password from
  `@pulumi/random` → Vault. No human types a password into a web form. Automatable: **yes**.
  Deterministic: the *flow* is; the password is random-but-stored. (High confidence — Forgejo
  supports headless admin creation.)
- **First admin authentication:** operator reads the generated admin password from Vault (passphrase
  → Vault). No default/weak credential ever exists.
- **First SSH key trusted:** the operator key is planted by cloud-init (Phase 0) — this is the
  irreducible day-zero trust seed. Subsequent keys are managed in Forgejo.
- **Service identities:** each service gets its own Vault path + (later) AppRole, mirroring ADR-002.
- **OIDC/SSO:** introduce at Layer 1 (§8), **not** day-zero — avoids an identity dependency inside the
  egg.

### 9.4 Certificates & Networking
- **Initial TLS:** DNS-01 ACME via the **existing Cloudflare token** (already in the platform per
  `002_platform_architecture.md`), issued by Caddy — works even before the host is publicly
  reachable. Fallback: Caddy internal CA for day-zero, swap to real certs once DNS resolves.
- **Internal PKI:** not required day-zero; Vault PKI adopts it at Layer 1 (§8).
- **Cert rotation:** Caddy auto-renews ACME; Vault PKI handles internal rotation later.
- **DNS assumptions:** `forge.olsitec.de` (+ registry/host) **must resolve to the VM before handover**.
  Owner: Cloudflare zone. This is a hard prerequisite — list it in preflight.
- **Reverse proxy bootstrap:** `Caddyfile` rendered from template by Pulumi; routes web/API/registry on
  one host; Git-over-SSH exposed directly (port 22/2222) not via the HTTP proxy.

### 9.5 Forgejo
- **First repository / first commit / repo arrival:** the foundation repo is pushed from the local
  clone into Forgejo at **Phase 7 handover**; origin is switched to Forgejo; this is the
  self-hosting moment. Automatable: yes (scripted `git remote` + `git push`).
- **First CI runner & registration token:** token generated via
  `forgejo actions generate-runner-token` (or admin API) → stored in Vault → consumed by `runner.ts`.
  Automatable: **yes**. Deterministic flow.
- **When CI owns deployments:** only after handover + runner registration + a proven self-`pulumi up`.
  Until then, manual `pulumi up` (§5.2, §7.4).

### 9.6 Storage
- **Postgres init:** container with generated superuser pw; `pg-init.sql` creates Forgejo role+DB.
  Automatable: yes.
- **RustFS init:** container with generated admin keys; `credentials.ts` creates service keys +
  buckets (`forgejo-packages`, `forgejo-artifacts`, `forgejo-lfs`, `foundation-backups`).
  Automatable: yes.
- **Bucket creation:** Pulumi (S3 provider against RustFS) — deterministic names.
- **Restore order after DR:** Vault → Postgres → RustFS data → **then** Forgejo (§6.2). Git repos
  (Forgejo data dir) are the irreplaceable core; restore before starting Forgejo.
- **Recreatable data:** images, indexes, caches (§6.3).

### 9.7 Backups & Recovery
- **First backup:** Phase 9, before "operational" is declared.
- **Where stored:** RustFS `foundation-backups` + offsite replica (§6.4).
- **Backup credential protection:** in Vault + mirrored to passphrase-encrypted config (R8/§4.5).
- **Required to recover everything:** repo + passphrase + {forgejo dump, pg_dump, vault snapshot,
  pulumi state}. **Disposable:** images, indexes, caches, runner state (§6.3).

### 9.8 Operations
- **Monitoring enabled:** minimal at bootstrap, full at Layer 1 (§7.3).
- **Alerting trusted:** only when it runs offsite (§7.3).
- **Upgrades before CI exists:** manual `pulumi up` with a pre-snapshot (§7.1).
- **Becomes self-hosting / all-changes-through-CI:** §7.4 milestone.

### 9.9 Chronological Day-Zero Timeline

```
T0  Fresh OS         Hetzner VM created (cloud-init: docker, ssh key, firewall, clock sync).
T1  First command    operator: git clone olsitec-foundation && ./preflight/preflight.sh
T2  Trust set        export PULUMI_CONFIG_PASSPHRASE (via pass); pulumi login (local file backend).
T3  Infra deploy     pulumi up → docker network + Postgres + RustFS + Vault(sealed) + Caddy.
T4  Secret init      vault operator init → capture keys → write to passphrase-encrypted config → unseal.
T5  Credentials      @pulumi/random → Vault; Postgres roles/DBs; RustFS keys+buckets.
T6  Services init    Forgejo up (app.ini ← secrets); headless first admin created.
T7  Operational      Web/API/registry reachable over TLS; admin password readable from Vault.
T8  Self-hosting     push foundation repo → Forgejo; switch origin; create org; register runner.
T9  First CI deploy  .forgejo/workflows runs pulumi preview → (approve) → up. CI now owns changes.
T10 Backup           backup.sh → RustFS → offsite. (first bundle)
T11 DR validated     restore-to-fresh-VM.sh rebuilds on a clean VM from offsite backup; smoke tests pass.
```

Goal achieved: **every step T1–T11 is scripted**; the only human actions are providing the passphrase
and approving the first CI deploy. No undocumented manual step remains.

---

## 10. AI Execution Plan

Work is split into low-coupling tasks. **Contracts are written first** (baseline §9) so tasks
parallelize without inventing incompatible interfaces. Each task: reviewable commit, explicit
acceptance criteria, conventional-commit subject.

### 10.0 Contracts (write before implementation tasks)
| Contract | Defines | Consumed by |
|----------|---------|-------------|
| **CONTRACT_001 — Config schema** | typed Pulumi config keys (hostnames, versions, sizes, feature flags) | every component |
| **CONTRACT_002 — Vault path layout** | `foundation/<service>/<type>-credentials` keys (camelCase, ADR-002 style) | credentials, forgejo, runner, backup |
| **CONTRACT_003 — Container network & DNS names** | network name, container names, internal ports | network, all services, proxy |
| **CONTRACT_004 — Backup artifact format** | bundle filenames, layout, restore order | backup, dr, backup-verify |

### 10.1 Tasks

| ID | Task | Depends on | Parallel? | Acceptance criteria |
|----|------|-----------|-----------|---------------------|
| **T00** | Contracts CONTRACT_001–004 + ADR_F001 (layered platform) | — | — | 4 contract docs + ADR committed; reviewed by human. |
| **T01** | Repo scaffold + `preflight/` + `VERSIONS` | T00 | yes | `preflight.sh` exits non-zero on any missing/mismatched tool; passes on a prepared host. |
| **T02** | Pulumi project skeleton + passphrase backend + `config.ts` (CONTRACT_001) | T00 | yes | `pulumi preview` runs with empty stack; config schema typed; secrets provider = passphrase. |
| **T03** | `network.ts` + `postgres.ts` | T02, C003 | yes | Postgres container up via `@pulumi/docker`; role+DB created; healthcheck green. |
| **T04** | `rustfs.ts` + bucket provisioning | T02, C002/C003 | yes | RustFS up; 4 buckets created; service key can put/get an object. |
| **T05** | `vault.ts` + `lib/vaultInitCapture` (reuse olsitec-core pattern) | T02 | yes | Vault inits; keys+root captured into encrypted config; unseal helper unseals after restart. |
| **T06** | `credentials.ts` (@pulumi/random → Vault, CONTRACT_002) | T05 | no (needs Vault) | All credential keys present in Vault at correct paths; idempotent on re-run. |
| **T07** | `proxy.ts` (Caddy) + TLS strategy (DNS-01 + internal-CA fallback) | T02, C003 | yes | HTTPS terminates for `forge.*`; cert from Let's Encrypt (or internal CA in dev). |
| **T08** | `forgejo.ts` — app.ini render, install-lock, S3+DB+Vault wiring | T03,T04,T06,T07 | no | Forgejo healthy; uses external Postgres + RustFS; web/API reachable via proxy. |
| **T09** | Forgejo headless first-admin + org + repo bootstrap | T08 | no | Admin created non-interactively; password in Vault; org exists; no default creds. |
| **T10** | `runner.ts` — registration-token flow + act_runner | T08,T09 | no | Runner registers via Vault token; a hello-world workflow runs to success. |
| **T11** | Self-hosting handover script (push repo, switch origin, mirror infra repos) | T09 | no | Foundation repo present in Forgejo; origin switched; `git push` works over SSH. |
| **T12** | `backup/` (backup.sh + restore.sh, CONTRACT_004) | T08 | yes | Bundle written to RustFS + offsite; restore.sh reconstructs into a scratch env. |
| **T13** | `dr/` runbook + `restore-to-fresh-vm.sh` | T12 | no | Automated rebuild on a clean VM passes smoke tests (clone, pipeline, registry push, vault read). |
| **T14** | `.forgejo/workflows/` (preflight, pulumi preview, pulumi up, backup-verify) | T10,T11 | yes | preview workflow posts plan; up workflow gated on approval; backup-verify restores+asserts. |
| **T15** | `index.ts` phase orchestration + Gate A/B + DAY-ZERO checklist | T03–T08 | no | `pulumi up` from empty → operational in one command (modulo passphrase + approval). |

### 10.2 Parallelization map
- **Wave 1 (parallel):** T01, T02 (after T00 contracts).
- **Wave 2 (parallel):** T03, T04, T05, T07 (all depend only on T02 + contracts).
- **Wave 3:** T06 (needs T05) ∥ start T12 design.
- **Wave 4:** T08 (integrates T03/04/06/07).
- **Wave 5:** T09 → T10 → T11 (sequential handover chain) ∥ T12 impl.
- **Wave 6:** T13, T14, T15.

### 10.3 Per-task prompt skeleton (baseline §7.1)
Each agent prompt must carry: Mission · Mode (BUILD or HIGH-RISK/INFRA) · the relevant **CONTRACT_00x** ·
the component file it owns · Non-goals (don't touch other components, don't edit generated/rendered
secrets, don't run `pulumi up` against the real VM without approval) · Acceptance criteria (above) ·
Escalation (stop if Vault/state/secret behavior diverges from this plan).

---

## Ratified Decisions (2026-06-30)

These four were decided by the human and are now binding (see ADR_004):

1. **Layered platform — RATIFIED.** Layer 0 = bare Docker on one VM via Pulumi; K8s/ArgoCD demoted
   to a Layer-1 consumer (§0). The whole plan stands on this.
2. **Vault unseal — passphrase-gated helper (§4.3 option 1).** No external KMS, no SaaS. Reboots
   require the master passphrase to be made available to the unseal step. Auto-unseal stays off until
   a Layer-1 trust anchor exists.
3. **Object storage — RustFS primary (§4 R3).** RustFS is the Layer-0 S3, matching the existing
   `rustfs` credential flag. **Hard rule:** the offsite replica is **non-RustFS**, so RustFS is never
   the only copy of a backup.
4. **Offsite backup — second self-hosted location (§6.4).** Different DC/failure domain, **no SaaS**
   dependency. Preferred seed: reuse `pulumi/hetzner-cloud` for both the Phase-0 VM and the offsite
   host.

### Remaining minor (reversible defaults — proceeding unless you object)
- **Reverse proxy:** defaulting to **Caddy** (auto-TLS, internal-CA fallback). Cheap to swap later.
- **Phase-0 VM seed:** defaulting to **`pulumi/hetzner-cloud`** for the foundation VM + the offsite host.

---

## Appendix — Mapping PLAN-001 → this plan
- PLAN-001 "StatefulSet/Helm/ArgoCD" → Layer-0 "container/named-volume/Pulumi resource."
- PLAN-001 data/state model (git on FS, Postgres, S3-for-blobs) → **reused unchanged.**
- PLAN-001 runner mapping (every job `runs-on: docker`, code_quality `dind`) → **reused for §T10.**
- PLAN-001 K8s HA topology → **§8 future HA path**, not bootstrap.
```