foundation/dr/RUNBOOK.md

# DR RUNBOOK — rebuild the foundation egg on a fresh VM (T13)

**Realises** CONTRACT_004 §4.4 (restore order) · **Companion**: `dr/restore-to-fresh-vm.sh`
(orchestrator) + `dr/restore-to-fresh-vm-remote.sh` (VM-side). This is the
**destructive** sibling of `backup/restore.sh` (the non-destructive scratch verifier).

## 0. When you are here

The Helsinki VM (or its Vault/data) is gone. You still have:

- **this git repo** (`olsitec/foundation`) — including `bootstrap/Pulumi.foundation.yaml`,
  whose passphrase-encrypted `secure:` values hold the Vault **OLD unseal keys + root
  token** (CONTRACT_002 §2.4) and the **age identity** (CONTRACT_004 §4.3);
- the **master passphrase** (`pass olsitec-foundation/PULUMI_CONFIG_PASSPHRASE`) — the
  one external secret; without it nothing below decrypts;
- the **offsite bundle** (self-hosted MinIO `olsitec-foundation/<TS>/`) — RustFS is
  assumed lost, so restore reads **offsite** (`--source off`).

`{repo + passphrase + offsite bundle}` is sufficient to fully reconstitute the egg.
Nothing else is needed — that is the whole point of the trust chain (PLAN-002 §4.1).

## 1. Pick the bundle

List candidates in the offsite store and choose the newest verified one:

```
ts=<UTC-YYYYMMDDTHHMMSSZ>     # e.g. the latest in olsitec-foundation/
```

A bundle is trustworthy only if `backup/restore.sh <ts> off` has passed for it (the
weekly verify job, CONTRACT_004 §4.6). Prefer the last green one.

## 2. Provision a fresh VM

**Real DR** — use the provision stack so the new VM becomes the managed home:

```
cd provision
export HCLOUD_TOKEN="$(pass olsicloud4/HCLOUD_TOKEN)"
# edit the server name if the old one still exists in Hetzner; the cloud-init already
# installs docker + age + zstd (jq is a base package) — backupTools in provision/index.ts.
PULUMI_CONFIG_PASSPHRASE=dev-validation-throwaway pulumi up
```

The VM must have, at first boot: **docker, age, zstd, jq** (the cloud-init provides
them). SSH is on **:222** for a provision-stack VM (the vendored cloud-init moves it);
pass `--port 222` below.

**Rehearsal** — a throwaway VM created directly via the Hetzner API (cx33, debian-12,
ssh key `foundation-test-ssh-key`, cloud-init installing docker+age+zstd+jq), sshd on
**:22**. Destroy it immediately after (`DELETE /v1/servers/<id>`).

## 3. Restore (Vault → Postgres → RustFS → Forgejo)

```
./dr/restore-to-fresh-vm.sh --host <new-ip> --port <22|222> --ts "$ts" --source off
```

What it does, in the **mandated order** (CONTRACT_004 §4.4 — starting Forgejo before
1–3 is a defect):

1. **Decrypt** the bundle with the age identity; verify every artifact's MANIFEST sha256.
2. **Vault** — start a fresh raft node, init a throwaway node, `raft snapshot restore
   -force`, then **unseal with the OLD keys** from config. Vault is now the source of
   truth again; the OLD root token authenticates. All other creds are read back out.
3. **Postgres** — start with the super-password from Vault, restore `postgres.sql.gz`
   (recreates the `forgejo` role + DB; asserts `"user"` rows ≥ 1).
4. **RustFS** — start with the admin keys from Vault, recreate the four buckets + the
   scoped service account (the exact `serviceKeyId/Secret` Forgejo's app.ini expects),
   sync `rustfs-blobs` back into the buckets.
5. **Forgejo** — extract `forgejo-repos.tar.zst` into the data volume (git repos +
   `app.ini`, which already carries DB/S3 creds + INTERNAL_TOKEN/JWTs), inject
   `SECRET_KEY` from Vault, start. Asserts healthz pass + `olsitec/foundation.git` present.

On success it prints `DR RESTORE OK (<ts>): …`.

## 4. What is NOT restored (recreatable — CONTRACT_004 §4.5)

Container images (re-pulled by digest), Caddy ACME data (re-issued), the runner's
ephemeral registration, search indexes/caches. These come back in §5.

## 5. Re-establish ingress + management

1. **DNS** — repoint `forge/git/s3/vault.olsitec.net` A records at the new IP
   (Cloudflare; the bootstrap's `deployDns` does this once the stack is re-adopted).
2. **Caddy + runner** — re-adopt the stack so IaC manages the new VM:
   ```
   cd bootstrap
   pulumi config set foundation:vm.host <new-ip>
   pulumi config set foundation:vm.sshPort <22|222>
   ./run.sh up        # creates Caddy (re-issues LE cert via DNS-01), re-registers the runner
   ```
   `up` is idempotent against the already-restored containers it can adopt, and creates
   the recreatable ones (Caddy, runner). Verify: `https://forge.olsitec.net` = 200,
   `git clone git@git.olsitec.net:olsitec/foundation.git`.
3. **Re-key reminder (D2)** — after a real disaster, rotate the Vault root token + the
   offsite creds (they were materialised on a possibly-compromised host): `pulumi up
   --replace` the relevant credential resources, then re-run a backup.

## 6. Gotchas (discovered during the T13 rehearsal)

- The **docker gid** on the host is host-specific; the runner mounts the host socket
  (PLAN-002 R5). On a fresh VM re-check the gid before trusting runner jobs.
- `raft snapshot restore -force` **re-seals** the node (it swaps in the snapshot's
  keyring) — you MUST unseal again with the OLD keys, not the throwaway init keys.
- Restore reads **offsite** by default. RustFS on the new VM starts EMPTY; its blobs
  come from the bundle, not from a surviving RustFS.