Self-contained prompt for a fresh Lead Agent context: required reads (incl. ADR-007),
current live state, operating essentials (run.sh / vault-unseal / backup), HIGH-RISK
watchouts (the refresh ipam diff), and the remaining PLAN-002 task order.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Data plane (postgres/rustfs/vault) → creds-in-Vault → Caddy DNS-01 → Forgejo →
admin/org/repo → runner → backup, all deployed live and validated. The goal is met:
git clone git@git.olsitec.net:olsitec/foundation.git works. Records state, the
ADR-007 control-plane mechanism, known gaps (age encryption, refresh ipam diff), and
the remaining PLAN-002 tasks (T11/T13/T14/T15).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backup/backup.sh (operator orchestrator) + backup-remote.sh (VM assembler) produce
a CONTRACT_004 bundle in RustFS foundation-backups/<TS>/ and replicate it to the
offsite olsitec-foundation bucket: pg_dumpall, forgejo git repos (tar.zst), vault
raft snapshot, pulumi state, rustfs blobs, MANIFEST.json (sha256 + restore order).
The timestamp is caller-supplied (§4.1); secrets travel on stdin (never argv,
ADR-007); mc runs containerized. restore.sh + restore-remote.sh are the §4.6
verifier: pull a bundle (rfs or offsite), check MANIFEST shas, then
NON-DESTRUCTIVELY reconstruct into scratch resources and assert (postgres users>0,
olsitec/foundation.git present, vault snapshot non-empty).
Live on cx33 Helsinki: bundle written to RustFS + offsite; restore-verify PASSES
from BOTH sources (forgejo.user rows=2, repo present, 16KB vault snapshot).
Known gap: at-rest age encryption (§4.3) not yet applied — both destinations are
private/access-controlled; adding age (generate key + encrypt-before-upload) is
the next hardening. Acceptance T12 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-runner (forgejo/runner:6, digest-pinned). Registration is idempotent
(ADR-007): it reuses /data/.runner if present, else mints a token via
`forgejo actions generate-runner-token` and consumes it with `forgejo-runner
register` (the token never leaves the VM). The daemon runs as uid 1000 with the
host docker group (gid 996) added for socket access — root-equivalent and
co-located, the documented day-zero compromise (PLAN-002 R5 / PLAN-001 §4a); a
fenced or separate runner VM is the steady state.
Live on cx33 Helsinki: runner declared (labels docker,dind) and polling; a
hello-world `runs-on: docker` workflow pushed to olsitec/foundation ran to
success (workflow run #1). Acceptance T10 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VM's admin sshd is on :222, so :22 is free for Forgejo's git-over-SSH. Opening
it makes the scp-form clone `git@git.olsitec.net:olsitec/...` work — Forgejo's sshd
ForceCommands `serv`, so :22 only ever does git (like github.com:22). :2222 stays
open too (CONTRACT_001 forgeSshPort).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bootstrapForgejo (idempotent, docker-exec — ADR-007) creates the headless admin
via `forgejo admin user create` (run as the git user; no web installer, no default
credentials — PLAN-002 §9.3), then via the image's own curl against the API: the
olsitec org, an auto-init'd olsitec/foundation repo, and the operator's SSH public
key. credentials.ts gains the forgejo admin slice (@pulumi/random) and
writeCredentialsToVault now also writes foundation/forgejo/service-credentials.
Live on cx33 Helsinki: admin + org + repo + key created. GOAL MET —
`git clone git@git.olsitec.net:olsitec/foundation.git` (scp-form, :22) and
`ssh://git@git.olsitec.net:2222/olsitec/foundation.git` both clone the repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-forgejo (forgejo:11, digest-pinned) on foundation-net: git repos on the
foundation-forgejo-data volume (the irreducible state), metadata in external
Postgres, blobs in RustFS (default storage + LFS over the minio API). Config is
seeded via FORGEJO__section__KEY env -> app.ini; INSTALL_LOCK skips the web
installer and the crypto secrets (SECRET_KEY/INTERNAL_TOKEN/JWT) auto-generate and
persist in the volume. HTTP 3000 is internal (Caddy fronts forge.olsitec.net); the
image's openssh sshd owns container :22 (START_SSH_SERVER=false — explicitly, so a
stale app.ini value can't keep Forgejo's built-in Go SSH server colliding on :22),
published on host :22 (scp-form goal) and :2222 (CONTRACT_003). A healthz-gated
ready command is GATE B for T09/T10.
Live on cx33 Helsinki: container healthy, https://forge.olsitec.net = 200 over a
valid Let's Encrypt cert, API 11.0.15, sshd reachable on :22 and :2222.
Acceptance T08 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-caddy — the only public ingress (80/443 published), automatic TLS via
Let's Encrypt DNS-01 over Cloudflare. Standard caddy:2 lacks the DNS plugin, so
the egg builds a custom image on the VM (containers/caddy-cloudflare/Dockerfile:
xcaddy + caddy-dns/cloudflare@v0.2.4, base digests pinned) via a remote.Command
(ADR-007) whose stdout image id the container runs. The Caddyfile carries no
secrets — the CF token is read from the container env ({env.CF_API_TOKEN}) — and
is rendered + bind-mounted from the host. Routes forge -> Forgejo:3000 and
s3 -> RustFS:9000; Vault is intentionally not proxied publicly (CONTRACT_003
"restricted").
Live on cx33 Helsinki: certs obtained for forge + s3; https://forge.olsitec.net
= 502 (Forgejo lands in T08) and https://s3.olsitec.net = 403 (RustFS), both over
valid Let's Encrypt certs (DNS-01). Acceptance T07 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
writeCredentialsToVault distributes the generated postgres + rustfs
service-credentials into the Vault `foundation` kv-v2 mount at the CONTRACT_002
paths, over docker-exec/SSH (ADR-007) since 8200 isn't reachable from the
operator. Secret values go in as a JSON object on the container's stdin (never
argv); the root token from the vault-init output authenticates. dependsOn
vault.init = GATE A. Idempotent: kv-v2 enable is guarded, `vault kv put`
overwrites. Forgejo crypto secrets, the runner token, registry tokens, and backup
creds are written by their own tasks (T08/T10/T12).
Live on cx33 Helsinki: foundation/{postgres,rustfs}/service-credentials present
with every CONTRACT_002 camelCase key non-empty; mount is kv v2. Acceptance T06
met for the data-plane slice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-vault (hashicorp/vault:1.18, digest-pinned) with integrated raft
storage in foundation-vault-data (-> /vault/file, which the entrypoint chowns to
the vault user), IPC_LOCK for mlock, internal only (8200 unpublished). Init +
unseal reuse the olsitec-core pattern but over docker-exec/SSH (ADR-007): the
foundation-vault-init command inits 1-of-1 Shamir, unseals, and emits keys + root
token on stdout — marked secret and NOT streamed (logging:Stderr) so they never
reach the terminal/logs (D2). run.sh captures them into vaultCredentials:* (the
one bootstrap secret that cannot live in Vault, CONTRACT_002 §2.4) with an
idempotent guard that avoids churning the config. vault-unseal.sh is the
passphrase-gated reboot helper (ADR-004): reads keys from config, unseals over an
SSH stdin pipe. run.sh also now pins the Pulumi backend per-process
(PULUMI_BACKEND_URL) instead of a global `pulumi login`.
Live on cx33 Helsinki: initialized + unsealed (raft 1.18.5), keys captured to
encrypted config, idempotent re-up reuses stored keys, container-restart reseal
recovered by vault-unseal.sh. Acceptance T05 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-rustfs (rustfs/rustfs digest-pinned) on foundation-net, internal only
(9000/9001 unpublished); named volume foundation-rustfs-data with retainOnDelete.
The four buckets (forgejo-packages/-artifacts/-lfs, foundation-backups) and a
scoped service account with generated keys (CONTRACT_002 rustfs slice) are
provisioned post-boot by an idempotent, readiness-gated remote.Command using a
throwaway mc container (ADR-007). RustFS speaks enough MinIO admin API for
`svcacct add`; `mc ready` is unreliable so readiness gates on `mc ls`; the mc
image's busybox lacks grep so existence checks use a shell `case`. Pins the
IMAGE_MC tool image in VERSIONS.
Live on cx33 Helsinki: 4 buckets present, service key registered, put/get
roundtrip OK, no published ports. Acceptance T04 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-postgres (postgres:17, digest-pinned in VERSIONS) on foundation-net,
internal only (5432 unpublished); named volume foundation-postgres-data with
retainOnDelete. The forgejo login role + database are created post-boot by an
idempotent, readiness-gated remote.Command (ADR-007), since 5432 isn't reachable
from the operator. Adds the generator half of credentials.ts (@pulumi/random →
CONTRACT_002 postgres keys) and lib/remote.ts (vmConnection over the VM SSH path).
Live on cx33 Helsinki: container healthy, role 'forgejo' + db 'forgejo' present,
no published ports. Acceptance T03 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Internal service ports (Postgres 5432, Vault 8200, RustFS 9000) are not
published off-host (CONTRACT_003), so the operator's Pulumi process cannot
reach them to run init/role/bucket/admin steps. Adopt @pulumi/command
remote.Command over the existing SSH path, acting through `docker exec`, for
every in-VM control-plane operation in Wave 2: idempotent, readiness-gated,
secrets passed on stdin (never inlined — the provider echoes the command on
error; D2). The vendored fetch()-based VaultInitialization is kept for
Layer-1, not used by the egg; the olsitec-core init→capture→unseal pattern is
reused, only the mechanism adapts to the remote VM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONTRACT_004 offsite target (ADR-004 'second self-hosted location'). @pulumi/minio
program (modeled on olsicloud4 modules/minio): bucket 'olsitec-foundation' +
scoped IAM user/policy + service account on minio.wob.olsitec.de:19000.
Verified: scoped SA can put/list/delete in its bucket, DENIED cross-bucket. Admin
creds + scoped creds via ENV/state only (gitignored), never committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cx33 only orderable in hel1-dc2; recreated there (new IP 204.168.234.72, old
nbg1 VM destroyed). Initial Hetzner home per 'deploy now, migrate later via
backup/restore'. Verified: Docker 29.6.1, docker-over-SSH OK, 4c/7.6G/75G usable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Composition substrate for Wave 2 (T03+):
- lib/context.ts: one Docker-over-SSH provider + DeployCtx threaded to component
factories; FOUNDATION_DOCKER_HOST override for ephemeral validation.
- lib/versions.ts: resolve pinned images from VERSIONS; FOUNDATION_ALLOW_UNPINNED
for local validation when digests are still PIN_DIGEST.
- components/network.ts: foundation-net (CONTRACT_003 §3.1).
- index.ts: phase-orchestration entrypoint with dependsOn gates; Wave-2 slots.
- ADR-006: shared-provider + per-component-factory model (egg does not route its
phased bootstrap through the monolithic vendored DockerDeployments).
Validated: pulumi up over Docker-over-SSH created+verified+destroyed foundation-net
on crunchy01 (x86_64); ephemeral, nothing persisted. tsc + preview clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Bun workspaces (packages/* + bootstrap); Pulumi nodejs runtime under
packagemanager: bun (no npm fallback needed).
- bootstrap/config.ts: typed FoundationConfig per CONTRACT_001; loadConfig()
fails closed, aggregating all missing+malformed keys in one error. Reads flat
dotted keys; image digests excluded (they live in VERSIONS, D5).
- bootstrap/Pulumi.foundation.yaml: non-secret placeholders only (RFC-5737 vm.host,
.invalid offsite); no encryptionsalt/secrets committed (D2). pulumi preview = 0
resources under the passphrase provider via gitignored file:// state backend.
- Stage-1 vendoring: packages/pulumi-{docker,vault} as @olsitec/* (source-only,
logic unchanged). vault's 5 type-only imports from modules/olsitec re-homed
verbatim into pulumi-vault/olsitec-types.ts to keep the egg self-contained.
Realizes PLAN-002 §10 T02; ADR-005 / 000_TOPOLOGY.md §5 Stage-1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo topology, baseline overlay, planning docs (PLAN-001/002), ADR-004/005,
and the bootstrap/packages/documentation skeleton. Implementation (T00+) not started.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
