Close the known gap. Docker auto-assigns the subnet's first host (.1) as the
bridge gateway — a field we never declared — so `pulumi up --refresh` surfaced
it as a spurious foundation-net ipamConfigs drift. `gateway` is a ForceNew
input, so reconciling it (whether by declaring it OR by applying the refreshed
diff) REPLACES the network and disconnects every container. (Verified: adding
the gateway turned a clean plan into a network + 6-container + commands
replacement.)
The IPAM is immutable by design (subnet fixed by CONTRACT_003), so ignore
drift on it: ignoreChanges:["ipamConfigs"]. Plain `up` stays clean (44
unchanged) and `up --refresh` no longer wants to recreate the network/containers.
Residual, NON-destructive: `preview --refresh` still shows pessimistic
"~triggers" replaces on the vault-init + credential-writer commands, because a
refreshed container.id resolves to [unknown] in the preview (a Pulumi
preview artifact). At real apply the id is known + unchanged; worst case the
commands re-run idempotently. Documented for CI (T14).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the crypto-secret mirror: Forgejo's [security] SECRET_KEY was
EMPTY because the bootstrap skips the web installer (INSTALL_LOCK), which is
what normally generates it. An empty SECRET_KEY weakens at-rest encryption of
2FA secrets, push-mirror/migration passwords, and OAuth app secrets.
Generate it with @pulumi/random (it is a plain high-entropy string, not a
format-constrained JWT — so unlike INTERNAL_TOKEN/JWT_SECRET it CAN be
random-generated, matching CONTRACT_002 §2.3) and inject via
FORGEJO__security__SECRET_KEY; env-to-ini overwrites it in the volume's
app.ini while leaving Forgejo's own INTERNAL_TOKEN + JWT secrets untouched.
The GATE-B mirror then captures the real value into Vault.
Done now while the egg is fresh (no encrypted data yet) → no re-encryption.
Validated live: app.ini + Vault forgejoSecretKey = 40 chars; forge healthz
pass + https 200; scp-form clone works; idempotent at 44 unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the known gap: foundation/forgejo/service-credentials held only the
admin user/pw; the crypto secrets Forgejo auto-generates into app.ini were
never captured. Make that path single-owned at GATE B and write admin +
crypto together.
- credentials.ts: drop the forgejo block from the GATE-A writer (its crypto
secrets don't exist until Forgejo first-starts) and add
writeForgejoCredentialsToVault — runs after forgejo.ready, reads SECRET_KEY,
INTERNAL_TOKEN, LFS_JWT_SECRET ([server]) and oauth2 JWT_SECRET straight off
the live app.ini via docker-exec (ADR-007), and puts the full path. One
writer per Vault path avoids a put/patch race on re-runs.
- index.ts: wire it at GATE B (dependsOn vault.init + forgejo.ready).
Keys: forgejoAdminUser, forgejoAdminPassword, forgejoSecretKey,
forgejoInternalToken, forgejoJwtSecret, forgejoOauth2JwtSecret.
Validated live: forgejo path now has all six; postgres/rustfs paths intact
through the GATE-A writer replacement; idempotent at 43 unchanged.
FINDING: forgejoSecretKey mirrors EMPTY — skipping the web installer
(INSTALL_LOCK) left Forgejo's [security] SECRET_KEY unset. Fixed next commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation/backup/backup-credentials was never populated in Vault. Add a
writer (same ADR-007 docker-exec-over-SSH pattern, GATE A / dependsOn
vault.init) that mirrors the config-seeded offsite S3 creds and the age key
into Vault, completing CONTRACT_002 §2.3 for in-Vault consumers (Layer-1
ESO, the weekly backup-verify job).
- config.ts: loadBackupSecrets() — single reader of the backup secret slice
(offsite creds + age recipient/identity), keeping components off raw Config.
- credentials.ts: writeBackupCredentialsToVault() — idempotent vault kv put;
secret values on stdin (D2), non-secrets as shell vars.
- index.ts: wire it beside the data-plane creds writer.
Keys written: offsiteEndpoint, offsiteAccessKey, offsiteSecretKey,
backupAgeRecipient, backupAgeIdentity. Validated live: +1 resource, then
42 unchanged (idempotent); vault kv get shows all five keys populated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the known gap: backup bundles were uploaded unencrypted, relying
solely on destination access control. Now every data artifact is
age-encrypted on the VM before upload and decrypted on restore.
- backup-remote.sh: assemble rustfs blobs into rustfs-blobs.tar.zst (so the
whole bundle is one encrypted unit), then age -r <recipient> each artifact
to <name>.age and drop the plaintext. MANIFEST.json stays cleartext — it
is the inventory + integrity gate and carries no secrets; it records each
artifact's PLAINTEXT sha256 so restore verifies after decrypt.
- restore-remote.sh: materialise the age identity to a 0600 file, decrypt
each .age, then run the existing sha + scratch-restore asserts; add a
rustfs-blobs extract+assert.
- backup.sh / restore.sh: pass the public recipient (arg) / secret identity
(stdin, never argv) from passphrase-encrypted config.
- provision/index.ts: install age + zstd on the VM via cloud-init so a fresh
DR VM (T13) has the backup tools from first boot.
- Pulumi.foundation.yaml: seed backup.ageRecipient (public) + backup.ageIdentity
(secure:). The identity lives in config so {repo + passphrase} can decrypt a
bundle even after total Vault loss (CONTRACT_004 §4.3).
Validated live: encrypted backup + restore-verify PASS from both RustFS and
offsite; bucket shows only *.age + cleartext MANIFEST.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-contained prompt for a fresh Lead Agent context: required reads (incl. ADR-007),
current live state, operating essentials (run.sh / vault-unseal / backup), HIGH-RISK
watchouts (the refresh ipam diff), and the remaining PLAN-002 task order.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Data plane (postgres/rustfs/vault) → creds-in-Vault → Caddy DNS-01 → Forgejo →
admin/org/repo → runner → backup, all deployed live and validated. The goal is met:
git clone git@git.olsitec.net:olsitec/foundation.git works. Records state, the
ADR-007 control-plane mechanism, known gaps (age encryption, refresh ipam diff), and
the remaining PLAN-002 tasks (T11/T13/T14/T15).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backup/backup.sh (operator orchestrator) + backup-remote.sh (VM assembler) produce
a CONTRACT_004 bundle in RustFS foundation-backups/<TS>/ and replicate it to the
offsite olsitec-foundation bucket: pg_dumpall, forgejo git repos (tar.zst), vault
raft snapshot, pulumi state, rustfs blobs, MANIFEST.json (sha256 + restore order).
The timestamp is caller-supplied (§4.1); secrets travel on stdin (never argv,
ADR-007); mc runs containerized. restore.sh + restore-remote.sh are the §4.6
verifier: pull a bundle (rfs or offsite), check MANIFEST shas, then
NON-DESTRUCTIVELY reconstruct into scratch resources and assert (postgres users>0,
olsitec/foundation.git present, vault snapshot non-empty).
Live on cx33 Helsinki: bundle written to RustFS + offsite; restore-verify PASSES
from BOTH sources (forgejo.user rows=2, repo present, 16KB vault snapshot).
Known gap: at-rest age encryption (§4.3) not yet applied — both destinations are
private/access-controlled; adding age (generate key + encrypt-before-upload) is
the next hardening. Acceptance T12 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-runner (forgejo/runner:6, digest-pinned). Registration is idempotent
(ADR-007): it reuses /data/.runner if present, else mints a token via
`forgejo actions generate-runner-token` and consumes it with `forgejo-runner
register` (the token never leaves the VM). The daemon runs as uid 1000 with the
host docker group (gid 996) added for socket access — root-equivalent and
co-located, the documented day-zero compromise (PLAN-002 R5 / PLAN-001 §4a); a
fenced or separate runner VM is the steady state.
Live on cx33 Helsinki: runner declared (labels docker,dind) and polling; a
hello-world `runs-on: docker` workflow pushed to olsitec/foundation ran to
success (workflow run #1). Acceptance T10 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VM's admin sshd is on :222, so :22 is free for Forgejo's git-over-SSH. Opening
it makes the scp-form clone `git@git.olsitec.net:olsitec/...` work — Forgejo's sshd
ForceCommands `serv`, so :22 only ever does git (like github.com:22). :2222 stays
open too (CONTRACT_001 forgeSshPort).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bootstrapForgejo (idempotent, docker-exec — ADR-007) creates the headless admin
via `forgejo admin user create` (run as the git user; no web installer, no default
credentials — PLAN-002 §9.3), then via the image's own curl against the API: the
olsitec org, an auto-init'd olsitec/foundation repo, and the operator's SSH public
key. credentials.ts gains the forgejo admin slice (@pulumi/random) and
writeCredentialsToVault now also writes foundation/forgejo/service-credentials.
Live on cx33 Helsinki: admin + org + repo + key created. GOAL MET —
`git clone git@git.olsitec.net:olsitec/foundation.git` (scp-form, :22) and
`ssh://git@git.olsitec.net:2222/olsitec/foundation.git` both clone the repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-forgejo (forgejo:11, digest-pinned) on foundation-net: git repos on the
foundation-forgejo-data volume (the irreducible state), metadata in external
Postgres, blobs in RustFS (default storage + LFS over the minio API). Config is
seeded via FORGEJO__section__KEY env -> app.ini; INSTALL_LOCK skips the web
installer and the crypto secrets (SECRET_KEY/INTERNAL_TOKEN/JWT) auto-generate and
persist in the volume. HTTP 3000 is internal (Caddy fronts forge.olsitec.net); the
image's openssh sshd owns container :22 (START_SSH_SERVER=false — explicitly, so a
stale app.ini value can't keep Forgejo's built-in Go SSH server colliding on :22),
published on host :22 (scp-form goal) and :2222 (CONTRACT_003). A healthz-gated
ready command is GATE B for T09/T10.
Live on cx33 Helsinki: container healthy, https://forge.olsitec.net = 200 over a
valid Let's Encrypt cert, API 11.0.15, sshd reachable on :22 and :2222.
Acceptance T08 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-caddy — the only public ingress (80/443 published), automatic TLS via
Let's Encrypt DNS-01 over Cloudflare. Standard caddy:2 lacks the DNS plugin, so
the egg builds a custom image on the VM (containers/caddy-cloudflare/Dockerfile:
xcaddy + caddy-dns/cloudflare@v0.2.4, base digests pinned) via a remote.Command
(ADR-007) whose stdout image id the container runs. The Caddyfile carries no
secrets — the CF token is read from the container env ({env.CF_API_TOKEN}) — and
is rendered + bind-mounted from the host. Routes forge -> Forgejo:3000 and
s3 -> RustFS:9000; Vault is intentionally not proxied publicly (CONTRACT_003
"restricted").
Live on cx33 Helsinki: certs obtained for forge + s3; https://forge.olsitec.net
= 502 (Forgejo lands in T08) and https://s3.olsitec.net = 403 (RustFS), both over
valid Let's Encrypt certs (DNS-01). Acceptance T07 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
writeCredentialsToVault distributes the generated postgres + rustfs
service-credentials into the Vault `foundation` kv-v2 mount at the CONTRACT_002
paths, over docker-exec/SSH (ADR-007) since 8200 isn't reachable from the
operator. Secret values go in as a JSON object on the container's stdin (never
argv); the root token from the vault-init output authenticates. dependsOn
vault.init = GATE A. Idempotent: kv-v2 enable is guarded, `vault kv put`
overwrites. Forgejo crypto secrets, the runner token, registry tokens, and backup
creds are written by their own tasks (T08/T10/T12).
Live on cx33 Helsinki: foundation/{postgres,rustfs}/service-credentials present
with every CONTRACT_002 camelCase key non-empty; mount is kv v2. Acceptance T06
met for the data-plane slice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-vault (hashicorp/vault:1.18, digest-pinned) with integrated raft
storage in foundation-vault-data (-> /vault/file, which the entrypoint chowns to
the vault user), IPC_LOCK for mlock, internal only (8200 unpublished). Init +
unseal reuse the olsitec-core pattern but over docker-exec/SSH (ADR-007): the
foundation-vault-init command inits 1-of-1 Shamir, unseals, and emits keys + root
token on stdout — marked secret and NOT streamed (logging:Stderr) so they never
reach the terminal/logs (D2). run.sh captures them into vaultCredentials:* (the
one bootstrap secret that cannot live in Vault, CONTRACT_002 §2.4) with an
idempotent guard that avoids churning the config. vault-unseal.sh is the
passphrase-gated reboot helper (ADR-004): reads keys from config, unseals over an
SSH stdin pipe. run.sh also now pins the Pulumi backend per-process
(PULUMI_BACKEND_URL) instead of a global `pulumi login`.
Live on cx33 Helsinki: initialized + unsealed (raft 1.18.5), keys captured to
encrypted config, idempotent re-up reuses stored keys, container-restart reseal
recovered by vault-unseal.sh. Acceptance T05 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-rustfs (rustfs/rustfs digest-pinned) on foundation-net, internal only
(9000/9001 unpublished); named volume foundation-rustfs-data with retainOnDelete.
The four buckets (forgejo-packages/-artifacts/-lfs, foundation-backups) and a
scoped service account with generated keys (CONTRACT_002 rustfs slice) are
provisioned post-boot by an idempotent, readiness-gated remote.Command using a
throwaway mc container (ADR-007). RustFS speaks enough MinIO admin API for
`svcacct add`; `mc ready` is unreliable so readiness gates on `mc ls`; the mc
image's busybox lacks grep so existence checks use a shell `case`. Pins the
IMAGE_MC tool image in VERSIONS.
Live on cx33 Helsinki: 4 buckets present, service key registered, put/get
roundtrip OK, no published ports. Acceptance T04 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
foundation-postgres (postgres:17, digest-pinned in VERSIONS) on foundation-net,
internal only (5432 unpublished); named volume foundation-postgres-data with
retainOnDelete. The forgejo login role + database are created post-boot by an
idempotent, readiness-gated remote.Command (ADR-007), since 5432 isn't reachable
from the operator. Adds the generator half of credentials.ts (@pulumi/random →
CONTRACT_002 postgres keys) and lib/remote.ts (vmConnection over the VM SSH path).
Live on cx33 Helsinki: container healthy, role 'forgejo' + db 'forgejo' present,
no published ports. Acceptance T03 met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Internal service ports (Postgres 5432, Vault 8200, RustFS 9000) are not
published off-host (CONTRACT_003), so the operator's Pulumi process cannot
reach them to run init/role/bucket/admin steps. Adopt @pulumi/command
remote.Command over the existing SSH path, acting through `docker exec`, for
every in-VM control-plane operation in Wave 2: idempotent, readiness-gated,
secrets passed on stdin (never inlined — the provider echoes the command on
error; D2). The vendored fetch()-based VaultInitialization is kept for
Layer-1, not used by the egg; the olsitec-core init→capture→unseal pattern is
reused, only the mechanism adapts to the remote VM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONTRACT_004 offsite target (ADR-004 'second self-hosted location'). @pulumi/minio
program (modeled on olsicloud4 modules/minio): bucket 'olsitec-foundation' +
scoped IAM user/policy + service account on minio.wob.olsitec.de:19000.
Verified: scoped SA can put/list/delete in its bucket, DENIED cross-bucket. Admin
creds + scoped creds via ENV/state only (gitignored), never committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cx33 only orderable in hel1-dc2; recreated there (new IP 204.168.234.72, old
nbg1 VM destroyed). Initial Hetzner home per 'deploy now, migrate later via
backup/restore'. Verified: Docker 29.6.1, docker-over-SSH OK, 4c/7.6G/75G usable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Composition substrate for Wave 2 (T03+):
- lib/context.ts: one Docker-over-SSH provider + DeployCtx threaded to component
factories; FOUNDATION_DOCKER_HOST override for ephemeral validation.
- lib/versions.ts: resolve pinned images from VERSIONS; FOUNDATION_ALLOW_UNPINNED
for local validation when digests are still PIN_DIGEST.
- components/network.ts: foundation-net (CONTRACT_003 §3.1).
- index.ts: phase-orchestration entrypoint with dependsOn gates; Wave-2 slots.
- ADR-006: shared-provider + per-component-factory model (egg does not route its
phased bootstrap through the monolithic vendored DockerDeployments).
Validated: pulumi up over Docker-over-SSH created+verified+destroyed foundation-net
on crunchy01 (x86_64); ephemeral, nothing persisted. tsc + preview clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Bun workspaces (packages/* + bootstrap); Pulumi nodejs runtime under
packagemanager: bun (no npm fallback needed).
- bootstrap/config.ts: typed FoundationConfig per CONTRACT_001; loadConfig()
fails closed, aggregating all missing+malformed keys in one error. Reads flat
dotted keys; image digests excluded (they live in VERSIONS, D5).
- bootstrap/Pulumi.foundation.yaml: non-secret placeholders only (RFC-5737 vm.host,
.invalid offsite); no encryptionsalt/secrets committed (D2). pulumi preview = 0
resources under the passphrase provider via gitignored file:// state backend.
- Stage-1 vendoring: packages/pulumi-{docker,vault} as @olsitec/* (source-only,
logic unchanged). vault's 5 type-only imports from modules/olsitec re-homed
verbatim into pulumi-vault/olsitec-types.ts to keep the egg self-contained.
Realizes PLAN-002 §10 T02; ADR-005 / 000_TOPOLOGY.md §5 Stage-1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo topology, baseline overlay, planning docs (PLAN-001/002), ADR-004/005,
and the bootstrap/packages/documentation skeleton. Implementation (T00+) not started.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
