olsitec/foundation
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(backup): age at-rest encryption of bundles (CONTRACT_004 §4.3)

Browse source 
Close the known gap: backup bundles were uploaded unencrypted, relying
solely on destination access control. Now every data artifact is
age-encrypted on the VM before upload and decrypted on restore.

- backup-remote.sh: assemble rustfs blobs into rustfs-blobs.tar.zst (so the
  whole bundle is one encrypted unit), then age -r <recipient> each artifact
  to <name>.age and drop the plaintext. MANIFEST.json stays cleartext — it
  is the inventory + integrity gate and carries no secrets; it records each
  artifact's PLAINTEXT sha256 so restore verifies after decrypt.
- restore-remote.sh: materialise the age identity to a 0600 file, decrypt
  each .age, then run the existing sha + scratch-restore asserts; add a
  rustfs-blobs extract+assert.
- backup.sh / restore.sh: pass the public recipient (arg) / secret identity
  (stdin, never argv) from passphrase-encrypted config.
- provision/index.ts: install age + zstd on the VM via cloud-init so a fresh
  DR VM (T13) has the backup tools from first boot.
- Pulumi.foundation.yaml: seed backup.ageRecipient (public) + backup.ageIdentity
  (secure:). The identity lives in config so {repo + passphrase} can decrypt a
  bundle even after total Vault loss (CONTRACT_004 §4.3).

Validated live: encrypted backup + restore-verify PASS from both RustFS and
offsite; bucket shows only *.age + cleartext MANIFEST.json.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andreas Niemann 2026-06-30 23:23:38 +02:00
parent aabb50fb3b
commit 92e8f978a5
6 changed files with 79 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

30
backup/restore-remote.sh
View file

@ -6,17 +6,24 @@
# disaster restore (overwriting live, restore order Vault->Postgres->RustFS->Forgejo)
# is dr/restore-to-fresh-vm.sh (T13), out of scope here.
#
# Secrets on stdin; non-secrets ($TS, $MC_IMAGE, $PG_IMAGE, $SRC) as args.
# Secrets on stdin (OFF_* offsite creds + the age IDENTITY); non-secrets ($TS,
# $MC_IMAGE, $PG_IMAGE, $SRC) as args. The bundle is age-encrypted (CONTRACT_004
# §4.3): every artifact is pulled as <name>.age and decrypted with the identity
# BEFORE its MANIFEST sha256 (a PLAINTEXT sha) is verified.
set -eu
IFS= read -r OFF_EP
IFS= read -r OFF_AK
IFS= read -r OFF_SK
IFS= read -r BUCKET
IFS= read -r AGE_IDENTITY
TS="$1"; MC_IMAGE="$2"; PG_IMAGE="$3"; SRC="${4:-rfs}"
OFFSITE_BUCKET=olsitec-foundation
W="/tmp/foundation-restore-$TS"
rm -rf "$W"; mkdir -p "$W"
fail() { echo "RESTORE VERIFY FAIL: $1" >&2; docker rm -f foundation-restore-pg >/dev/null 2>&1 || true; exit 1; }
rm -rf "$W"; (umask 077; mkdir -p "$W")
fail() { echo "RESTORE VERIFY FAIL: $1" >&2; docker rm -f foundation-restore-pg >/dev/null 2>&1 || true; rm -f "$W/age.key" 2>/dev/null || true; exit 1; }
# Materialise the age identity to a 0600 file for `age -d -i` (removed on exit).
( umask 077; printf '%s\n' "$AGE_IDENTITY" > "$W/age.key" )
RAK=$(docker inspect foundation-rustfs --format '{{range .Config.Env}}{{println .}}{{end}}' | sed -n 's/^RUSTFS_ACCESS_KEY=//p')
RSK=$(docker inspect foundation-rustfs --format '{{range .Config.Env}}{{println .}}{{end}}' | sed -n 's/^RUSTFS_SECRET_KEY=//p')
@ -37,8 +44,14 @@ docker run --rm --network foundation-net --entrypoint sh -v "$W":/w \
[ -f "$W/MANIFEST.json" ] || { [ -d "$W/$TS" ] && mv "$W/$TS"/* "$W"/; }
[ -f "$W/MANIFEST.json" ] || fail "MANIFEST.json missing from pulled bundle"
echo "[restore] verify MANIFEST sha256" >&2
cd "$W"
echo "[restore] age-decrypt artifacts" >&2
for name in $(jq -r '.artifacts[].name' MANIFEST.json); do
[ -f "$name.age" ] || fail "$name.age missing from bundle"
age -d -i age.key -o "$name" "$name.age" 2>/dev/null || fail "age decrypt failed: $name"
done
echo "[restore] verify MANIFEST sha256 (plaintext)" >&2
jq -r '.artifacts[] | "\(.sha256) \(.name)"' MANIFEST.json | while read -r sha name; do
[ -f "$name" ] || { echo "missing $name" >&2; exit 1; }
got=$(sha256sum "$name" | cut -d' ' -f1)
@ -62,9 +75,16 @@ zstd -dc forgejo-repos.tar.zst | tar -C repos -xf - 2>/dev/null || fail "forgejo
[ -d repos/git/repositories/olsitec/foundation.git ] || fail "olsitec/foundation.git not in repo bundle"
echo "[restore] forgejo repos OK: olsitec/foundation.git present" >&2
echo "[restore] extract rustfs blobs + assert packages present" >&2
mkdir -p blobs
zstd -dc rustfs-blobs.tar.zst | tar -C blobs -xf - 2>/dev/null || fail "rustfs-blobs tar extract failed"
[ -d blobs/forgejo-packages ] || fail "forgejo-packages not in blob bundle"
echo "[restore] rustfs blobs OK: $(find blobs -type f | wc -l | tr -d ' ') object(s)" >&2
echo "[restore] vault snapshot sanity" >&2
[ -s vault-raft.snap ] || fail "vault-raft.snap empty"
echo "[restore] vault snapshot OK: $(stat -c %s vault-raft.snap) bytes" >&2
rm -f "$W/age.key"
rm -rf "$W"
echo "RESTORE VERIFY PASS ($TS from $SRC)"
echo "RESTORE VERIFY PASS ($TS from $SRC, age-decrypted)"