explewd/keep
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revise announce+agent proposal against the real deploy script

Browse Source 
The original draft speced a hypothetical per-project agent.sh on a
per-project cron. Reality turned out different: there's already a
production bulk deploy script on the host (fixed project array,
docker compose up -d per project, no explicit pull — freshness comes
entirely from pull_policy: always) — so this patches that existing
loop in place instead of introducing a parallel per-project script
that would duplicate its array/failure-tracking logic.

Also folds in a real prerequisite gap found while reconciling the
proposal with the actual script: pull_policy: always isn't universal
across the fleet yet, independent of keep entirely — the bulk script
can't redeploy anything missing that line regardless of keep adoption.
One instance of this was already fixed this session (a sibling
project's docker-compose.yml missed the line when copying another
project's compose pattern).

Reframes §5's announce/watch benefit accordingly: since the real
script is one all-or-nothing sweep, not N independent watchers, the
natural fit is waking the whole sweep early (watch-any), not
per-project selective redeploy — still deliberately left unbuilt
until there's a real signal it's worth it over the already-scheduled
sweep.

Project names anonymized per policy — generic proj1..proj12
placeholders instead of the real fleet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Fredrik Johansson
2026-07-12 20:50:15 +02:00
parent e505fbf9b4
commit 114f234cd3
1 changed files with 129 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files

221
PROPOSAL-announce-and-agent.md
View File

@@ -23,13 +23,50 @@ code. The relevant existing pieces for this proposal:
- `src/cli/` — `identity.ts`, `recipient.ts`, `vault.ts`, - `src/cli/` — `identity.ts`, `recipient.ts`, `vault.ts`,
`signedFetch.ts` (note: `signedFetch`'s `path` param must never `signedFetch.ts` (note: `signedFetch`'s `path` param must never
include a query string — see the comment there for why). include a query string — see the comment there for why).
- `scripts/deploy.sh` — pulls `<project>/<env>` from `keep` into - `scripts/deploy.sh` (this repo) — pulls `<project>/<env>` from `keep`
`$DEPLOY_ROOT/<project>/.env`, then `docker compose up -d`. Assumes into `$DEPLOY_ROOT/<project>/.env`, then `docker compose up -d` for
the vault already exists and this machine already has a grant on it. one project. Assumes the vault already exists and this machine
- `.gitea/workflows/docker.yml` (present in `keep`, `wisp`, already has a grant on it. **Superseded by the real deploy script
`npm-statuspage`) — builds and pushes an image to the Gitea container below as the thing this proposal actually patches** — kept in this
repo as a single-project example/fallback, not the production path.
- **The real, already-running deploy script — on the host, not in any
repo.** A fixed-array bulk script, already in production use, looping
over roughly a dozen project directories:
```bash
ROOT="$(cd "$(dirname "$0")" && pwd)"
PROJECTS=(proj1 proj2 proj3 proj4 proj5 proj6 proj7 proj8 proj9 proj10 proj11 proj12)
for project in "${PROJECTS[@]}"; do
dir="$ROOT/$project"
[ -f "$dir/docker-compose.yml" ] || { echo "skip: $project"; continue; }
(cd "$dir" && docker compose up -d) || FAILED+=("$project")
done
docker image prune -f
```
Two things about it that matter for this proposal: **it never runs
`docker compose pull`** — freshness comes entirely from each
project's own `pull_policy: always` in its `docker-compose.yml`
(`up -d` re-pulls implicitly). And **it loops over a fixed, explicit
array**, not directory auto-discovery — adding a project means
editing this array by hand. This is the actual thing to patch, not a
new per-project script; see Design §4.
- **Prerequisite gap, found while writing this proposal, not yet fixed
everywhere**: `pull_policy: always` isn't universal across the
fleet — confirmed present in roughly half the projects checked,
confirmed missing in several others (one of which was fixed in this
session — a sibling project's `docker-compose.yml` didn't have it, a
regression from copying another project's compose file without
carrying this line over), and unchecked for the rest (no local
checkout available at proposal time). **This is independent of
`keep` entirely** — the bulk script can't redeploy any project
missing this line regardless of whether that project ever adopts
`keep`, and it should be swept fleet-wide before or alongside this
work, not as part of it.
- `.gitea/workflows/docker.yml` (present in `keep` and other sibling
projects) — builds and pushes an image to the Gitea container
registry on every push to `main`. Does **not** currently deploy registry on every push to `main`. Does **not** currently deploy
anything — that's a manual step today. anything — that's the manual/scheduled bulk script above today.
## Motivation ## Motivation
@@ -38,7 +75,7 @@ is still a manual "SSH in and run the deploy script" step. The obvious
fix — CI SSHes into the VPS and runs the deploy — was considered and fix — CI SSHes into the VPS and runs the deploy — was considered and
rejected: it means a standing SSH credential with production-deploy rejected: it means a standing SSH credential with production-deploy
rights sits in Gitea's CI secrets, reachable by whatever's executing on rights sits in Gitea's CI secrets, reachable by whatever's executing on
the shared runner across all ~17 projects it builds. A compromised build the shared runner across every project it builds. A compromised build
step in any one project could then reach every other project's deploy step in any one project could then reach every other project's deploy
target. That's a real regression from where `keep` already left things target. That's a real regression from where `keep` already left things
(see IMPLEMENTATION.md's resolved "read no longer implies write" — the (see IMPLEMENTATION.md's resolved "read no longer implies write" — the
@@ -55,12 +92,11 @@ relay, same posture as everything else it does.
**Not every project will have been migrated to `keep` yet.** Bootstrapping **Not every project will have been migrated to `keep` yet.** Bootstrapping
a project (register a recipient, grant it, push the first vault) is a a project (register a recipient, grant it, push the first vault) is a
real, deliberate step per project, and there'll be a period — maybe a real, deliberate step per project, and there'll be a period — maybe a
long one — where some of the ~17 projects have it and some don't. The long one — where some projects have it and some don't. The redeploy path
self-deploying agent has to work identically well for both, defaulting to has to work identically well for both, defaulting to exactly what
exactly what happens today (a plain `docker compose pull && up -d` happens today (a plain `docker compose up -d`, relying on that project's
against whatever's already on disk) for anything not yet bootstrapped, own `pull_policy: always`) for anything not yet bootstrapped, rather than
rather than either blocking deploys entirely or hard-failing on missing either blocking deploys entirely or hard-failing on missing vaults.
vaults.
## Design ## Design
@@ -113,91 +149,93 @@ keep watch <vault> [--interval 60] -- polls GET .../announce every N seconds,
``` ```
`keep watch` is deliberately dumb — plain polling, not a websocket or `keep watch` is deliberately dumb — plain polling, not a websocket or
long-poll. Matches `npm-statuspage`'s existing polling pattern, and a long-poll. Matches the polling pattern already used elsewhere in this
60-second detection lag for a deploy is fine. Don't build push-based project family, and a 60-second detection lag for a deploy is fine.
notification for this; it's more machinery for a problem polling already Don't build push-based notification for this; it's more machinery for a
solves adequately. problem polling already solves adequately.
### 4. The agent script and its fallback ### 4. Patch the existing loop — don't build a parallel script
This is the part the fallback requirement is actually about. One script This is the part the fallback requirement is actually about. The real
per VPS, running on a timer (systemd timer or cron, not a long-lived deploy script already does the one thing that matters (a loop over a
daemon — matches this project's bias toward simple, restartable, fixed project list, `docker compose up -d` per project, relying on
stateless-between-runs tooling), one invocation per project it's `pull_policy: always` for freshness) — the change is inserting one
responsible for: `keep`-with-fallback step per iteration, in place, not writing a new
per-project `agent.sh` that duplicates the loop/array/failure-tracking
logic the real script already has:
```bash ```bash
#!/usr/bin/env bash for project in "${PROJECTS[@]}"; do
# agent.sh <project> [env] — self-deploy check for one project. Meant to dir="$ROOT/$project"
# run on a schedule (cron/systemd timer), not as a daemon. [ -f "$dir/docker-compose.yml" ] || { echo "skip: $project"; continue; }
set -euo pipefail
project="${1:?usage: agent.sh <project> [env]}" echo ""
env="${2:-production}" echo "▶ $project"
root="${DEPLOY_ROOT:-/opt/docker}"
dir="$root/$project"
vault="$project/$env"
if [ ! -d "$dir" ]; then # Best-effort: does this machine have a working keep grant for
echo "error: $dir does not exist" >&2 # "$project/production"? If not — not bootstrapped onto keep yet, or
exit 1 # never will be (a static site with no secrets) — fall through to
fi # exactly what this script already did before keep existed. Every
# keep call here is allowed to fail; failure means "keep doesn't
# apply to this project," never "stop the deploy."
if keep pull "$project/production" --out "$dir/.env.new" 2>/dev/null; then
mv "$dir/.env.new" "$dir/.env"
else
rm -f "$dir/.env.new"
fi
cd "$dir" if (cd "$dir" && docker compose up -d 2>&1); then
echo " ✓ $project up"
# Best-effort: does this machine have a working keep grant on this else
# vault at all? If not — project not yet bootstrapped onto keep, or echo " ✗ $project failed"
# this box was never granted access — fall through to the plain path. FAILED+=("$project")
# Every keep call below is allowed to fail; failure just means "keep fi
# isn't available for this project," not "the deploy is broken." done
if keep pull "$vault" --out .env.new 2>/dev/null; then
mv .env.new .env
docker compose pull -q
docker compose up -d
else
rm -f .env.new
# No keep vault (or no grant) for this project — deploy exactly as it
# would have worked before keep existed: whatever .env is already
# sitting here (possibly none, if the compose file doesn't need one),
# plain pull + up. This path must never hard-fail just because keep
# doesn't know about this project.
docker compose pull -q
docker compose up -d
fi
``` ```
Vault key convention: `<project>/production`, using the exact directory
name already in `PROJECTS` — no separate mapping table to keep in sync.
Notice `announce`/`watch` aren't actually load-bearing in this version — Notice `announce`/`watch` aren't actually load-bearing in this version —
the agent above just runs on a timer and lets `docker compose pull` whatever already triggers this script (cron, systemd timer, or still
figure out for itself whether there's anything new (a no-op if the run by hand) keeps triggering it exactly as before, and `keep pull`
digest hasn't changed, idempotent either way). That's deliberate: **a either updates `.env` or silently doesn't. That's deliberate: **the
dumb periodic `compose pull && up -d` is already a complete, correct existing script, patched this way, is already a complete, correct
baseline deploy strategy on its own**, keep-bootstrapped or not. `announce` baseline** — keep-bootstrapped projects get fresh secrets on every run,
sits on top as an optimization for bootstrapped projects, not a everything else behaves identically to today. `announce` sits on top as
requirement for the agent to function — see the next section. a latency optimization *for however this script gets triggered*, not a
requirement for this patch to be complete — see the next section.
### 5. Where `announce`/`watch` actually earn their keep ### 5. Where `announce`/`watch` actually earn their keep
For a project that *has* been bootstrapped, there are two independent Two independent benefits, worth keeping conceptually separate — and the
benefits, worth keeping conceptually separate: second one looks different than it would for a one-agent-per-project
design, because the real script is one bulk all-or-nothing sweep, not N
independent watchers:
- **Secrets sourcing** — `keep pull` instead of a hand-placed `.env`. - **Secrets sourcing** — `keep pull` instead of a hand-placed `.env`.
Available the moment a vault exists and this machine has a grant. Available per-project the moment its vault exists and this machine
Already true of `deploy.sh` today; unchanged by this proposal. has a grant on it. This is what §4's patch already delivers, on its
- **Fast, precise redeploy** — instead of every VPS blindly polling own, with zero dependency on `announce`.
`docker compose pull` on a fixed timer for every project it hosts - **Waking the sweep early** — however the bulk script currently gets
(harmless, but imprecise and up to a full interval late), a triggered (cron, systemd timer, or still by hand), `announce` could
bootstrapped project's agent can `keep watch` and react within let it wake immediately on a push instead of waiting for the next
seconds of the CI `announce` step, and only for the project that scheduled run. The natural fit given the script's actual shape: `keep
actually changed rather than sweeping all of them. watch-any <vault1> <vault2> ...` blocking until *any* watched vault's
tag changes, used as the trigger for a full sweep — not a per-project
selective redeploy. Selective ("only redeploy the one project that
actually changed") would require reshaping the script away from its
current fixed-array-sweep model into something that takes a target
project as an argument, which is a real behavior change to something
already in production use, not a small addition — out of scope here.
A fancier version of the agent script could branch on `keep watch` Deliberately **not** building the wake-the-sweep-early mechanism in this
succeeding vs. not to decide between "wait for a watch event" and "just pass. Ship §4's patch first — it already satisfies the fallback
poll on the dumb timer" — deliberately **not** speccing that split here. requirement completely and delivers the secrets-sourcing benefit on its
Ship the dumb-timer-plus-opportunistic-`keep-pull` version first (above); own. Whether waiting up to one scheduled interval for a redeploy is
it already satisfies the fallback requirement completely, and whether actually a problem worth solving with `announce`/`watch-any` should be
the watch-driven fast path is worth the added complexity should be judged after living with the patched bulk script for a while, not
judged after living with the dumb version for a while, not designed in designed in advance of any real signal that the wait matters.
advance of any real signal that polling latency is actually a problem.
### 6. CI workflow addition ### 6. CI workflow addition
@@ -249,16 +287,15 @@ not-yet-bootstrapped project, not a red X on the workflow.
## Open questions ## Open questions
- **Per-project agent scheduling** — one cron line per project, or one - **Fleet-wide `pull_policy: always` sweep** — a prerequisite this
script that loops over every directory under `$DEPLOY_ROOT`? Leaning proposal depends on but doesn't itself fix (see Context). Should
toward one-line-per-project (explicit, greppable in crontab, no happen before or alongside this work, as its own small pass, not
surprise when a new project directory silently starts getting picked bundled into this one.
up) over auto-discovery. - **Does `watch`/`watch-any` ever get used**, or does the
- **Does `watch` ever get used**, or does the dumb-timer version turn already-scheduled bulk sweep turn out to be good enough forever?
out to be good enough forever? Deliberately left unbuilt (see "Where Deliberately left unbuilt (see §5) until there's a real signal the
announce/watch actually earn their keep") until there's a real signal wait between scheduled runs is worth solving.
it's worth the added complexity.
- **Multi-VPS fan-out for one project** — if a project ever runs on more - **Multi-VPS fan-out for one project** — if a project ever runs on more
than one box, `announce` already supports that for free (any number of than one box, `announce` already supports that for free (any number of
agents can poll the same vault's announce endpoint), but this hasn't watchers can poll the same vault's announce endpoint), but this hasn't
been exercised and might reveal something not accounted for here. been exercised and might reveal something not accounted for here.