1b4cd9b826
Caught live, deployed to the actual target -- the previous version failed on the real host with "Cannot use import statement outside a module" (Node 18, no ancestor package.json to signal ESM for an extensionless file). That surfaced two stacked bugs my own testing had missed by running the bundle from inside this repo's own directory tree, which coincidentally supplied both things the standalone artifact was silently depending on: 1. Module-format ambiguity: an extensionless file with no controlling package.json defaults to CommonJS on older Node (no auto-detection heuristic before recent versions). Fixed by naming the bundle output keep.mjs -- unconditionally ESM on any Node version, regardless of extension-less-file heuristics or nearby package.json. 2. The bigger one: crypto.ts's createRequire(import.meta.url) trick (needed because libsodium-wrappers' published ESM build follows a broken relative import) is opaque to esbuild's static bundler -- it's a runtime-obtained require reference, not the literal `require` token esbuild's bundling recognizes. The previous "18kb bundle" never actually contained libsodium-wrappers at all; it silently relied on real node_modules being nearby on disk, which was only ever true by accident when testing from within this repo. On a clean host it threw "Cannot find module 'libsodium-wrappers'". Tried forcing static inlining via a literal require() call (esbuild does special-case that even in ESM-format output, unlike `import`, which is permanently pinned to the "import" resolution condition and can't be routed to the package's working CJS build no matter what --conditions/--main-fields/--alias combination is tried this was tried and confirmed exhaustively). That got real resolution and a real 1.7MB bundle, but broke at runtime with "No secure random number generator found" -- inlining the WASM engine breaks its own Node-crypto feature detection. Landed on: ship libsodium-wrappers (and its own dependency, libsodium) as real, unmodified package files alongside the ~18kb bundle, via a new copy-cli-deps.mjs step, rather than fighting further to force single-file inlining. ~1.6MB total, still one tar stream over SSH, still the entire deploy step. deploy-cli.sh now tars dist-bundle/ (bundle + node_modules) instead of catting a single file, extracts to ~/.keep-cli/ on the target, and symlinks ~/bin/keep to the entry point inside it. Verified this time in conditions that actually match the failure: full isolation (mktemp'd HOME and install dir, no ancestor package.json, no adjacent node_modules from this repo) for both `node keep.mjs` and direct shebang execution, plus the exact tar/extract cycle deploy-cli.sh performs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
167 lines
6.9 KiB
Markdown
167 lines
6.9 KiB
Markdown
# Incorporating keep into an existing deploy setup
|
|
|
|
Concrete, by deploy *pattern* rather than by naming specific projects —
|
|
the shape of the fit depends on how a project currently gets its config
|
|
onto a target machine, not on what the project is.
|
|
|
|
## The general shape
|
|
|
|
Wherever a deploy script or a manually-maintained `.env` exists today, the
|
|
change is the same: replace "assume `.env` is already sitting there,
|
|
hand-copied at some point" with a `keep pull` at the top of the script.
|
|
|
|
```bash
|
|
# before — a deploy script assumes .env already exists on this machine
|
|
set -euo pipefail
|
|
source .env # or docker-compose reads it directly
|
|
|
|
# after
|
|
set -euo pipefail
|
|
keep pull <project>/<env> --out .env
|
|
```
|
|
|
|
The one-time setup cost per machine: `keep identity init` once, send the
|
|
printed public key to whoever runs `keep recipient add`, then
|
|
`keep identity set-id <id>` with what comes back. After that, every future
|
|
deploy from that machine just works — no more "wait, what's the DB
|
|
password again" when setting up a new box.
|
|
|
|
A deploy identity should almost always be granted **read-only** access
|
|
(`keep grant <vault> <deploy-recipient-id> --read-only`) — it only ever
|
|
needs to `pull`, and a read-only grant means a compromised deploy box
|
|
can't also rotate the vault or add itself more recipients.
|
|
|
|
## Getting the `keep` CLI onto a machine
|
|
|
|
Chicken-and-egg: every pattern below assumes `keep` is already a runnable
|
|
command on the machine doing the pulling. Two ways to get there, pick
|
|
based on whether you're fine with a git checkout existing on that
|
|
machine:
|
|
|
|
**A machine you're already developing on, or don't mind cloning onto**
|
|
(your laptop, a VPS you administer directly):
|
|
|
|
```bash
|
|
git clone <this repo>
|
|
cd keep
|
|
npm install
|
|
./scripts/install-cli.sh # builds, symlinks dist/cli/index.js onto PATH as `keep`
|
|
```
|
|
|
|
Symlinked rather than copied — a future `git pull && npm run build` is
|
|
the entire upgrade story, no need to re-run the install script.
|
|
|
|
**A deploy target you'd rather not clone a git repo onto** — bundle the
|
|
CLI and ship it over SSH, the same way flit/waste-go's `deploy-*.sh`
|
|
scripts ship a compiled binary:
|
|
|
|
```bash
|
|
HOST=user@your-vps ./deploy-cli.sh
|
|
```
|
|
|
|
Not quite a single file: the bundle itself (esbuild, ~18kb) covers
|
|
everything except `libsodium-wrappers`, which ships alongside it as the
|
|
real, unmodified package (~1.6MB total with its own dependency,
|
|
`libsodium`) rather than being force-inlined. That package's published
|
|
ESM build follows a broken relative import (see `src/shared/crypto.ts`'s
|
|
own comment on this), and statically inlining it via esbuild — technically
|
|
possible by routing through a literal `require()` call instead of
|
|
`import` — breaks its WASM engine's own Node-crypto feature detection at
|
|
runtime ("No secure random number generator found"), confirmed by
|
|
testing, not assumed. Shipping the real package is more reliable than
|
|
fighting that.
|
|
|
|
The script tars `dist-bundle/` (the esbuild output plus that one
|
|
dependency) and extracts it to `~/.keep-cli/` on the target over SSH,
|
|
then symlinks `~/bin/keep` to the entry point inside it — no git, no npm
|
|
install on that machine at all, just a `node` runtime (already there, if
|
|
it's running any of the other Node-based projects in this family). Re-run
|
|
it any time the CLI changes; safe to overwrite.
|
|
|
|
## Pattern: several docker-compose projects on one or more VPSes
|
|
|
|
The common shape once more than one or two projects are on `keep`:
|
|
each project lives in its own directory on the VPS
|
|
(`/opt/docker/<project>/docker-compose.yml`), and "deploy" means pulling
|
|
that project's env and running `docker compose up -d` in its directory.
|
|
`scripts/deploy.sh` in this repo is a small generic wrapper for exactly
|
|
that:
|
|
|
|
```bash
|
|
./deploy.sh myapp # pulls myapp/production, runs compose in /opt/docker/myapp
|
|
./deploy.sh myapp staging # pulls myapp/staging instead
|
|
DEPLOY_ROOT=/srv/apps ./deploy.sh myapp # different project root
|
|
```
|
|
|
|
One-time setup per VPS: `keep identity init` on that machine, register
|
|
its public key with `keep recipient add`, `keep identity set-id`, then
|
|
grant it **read-only** access to each project vault it deploys
|
|
(`keep grant <project>/production <vps-recipient-id> --read-only`). After
|
|
that, `./deploy.sh <project>` is the whole deploy step — no `.env`
|
|
hand-copied onto the box first.
|
|
|
|
For "which machine can deploy which project," `keep overview` (admin)
|
|
prints every vault and every recipient's access in one screen — the
|
|
thing to reach for instead of checking each vault's grants one at a
|
|
time.
|
|
|
|
## Pattern: docker-compose reading a local `.env`
|
|
|
|
A common shape — `docker compose up -d` reads `environment:` values from
|
|
a `.env` file sitting next to `docker-compose.yml` on the deploy machine,
|
|
hand-maintained and containing real secrets.
|
|
|
|
```bash
|
|
keep pull myapp/production --out .env
|
|
docker compose up -d
|
|
```
|
|
|
|
No code changes needed in the target project — `keep` only changes *how
|
|
the `.env` file gets there*, not what reads it. This is the easiest
|
|
category to retrofit.
|
|
|
|
## Pattern: cross-compile + scp + SSH-restart deploy scripts
|
|
|
|
A shape where a deploy script builds a binary locally, ships it to a VPS,
|
|
and restarts a service over SSH — reading a local `.env` next to the
|
|
script for things like shared secrets that shouldn't be hardcoded or
|
|
re-typed on every invocation.
|
|
|
|
```bash
|
|
keep pull myapp/production --out .env
|
|
set -a && source .env && set +a
|
|
# ...rest of the deploy script unchanged
|
|
```
|
|
|
|
Same retrofit as the docker-compose case, just applied before whatever
|
|
build/ship step already exists.
|
|
|
|
## Pattern: a plaintext config file served to a browser
|
|
|
|
Not every `config.js`/similar file that *looks* like environment config
|
|
actually holds secrets — some of it is public-by-design (an API base URL,
|
|
a feature flag) that's explicitly meant to be visible to anyone loading
|
|
the page. `keep` solves a confidentiality problem; a file with nothing
|
|
confidential in it doesn't have one. Worth checking this distinction
|
|
before wiring `keep` into anything — retrofitting a non-secret config
|
|
file would be solving a problem that file doesn't have.
|
|
|
|
## A vault server's own configuration
|
|
|
|
Whatever runs `keep` itself needs its own `ADMIN_PASSWORD` to be a real,
|
|
manually-set secret on its own host — it can't bootstrap its own trust
|
|
root by pulling itself from itself. This is the same "root of trust
|
|
starts somewhere unmanaged" fact every secrets system has (Vault's own
|
|
unseal keys aren't stored in Vault either). Not a gap, just worth being
|
|
explicit about: `keep`'s own deploy stays on a plain `.env`/
|
|
`docker-compose.yml` environment variables.
|
|
|
|
## New projects going forward
|
|
|
|
The actual best use case isn't retrofitting existing deploys — it's never
|
|
having the "hand-reconstruct a `.env` from memory" moment for a *new*
|
|
deploy target in the first place. For whatever gets built next: register
|
|
its production host as a `keep` recipient as part of first setup, write
|
|
the deploy script to `keep pull` from day one, and the annoyance this was
|
|
built to fix never has a chance to happen.
|