Files
keep/INTEGRATION.md

174 lines
7.3 KiB
Markdown
Raw Normal View History

Implement keep: self-hosted E2E encrypted secrets sync Server: Express + better-sqlite3 (WAL), multi-recipient key-wrapping per IMPLEMENTATION.md's design — vaults/recipients/vault_grants/ access_log. Two auth paths: ADMIN_PASSWORD header for recipient management and revoke (pure metadata operations), signed-request auth (Ed25519 signature over method+path+timestamp+body-hash) for push/pull/grant, mirroring the spirit of this project family's other signed-handshake patterns without naming them. CLI: identity init/show/set-id, push/pull/grant/log, admin recipient add/list/remove and revoke. Grant is a client-side crypto operation (the granter unwraps the vault's current key locally and reseals it for the new recipient) rather than a server-side operation, since the server never holds an unwrapped key to grant with. Verified end-to-end with two independent local identities against a live server and separately against the built Docker image: register, push, pull (granted and ungranted), grant without re-pushing, admin revoke, a subsequent rotation confirming the revoked recipient stays excluded, and rejection of missing/malformed signed-request auth. Two real bugs caught during verification, not just written up: - libsodium-wrappers' published ESM build does a relative import only resolvable under bundler-style resolution — broken under plain Node ESM. Fixed via createRequire to force the CJS build. - Express's req.path inside a sub-router is relative to the mount point, which would have silently mismatched a client signing the full request path. Fixed by verifying against req.originalUrl. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 19:38:24 +02:00
# Incorporating keep into an existing deploy setup
Concrete, by deploy *pattern* rather than by naming specific projects —
the shape of the fit depends on how a project currently gets its config
onto a target machine, not on what the project is.
## The general shape
Wherever a deploy script or a manually-maintained `.env` exists today, the
change is the same: replace "assume `.env` is already sitting there,
hand-copied at some point" with a `keep pull` at the top of the script.
```bash
# before — a deploy script assumes .env already exists on this machine
set -euo pipefail
source .env # or docker-compose reads it directly
# after
set -euo pipefail
keep pull <project>/<env> --out .env
```
The one-time setup cost per machine: `keep identity init` once, send the
printed public key to whoever runs `keep recipient add`, then
`keep identity set-id <id>` with what comes back. After that, every future
deploy from that machine just works — no more "wait, what's the DB
password again" when setting up a new box.
A deploy identity should almost always be granted **read-only** access
(`keep grant <vault> <deploy-recipient-id> --read-only`) — it only ever
needs to `pull`, and a read-only grant means a compromised deploy box
can't also rotate the vault or add itself more recipients.
When you do the first `keep push` for a project, push the *real*,
hand-maintained `.env` rather than a stripped-down one — any `# get this
from X` / `# rotate quarterly` comments in it get preserved (values
redacted) as a template alongside the secrets, retrievable later with
`keep pull <vault> --template`. That's the one moment this context is
available at all; a `.env` reconstructed later from memory won't have it.
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed The CLI only touches Node builtins and libsodium-wrappers (no native addons like better-sqlite3, which the server needs but the CLI never imports) -- confirmed by checking its actual import graph, not assumed. That makes it a good candidate for a single-file esbuild bundle rather than requiring a full git checkout + npm install on every machine that needs to run `keep pull`. deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new convention. No git, no npm install, no node_modules on the target at all, just the one file plus a `node` runtime already there from whatever else is deployed on that box. Caught a real bug while building this: --banner:js="#!/usr/bin/env node" duplicated the shebang esbuild already preserves from the source file automatically, landing it on line 2 instead of line 1 -- Node only special-cases a shebang on line 1, so the bundle threw a syntax error on every invocation until the redundant banner flag was removed. Verified the bundle for real, not just that it builds: ran `keep identity init` / `identity show` against it directly and confirmed a real Ed25519 keypair comes out the other end -- the specific thing worth checking given libsodium-wrappers needed a createRequire workaround for its own broken ESM packaging, and bundling could easily have broken that differently. Also documents both CLI-provisioning paths in INTEGRATION.md (scripts/install-cli.sh for a machine you're fine cloning onto, deploy-cli.sh for one you'd rather not) as a "getting keep onto a machine" prerequisite section, ahead of the existing deploy-pattern docs that all assume it's already there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
## Getting the `keep` CLI onto a machine
Chicken-and-egg: every pattern below assumes `keep` is already a runnable
command on the machine doing the pulling. Two ways to get there, pick
based on whether you're fine with a git checkout existing on that
machine:
**A machine you're already developing on, or don't mind cloning onto**
(your laptop, a VPS you administer directly):
```bash
git clone <this repo>
cd keep
npm install
./scripts/install-cli.sh # builds, symlinks dist/cli/index.js onto PATH as `keep`
```
Symlinked rather than copied — a future `git pull && npm run build` is
the entire upgrade story, no need to re-run the install script.
**A deploy target you'd rather not clone a git repo onto** — bundle the
Fix deploy-cli.sh: bundle was completely broken on a real host Caught live, deployed to the actual target -- the previous version failed on the real host with "Cannot use import statement outside a module" (Node 18, no ancestor package.json to signal ESM for an extensionless file). That surfaced two stacked bugs my own testing had missed by running the bundle from inside this repo's own directory tree, which coincidentally supplied both things the standalone artifact was silently depending on: 1. Module-format ambiguity: an extensionless file with no controlling package.json defaults to CommonJS on older Node (no auto-detection heuristic before recent versions). Fixed by naming the bundle output keep.mjs -- unconditionally ESM on any Node version, regardless of extension-less-file heuristics or nearby package.json. 2. The bigger one: crypto.ts's createRequire(import.meta.url) trick (needed because libsodium-wrappers' published ESM build follows a broken relative import) is opaque to esbuild's static bundler -- it's a runtime-obtained require reference, not the literal `require` token esbuild's bundling recognizes. The previous "18kb bundle" never actually contained libsodium-wrappers at all; it silently relied on real node_modules being nearby on disk, which was only ever true by accident when testing from within this repo. On a clean host it threw "Cannot find module 'libsodium-wrappers'". Tried forcing static inlining via a literal require() call (esbuild does special-case that even in ESM-format output, unlike `import`, which is permanently pinned to the "import" resolution condition and can't be routed to the package's working CJS build no matter what --conditions/--main-fields/--alias combination is tried this was tried and confirmed exhaustively). That got real resolution and a real 1.7MB bundle, but broke at runtime with "No secure random number generator found" -- inlining the WASM engine breaks its own Node-crypto feature detection. Landed on: ship libsodium-wrappers (and its own dependency, libsodium) as real, unmodified package files alongside the ~18kb bundle, via a new copy-cli-deps.mjs step, rather than fighting further to force single-file inlining. ~1.6MB total, still one tar stream over SSH, still the entire deploy step. deploy-cli.sh now tars dist-bundle/ (bundle + node_modules) instead of catting a single file, extracts to ~/.keep-cli/ on the target, and symlinks ~/bin/keep to the entry point inside it. Verified this time in conditions that actually match the failure: full isolation (mktemp'd HOME and install dir, no ancestor package.json, no adjacent node_modules from this repo) for both `node keep.mjs` and direct shebang execution, plus the exact tar/extract cycle deploy-cli.sh performs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:59:44 +02:00
CLI and ship it over SSH, the same way flit/waste-go's `deploy-*.sh`
scripts ship a compiled binary:
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed The CLI only touches Node builtins and libsodium-wrappers (no native addons like better-sqlite3, which the server needs but the CLI never imports) -- confirmed by checking its actual import graph, not assumed. That makes it a good candidate for a single-file esbuild bundle rather than requiring a full git checkout + npm install on every machine that needs to run `keep pull`. deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new convention. No git, no npm install, no node_modules on the target at all, just the one file plus a `node` runtime already there from whatever else is deployed on that box. Caught a real bug while building this: --banner:js="#!/usr/bin/env node" duplicated the shebang esbuild already preserves from the source file automatically, landing it on line 2 instead of line 1 -- Node only special-cases a shebang on line 1, so the bundle threw a syntax error on every invocation until the redundant banner flag was removed. Verified the bundle for real, not just that it builds: ran `keep identity init` / `identity show` against it directly and confirmed a real Ed25519 keypair comes out the other end -- the specific thing worth checking given libsodium-wrappers needed a createRequire workaround for its own broken ESM packaging, and bundling could easily have broken that differently. Also documents both CLI-provisioning paths in INTEGRATION.md (scripts/install-cli.sh for a machine you're fine cloning onto, deploy-cli.sh for one you'd rather not) as a "getting keep onto a machine" prerequisite section, ahead of the existing deploy-pattern docs that all assume it's already there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
```bash
HOST=user@your-vps ./deploy-cli.sh
```
Fix deploy-cli.sh: bundle was completely broken on a real host Caught live, deployed to the actual target -- the previous version failed on the real host with "Cannot use import statement outside a module" (Node 18, no ancestor package.json to signal ESM for an extensionless file). That surfaced two stacked bugs my own testing had missed by running the bundle from inside this repo's own directory tree, which coincidentally supplied both things the standalone artifact was silently depending on: 1. Module-format ambiguity: an extensionless file with no controlling package.json defaults to CommonJS on older Node (no auto-detection heuristic before recent versions). Fixed by naming the bundle output keep.mjs -- unconditionally ESM on any Node version, regardless of extension-less-file heuristics or nearby package.json. 2. The bigger one: crypto.ts's createRequire(import.meta.url) trick (needed because libsodium-wrappers' published ESM build follows a broken relative import) is opaque to esbuild's static bundler -- it's a runtime-obtained require reference, not the literal `require` token esbuild's bundling recognizes. The previous "18kb bundle" never actually contained libsodium-wrappers at all; it silently relied on real node_modules being nearby on disk, which was only ever true by accident when testing from within this repo. On a clean host it threw "Cannot find module 'libsodium-wrappers'". Tried forcing static inlining via a literal require() call (esbuild does special-case that even in ESM-format output, unlike `import`, which is permanently pinned to the "import" resolution condition and can't be routed to the package's working CJS build no matter what --conditions/--main-fields/--alias combination is tried this was tried and confirmed exhaustively). That got real resolution and a real 1.7MB bundle, but broke at runtime with "No secure random number generator found" -- inlining the WASM engine breaks its own Node-crypto feature detection. Landed on: ship libsodium-wrappers (and its own dependency, libsodium) as real, unmodified package files alongside the ~18kb bundle, via a new copy-cli-deps.mjs step, rather than fighting further to force single-file inlining. ~1.6MB total, still one tar stream over SSH, still the entire deploy step. deploy-cli.sh now tars dist-bundle/ (bundle + node_modules) instead of catting a single file, extracts to ~/.keep-cli/ on the target, and symlinks ~/bin/keep to the entry point inside it. Verified this time in conditions that actually match the failure: full isolation (mktemp'd HOME and install dir, no ancestor package.json, no adjacent node_modules from this repo) for both `node keep.mjs` and direct shebang execution, plus the exact tar/extract cycle deploy-cli.sh performs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:59:44 +02:00
Not quite a single file: the bundle itself (esbuild, ~18kb) covers
everything except `libsodium-wrappers`, which ships alongside it as the
real, unmodified package (~1.6MB total with its own dependency,
`libsodium`) rather than being force-inlined. That package's published
ESM build follows a broken relative import (see `src/shared/crypto.ts`'s
own comment on this), and statically inlining it via esbuild — technically
possible by routing through a literal `require()` call instead of
`import` — breaks its WASM engine's own Node-crypto feature detection at
runtime ("No secure random number generator found"), confirmed by
testing, not assumed. Shipping the real package is more reliable than
fighting that.
The script tars `dist-bundle/` (the esbuild output plus that one
dependency) and extracts it to `~/.keep-cli/` on the target over SSH,
then symlinks `~/bin/keep` to the entry point inside it — no git, no npm
install on that machine at all, just a `node` runtime (already there, if
it's running any of the other Node-based projects in this family). Re-run
it any time the CLI changes; safe to overwrite.
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed The CLI only touches Node builtins and libsodium-wrappers (no native addons like better-sqlite3, which the server needs but the CLI never imports) -- confirmed by checking its actual import graph, not assumed. That makes it a good candidate for a single-file esbuild bundle rather than requiring a full git checkout + npm install on every machine that needs to run `keep pull`. deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new convention. No git, no npm install, no node_modules on the target at all, just the one file plus a `node` runtime already there from whatever else is deployed on that box. Caught a real bug while building this: --banner:js="#!/usr/bin/env node" duplicated the shebang esbuild already preserves from the source file automatically, landing it on line 2 instead of line 1 -- Node only special-cases a shebang on line 1, so the bundle threw a syntax error on every invocation until the redundant banner flag was removed. Verified the bundle for real, not just that it builds: ran `keep identity init` / `identity show` against it directly and confirmed a real Ed25519 keypair comes out the other end -- the specific thing worth checking given libsodium-wrappers needed a createRequire workaround for its own broken ESM packaging, and bundling could easily have broken that differently. Also documents both CLI-provisioning paths in INTEGRATION.md (scripts/install-cli.sh for a machine you're fine cloning onto, deploy-cli.sh for one you'd rather not) as a "getting keep onto a machine" prerequisite section, ahead of the existing deploy-pattern docs that all assume it's already there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
## Pattern: several docker-compose projects on one or more VPSes
The common shape once more than one or two projects are on `keep`:
each project lives in its own directory on the VPS
(`/opt/docker/<project>/docker-compose.yml`), and "deploy" means pulling
that project's env and running `docker compose up -d` in its directory.
`scripts/deploy.sh` in this repo is a small generic wrapper for exactly
that:
```bash
./deploy.sh myapp # pulls myapp/production, runs compose in /opt/docker/myapp
./deploy.sh myapp staging # pulls myapp/staging instead
DEPLOY_ROOT=/srv/apps ./deploy.sh myapp # different project root
```
One-time setup per VPS: `keep identity init` on that machine, register
its public key with `keep recipient add`, `keep identity set-id`, then
grant it **read-only** access to each project vault it deploys
(`keep grant <project>/production <vps-recipient-id> --read-only`). After
that, `./deploy.sh <project>` is the whole deploy step — no `.env`
hand-copied onto the box first.
For "which machine can deploy which project," `keep overview` (admin)
prints every vault and every recipient's access in one screen — the
thing to reach for instead of checking each vault's grants one at a
time.
Implement keep: self-hosted E2E encrypted secrets sync Server: Express + better-sqlite3 (WAL), multi-recipient key-wrapping per IMPLEMENTATION.md's design — vaults/recipients/vault_grants/ access_log. Two auth paths: ADMIN_PASSWORD header for recipient management and revoke (pure metadata operations), signed-request auth (Ed25519 signature over method+path+timestamp+body-hash) for push/pull/grant, mirroring the spirit of this project family's other signed-handshake patterns without naming them. CLI: identity init/show/set-id, push/pull/grant/log, admin recipient add/list/remove and revoke. Grant is a client-side crypto operation (the granter unwraps the vault's current key locally and reseals it for the new recipient) rather than a server-side operation, since the server never holds an unwrapped key to grant with. Verified end-to-end with two independent local identities against a live server and separately against the built Docker image: register, push, pull (granted and ungranted), grant without re-pushing, admin revoke, a subsequent rotation confirming the revoked recipient stays excluded, and rejection of missing/malformed signed-request auth. Two real bugs caught during verification, not just written up: - libsodium-wrappers' published ESM build does a relative import only resolvable under bundler-style resolution — broken under plain Node ESM. Fixed via createRequire to force the CJS build. - Express's req.path inside a sub-router is relative to the mount point, which would have silently mismatched a client signing the full request path. Fixed by verifying against req.originalUrl. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 19:38:24 +02:00
## Pattern: docker-compose reading a local `.env`
A common shape — `docker compose up -d` reads `environment:` values from
a `.env` file sitting next to `docker-compose.yml` on the deploy machine,
hand-maintained and containing real secrets.
```bash
keep pull myapp/production --out .env
docker compose up -d
```
No code changes needed in the target project — `keep` only changes *how
the `.env` file gets there*, not what reads it. This is the easiest
category to retrofit.
## Pattern: cross-compile + scp + SSH-restart deploy scripts
A shape where a deploy script builds a binary locally, ships it to a VPS,
and restarts a service over SSH — reading a local `.env` next to the
script for things like shared secrets that shouldn't be hardcoded or
re-typed on every invocation.
```bash
keep pull myapp/production --out .env
set -a && source .env && set +a
# ...rest of the deploy script unchanged
```
Same retrofit as the docker-compose case, just applied before whatever
build/ship step already exists.
## Pattern: a plaintext config file served to a browser
Not every `config.js`/similar file that *looks* like environment config
actually holds secrets — some of it is public-by-design (an API base URL,
a feature flag) that's explicitly meant to be visible to anyone loading
the page. `keep` solves a confidentiality problem; a file with nothing
confidential in it doesn't have one. Worth checking this distinction
before wiring `keep` into anything — retrofitting a non-secret config
file would be solving a problem that file doesn't have.
## A vault server's own configuration
Whatever runs `keep` itself needs its own `ADMIN_PASSWORD` to be a real,
manually-set secret on its own host — it can't bootstrap its own trust
root by pulling itself from itself. This is the same "root of trust
starts somewhere unmanaged" fact every secrets system has (Vault's own
unseal keys aren't stored in Vault either). Not a gap, just worth being
explicit about: `keep`'s own deploy stays on a plain `.env`/
`docker-compose.yml` environment variables.
## New projects going forward
The actual best use case isn't retrofitting existing deploys — it's never
having the "hand-reconstruct a `.env` from memory" moment for a *new*
deploy target in the first place. For whatever gets built next: register
its production host as a `keep` recipient as part of first setup, write
the deploy script to `keep pull` from day one, and the annoyance this was
built to fix never has a chance to happen.