Implement keep: self-hosted E2E encrypted secrets sync
Server: Express + better-sqlite3 (WAL), multi-recipient key-wrapping
per IMPLEMENTATION.md's design — vaults/recipients/vault_grants/
access_log. Two auth paths: ADMIN_PASSWORD header for recipient
management and revoke (pure metadata operations), signed-request
auth (Ed25519 signature over method+path+timestamp+body-hash) for
push/pull/grant, mirroring the spirit of this project family's other
signed-handshake patterns without naming them.
CLI: identity init/show/set-id, push/pull/grant/log, admin recipient
add/list/remove and revoke. Grant is a client-side crypto operation
(the granter unwraps the vault's current key locally and reseals it
for the new recipient) rather than a server-side operation, since the
server never holds an unwrapped key to grant with.
Verified end-to-end with two independent local identities against a
live server and separately against the built Docker image: register,
push, pull (granted and ungranted), grant without re-pushing, admin
revoke, a subsequent rotation confirming the revoked recipient stays
excluded, and rejection of missing/malformed signed-request auth.
Two real bugs caught during verification, not just written up:
- libsodium-wrappers' published ESM build does a relative import only
resolvable under bundler-style resolution — broken under plain Node
ESM. Fixed via createRequire to force the CJS build.
- Express's req.path inside a sub-router is relative to the mount
point, which would have silently mismatched a client signing the
full request path. Fixed by verifying against req.originalUrl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 19:38:24 +02:00
|
|
|
# Incorporating keep into an existing deploy setup
|
|
|
|
|
|
|
|
|
|
Concrete, by deploy *pattern* rather than by naming specific projects —
|
|
|
|
|
the shape of the fit depends on how a project currently gets its config
|
|
|
|
|
onto a target machine, not on what the project is.
|
|
|
|
|
|
|
|
|
|
## The general shape
|
|
|
|
|
|
|
|
|
|
Wherever a deploy script or a manually-maintained `.env` exists today, the
|
|
|
|
|
change is the same: replace "assume `.env` is already sitting there,
|
|
|
|
|
hand-copied at some point" with a `keep pull` at the top of the script.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# before — a deploy script assumes .env already exists on this machine
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
source .env # or docker-compose reads it directly
|
|
|
|
|
|
|
|
|
|
# after
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
keep pull <project>/<env> --out .env
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The one-time setup cost per machine: `keep identity init` once, send the
|
|
|
|
|
printed public key to whoever runs `keep recipient add`, then
|
|
|
|
|
`keep identity set-id <id>` with what comes back. After that, every future
|
|
|
|
|
deploy from that machine just works — no more "wait, what's the DB
|
|
|
|
|
password again" when setting up a new box.
|
|
|
|
|
|
2026-07-12 19:54:45 +02:00
|
|
|
A deploy identity should almost always be granted **read-only** access
|
|
|
|
|
(`keep grant <vault> <deploy-recipient-id> --read-only`) — it only ever
|
|
|
|
|
needs to `pull`, and a read-only grant means a compromised deploy box
|
|
|
|
|
can't also rotate the vault or add itself more recipients.
|
|
|
|
|
|
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed
The CLI only touches Node builtins and libsodium-wrappers (no native
addons like better-sqlite3, which the server needs but the CLI never
imports) -- confirmed by checking its actual import graph, not assumed.
That makes it a good candidate for a single-file esbuild bundle rather
than requiring a full git checkout + npm install on every machine that
needs to run `keep pull`.
deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host
over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's
deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new
convention. No git, no npm install, no node_modules on the target at
all, just the one file plus a `node` runtime already there from
whatever else is deployed on that box.
Caught a real bug while building this: --banner:js="#!/usr/bin/env
node" duplicated the shebang esbuild already preserves from the source
file automatically, landing it on line 2 instead of line 1 -- Node
only special-cases a shebang on line 1, so the bundle threw a syntax
error on every invocation until the redundant banner flag was removed.
Verified the bundle for real, not just that it builds: ran `keep
identity init` / `identity show` against it directly and confirmed a
real Ed25519 keypair comes out the other end -- the specific thing
worth checking given libsodium-wrappers needed a createRequire
workaround for its own broken ESM packaging, and bundling could easily
have broken that differently.
Also documents both CLI-provisioning paths in INTEGRATION.md
(scripts/install-cli.sh for a machine you're fine cloning onto,
deploy-cli.sh for one you'd rather not) as a "getting keep onto a
machine" prerequisite section, ahead of the existing deploy-pattern
docs that all assume it's already there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
|
|
|
## Getting the `keep` CLI onto a machine
|
|
|
|
|
|
|
|
|
|
Chicken-and-egg: every pattern below assumes `keep` is already a runnable
|
|
|
|
|
command on the machine doing the pulling. Two ways to get there, pick
|
|
|
|
|
based on whether you're fine with a git checkout existing on that
|
|
|
|
|
machine:
|
|
|
|
|
|
|
|
|
|
**A machine you're already developing on, or don't mind cloning onto**
|
|
|
|
|
(your laptop, a VPS you administer directly):
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
git clone <this repo>
|
|
|
|
|
cd keep
|
|
|
|
|
npm install
|
|
|
|
|
./scripts/install-cli.sh # builds, symlinks dist/cli/index.js onto PATH as `keep`
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Symlinked rather than copied — a future `git pull && npm run build` is
|
|
|
|
|
the entire upgrade story, no need to re-run the install script.
|
|
|
|
|
|
|
|
|
|
**A deploy target you'd rather not clone a git repo onto** — bundle the
|
Fix deploy-cli.sh: bundle was completely broken on a real host
Caught live, deployed to the actual target -- the previous version
failed on the real host with "Cannot use import statement outside a
module" (Node 18, no ancestor package.json to signal ESM for an
extensionless file). That surfaced two stacked bugs my own testing had
missed by running the bundle from inside this repo's own directory
tree, which coincidentally supplied both things the standalone
artifact was silently depending on:
1. Module-format ambiguity: an extensionless file with no controlling
package.json defaults to CommonJS on older Node (no auto-detection
heuristic before recent versions). Fixed by naming the bundle
output keep.mjs -- unconditionally ESM on any Node version,
regardless of extension-less-file heuristics or nearby package.json.
2. The bigger one: crypto.ts's createRequire(import.meta.url) trick
(needed because libsodium-wrappers' published ESM build follows a
broken relative import) is opaque to esbuild's static bundler --
it's a runtime-obtained require reference, not the literal `require`
token esbuild's bundling recognizes. The previous "18kb bundle"
never actually contained libsodium-wrappers at all; it silently
relied on real node_modules being nearby on disk, which was only
ever true by accident when testing from within this repo. On a
clean host it threw "Cannot find module 'libsodium-wrappers'".
Tried forcing static inlining via a literal require() call (esbuild
does special-case that even in ESM-format output, unlike `import`,
which is permanently pinned to the "import" resolution condition
and can't be routed to the package's working CJS build no matter
what --conditions/--main-fields/--alias combination is tried this
was tried and confirmed exhaustively). That got real resolution and
a real 1.7MB bundle, but broke at runtime with "No secure random
number generator found" -- inlining the WASM engine breaks its own
Node-crypto feature detection.
Landed on: ship libsodium-wrappers (and its own dependency,
libsodium) as real, unmodified package files alongside the ~18kb
bundle, via a new copy-cli-deps.mjs step, rather than fighting
further to force single-file inlining. ~1.6MB total, still one tar
stream over SSH, still the entire deploy step.
deploy-cli.sh now tars dist-bundle/ (bundle + node_modules) instead of
catting a single file, extracts to ~/.keep-cli/ on the target, and
symlinks ~/bin/keep to the entry point inside it.
Verified this time in conditions that actually match the failure: full
isolation (mktemp'd HOME and install dir, no ancestor package.json, no
adjacent node_modules from this repo) for both `node keep.mjs` and
direct shebang execution, plus the exact tar/extract cycle
deploy-cli.sh performs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:59:44 +02:00
|
|
|
CLI and ship it over SSH, the same way flit/waste-go's `deploy-*.sh`
|
|
|
|
|
scripts ship a compiled binary:
|
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed
The CLI only touches Node builtins and libsodium-wrappers (no native
addons like better-sqlite3, which the server needs but the CLI never
imports) -- confirmed by checking its actual import graph, not assumed.
That makes it a good candidate for a single-file esbuild bundle rather
than requiring a full git checkout + npm install on every machine that
needs to run `keep pull`.
deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host
over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's
deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new
convention. No git, no npm install, no node_modules on the target at
all, just the one file plus a `node` runtime already there from
whatever else is deployed on that box.
Caught a real bug while building this: --banner:js="#!/usr/bin/env
node" duplicated the shebang esbuild already preserves from the source
file automatically, landing it on line 2 instead of line 1 -- Node
only special-cases a shebang on line 1, so the bundle threw a syntax
error on every invocation until the redundant banner flag was removed.
Verified the bundle for real, not just that it builds: ran `keep
identity init` / `identity show` against it directly and confirmed a
real Ed25519 keypair comes out the other end -- the specific thing
worth checking given libsodium-wrappers needed a createRequire
workaround for its own broken ESM packaging, and bundling could easily
have broken that differently.
Also documents both CLI-provisioning paths in INTEGRATION.md
(scripts/install-cli.sh for a machine you're fine cloning onto,
deploy-cli.sh for one you'd rather not) as a "getting keep onto a
machine" prerequisite section, ahead of the existing deploy-pattern
docs that all assume it's already there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
HOST=user@your-vps ./deploy-cli.sh
|
|
|
|
|
```
|
|
|
|
|
|
Fix deploy-cli.sh: bundle was completely broken on a real host
Caught live, deployed to the actual target -- the previous version
failed on the real host with "Cannot use import statement outside a
module" (Node 18, no ancestor package.json to signal ESM for an
extensionless file). That surfaced two stacked bugs my own testing had
missed by running the bundle from inside this repo's own directory
tree, which coincidentally supplied both things the standalone
artifact was silently depending on:
1. Module-format ambiguity: an extensionless file with no controlling
package.json defaults to CommonJS on older Node (no auto-detection
heuristic before recent versions). Fixed by naming the bundle
output keep.mjs -- unconditionally ESM on any Node version,
regardless of extension-less-file heuristics or nearby package.json.
2. The bigger one: crypto.ts's createRequire(import.meta.url) trick
(needed because libsodium-wrappers' published ESM build follows a
broken relative import) is opaque to esbuild's static bundler --
it's a runtime-obtained require reference, not the literal `require`
token esbuild's bundling recognizes. The previous "18kb bundle"
never actually contained libsodium-wrappers at all; it silently
relied on real node_modules being nearby on disk, which was only
ever true by accident when testing from within this repo. On a
clean host it threw "Cannot find module 'libsodium-wrappers'".
Tried forcing static inlining via a literal require() call (esbuild
does special-case that even in ESM-format output, unlike `import`,
which is permanently pinned to the "import" resolution condition
and can't be routed to the package's working CJS build no matter
what --conditions/--main-fields/--alias combination is tried this
was tried and confirmed exhaustively). That got real resolution and
a real 1.7MB bundle, but broke at runtime with "No secure random
number generator found" -- inlining the WASM engine breaks its own
Node-crypto feature detection.
Landed on: ship libsodium-wrappers (and its own dependency,
libsodium) as real, unmodified package files alongside the ~18kb
bundle, via a new copy-cli-deps.mjs step, rather than fighting
further to force single-file inlining. ~1.6MB total, still one tar
stream over SSH, still the entire deploy step.
deploy-cli.sh now tars dist-bundle/ (bundle + node_modules) instead of
catting a single file, extracts to ~/.keep-cli/ on the target, and
symlinks ~/bin/keep to the entry point inside it.
Verified this time in conditions that actually match the failure: full
isolation (mktemp'd HOME and install dir, no ancestor package.json, no
adjacent node_modules from this repo) for both `node keep.mjs` and
direct shebang execution, plus the exact tar/extract cycle
deploy-cli.sh performs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:59:44 +02:00
|
|
|
Not quite a single file: the bundle itself (esbuild, ~18kb) covers
|
|
|
|
|
everything except `libsodium-wrappers`, which ships alongside it as the
|
|
|
|
|
real, unmodified package (~1.6MB total with its own dependency,
|
|
|
|
|
`libsodium`) rather than being force-inlined. That package's published
|
|
|
|
|
ESM build follows a broken relative import (see `src/shared/crypto.ts`'s
|
|
|
|
|
own comment on this), and statically inlining it via esbuild — technically
|
|
|
|
|
possible by routing through a literal `require()` call instead of
|
|
|
|
|
`import` — breaks its WASM engine's own Node-crypto feature detection at
|
|
|
|
|
runtime ("No secure random number generator found"), confirmed by
|
|
|
|
|
testing, not assumed. Shipping the real package is more reliable than
|
|
|
|
|
fighting that.
|
|
|
|
|
|
|
|
|
|
The script tars `dist-bundle/` (the esbuild output plus that one
|
|
|
|
|
dependency) and extracts it to `~/.keep-cli/` on the target over SSH,
|
|
|
|
|
then symlinks `~/bin/keep` to the entry point inside it — no git, no npm
|
|
|
|
|
install on that machine at all, just a `node` runtime (already there, if
|
|
|
|
|
it's running any of the other Node-based projects in this family). Re-run
|
|
|
|
|
it any time the CLI changes; safe to overwrite.
|
Add deploy-cli.sh: ship the CLI as a single bundled file, no git clone needed
The CLI only touches Node builtins and libsodium-wrappers (no native
addons like better-sqlite3, which the server needs but the CLI never
imports) -- confirmed by checking its actual import graph, not assumed.
That makes it a good candidate for a single-file esbuild bundle rather
than requiring a full git checkout + npm install on every machine that
needs to run `keep pull`.
deploy-cli.sh bundles src/cli (~18kb) and ships it straight to a host
over SSH -- HOST=user@vps ./deploy-cli.sh -- mirroring flit's
deploy-daemon.sh ssh-cat-chmod pattern rather than inventing a new
convention. No git, no npm install, no node_modules on the target at
all, just the one file plus a `node` runtime already there from
whatever else is deployed on that box.
Caught a real bug while building this: --banner:js="#!/usr/bin/env
node" duplicated the shebang esbuild already preserves from the source
file automatically, landing it on line 2 instead of line 1 -- Node
only special-cases a shebang on line 1, so the bundle threw a syntax
error on every invocation until the redundant banner flag was removed.
Verified the bundle for real, not just that it builds: ran `keep
identity init` / `identity show` against it directly and confirmed a
real Ed25519 keypair comes out the other end -- the specific thing
worth checking given libsodium-wrappers needed a createRequire
workaround for its own broken ESM packaging, and bundling could easily
have broken that differently.
Also documents both CLI-provisioning paths in INTEGRATION.md
(scripts/install-cli.sh for a machine you're fine cloning onto,
deploy-cli.sh for one you'd rather not) as a "getting keep onto a
machine" prerequisite section, ahead of the existing deploy-pattern
docs that all assume it's already there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:45:31 +02:00
|
|
|
|
2026-07-12 20:16:29 +02:00
|
|
|
## Pattern: several docker-compose projects on one or more VPSes
|
|
|
|
|
|
|
|
|
|
The common shape once more than one or two projects are on `keep`:
|
|
|
|
|
each project lives in its own directory on the VPS
|
|
|
|
|
(`/opt/docker/<project>/docker-compose.yml`), and "deploy" means pulling
|
|
|
|
|
that project's env and running `docker compose up -d` in its directory.
|
|
|
|
|
`scripts/deploy.sh` in this repo is a small generic wrapper for exactly
|
|
|
|
|
that:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
./deploy.sh myapp # pulls myapp/production, runs compose in /opt/docker/myapp
|
|
|
|
|
./deploy.sh myapp staging # pulls myapp/staging instead
|
|
|
|
|
DEPLOY_ROOT=/srv/apps ./deploy.sh myapp # different project root
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
One-time setup per VPS: `keep identity init` on that machine, register
|
|
|
|
|
its public key with `keep recipient add`, `keep identity set-id`, then
|
|
|
|
|
grant it **read-only** access to each project vault it deploys
|
|
|
|
|
(`keep grant <project>/production <vps-recipient-id> --read-only`). After
|
|
|
|
|
that, `./deploy.sh <project>` is the whole deploy step — no `.env`
|
|
|
|
|
hand-copied onto the box first.
|
|
|
|
|
|
|
|
|
|
For "which machine can deploy which project," `keep overview` (admin)
|
|
|
|
|
prints every vault and every recipient's access in one screen — the
|
|
|
|
|
thing to reach for instead of checking each vault's grants one at a
|
|
|
|
|
time.
|
|
|
|
|
|
Implement keep: self-hosted E2E encrypted secrets sync
Server: Express + better-sqlite3 (WAL), multi-recipient key-wrapping
per IMPLEMENTATION.md's design — vaults/recipients/vault_grants/
access_log. Two auth paths: ADMIN_PASSWORD header for recipient
management and revoke (pure metadata operations), signed-request
auth (Ed25519 signature over method+path+timestamp+body-hash) for
push/pull/grant, mirroring the spirit of this project family's other
signed-handshake patterns without naming them.
CLI: identity init/show/set-id, push/pull/grant/log, admin recipient
add/list/remove and revoke. Grant is a client-side crypto operation
(the granter unwraps the vault's current key locally and reseals it
for the new recipient) rather than a server-side operation, since the
server never holds an unwrapped key to grant with.
Verified end-to-end with two independent local identities against a
live server and separately against the built Docker image: register,
push, pull (granted and ungranted), grant without re-pushing, admin
revoke, a subsequent rotation confirming the revoked recipient stays
excluded, and rejection of missing/malformed signed-request auth.
Two real bugs caught during verification, not just written up:
- libsodium-wrappers' published ESM build does a relative import only
resolvable under bundler-style resolution — broken under plain Node
ESM. Fixed via createRequire to force the CJS build.
- Express's req.path inside a sub-router is relative to the mount
point, which would have silently mismatched a client signing the
full request path. Fixed by verifying against req.originalUrl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 19:38:24 +02:00
|
|
|
## Pattern: docker-compose reading a local `.env`
|
|
|
|
|
|
|
|
|
|
A common shape — `docker compose up -d` reads `environment:` values from
|
|
|
|
|
a `.env` file sitting next to `docker-compose.yml` on the deploy machine,
|
|
|
|
|
hand-maintained and containing real secrets.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
keep pull myapp/production --out .env
|
|
|
|
|
docker compose up -d
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
No code changes needed in the target project — `keep` only changes *how
|
|
|
|
|
the `.env` file gets there*, not what reads it. This is the easiest
|
|
|
|
|
category to retrofit.
|
|
|
|
|
|
|
|
|
|
## Pattern: cross-compile + scp + SSH-restart deploy scripts
|
|
|
|
|
|
|
|
|
|
A shape where a deploy script builds a binary locally, ships it to a VPS,
|
|
|
|
|
and restarts a service over SSH — reading a local `.env` next to the
|
|
|
|
|
script for things like shared secrets that shouldn't be hardcoded or
|
|
|
|
|
re-typed on every invocation.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
keep pull myapp/production --out .env
|
|
|
|
|
set -a && source .env && set +a
|
|
|
|
|
# ...rest of the deploy script unchanged
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Same retrofit as the docker-compose case, just applied before whatever
|
|
|
|
|
build/ship step already exists.
|
|
|
|
|
|
|
|
|
|
## Pattern: a plaintext config file served to a browser
|
|
|
|
|
|
|
|
|
|
Not every `config.js`/similar file that *looks* like environment config
|
|
|
|
|
actually holds secrets — some of it is public-by-design (an API base URL,
|
|
|
|
|
a feature flag) that's explicitly meant to be visible to anyone loading
|
|
|
|
|
the page. `keep` solves a confidentiality problem; a file with nothing
|
|
|
|
|
confidential in it doesn't have one. Worth checking this distinction
|
|
|
|
|
before wiring `keep` into anything — retrofitting a non-secret config
|
|
|
|
|
file would be solving a problem that file doesn't have.
|
|
|
|
|
|
|
|
|
|
## A vault server's own configuration
|
|
|
|
|
|
|
|
|
|
Whatever runs `keep` itself needs its own `ADMIN_PASSWORD` to be a real,
|
|
|
|
|
manually-set secret on its own host — it can't bootstrap its own trust
|
|
|
|
|
root by pulling itself from itself. This is the same "root of trust
|
|
|
|
|
starts somewhere unmanaged" fact every secrets system has (Vault's own
|
|
|
|
|
unseal keys aren't stored in Vault either). Not a gap, just worth being
|
|
|
|
|
explicit about: `keep`'s own deploy stays on a plain `.env`/
|
|
|
|
|
`docker-compose.yml` environment variables.
|
|
|
|
|
|
|
|
|
|
## New projects going forward
|
|
|
|
|
|
|
|
|
|
The actual best use case isn't retrofitting existing deploys — it's never
|
|
|
|
|
having the "hand-reconstruct a `.env` from memory" moment for a *new*
|
|
|
|
|
deploy target in the first place. For whatever gets built next: register
|
|
|
|
|
its production host as a `keep` recipient as part of first setup, write
|
|
|
|
|
the deploy script to `keep pull` from day one, and the annoyance this was
|
|
|
|
|
built to fix never has a chance to happen.
|