Fredrik JohanssonandClaude Sonnet 5
e37a1d5798
Docker / test-build-push (push) Successful in 1m47s
The known-rough-edge section described POST /api/receipts as still present but scheduled for removal — the refactor already removed it. Replaced with a table of what each file now owns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
132 lines
10 KiB
Markdown
132 lines
10 KiB
Markdown
# trace
|
||
|
||
Private incident notebook, Git fix-candidate inbox, deployment-receipt evidence store, and reviewed Astro Markdown exporter for goonk's `/failures` page. One ~70-line HTTP server (`src/server.mjs`), Node's built-in `node:sqlite`, no framework, no runtime dependencies.
|
||
|
||
```bash
|
||
cp .env.example .env # set TRACE_PASSWORD and TRACE_RECEIPT_TOKENS at minimum
|
||
npm start
|
||
```
|
||
|
||
Node 24+. `npm run dev` for `--watch` during local work; `npm test` runs the receipt-contract tests; `npm run build` is a syntax check (`node --check`), not a bundle — there's nothing to bundle.
|
||
|
||
## What trace actually is
|
||
|
||
Two things that get conflated everywhere else, kept structurally separate on purpose:
|
||
|
||
1. **A private incident notebook.** Record what broke, what you tried, what was actually true, and what you'd do differently — while you're still in it, not reconstructed from memory afterward. Nothing here is public until a human explicitly publishes it.
|
||
2. **An operational receipts ledger.** CI says what it built; a deploy host says what it actually started and verified. Trace stores both as separate, versioned, authenticated event types and never lets a green build masquerade as a live deployment.
|
||
|
||
The two features share a database and a process, but not a purpose — the incident notebook is where a human writes a story, the receipts ledger is where machines report bare facts.
|
||
|
||
## The private incident notebook
|
||
|
||
Everything under `/` (Basic Auth, `trace:$TRACE_PASSWORD`) is private by default:
|
||
|
||
- **`/`** — every incident, newest first, with a live status breakdown.
|
||
- **`/incidents/new`** → **`/incidents/:id`** — title, project, status (`investigating` / `mitigated` / `resolved` / `abandoned`), severity (`annoyance` → `security-risk`), and the fields that actually make a postmortem useful: symptom, impact, root cause, confidence, fix, verification, prevention, remaining risk, lesson. Confidence is tracked separately from status on purpose — "resolved, high confidence" and "resolved, we're not totally sure why" are different claims and shouldn't collapse into one field.
|
||
- **Timeline** — append-only entries per incident, each tagged `observation` / `hypothesis` / `action` / `result` / `decision` / `deploy` / `note`. This is the actual investigation log, in order, not reconstructed after the fact.
|
||
- **Hypotheses** — a statement, a status (`untested` / `supported` / `rejected` / `inconclusive`), and evidence. Explicitly tracking rejected hypotheses is the point — "we thought it was X, it wasn't, here's how we knew" is worth as much as the real answer.
|
||
- **`/candidates`** (the Git inbox) — click "scan configured repositories" to pull commits whose subject line matches fix-shaped keywords (`fix`, `bug`, `broken`, `race`, `crash`, `regression`, `rollback`, `restore`, `duplicate`, `auth`, `fail`) from either locally mounted read-only repo mirrors (`TRACE_REPOS`) or Gitea's API with no checkout at all (`TRACE_GITEA_URL` + `TRACE_GITEA_REPOS`). Link a candidate to an incident's UUID to connect "here's the bug" with "here's the commit that caused or fixed it."
|
||
|
||
### Publishing a real postmortem
|
||
|
||
An incident page has an **export public draft** action (writes reviewed Markdown to `data/exports/` for hand-editing) and, once you're happy with it, **publish / update reviewed article** — this sets a public slug and makes the incident visible through the read-only public API below. **Unpublish** reverses it instantly. Nothing crosses from private to public without that explicit action; a `symptom` field with an internal hostname in it never accidentally becomes a blog post.
|
||
|
||
```bash
|
||
curl -u trace:$TRACE_PASSWORD -X POST http://localhost:3082/incidents/new \
|
||
--data-urlencode "title=under-the-hood showed a stale deploy commit" \
|
||
--data-urlencode "project=goonk" \
|
||
--data-urlencode "detected_at=2026-07-21T01:00" \
|
||
--data-urlencode "status=resolved" \
|
||
--data-urlencode "severity=annoyance" \
|
||
--data-urlencode "symptom=Deployed showed an older commit and earlier timestamp than Built." \
|
||
--data-urlencode "root_cause=Nothing on the deploy host ever submitted a deployment receipt." \
|
||
--data-urlencode "fix=Added best-effort receipt submission to the shared deploy wrapper." \
|
||
--data-urlencode "public_summary=A deploy dashboard kept showing an old commit as current."
|
||
# → 303 redirect to /incidents/<new-uuid>
|
||
```
|
||
|
||
The form works the same from a browser; this is just the same POST scripted, useful for backfilling incidents from notes taken elsewhere.
|
||
|
||
## Deployment receipts (v1)
|
||
|
||
Two separate authenticated write paths, one shared authenticated read-only view:
|
||
|
||
| Endpoint | Method | Auth | Purpose |
|
||
|---|---|---|---|
|
||
| `/api/v1/events` | `POST` | `Authorization: Bearer <project-scoped token>` | Submit a build/deployment/release/rollback receipt |
|
||
| `/events` | `GET` | Basic (`trace:$TRACE_PASSWORD`) | Private, human-readable feed of every received event, raw payload included |
|
||
| `/api/v1/public/projects/:project/latest` | `GET` | none | Latest **deployment** event only, allowlisted fields |
|
||
| `/api/v1/public/projects` | `GET` | none | Latest deployment per project, same allowlisted shape |
|
||
| `/api/health` | `GET` | none | Liveness probe (used by the Dockerfile's `HEALTHCHECK`) |
|
||
|
||
A receipt (`schemaVersion: 1`) needs `eventType` (`build`/`deployment`/`release`/`rollback`), `eventId`, `project`, `occurredAt` (ISO-8601), `result` (`success`/`failed`/`cancelled`/`unknown`), and `source.commit` (a 7–64 char hex SHA). `eventId` is the idempotency key — resubmitting the same one returns `{"ok":true,"duplicate":true}` rather than a second row, which is what makes "retry the CI step that failed to reach trace" safe.
|
||
|
||
```bash
|
||
# CI, right after pushing images (a build receipt)
|
||
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
|
||
--data-binary '{"schemaVersion":1,"eventType":"build","eventId":"goonk-build-713-3777586","project":"goonk",
|
||
"occurredAt":"2026-07-21T03:06:00Z","result":"success",
|
||
"source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
|
||
"artifacts":[{"component":"web","digest":"sha256:fe9e15f73804..."}],
|
||
"verification":{"build":"success"}}' \
|
||
https://trace.dev.xplwd.com/api/v1/events
|
||
|
||
# A deploy host, after docker compose up -d and a real health check
|
||
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
|
||
--data-binary '{"schemaVersion":1,"eventType":"deployment","eventId":"goonk-production-3777586-20260721T013412Z",
|
||
"project":"goonk","environment":"production","occurredAt":"2026-07-21T01:34:12Z","result":"success",
|
||
"source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
|
||
"verification":{"health":"healthy"}}' \
|
||
https://trace.dev.xplwd.com/api/v1/events
|
||
|
||
# Check what a public consumer sees
|
||
curl https://trace.dev.xplwd.com/api/v1/public/projects/goonk/latest
|
||
```
|
||
|
||
Full producer onboarding — registering a project, per-project tokens, the exact CI step shape, and the deploy-host responsibilities above — is in [`docs/INTEGRATING_PROJECTS.md`](docs/INTEGRATING_PROJECTS.md). `keep/scripts/deploy.sh` is the reference deploy-host implementation of the second `curl` above: after `docker compose up -d`, it polls the app's own embedded build identity until healthy, then submits exactly this receipt — opt-in per project via a `.trace-meta` marker file, never assumed.
|
||
|
||
## Code layout
|
||
|
||
`src/server.mjs` is composition only — open the database, build each route module, wire the dispatch order, listen. Everything else lives by concern:
|
||
|
||
| File | Owns |
|
||
|---|---|
|
||
| `src/db.mjs` | Schema and migrations for the incident notebook (`openDb()`) |
|
||
| `src/views.mjs` | Pure HTML/Markdown rendering — no database or network access |
|
||
| `src/style.mjs` | The one shared stylesheet every page links to |
|
||
| `src/http-utils.mjs` | `form()`/`redirect()` request helpers |
|
||
| `src/routes/incidents.mjs` | The private notebook, the Git inbox, publish/unpublish — auth-gated once at the top |
|
||
| `src/routes/public-failures.mjs` | The read-only `/api/v1/public/failures*` projection |
|
||
| `src/receipts.mjs` | The deployment-receipts ledger and `/events` |
|
||
| `src/gitea.mjs` | Gitea-API-based commit scanning, no checkout needed |
|
||
|
||
## Configuration reference
|
||
|
||
| Variable | Required | Purpose |
|
||
|---|---|---|
|
||
| `PORT` | no (default `3082`) | HTTP listen port |
|
||
| `TRACE_PASSWORD` | effectively yes | Basic Auth password for `/`, `/events`, and the publish/unpublish actions. Empty disables auth entirely — fine for a throwaway local instance, never for anything reachable outside localhost |
|
||
| `TRACE_RECEIPT_TOKENS` | yes, for receipt ingestion | `project:token,project2:token2` — one random token per producer (`openssl rand -hex 32`), never shared across projects |
|
||
| `TRACE_REPOS` | no | `id:/absolute/path,id2:/absolute/path2` — locally mounted **read-only** repo mirrors for the Git inbox |
|
||
| `TRACE_GITEA_URL` | no (default `https://repo.explewd.com`) | Base URL for Gitea-API-based candidate scanning, no checkout needed |
|
||
| `TRACE_GITEA_REPOS` | no | Comma-separated `owner/repo` list to scan via the Gitea API |
|
||
| `TRACE_GITEA_TOKEN` | no | Only needed for private Gitea repositories; public ones scan with no token |
|
||
|
||
## Deploying
|
||
|
||
```bash
|
||
cp .env.example .env # fill in real values
|
||
docker compose up -d
|
||
```
|
||
|
||
Single service, a named volume for `data/` (SQLite + exports), a `HEALTHCHECK` hitting `/api/health`. CI (`.gitea/workflows/docker.yml`) runs `npm test` and `npm run build` before it will push an image — a broken receipt-validation contract or a syntax error never reaches the registry.
|
||
|
||
## Security boundaries
|
||
|
||
- One bearer token per project; a `retur` token cannot submit a `goonk` receipt (checked with `crypto.timingSafeEqual`, not `===`).
|
||
- `TRACE_PASSWORD` gates every private route in one place — the notebook, the raw events feed, and publish/unpublish.
|
||
- The public API (`/api/v1/public/*`) never includes raw payloads, tokens, internal hostnames, or full registry references — only the allowlisted fields in `publicEvent()`/`publicIncident()`.
|
||
- `TRACE_REPOS` should be empty in any deployment with no deliberate read-only mirrors mounted; receipt ingestion never requires repository access at all.
|
||
- Trace being unreachable should never break a producer's build or deploy — every documented integration point treats submission as best-effort.
|