trace/README.md

trace

Private incident notebook, Git fix-candidate inbox, deployment-receipt evidence store, and reviewed Astro Markdown exporter for goonk's /failures page. One ~70-line HTTP server (src/server.mjs), Node's built-in node:sqlite, no framework, no runtime dependencies.

cp .env.example .env   # set TRACE_PASSWORD and TRACE_RECEIPT_TOKENS at minimum
npm start

Node 24+. npm run dev for --watch during local work; npm test runs the receipt-contract tests; npm run build is a syntax check (node --check), not a bundle — there's nothing to bundle.

What trace actually is

Two things that get conflated everywhere else, kept structurally separate on purpose:

1. A private incident notebook. Record what broke, what you tried, what was actually true, and what you'd do differently — while you're still in it, not reconstructed from memory afterward. Nothing here is public until a human explicitly publishes it.
2. An operational receipts ledger. CI says what it built; a deploy host says what it actually started and verified. Trace stores both as separate, versioned, authenticated event types and never lets a green build masquerade as a live deployment.

The two features share a database and a process, but not a purpose — the incident notebook is where a human writes a story, the receipts ledger is where machines report bare facts.

The private incident notebook

Everything under / (Basic Auth, trace:$TRACE_PASSWORD) is private by default:

• / — every incident, newest first, with a live status breakdown.
• /incidents/new → /incidents/:id — title, project, status (investigating / mitigated / resolved / abandoned), severity (annoyance → security-risk), and the fields that actually make a postmortem useful: symptom, impact, root cause, confidence, fix, verification, prevention, remaining risk, lesson. Confidence is tracked separately from status on purpose — "resolved, high confidence" and "resolved, we're not totally sure why" are different claims and shouldn't collapse into one field.
• Timeline — append-only entries per incident, each tagged observation / hypothesis / action / result / decision / deploy / note. This is the actual investigation log, in order, not reconstructed after the fact.
• Hypotheses — a statement, a status (untested / supported / rejected / inconclusive), and evidence. Explicitly tracking rejected hypotheses is the point — "we thought it was X, it wasn't, here's how we knew" is worth as much as the real answer.
• /candidates (the Git inbox) — click "scan configured repositories" to pull commits whose subject line matches fix-shaped keywords (fix, bug, broken, race, crash, regression, rollback, restore, duplicate, auth, fail) from either locally mounted read-only repo mirrors (TRACE_REPOS) or Gitea's API with no checkout at all (TRACE_GITEA_URL + TRACE_GITEA_REPOS). Link a candidate to an incident's UUID to connect "here's the bug" with "here's the commit that caused or fixed it."

Publishing a real postmortem

An incident page has an export public draft action (writes reviewed Markdown to data/exports/ for hand-editing) and, once you're happy with it, publish / update reviewed article — this sets a public slug and makes the incident visible through the read-only public API below. Unpublish reverses it instantly. Nothing crosses from private to public without that explicit action; a symptom field with an internal hostname in it never accidentally becomes a blog post.

curl -u trace:$TRACE_PASSWORD -X POST http://localhost:3082/incidents/new \
  --data-urlencode "title=under-the-hood showed a stale deploy commit" \
  --data-urlencode "project=goonk" \
  --data-urlencode "detected_at=2026-07-21T01:00" \
  --data-urlencode "status=resolved" \
  --data-urlencode "severity=annoyance" \
  --data-urlencode "symptom=Deployed showed an older commit and earlier timestamp than Built." \
  --data-urlencode "root_cause=Nothing on the deploy host ever submitted a deployment receipt." \
  --data-urlencode "fix=Added best-effort receipt submission to the shared deploy wrapper." \
  --data-urlencode "public_summary=A deploy dashboard kept showing an old commit as current."
# → 303 redirect to /incidents/<new-uuid>

The form works the same from a browser; this is just the same POST scripted, useful for backfilling incidents from notes taken elsewhere.

Deployment receipts (v1)

Two separate authenticated write paths, one shared authenticated read-only view:

Endpoint	Method	Auth	Purpose
/api/v1/events	POST	Authorization: Bearer <project-scoped token>	Submit a build/deployment/release/rollback receipt
/events	GET	Basic (trace:$TRACE_PASSWORD)	Private, human-readable feed of every received event, raw payload included
/api/v1/public/projects/:project/latest	GET	none	Latest deployment event only, allowlisted fields
/api/v1/public/projects	GET	none	Latest deployment per project, same allowlisted shape
/api/health	GET	none	Liveness probe (used by the Dockerfile's HEALTHCHECK)

A receipt (schemaVersion: 1) needs eventType (build/deployment/release/rollback), eventId, project, occurredAt (ISO-8601), result (success/failed/cancelled/unknown), and source.commit (a 7–64 char hex SHA). eventId is the idempotency key — resubmitting the same one returns {"ok":true,"duplicate":true} rather than a second row, which is what makes "retry the CI step that failed to reach trace" safe.

# CI, right after pushing images (a build receipt)
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
  --data-binary '{"schemaVersion":1,"eventType":"build","eventId":"goonk-build-713-3777586","project":"goonk",
    "occurredAt":"2026-07-21T03:06:00Z","result":"success",
    "source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
    "artifacts":[{"component":"web","digest":"sha256:fe9e15f73804..."}],
    "verification":{"build":"success"}}' \
  https://trace.dev.xplwd.com/api/v1/events

# A deploy host, after docker compose up -d and a real health check
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
  --data-binary '{"schemaVersion":1,"eventType":"deployment","eventId":"goonk-production-3777586-20260721T013412Z",
    "project":"goonk","environment":"production","occurredAt":"2026-07-21T01:34:12Z","result":"success",
    "source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
    "verification":{"health":"healthy"}}' \
  https://trace.dev.xplwd.com/api/v1/events

# Check what a public consumer sees
curl https://trace.dev.xplwd.com/api/v1/public/projects/goonk/latest

Full producer onboarding — registering a project, per-project tokens, the exact CI step shape, and the deploy-host responsibilities above — is in docs/INTEGRATING_PROJECTS.md. keep/scripts/deploy.sh is the reference deploy-host implementation of the second curl above: after docker compose up -d, it polls the app's own embedded build identity until healthy, then submits exactly this receipt — opt-in per project via a .trace-meta marker file, never assumed.

A known rough edge

POST /api/receipts (Basic Auth, no /v1, no bearer token) still exists in src/server.mjs as a leftover from before the v1 events API was built. It writes to a receipts table nothing else ever reads from — dead code, not a second real API. Use /api/v1/events for everything; the old path is scheduled for removal, not a documented feature.

Configuration reference

Variable	Required	Purpose
PORT	no (default 3082)	HTTP listen port
TRACE_PASSWORD	effectively yes	Basic Auth password for /, /events, and the publish/unpublish actions. Empty disables auth entirely — fine for a throwaway local instance, never for anything reachable outside localhost
TRACE_RECEIPT_TOKENS	yes, for receipt ingestion	project:token,project2:token2 — one random token per producer (openssl rand -hex 32), never shared across projects
TRACE_REPOS	no	id:/absolute/path,id2:/absolute/path2 — locally mounted read-only repo mirrors for the Git inbox
TRACE_GITEA_URL	no (default https://repo.explewd.com)	Base URL for Gitea-API-based candidate scanning, no checkout needed
TRACE_GITEA_REPOS	no	Comma-separated owner/repo list to scan via the Gitea API
TRACE_GITEA_TOKEN	no	Only needed for private Gitea repositories; public ones scan with no token

Deploying

cp .env.example .env   # fill in real values
docker compose up -d

Single service, a named volume for data/ (SQLite + exports), a HEALTHCHECK hitting /api/health. CI (.gitea/workflows/docker.yml) runs npm test and npm run build before it will push an image — a broken receipt-validation contract or a syntax error never reaches the registry.

Security boundaries

• One bearer token per project; a retur token cannot submit a goonk receipt (checked with crypto.timingSafeEqual, not ===).
• TRACE_PASSWORD gates every private route in one place — the notebook, the raw events feed, and publish/unpublish.
• The public API (/api/v1/public/*) never includes raw payloads, tokens, internal hostnames, or full registry references — only the allowlisted fields in publicEvent()/publicIncident().
• TRACE_REPOS should be empty in any deployment with no deliberate read-only mirrors mounted; receipt ingestion never requires repository access at all.
• Trace being unreachable should never break a producer's build or deploy — every documented integration point treats submission as best-effort.