trace/README.md

# trace

Private incident notebook, Git fix-candidate inbox, deployment-receipt evidence store, and reviewed Astro Markdown exporter for goonk's `/failures` page. One ~70-line HTTP server (`src/server.mjs`), Node's built-in `node:sqlite`, no framework, no runtime dependencies.

```bash
cp .env.example .env   # set TRACE_PASSWORD and TRACE_RECEIPT_TOKENS at minimum
npm start
```

Node 24+. `npm run dev` for `--watch` during local work; `npm test` runs the receipt-contract tests; `npm run build` is a syntax check (`node --check`), not a bundle — there's nothing to bundle.

## What trace actually is

Two things that get conflated everywhere else, kept structurally separate on purpose:

1. **A private incident notebook.** Record what broke, what you tried, what was actually true, and what you'd do differently — while you're still in it, not reconstructed from memory afterward. Nothing here is public until a human explicitly publishes it.
2. **An operational receipts ledger.** CI says what it built; a deploy host says what it actually started and verified. Trace stores both as separate, versioned, authenticated event types and never lets a green build masquerade as a live deployment.

The two features share a database and a process, but not a purpose — the incident notebook is where a human writes a story, the receipts ledger is where machines report bare facts.

## The private incident notebook

Everything under `/` (Basic Auth, `trace:$TRACE_PASSWORD`) is private by default:

- **`/`** — every incident, newest first, with a live status breakdown.
- **`/incidents/new`** → **`/incidents/:id`** — title, project, status (`investigating` / `mitigated` / `resolved` / `abandoned`), severity (`annoyance` → `security-risk`), and the fields that actually make a postmortem useful: symptom, impact, root cause, confidence, fix, verification, prevention, remaining risk, lesson. Confidence is tracked separately from status on purpose — "resolved, high confidence" and "resolved, we're not totally sure why" are different claims and shouldn't collapse into one field.
- **Timeline** — append-only entries per incident, each tagged `observation` / `hypothesis` / `action` / `result` / `decision` / `deploy` / `note`. This is the actual investigation log, in order, not reconstructed after the fact.
- **Hypotheses** — a statement, a status (`untested` / `supported` / `rejected` / `inconclusive`), and evidence. Explicitly tracking rejected hypotheses is the point — "we thought it was X, it wasn't, here's how we knew" is worth as much as the real answer.
- **`/candidates`** (the Git inbox) — click "scan configured repositories" to pull commits whose subject line matches fix-shaped keywords (`fix`, `bug`, `broken`, `race`, `crash`, `regression`, `rollback`, `restore`, `duplicate`, `auth`, `fail`) from either locally mounted read-only repo mirrors (`TRACE_REPOS`) or Gitea's API with no checkout at all (`TRACE_GITEA_URL` + `TRACE_GITEA_REPOS`). Link a candidate to an incident's UUID to connect "here's the bug" with "here's the commit that caused or fixed it."

### Publishing a real postmortem

An incident page has an **export public draft** action (writes reviewed Markdown to `data/exports/` for hand-editing) and, once you're happy with it, **publish / update reviewed article** — this sets a public slug and makes the incident visible through the read-only public API below. **Unpublish** reverses it instantly. Nothing crosses from private to public without that explicit action; a `symptom` field with an internal hostname in it never accidentally becomes a blog post.

```bash
curl -u trace:$TRACE_PASSWORD -X POST http://localhost:3082/incidents/new \
  --data-urlencode "title=under-the-hood showed a stale deploy commit" \
  --data-urlencode "project=goonk" \
  --data-urlencode "detected_at=2026-07-21T01:00" \
  --data-urlencode "status=resolved" \
  --data-urlencode "severity=annoyance" \
  --data-urlencode "symptom=Deployed showed an older commit and earlier timestamp than Built." \
  --data-urlencode "root_cause=Nothing on the deploy host ever submitted a deployment receipt." \
  --data-urlencode "fix=Added best-effort receipt submission to the shared deploy wrapper." \
  --data-urlencode "public_summary=A deploy dashboard kept showing an old commit as current."
# → 303 redirect to /incidents/<new-uuid>
```

The form works the same from a browser; this is just the same POST scripted, useful for backfilling incidents from notes taken elsewhere.

## Deployment receipts (v1)

Two separate authenticated write paths, one shared authenticated read-only view:

| Endpoint | Method | Auth | Purpose |
|---|---|---|---|
| `/api/v1/events` | `POST` | `Authorization: Bearer <project-scoped token>` | Submit a build/deployment/release/rollback receipt |
| `/events` | `GET` | Basic (`trace:$TRACE_PASSWORD`) | Private, human-readable feed of every received event, raw payload included |
| `/api/v1/public/projects/:project/latest` | `GET` | none | Latest **deployment** event only, allowlisted fields |
| `/api/v1/public/projects` | `GET` | none | Latest deployment per project, same allowlisted shape |
| `/api/health` | `GET` | none | Liveness probe (used by the Dockerfile's `HEALTHCHECK`) |

A receipt (`schemaVersion: 1`) needs `eventType` (`build`/`deployment`/`release`/`rollback`), `eventId`, `project`, `occurredAt` (ISO-8601), `result` (`success`/`failed`/`cancelled`/`unknown`), and `source.commit` (a 7–64 char hex SHA). `eventId` is the idempotency key — resubmitting the same one returns `{"ok":true,"duplicate":true}` rather than a second row, which is what makes "retry the CI step that failed to reach trace" safe.

```bash
# CI, right after pushing images (a build receipt)
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
  --data-binary '{"schemaVersion":1,"eventType":"build","eventId":"goonk-build-713-3777586","project":"goonk",
    "occurredAt":"2026-07-21T03:06:00Z","result":"success",
    "source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
    "artifacts":[{"component":"web","digest":"sha256:fe9e15f73804..."}],
    "verification":{"build":"success"}}' \
  https://trace.dev.xplwd.com/api/v1/events

# A deploy host, after docker compose up -d and a real health check
curl -H "Authorization: Bearer $TRACE_RECEIPT_TOKEN" -H "Content-Type: application/json" \
  --data-binary '{"schemaVersion":1,"eventType":"deployment","eventId":"goonk-production-3777586-20260721T013412Z",
    "project":"goonk","environment":"production","occurredAt":"2026-07-21T01:34:12Z","result":"success",
    "source":{"commit":"37775868ab303f9f8a7da09b354dd0cc31bb0c6f","branch":"main"},
    "verification":{"health":"healthy"}}' \
  https://trace.dev.xplwd.com/api/v1/events

# Check what a public consumer sees
curl https://trace.dev.xplwd.com/api/v1/public/projects/goonk/latest
```

Full producer onboarding — registering a project, per-project tokens, the exact CI step shape, and the deploy-host responsibilities above — is in [`docs/INTEGRATING_PROJECTS.md`](docs/INTEGRATING_PROJECTS.md). `keep/scripts/deploy.sh` is the reference deploy-host implementation of the second `curl` above: after `docker compose up -d`, it polls the app's own embedded build identity until healthy, then submits exactly this receipt — opt-in per project via a `.trace-meta` marker file, never assumed.

### A known rough edge

`POST /api/receipts` (Basic Auth, no `/v1`, no bearer token) still exists in `src/server.mjs` as a leftover from before the v1 events API was built. It writes to a `receipts` table nothing else ever reads from — dead code, not a second real API. Use `/api/v1/events` for everything; the old path is scheduled for removal, not a documented feature.

## Configuration reference

| Variable | Required | Purpose |
|---|---|---|
| `PORT` | no (default `3082`) | HTTP listen port |
| `TRACE_PASSWORD` | effectively yes | Basic Auth password for `/`, `/events`, and the publish/unpublish actions. Empty disables auth entirely — fine for a throwaway local instance, never for anything reachable outside localhost |
| `TRACE_RECEIPT_TOKENS` | yes, for receipt ingestion | `project:token,project2:token2` — one random token per producer (`openssl rand -hex 32`), never shared across projects |
| `TRACE_REPOS` | no | `id:/absolute/path,id2:/absolute/path2` — locally mounted **read-only** repo mirrors for the Git inbox |
| `TRACE_GITEA_URL` | no (default `https://repo.explewd.com`) | Base URL for Gitea-API-based candidate scanning, no checkout needed |
| `TRACE_GITEA_REPOS` | no | Comma-separated `owner/repo` list to scan via the Gitea API |
| `TRACE_GITEA_TOKEN` | no | Only needed for private Gitea repositories; public ones scan with no token |

## Deploying

```bash
cp .env.example .env   # fill in real values
docker compose up -d
```

Single service, a named volume for `data/` (SQLite + exports), a `HEALTHCHECK` hitting `/api/health`. CI (`.gitea/workflows/docker.yml`) runs `npm test` and `npm run build` before it will push an image — a broken receipt-validation contract or a syntax error never reaches the registry.

## Security boundaries

- One bearer token per project; a `retur` token cannot submit a `goonk` receipt (checked with `crypto.timingSafeEqual`, not `===`).
- `TRACE_PASSWORD` gates every private route in one place — the notebook, the raw events feed, and publish/unpublish.
- The public API (`/api/v1/public/*`) never includes raw payloads, tokens, internal hostnames, or full registry references — only the allowlisted fields in `publicEvent()`/`publicIncident()`.
- `TRACE_REPOS` should be empty in any deployment with no deliberate read-only mirrors mounted; receipt ingestion never requires repository access at all.
- Trace being unreachable should never break a producer's build or deploy — every documented integration point treats submission as best-effort.