Conformance tracking: cited-evidence overlay, CI gate, and generated readiness render

Status: accepted (2026-06-17)

We target many standards (docs/architecture/standards.md) and intend to apply for full certification later and have it just pass. Today the truth is scattered: standards.md is hand-typed prose with coarse per-spec tags that drift the instant code moves; the 26 ADRs hold rationale but give no bird’s-eye view and no point-by-point checklist. There is no single, non-drifting, certification-aligned answer to “where exactly are we on every part of every spec” — for humans, for AI assistants, or for an eventual certifier hand-off.

Builds on conform-ed ADR-0013, which publishes a per-spec:version Coverage Map (L1 information-model inventory walked from the literal schema ⊕ L2 “modelled in conform-ed?” ⊕ C hand-curated conformance catalog, profile-tagged). This ADR covers emergent’s half.

Decision

emergent depends on the published @conform-ed/coverage Coverage Map and adds the product overlay on top, with a CI gate that keeps it honest and a generated render for the bird’s-eye view.

• L3 overlay. Per-item product status + cited evidence + gotchas/notes, in git-tracked YAML files (one per spec:version/profile), keyed by the Coverage Map’s literal-schema item keys and validated against a Zod schema. YAML for prose-heavy gotchas/reason fields (block scalars) and clean one-key-per-line diffs; Zod validation neutralises YAML’s coercion footguns. Not a database — git-tracked files keep status changes inside the same PR review as the code that caused them (that is what stops drift), stay greppable by AI assistants, and make git history the audit trail (the “which commit became conformant?” question, for free).
• Status is declared-with-cited-evidence, never free percentages. A done/partial claim must cite a resolvable test/lane (+ assertion/case where possible); deferred/n-a carries a reason and ADR link. Percentages are computed rollups, per certification profile — you mark items, the rollups fall out; you never type a number.
• CI gate (four checks), in validate:full:
  1. Referential integrity — every overlay key still exists in the pinned Coverage Map; an orphaned annotation (after a spec re-fetch / Zod change upstream) fails the build.
  2. Evidence verified against actual pass/fail — the gate runs after the suite, consumes the runner’s results, and asserts every done/partial’s cited evidence actually passed this run. Evidence is bound via a greppable [covers: <key>(, <key>)*] marker in conformance/integration test names (runner-agnostic; may evolve to a structured registry).
  3. Untriaged-gap detection — newly-surfaced L1-without-L2 residue or new conformance requirements must be explicitly marked (even just todo/deferred); unmarked = build failure.
  4. Render freshness — regenerate the Markdown and diff; stale committed render fails.
• Render. A generated Markdown tree (canonical): top-level per-certification-profile readiness rollups → per-spec → per-section item tables with inline bars, joining the published Coverage Map with the local overlay. HTML/PDF certifier polish is deferred until we actually approach a certifier.
• Rollout. Prove the full vertical (both axes, both repos, the gate, the render) on the Open Badges 3.0 pilot (already validated, has test:ob:conformance, JSON-Schema family), then fan out by schema-language family → QTI 3.0.1 (XSD, the highest-value target) → CC + QTI 2.x → OneRoster (OpenAPI) → rest.

docs/architecture/standards.md is superseded as the status tracker: it reduces to a narrative index that points at the generated render.

Considered and rejected

• Everything in emergent. Violates the standing rule that reusable conformance tooling lives in conform-ed; the schema walkers and L1⊕L2 reconciliation belong where the schemas and Zod already are (hence the ADR-0013 split).
• Database backing. Divorces status from PR review (the change wouldn’t show in the diff of the code that caused it — reintroducing drift), needs infra, isn’t greppable by an AI assistant.
• Typed percentages. Subjective and drift like any hand-typed value; rollups are honest and non-gameable.
• Existence-only evidence checks. A done could cite a currently-red test and still pass the gate — lets the artifact lie, defeating its purpose.
• Annotating against our own Zod model instead of the literal spec. Rejected upstream in ADR-0013: structurally blind to silent omissions.

Consequences

• Cross-repo release coupling: emergent pins a Coverage Map version; upstream regeneration is a deliberate bump.
• A test-tagging convention ([covers: …]) on conformance/integration tests.
• A real upkeep cost on every spec-touching PR — the gate enforces it. That cost is the mechanism; without the gate this decays into another standards.md.