LRS Conformance Suite Runbook

Intent

Run the upstream ADL LRS Conformance Test Suite against a disposable Emergent LRS environment early and repeatedly.

This runbook turns the suite into an operational requirement, not a release-week smoke test.

See also: LRS Operational Readiness Runbook

Hard Requirement

• The public xAPI LRS surface is not complete until the relevant upstream ADL batteries are green for the active implementation slice.
• The LRS must not be marked validated until a full upstream suite run is green.
• A failing upstream battery blocks readiness even if internal unit, contract, and integration tests are green.

Constraints

• Run only against a disposable non-production environment.
• The upstream suite persists data and does not clean up by voiding everything it writes.
• The upstream suite currently focuses on Basic auth, so Basic-auth compatibility must remain first-wave scope.
• Point the suite at the public xAPI endpoint.
• For local direct runs against the standalone service, this is typically the /xapi base path.
• For the Kubernetes gateway deployment, this is the Supabase-compatible /functions/v1/xapi path on the API host.

Development Lifecycle Policy

• PR 2 onward: wire the harness in as an expected-red executable contract.
• PR 5 onward: require the relevant subset runs for each LRS-facing protocol slice.
• Milestone and release gates: require a full-suite green run.

Upstream Reuse Policy

• The upstream suite is MIT licensed.
• Reuse or adapt narrowly useful upstream helpers into repository TypeScript when they reduce spec ambiguity or duplicated protocol logic.
• Preserve the required license notice for copied or derived upstream material.
• Prefer selective adaptation of fixtures, request builders, requirement mappings, and assertion helpers over a wholesale fork.

Local Execution

Default setup:

1. Ensure podman is available locally.
2. Ensure the conform-ed LRS runner image is available (or buildable) for the local lane.
3. Use the repository wrapper command:

Run the conform-ed-backed image lane (validates conform-ed lrs runner config from image, then executes the current emergent lane):

bun run test:lrs:conformance:conform-ed

Equivalent explicit image alias:

bun run test:lrs:conformance:conform-ed:image

By default, this wrapper now does all of the local orchestration work itself:

• starts the dedicated postgres service from infra/compose/podman-compose.lrs-conformance.yaml
• runs packages/db migrations against that disposable database
• starts the LRS service on http://127.0.0.1:4001/xapi
• runs the conform-ed dual-version suite runner image through podman
• tears the stack down with volumes after the run

Useful local-stack options:

• --xapi-version <value> to run a single target version through the conform-ed lane
• --keep-stack to leave postgres plus LRS running after the suite exits for local debugging
• --postgres-port <num> and --lrs-port <num> to override default local ports
• --endpoint <url>, --username <value>, --password <value> for external endpoint runs

Triage Order

1. Fix protocol-foundation failures first: version header handling, auth status codes, HEAD, Last-Modified, and error semantics.
2. Then fix statement ingestion and retrieval semantics.
3. Then fix document-resource preconditions, merge behavior, attachments, and signing.
4. Only after the upstream suite is stable should you treat browser-level H5P playback issues as likely frontend bugs.