LRS Operational Readiness Runbook

Purpose

Document the current runtime requirements, storage behavior, verification commands, and operational caveats for the Emergent LRS package.

This is the companion runbook to the conformance harness. The conformance runbook explains how to execute the upstream ADL suite. This runbook explains what the LRS actually requires and guarantees today.

Current State

• The real server path uses Postgres-backed storage. apps/lrs/src/server.ts always injects apps/lrs/src/db/storage.ts before listening.
• Direct createApp() construction still falls back to in-memory storage and default Basic-auth credentials. That behavior exists for tests and local seams, not for the real server entrypoint.
• Statement collection reads are now store-backed instead of application-memory full scans for the main protocol paths.
• /xapi/activities now resolves activity payloads through a targeted store lookup instead of loading the full statement set.
• The package is suitable for local development, integration, and bounded conformance work. It is not yet a broad production-ready deployment target.

Required Runtime Configuration

The real server path fails fast if these values are missing or invalid.

• DATABASE_URL: required. Server startup and Postgres storage initialization both reject startup without it.
• LRS_BASIC_AUTH_USERNAME: required for the real server path.
• LRS_BASIC_AUTH_PASSWORD: required for the real server path.
• PORT: optional. Defaults to 3011. Must be an integer between 1 and 65535.

Optional settings:

• LRS_AUTH_FAILURE_WINDOW_MS: defaults to 60000.
• LRS_AUTH_FAILURE_MAX_ATTEMPTS: defaults to 10.
• LRS_AUTH_FAILURE_LOCKOUT_MS: defaults to 60000.
• LRS_DEBUG_LOGGING: enables console debug logging when set to 1 or true.

Example local startup:

DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres \
PORT=4001 \
LRS_BASIC_AUTH_USERNAME=emergent-conformance \
LRS_BASIC_AUTH_PASSWORD=local-secret \
bun run --cwd apps/lrs start

Auth And Protocol Assumptions

• The current public auth model is Basic auth only.
• /xapi/about can be read without auth when no Authorization header is sent.
• All other /xapi/* routes require valid Basic auth.
• Non-/xapi/about routes require a supported X-Experience-API-Version header.
• Failed auth attempts are throttled in-process and return 429 with Retry-After once the lockout threshold is reached.
• Statement authority is derived from the authenticated Basic-auth username.

Operational implication:

• The current auth throttle is single-process memory state. It is useful for local and single-instance deployments, but it is not a distributed rate-limiting system.

Public Endpoint Shape

• The standalone LRS service still serves its protocol surface directly at /xapi/*.
• In the Kubernetes gateway deployment, that surface is published through the existing Supabase-compatible host as /functions/v1/xapi/*.
• The gateway rewrites /functions/v1/xapi/* back to /xapi/* before the request reaches the LRS service.
• Local direct runs and disposable conformance stacks may still target the standalone /xapi base path without the gateway in front.

Health And Readiness Semantics

• /health is a process-liveness check only. It does not touch the database.
• /ready is storage-backed. It returns success only when the storage readiness check succeeds.
• Server startup performs the same readiness check before binding the Bun server.
• Postgres readiness currently does two things:
  1. Executes select 1 against the database.
  2. Backfills missing statement-relation rows for older statements that predate the current relation-backed query path.

Operational implication:

• /ready is the correct probe for deploy readiness and database availability.
• /health is the correct probe for basic process survival.

Persistence And Query Behavior

Current persistence behavior:

• Statements are stored in Postgres.
• State, activity-profile, and agent-profile documents are stored in Postgres.
• Document bytes are persisted inline in the document table, not in a separate object store.
• Statement writes populate relation rows used for relation-backed collection queries.
• Statement voiding state is persisted and respected by collection visibility logic.

Current read/query behavior:

• Unfiltered and time-bounded statement collections use a DB-backed visibility and paging path.
• Direct statement filters for verb, activity, and registration use targeted SQL instead of loading the full statement set.
• agent, related_agents, and related_activities collection queries use relation rows in the database.
• /xapi/activities resolves merged activity payloads through a targeted store lookup using direct object matches plus related-activity rows.
• Matched voided statements still expand to statements that target them through StatementRef, preserving the existing xAPI visibility rules.

Current database client assumptions:

• postgres.js client max is 1.
• Prepared statements are disabled.
• connect_timeout is 10 seconds.
• idle_timeout is 20 seconds.

Operational implication:

• The package is intentionally conservative and deterministic right now.
• It is not yet tuned for sustained high-concurrency or multi-worker traffic.

Failure And Debugging Behavior

Startup behavior:

• Missing DATABASE_URL or missing Basic-auth credentials abort startup.
• If readiness fails before listen, startup logs the failure, closes storage, and exits non-zero.

Request-time behavior:

• Storage failures are mapped to 503 with storage_unavailable.
• Unexpected unhandled request errors are mapped to 500 with internal_error.
• Auth failures return 401 until throttling activates, then 429 with Retry-After.

Shutdown behavior:

• SIGTERM and SIGINT stop the Bun server and close the SQL client.
• uncaughtException and unhandledRejection are logged and then trigger shutdown.

Debug logging behavior:

• LRS_DEBUG_LOGGING=true emits request-start, request-end, request-error, and storage retry/debug logs to console.log.
• Request logs include a generated request id, but that id is not currently returned to clients or propagated across services.
• Logging is still console-based. There is no dedicated structured log sink, trace exporter, or metrics pipeline in the package itself.

Verification Commands

Use the narrowest command that matches the change, then close with the workspace gate.

1. Docs or root-only changes:

bun run validate:root

2. LRS package typecheck:

bun run --cwd apps/lrs typecheck

3. LRS package coverage gate:

bun run --cwd apps/lrs test:coverage

4. DB-backed LRS storage integration:

podman ps
podman pod ps
bun tools/scripts/run-integration-suite.ts tests/integration/lrs-storage.integration.test.ts

5. Relevant ADL subset during semantic work:

podman ps
podman pod ps
bun run test:lrs:conformance:conform-ed --xapi-version 1.0.3

6. Final workspace gate:

bun run validate:full

7. Full validation status requirement:

• Internal validation is not enough to call the LRS validated.
• A full upstream ADL suite run is still required before the package should be treated as validated for production-facing claims.

Good-Enough Acceptance Checklist

Enforced In Code

• Required database and Basic-auth configuration for the real server entrypoint.
• Version-header enforcement on spec routes.
• Basic-auth lockout and Retry-After behavior.
• Storage-backed readiness checks before listen.
• 503 mapping for storage failures.
• DB-backed collection filtering for the main statement query paths.
• Targeted store-backed resolution for /xapi/activities.

Validated In Automated Checks

• Package-local typecheck, tests, and coverage.
• DB-backed integration coverage for Postgres storage behavior.
• Relevant ADL subset runs for semantic slices.
• Workspace validation through bun run validate:full.

Still Operational Requirements

• Run conformance only against disposable environments.
• Keep real credentials out of source control and local command history where possible.
• Treat full upstream ADL suite success as the real spec-validation gate.
• Monitor 503 spikes, startup failures, and auth-throttle responses externally.
• Capacity test before exposing this package to materially higher traffic.

Not Yet Production-Ready For

• Non-Basic-auth public deployments.
• Multi-instance shared auth throttling.
• High-throughput statement ingestion or read-heavy traffic.
• Large-payload protection through explicit request-size limits.
• First-class observability with centralized logs, metrics, and traces.
• Formal incident workflows beyond the current lightweight console/debug path.

Until those gaps are closed and the full ADL suite is green, the honest label for this package is: development-ready, integration-ready, and conformance-harness-ready, but not broadly production-ready.