Skip to content
emergent engineering

Basic computer-adaptive testing: in-process Rasch engine behind the CAT 1.0 seam

Status: accepted (supersedes ADR-0025 §4)

We will ship a basic but genuine Computerized Adaptive Test (CAT) before go-live: a small, owned TypeScript engine implementing common, simple psychometric algorithms, sitting behind the exact seam an industrial engine would later occupy. This reverses ADR-0025 §4 (“CAT out of scope; revisit via a dedicated ADR”) — this is that ADR.

CAT here is not a QTI feature. Per ADR-0005 it is the measurement-precision mode of the single Selection Policy: given learner state + an item pool + an objective, choose the next item. We build that one mode now; spaced repetition and experiment assignment are sibling modes, deferred.

The decisions (11)

Section titled “The decisions (11)”

  1. The seam is conform-ed’s CatV1_0 contract, called in-process. The engine consumes SectionData + AssessmentResult and returns ItemStage + CatEngineResultReport — the real Zod contracts from @conform-ed/contracts/cat/v1_0 — but is invoked as a TS module, not over HTTP. Swapping an industrial engine in later is “point the same typed boundary at a REST adapter,” which is config, not a rewrite. This buys CAT 1.0 conformance now and the swap option without a premature network service.

  2. Rasch / 1PL is the measurement spine. One difficulty b per item; ability θ on the same logit scale; item information peaks at b ≈ θ. The smallest model that is genuinely CAT, needs only one author-suppliable number per item, and speaks the same θ/info vocabulary as any engine we’d swap in. Multistage and staircase patterns can be added later as alternative selection strategies behind the same seam.

  3. Item parameters are author/import-supplied and frozen. No response data exists to calibrate from yet. A new immutable item_parameter_set (b, provenance ∈ {author, imported, calibrated}) is bound to one Item Version; each item_session pins the exact set it used (params_snapshot_ref), so a θ is always reproducible. A CAT pool is gated to require a parameter set on every item-version — a missing b is an authoring error, never a silent default. calibrated is the scaffold for the future calibrate-from-interaction_event path.

  4. EAP is the estimator. Expected a posteriori with a N(0,1) prior over a fixed quadrature grid. Chosen over MLE because MLE diverges on all-correct/all-incorrect patterns — i.e. every CAT’s first items — whereas EAP is robust from item one, never diverges, and the posterior SD is the SE the stopping rule needs. Cost: mild shrinkage toward 0, which vanishes as the test lengthens. Grid and prior are hardcoded sane defaults now, tunable later.

  5. Selection is max-information + randomesque. For Rasch, max-info = the unused item whose b is nearest current θ. To avoid over-exposing mid-difficulty items we pick randomly among the top-k most-informative (default k=3), seeded by the attempt’s replay key so a session stays reproducible. The first item (θ = prior mean 0) starts at median difficulty. Sympson–Hetter, content-balancing and constraint-based selection are the industrial-engine carrot — deferred.

  6. Stopping combines three authored constraints: stop when (SE ≤ target OR items ≥ max) AND items ≥ min, defaults min=5 / max=20 / SE≤0.3, with pool exhaustion as an implicit terminator. Carried in SectionData.constraints[]. Fixed-length is min=max. The θ→reported-score/band transform is deferred; v1 reports raw θ + SE.

  7. Server-anchored only. CAT is structurally incompatible with offline-complete: it would require shipping the learner the whole pool + every answer key + every parameter, defeating key-withholding and enabling gaming. Each “respond → next item” round-trip is a server step the client blocks on — exactly what server-anchored already means (ADR-0006). Authoring a CAT under an offline-complete profile is a validation error.

  8. CAT/measurement only, behind a generic SelectionPolicy seam. Experiment assignment (arms, bandits) and its governance (Study/Consent — ADR-0008) are deferred; item_session.variant/condition stay dormant this phase. A later experiment-mode policy lights them up. (condition = the experimental arm; distinct from QTI feedback’s “condition” predicate — see CONTEXT.md.)

  9. emergent claims the platform/delivery conformance role and ships a reference engine. Proven by an in-process interop lane (the ADR-0033 “interop proof = done” pattern): a full adaptive session drives the real engine against the real CatV1_0 Zod contracts, asserting both shape-validity and psychometric behaviour (θ converges to true-θ, SE shrinks monotonically, stop fires in bounds). cat/v1_0 is pinned in the conformance overlay (ADR-0028); the REST/OAuth transport binding is deliberately deferred (it lights up only when an external engine is wired — the coverage map treats CAT operations as payload-constraining requirements, not a walked transport axis).

  10. A CAT is a first-class pool delivered via a Practice Session, scaled by a competency node. Not an assessmentTest section — that would entangle CAT with QTI sequencing and the §4 “separate spec layer” boundary. A CAT Pool is an explicit, author-curated set of Item Versions declared to measure one latent trait, anchored to a CASE competency node (which gives θ meaning and reuse). Unidimensional in v1. SectionData is assembled 1:1 from the pool.

  11. The reference engine is stateless; caching is a sanctioned later optimisation. θ is a pure function of (persisted item_session responses + each row’s params_snapshot_ref

    • pool + selection_policy_version + grid/prior config). The persisted rows are the source of truth — the engine holds nothing between round-trips, making it a pure function: trivially testable, perfectly reproducible, crash-safe, and clean to swap. SessionInfo is a thin handle onto the practice_session. A Dragonfly (Redis-compatible) memo of the computed posterior is allowed later as a non-authoritative, fully-keyed cache (key includes every input, so a rescore or recalibration can never serve a stale θ; audit/DIF/research always recompute from rows). It does not make the engine stateful in the problematic sense.

Consequences

Section titled “Consequences”
  • New domain code (pure, no DB): a Rasch/EAP psychometrics module + a stateless CAT engine implementing CatV1_0 + a generic SelectionPolicy seam, all in packages/domain.
  • New schema (Anton’s DB-apply step): item_parameter_set (gives params_snapshot_ref a target) and a CAT-pool authoring surface; both regenerate through the migrations runbook.
  • The conformance board gains a pinned cat/v1_0 map with an honest triage: information model + operation semantics covered in-process; transport binding deferred; vocabularies verified as value-sets.
  • The swap to an industrial engine is reduced to writing a REST adapter behind the same contract — no change to session-writing, pool assembly, or persistence.
  • Statelessness leaves a clean optimisation lever (Dragonfly) for if/when recomputation ever shows up in latency — not before.

Decisions for Anton

Section titled “Decisions for Anton”
  1. CAT ships basic-but-real now, behind the CAT 1.0 seam, reversing ADR-0025 §4. (accepted)
  2. Rasch/1PL + EAP + max-info/randomesque + 3-way stopping, all sane-defaulted and tunable later. (accepted)
  3. Server-anchored only; offline-complete CAT is a validation error. (accepted)
  4. New tables item_parameter_set + CAT-pool surface regenerate via the migrations runbook; DB reset+apply remains Anton’s step. (pending operator step)
  5. Experiment assignment + governance stay deferred; this phase is measurement only. (accepted)