Fine-grained knowledge items as first-class competency referents

Status: accepted

Date: 2026-07-16

Relates to: ADR-0005 (unified learning/research construct axes — Selection Policy), ADR-0007 (state-canonical session/telemetry model), ADR-0026 (EPUB/PDF reading — demarcates the token-grained Language Reader lane this ADR now models the data side of), ADR-0050 (surface layout and subject packs), docs/architecture/core-model.md (competency core, personal offering intent), docs/architecture/language-module-migration-v1.md (the migration record this decision emerged from).

Context

Emergent’s competency core (competency_framework/competency_node, append-oriented person_competency_evidence → derived person_competency_state) was built for human-curated outcomes frameworks: hundreds to low thousands of nodes, edition-immutable, CASE-exchanged, synced to clients only through narrow projections.

The language-learning specialization — and any future “objective knowledge” domain (arithmetic facts, geography, music skills) — needs mastery tracking over tens of thousands of fine-grained knowledge items per person: vocabulary words, characters, grammar patterns. Evidence arrives from every learner action, on a spectrum of strength and intent: passive encounters during normal language use (reading, watching), learning activities (spaced-repetition reviews), manual declarations (“I know this word”), quizzes, and formal assessments. An assessment is an activity engineered to elicit high-quality evidence over a targeted competency set; a goal is a desired state over a competency set by a date. The platform’s central loop is: evidence → derived learner model → gap identification → Selection Policy tailoring.

Two properties of the domain stress the existing core:

• Scale asymmetry. One committed learner’s real numbers (measured in the predecessor system, transcrobes): ~41k tracked words, ~104k spaced-repetition cards, review throughput on the order of 10⁵ acts/year — against a core designed for per-node governance and human-meaningful evidence volume.
• The lexicon is an open set. No complete dictionary exists for any language; segmentation is imperfect (the same corpus held ~720k cached gloss lookups, most machine-segmented and many wrong-but-useful); novel usages are coined continuously; an AI translation engine permanently serves the tail. There is no closed inventory to enumerate up front.

Decisions

1. One referent space. Every knowledge item — word, character, grammar pattern, discrete fact/skill — is a first-class competency_node. Goals, alignments, evidence provenance, and exports all reference this single ID space. No parallel module-local referent IDs, no bridge layer.

2. Tiered operational machinery. High-frequency learner-model machinery (encounter running totals, spaced-repetition scheduling state, token-grained events) lives in specialized extension tables keyed by node ID. These tables implement fine-grained competency state; they do not duplicate its identity. They ride the pgxsinkit sync rail and are retention-managed.

3. Evidence summarisation boundary. Raw events (token encounters, individual review grades) stay in the event lane: append-only rows on the sync rail as a retention-trimmed buffer, forwarded to a separable archive schema. person_competency_evidence receives only meaningful transitions and summaries emitted by a single server-side projector (“crossed into known”, session rollups), with provenance pointing into the archive. Core evidence volume stays human-meaningful; the projector is the only reader of raw events and therefore the swap seam if the event store ever moves off Postgres.

4. Lazy minting — nodes are evidenced into existence. Gloss/translation lookup caches are enrichment-pipeline service data, never knowledge items. A surface form becomes a node the moment the learner model first has something to say about it: first tracked encounter, first card, first declaration. The node space is bounded by learner activity plus curated inventories, not by lexicon size (observed ratio: ~720k cached lookups vs ~41k tracked items for a committed learner).

5. Provenance tiers and a merge lifecycle. Nodes carry provenance (curated-dictionary / list-member / machine-segmented / learner-coined). Quality is a property, not an admission criterion — a mis-segmented form a learner genuinely drilled is legitimately their knowledge item. When segmentation is corrected or duplicates are recognised, nodes alias/merge and evidence + state re-map to the survivor (the identity_binding merge pattern, applied to referents).

6. Bulk authoring path for machine-generated inventories. Pipeline-generated node inventories are versioned in bulk; per-node revision/edition workflows remain for human-curated frameworks.

7. Client projection for the node inventory. The language node inventory syncs to learner clients via a dedicated projection (the CAT-pool precedent, larger), overriding the default DENY_ALL node posture. Definitions/glosses hydrate on demand, not as a full-lexicon sync.

8. CASE at curation granularity. Curated sets (HSK levels, CEFR inventories) export as CASE frameworks; the full lexicon is exportable on demand, never maintained as a live governed edition.

9. Goals are targets over competency sets, in core. A goal is (person, competency-set referent, success criterion, due date). Referents include curated sets, module-defined sets (word lists are competency-set definitions), and assessments (“be ready for assessment X” is a goal referencing an assessment, not an assessment). Goals are a core construct, not a module one: nothing language-specific exists in “desired state over a competency set by a date”, and progress derivation reuses the evidence → state machinery.

Alternatives rejected

• Full residence: all machinery (counts, scheduling state) on core competency tables — dies on evidence volume (drill telemetry in an OLTP evidence table), per-node governance weight (edition/revision workflows over 50k+ machine-generated rows), and node sync posture.
• Separate ID space + bridge: module-native word/list IDs mapped onto competencies — two referent spaces, a permanent translation layer, and fragmentation of goals, evidence, and alignment semantics.
• Eager enumeration: minting nodes for the whole lexicon up front — imports hundreds of thousands of segmentation artefacts per language as “durable competency referents” and still never closes the set.

Consequences

• Scale hazards are addressed structurally rather than by capping granularity: node counts are trivial; per-person state is sparse (only evidenced items); evidence volume is gated by the summarisation boundary; governance weight by the bulk path; sync fan-out by the projection.
• The competency core’s derived-state pattern (person_competency_evidence → person_competency_state) becomes the platform-wide template for learner models; word-level state aggregates upward into ordinary competency state (list/level mastery) via the same projector.
• Per-person operational state at real scale (~10⁵ rows) syncs to PGlite clients; this is proven practice in the predecessor system, not a new risk.
• The retention Selection Policy objective (reserved in ADR-0005) acquires its data substrate; the scheduler itself remains module code behind that seam.
• Erasure and consent governance apply to the new person-scoped tiers (event lane, operational tables, archive) exactly as to existing learner state; the archive must be registered with the erasure machinery.