In-app EPUB/PDF reading: opaque External-Package delivery + a generic Reading State overlay

Status: accepted

Builds on ADR-0009 (imported-standard boundary: QTI is Ingested, everything else stays opaque), ADR-0022 (CC 1.4 decompose; the webcontent → static-package precedent), ADR-0002 (headless core, Mantine skin), ADR-0007 (session/telemetry), ADR-0008 (consent/research governance), and ADR-0014 (erasure-is-anonymization). Pre-launch and conform-ed is ours ([[feedback_no_legacy_pre_launch]], [[feedback_internal_lib_release_all_packages]]).

The trigger (and the premise it falsifies)

The question arrived as “if we fully support Common Cartridge 1.4, shouldn’t we also read EPUBs?” Grounding falsifies the implication:

• CC 1.4 does carry EPUBs — a dedicated resource type idpfepub_epubv3p0, added in v1.3 alongside IWB, APIP, and Open Video. ADR-0022’s routing table never mentions it (a gap this ADR closes).
• But rendering EPUB is optional for conformance. The CC 1.4 spec lists EPUB3/IWB/APIP/QTI-3 zips as “allowed as exception in Core CC,” and requires only that a system “fail gracefully and indicate to the user that they do not support the feature.” That is exactly ADR-0022 §5 and the CONTEXT.md definition of a “full” import: nothing silently dropped — quarantined-with-reason in the Cartridge Import report. A cartridge carrying an EPUB imports fully and conformantly with the EPUB quarantined, just like an LTI link.

So CC is not a forcing reason. The real decision is standalone: do we want in-app reading of learner documents (EPUB, and PDF)? The justification is domain fit — extensive/graded reading is core to language learning (the initial focus), and general texts (a chemistry book) are common LMS content. If we build the reader, CC’s idpfepub_epubv3p0 gets a home for free; if we don’t, CC stays conformant regardless.

Two readers, deliberately demarcated

There will be two ebook readers, and conflating them is a security mistake, not just an aesthetic one:

• Generic Document Reader (this ADR, now): renders arbitrary, untrusted books arriving from CC imports or teacher uploads. An arbitrary EPUB is arbitrary HTML/CSS/JS; the dominant requirement is robustness + isolation, not features.
• Language Reader (future, separate): a bespoke, already-built reader over reprocessed, trusted content — per-word events, timers, computed/dynamic glosses. Operates on cleaned input and assumes it.

The line between them is anchoring granularity (locator-grained vs token-grained) and trust model (arbitrary/untrusted vs cleaned/controlled). They never share a delivery path or a state model: the Language Reader’s clean-input optimisations are a hole if pointed at a hostile book, and the generic reader’s defensive overhead is dead weight on reprocessed content. The features that make Readium “too heavy” for the controlled Language Reader are disproportionately the ones the generic reader needs (odd-book tolerance, full nav, a11y) — “wrong for B” does not transfer to A.

Decision

1. Decouple from CC conformance

In-app reading is an optional learner-content capability, justified by reading practice — not a CC obligation. CC’s idpfepub_epubv3p0 routes to a document External Package iff reading is enabled, else it quarantines (ADR-0022 §5 unchanged). This closes the EPUB3 gap in the ADR-0022 routing table; IWB / APIP / Open Video remain quarantined-with-reason (named follow-ups).

2. Delivery = an opaque, sandboxed External Package

New external_package_kind epub (and pdf), a matching launch kind, and a new runtime provider. The document renders in-app in a sandboxed iframe with strict CSP (block all scripts except 'self') so the book’s own scripts never execute. The content stays opaque to the canonical model — it is not Ingested the way QTI is (the ADR-0009 line); the reader library parses spine/nav purely to display it, exactly as a SCORM player parses its package inside its iframe. This reuses the existing External Package machinery — Version (an immutable snapshot, which is what makes locator anchoring safe), Assignment, and Launch Session. It is the direct sibling of ADR-0022’s webcontent → static-package: rendered honestly as-is, not pretending to be transparent.

3. Library = foliate-js (MIT), vendored and CSP-sandboxed

Option	License	Why
foliate-js	MIT (vendors zip.js BSD-3, PDF.js Apache-2.0)	Chosen. Pure client-side, no server; EPUB/MOBI/AZW3/FB2/CBZ/PDF; hooks-only (no bundled storage — see §5). Self-declares an unstable API → vendor + pin. Mandates CSP, which is the isolation posture we want anyway.
epub.ts (LikeCoin)	BSD	Named fallback — maintained TS fork of epub.js, EPUB-only, more stable API in the epub.js idiom.
Vivliostyle	AGPL-3.0	Rejected. Copyleft trap for a closed-source SaaS — §13 network-use reaches the combined work, and vendoring strengthens the obligation, it doesn’t dodge it. Also the wrong tool: a CSS Paged-Media print/PDF typesetter, not a screen reflow reader.
Readium ts-toolkit / Readium Web	BSD-3	Rejected for this lane. The web navigator renders Readium Web Publications, not raw EPUB — it needs a server-side streamer to crack the package first. That re-introduces exactly the infra weight already rejected for the Language Reader, for a nice-to-have.

PDF rendering is PDF.js (Apache-2.0) — already foliate-js’s PDF backend.

4. Reading State = a new generic, format-agnostic overlay (NOT Package Progress)

A learner’s reading state is a new first-class model, separate from the External Package’s xAPI-only lane and from Package Progress. It is locator-anchored — EPUB CFI, PDF page+region — never page numbers (EPUB reflows; “page” is a viewport artifact). It carries three things keyed by (Person, External Package Version, locator):

1. progress / dwell — also mirrored to xAPI for the teacher analytics view (“3 min on chapter 4”);
2. bookmarks;
3. annotations (highlight + note).

It is document-type-agnostic: the locator is the only format-specific part, so the same model serves EPUB and PDF. This is the part that genuinely exceeds “opaque, observed only through xAPI” — you cannot reconstruct “show my 4 bookmarks” by replaying an LRS statement stream; it needs a queryable application store.

5. The library/host seam — no parallel rebuild

foliate-js owns anchoring + rendering only and ships no storage (“storage is entirely the host application’s responsibility”). The seam is therefore the natural division of labour, not duplication:

foliate-js owns	emergent owns
parse, paginate, render-in-sandbox	persist { locator, kind, note, person, packageVersion }
relocate event → { range, index, fraction } (current locator)	consent gating + erasure registry wiring
CFI.fromRange(range) (selection/position → CFI)	teacher rollup / analytics projection
Overlayer + create-overlayer.attach(overlay) (render highlight from a stored CFI)	the Mantine reader chrome (TOC, bookmark control, annotation list)
.goTo({ index, anchor }) (restore to a stored locator)

A library that shipped its own opinionated storage (Readium-style) would be the thing we’d fight; foliate-js’s hooks-only design is the clean seam.

6. Governance

Reading State is learner PII. It is consent-gated (ADR-0008) and wired into the erasure registry / anonymised (ADR-0014) like any other learner-authored data. Teacher visibility (dwell, bookmark counts) is consented analytics, not ambient surveillance.

Tiering (so the cheap part is not hostage to the expensive part)

• Tier 1 — the genuine nice-to-have. Vendor + sandbox foliate-js; render EPUB (+PDF); route CC idpfepub_epubv3p0; emit coarse locator progress/dwell → xAPI / Package-Progress teacher view. No new PII model beyond governed telemetry.
• Tier 2 — separately justified. The Reading State overlay: bookmarks + annotations + teacher rollup, consent + erasure wired. This is where “first-class” and the real build live.

Considered and rejected

• Tie EPUB to CC conformance — rejected; optional per spec, quarantine already conformant.
• Vivliostyle — rejected; AGPL copyleft + wrong tool (typesetter, not reader).
• Readium ts-toolkit / Web for the generic reader — rejected; client/streamer split = server weight; overkill; already rejected for the Language Reader.
• A native reader content type owning delivery and state — rejected; forfeits iframe isolation + External Package reuse and pre-empts the Language Reader.
• Store annotations as xAPI statements only — rejected; “list my bookmarks” cannot be a log-replay; it needs a first-class queryable store.
• Page-based locations — rejected; EPUB reflows, so CFI/locator is the only stable anchor (immutable Version bytes are what keep CFIs valid under stored highlights).
• Extend the Language Reader to cover generic books (or vice versa) — rejected; different trust model + granularity; coupling is a security hazard.

Consequences

• New external_package_kind epub / pdf + launch kind + runtime provider; vendored foliate-js
  • PDF.js; CSP/iframe hardening for untrusted documents.
• A new Reading State model (tables) with consent gating and erasure-registry entries.
• ADR-0022’s routing table gains a conditional EPUB3 home; IWB / APIP / Open Video stay quarantined (named follow-ups). ADR-0022 carries a pointer here.
• The Language Reader remains a separate, future, token-grained language-learning feature; this ADR deliberately does not model it, only demarcates it.
• Executed incrementally (feature branch per slice, rebased to land — [[feedback_rebase_never_merge]]); a docs/design/epub-reading-plan.md slice plan follows.

Implementation note (2026-06-16) — the script-blocking mechanism is CSP, not the sandbox token

§2 framed isolation as “strict CSP (block all scripts except self)”. The browser-verified build sharpens this: book script is blocked by a per-section CSP of script-src 'none', injected through foliate’s official transformTarget content hook — with no foliate patch. We deliberately keep foliate’s allow-scripts sandbox token. The reflex to “harden” by dropping allow-scripts is a trap: the e2e lane proved it breaks in-content navigation in WebKit/iOS (WebKit bug 218086 — an in-content link click does nothing without the token), which would silently drop every iOS browser. So the security boundary is the CSP; the sandbox is left as upstream. The lane runs in chromium and webkit and asserts render + in-content navigation + script-blocked in both.

Implementation note (2026-06-16) — delivery is lean, NOT the cmi5 launch ceremony

§2 says delivery “reuses the existing External Package machinery — Version, Assignment, and Launch Session.” The build sharpens what “Launch Session” meant. It reuses the entitlement + assignment + Version + PackageAsset model, but a document is not a cmi5 AU, so it deliberately bypasses the cmi5 launch ceremony (the LRS client credential, the fetch token, and the launched/satisfied/abandoned xAPI statements that createLaunchSession emits). Instead a lean endpoint resolves the assignment, checks offering entitlement, and signs the opaque bytes:

• Serving (GET /workflow/document-packages/:assignmentId/delivery) returns JSON { kind, title, externalPackageVersionId, url, expiresInSeconds } (not a 302 like the QTI asset serve — the reader is JS that needs the kind + version id alongside the bytes URL). Entitlement is the same offering-membership gate as the QTI asset pipeline (org/platform admin, offering teacher/assistant, or active member). The url is a short-lived signed Storage URL.
• Creation (POST /workflow/document-packages, org-admin) validates the upload by magic (PDF %PDF-; EPUB by the OCF-mandated uncompressed mimetype = application/epub+zip first entry — never the client’s declared type), stores it as one opaque object (not an unzipped multi-file package — the .epub/.pdf is the deliverable), then creates the row graph through the shared importExternalPackage seam (so document + SCORM/h5p imports share one creator). Storage reuses the existing private external-package-runtime-packages bucket (500 MiB, no mime restriction) — no new bucket, so no Storage migration.
• Schema: external_package_kind and external_package_launch_kind gain epub/pdf; package_runtime_provider gains foliate. The whole migration set was regenerated from the schema per the regenerate-migrations runbook (a DB-free file operation — the initial_baseline now carries the new enum values). Applying to the dev DB is Anton’s smoke-test step but gates nothing: the unit lane uses the PGlite pushSchema harness and the container integration lane applies the committed migrations to its own throwaway Postgres.

This is consistent with §4/§6, not a reversal: §4 already puts Reading State and progress on a separate lane from “the External Package’s xAPI-only lane,” and tiering defers progress. So the cmi5 statement ceremony was never the right vehicle for a book; progress (Slice 3) is the locator-anchored lane, where the xAPI-mirror-vs-direct-package_progress choice is decided.

Decisions for Anton (resolved in the 2026-06-16 grilling)

1. Reframe to a standalone learner-reading capability; CC conformance explicitly not a forcing reason (quarantine already satisfies it). (accepted)
2. Hybrid: opaque External-Package delivery + a generic Reading State overlay. (accepted)
3. foliate-js (MIT), vendored + pinned + CSP-sandboxed; Vivliostyle (AGPL) and Readium (split architecture) rejected; PDF via PDF.js; epub.ts the fallback. (accepted)
4. Reading State is locator-anchored, format-agnostic (EPUB + PDF), separate from Package Progress, and PII-governed (consent + erasure). (accepted)
5. The library owns anchoring + rendering; emergent owns persistence + governance + rollup — no parallel renderer; foliate-js’s hooks-only design is the seam. (accepted)
6. Tiered: render + coarse progress first; bookmarks/annotations as Tier 2. (accepted)
7. Two readers demarcated by anchoring granularity + trust model; never a shared path. (accepted)

Implementation note (2026-06-16) — progress rides the offline sync rail, NOT LRS HTTP

The prior two notes built delivery as a lean, online signed-URL fetch and left Slice 3 (progress) with an open vehicle. Starting to wire it surfaced the omission this whole ADR had quietly carried: offline. Emergent is an offline-first pgxsinkit/PGlite platform, but every progress vehicle on the table — the reader POSTing xAPI to the LRS, or the API server-authoring a statement and POSTing it to the LRS — is online-only. Building any of them now would be throwaway. Resolved:

• Progress (and, later, Reading State) are captured as learner-owned, append-only rows on the existing pgxsinkit sync rail — the precedent is already in sync-registry.ts readwriteEntries: interactionEvent (“append-only telemetry, cat 5”) and itemSession (“local-captured, cat 4”) are captured into local PGlite and synced up, applied server-side by sync-mutation-applier.ts. xAPI statements are exactly this — immutable, UUID-keyed, conflict-free — so they fit that rail with no new sync machinery. This is the literal cash-out of “xAPI is just an append log.”
• The server projects synced rows into package_progress through the same projection the LRS uses. lrs.ts is pgSchema("lrs") and apps/lrs/src/db/client.ts reads the same DATABASE_URL: the LRS is not a separate database, so package_progress and derivePackageProgressEvent / applyPackageProgress live in the Postgres the API already writes. The projection is extracted into a shared module so the API’s sync-applier and the LRS HTTP path call one implementation; no cross-service HTTP is structurally required.
• The LRS HTTP ingestion + Basic-credential model (launch-session / system-integration) stays for genuinely external, online xAPI producers (third-party cmi5 AUs, interop tools). Our own offline reader does not cosplay as an external AU. This is what dissolves the credential fork (no LRS Basic credential or server-authored-over-HTTP statement for the native reader) — it was an artifact of pushing an offline producer through an online ingestion door.
• Open follow-on (named, not solved): offline delivery. Delivery itself is still online (fetch opaque bytes from a short-lived signed URL on demand). True offline reading needs the document blob cached/synced locally — the same question CC/QTI External Packages face. Stance: online-first for v1, offline package-byte caching is a deliberate later decision, not a drift.

This resolves the “xAPI-mirror-vs-direct-package_progress” choice the prior note left open in §4/Slice 3: direct package_progress via the shared projection, fed by synced append-only rows.

Status: BUILT. mergePackageProgress extracted to @emergent/db (the LRS now calls it, behaviour unchanged); the learner-owned append-only document_progress_event synced table (RLS + erasure + sync registry); the sync mutation applier inserts the append row and projects package_progress in one transaction; the reader captures progress through a throttle (document-reader/lib/progress-capture) into local PGlite via recordDocumentProgress. Unit + integration (in-progress → completed → monotonic) + e2e (fraction fires for EPUB and PDF in chromium/webkit) green.

Implementation note (2026-06-16) — offline delivery: the bytes ride the Cache API

The follow-on the progress note named is now closed. Offline reading needs two things local — the opaque bytes and enough metadata to open them — on the storage each is suited to:

• Bytes → the Cache API, not PGlite/IndexedDB/OPFS and never the pgxsinkit rail. A book is a fetched HTTP resource, which is precisely what the Cache API exists for; it streams the response to disk (no 500 MiB buffered in JS) and match().blob() hands foliate exactly what it wants, on the main thread, in every engine. OPFS was considered and rejected for v1: its uniform cross-engine support (incl. iOS WebKit) only dates to ~mid-2023 and its portable write path is Worker-only (createSyncAccessHandle) — more surface, narrower reach, for a random-access capability we don’t use (we hand foliate a whole File). OPFS stays the documented escape hatch if we later range-load big PDFs via PDFDataRangeTransport. Bytes are keyed by the immutable externalPackageVersionId (content-addressed: re-pointing an assignment misses and re-fetches; the stale blob ages out by LRU), via a synthetic cache request — never the rotating signed URL.
• Descriptor → cached beside the bytes, keyed by assignmentId. We deliberately did not put external_package_assignment on the sync rail: the assignment row carries only the version id, not the kind/title the reader needs (those are on external_package / _version), so syncing it alone wouldn’t deliver the descriptor, and syncing all three tables is overkill with no document-list view yet. Caching the API’s already-joined delivery JSON is complete and simpler. (When a library/list view arrives, putting the assignment on the sync rail becomes worthwhile — for listing assignable documents offline — and is an additive change.)
• Two tiers from the two triggers. Incidental first-read caching happens in loadAssignedDocument for free (the reader already downloads the whole blob to open it); it is LRU-evicted under a byte budget. The “Save for offline” control pins a version (never evicted) and requests navigator.storage.persist() for durable origin storage. All web APIs are injected behind small interfaces (offline-cache.ts), so the LRU/pin policy is unit-tested without a browser and the cache degrades to a no-op where the Cache API is absent (insecure/legacy contexts).

loadAssignedDocument is now offline-capable: online it caches the descriptor + serves bytes cache-first (caching on first read); if delivery fails it falls back to a fully-cached copy. The whole read loop closes offline — descriptor + bytes from the Cache API, progress events queued in PGlite and flushed on reconnect.