Skip to content

Sprint 1: Mandala Images implementation (pilot)

Status: ◐ In progress — Step 1a ✅ complete (2026-07-08); 1b.2 ✅ complete (PR #27 + #28, full data migration 111,307/111,307); 1b.1 ✅ complete (PR #30 + PR #32 + solr-proxy CI/CD deployed 2026-08-12); 1b.3 ✅ complete — proven end-to-end 2026-08-20 (PRs #128–#132): all four OAuth2 defects are fixed and verified live on dev-0 — signing-key persistence, the proxy's missing Bearer header, the openid scope's missing permission grant, and the fourth session-handling redirect loop (fixed by the new mandala_saml_oauth module, which exempts OAuth2 Bearer requests from checkAuthStatus()'s SAML-liveness check). ADR 014's visibility filtering is proven, not merely functional, with a positive and a negative discriminator; two regression tests now live in scripts/; 1b.4 ✅ proven 2026-08-18 — paragraph access correctly inherits from the parent node, flat and nested, no code change needed. Both steps are now task-complete; what remains to close Sprint 1 is the deferred 1a.9 staging acceptance run (gated on staging prerequisites) and signing off the acceptance criteria below, none of which are ticked yet. Phase: Roadmap Phase 1 — mob-build the Images pilot (the spine) Lead: Yuji Shinozaki Mode: Mob-build (whole team), then individuals replicate the pattern Relates to: ADR 008 (migrate, not improve), ADR 009 (Images is the pilot), ADR 010 (remodeling permitted), Images Content-Model Audit

Goal

Build and migrate Mandala Images end-to-end on Drupal 11 as the first vertical slice of the 5→1 consolidation — proving the shared migration pattern (Migrate API consolidation, KMaps field productionization, Solr sync via reindeer_x, proxy auth, and a rollback story) once, together, before the codebase forks into per-site tracks.

A green Images pilot is the early demonstrable win. It deliberately isolates the hard content risks (footnotes, Kaltura, bibcite, Tibetan script) — those live in Texts and AV and are retired in their own Phase 2/3 tracks — but it does include the proxy-auth access-control foundation, because Images has substantial proxy-auth-gated content.

Scope boundary

Inherited from ADR 008 / ADR 010: faithful migration of user-facing behavior is the floor; internal data remodeling is permitted where it reduces risk.

In scope (Sprint 1) Out of scope (later phases)
Full shanti_image entity graph (primary + 3 satellites + scheme lookup) Texts footnotes, AV/Kaltura, Sources/bibcite
4 KMaps fields wired to kmassets (Spike 1 productionized) Improving Tibetan/Solr search quality (deferred)
Transliteration diacritic fidelity (NFC/NFD round-trip) New IIIF stack or shanti-image-NNN scheme change (IIIF stays as-is)
Content indexes and is retrievable via existing query patterns JSON/AJAX API parity (api/json/{nid}, api/ajax/{nid}) — Phase 5
Proxy-auth access path + Solr-proxy visibility filtering Auth redesign (hybrid proxy per ADR 014; D11 enforces access natively per ADR 013)
OG → Group collection membership for Images Sub-subcollection nesting beyond one level (deferred)

"Docs land in Solr" (✓) is not "Tibetan search works well" (deferred). The Solr success criterion is written narrowly — retrievable via existing query patterns, not search quality.

Two-step structure (per ADR 009)

  • Step 1a — public plumbing. Migrate the Images public subset; prove consolidation
  • KMaps + Solr sync + retrieval end-to-end. The win decoupled from auth risk.
  • Step 1b — auth increment. Wire D11 into the existing proxy-auth contract and prove the security path. Access-control coherence (Solr-proxy visibility vs. node/Group access) is an explicit integration concern — search-visible results and node-level access must agree on "who can see what."

Backlog

The Images content model is settled by the audit: the D7 "five content types" become a shanti_image content type with the three satellites as Paragraph types and external_classification_scheme as a taxonomy vocabulary. Production-data validation (111,340 images) confirms required fields are 0-missing / 0-out-of-list and agents are 99.8% per-image. Tasks are in dependency order.

Progress

Date Landed
2026-06-17 1a.1–1a.3 ✅ — shanti_image (48 fields), the three paragraph types, and the external_classification_scheme vocabulary built as committed CMI config (382-file baseline, zero-diff round-trip). Prereqs cleared: core bumped D10.6 → D11 (ADR 002) + paragraphs/ERR added; settings.php committed env-driven with config_sync_directory (drupal-dsf convention); Dockerfile → D11. New deferred notes: agent-name mapping, prod packaging. KMaps search-roots punted to 1a.4.
2026-06-18 1a.4 ✅ (on branch sprint-01/kmaps-search-root) — KMaps field productionized as the Spike 1 tail: new search_root_kmapid field setting + config schema; widget + autocomplete controller plumb the root through as an ancestor_id_path fq; field_kmap_collections wired to root 2823, field_language to root 301. New KmapsPathResolver service (Drupal cache.default, batched, Solr-fallback-safe) for non-widget code paths (migrations, programmatic save). KMaps D7→D11 migration mapping doc pins three policy decisions (preserve paths/headers verbatim; don't validate orphans at migration time) — feeds 1a.7. Live-Solr finding: the prior build's kmap_domain: terms on both fields was wrong; Solr probes proved both roots live in subjects, fix applied. Verified end-to-end against live kmterms index. "Wire to kmassets" remains the 1a.8 / reindeer_x handoff.
2026-06-22 1a.6 ✅ — First migration shipped. Establishes the Migrate API plumbing for everything that follows. New mandala_migrations module hosts a mandala_images migration group and the d7_images_external_classification_scheme migration (D7 external_classification_scheme nodes → D11 taxonomy terms in the 1a.3 vocabulary). Composer adds drupal/migrate_plus ^6.0 + drupal/migrate_tools ^6.1; enabled core migrate + migrate_drupal + plus/tools. Source DB: D7 prod images dump (data/mandala-prod-images-db_2026-06-11.sql.gz, 70MB) loaded into secondary DDEV database d7_images; Drupal migrate connection key wired in settings.php inside the DDEV-conditional include block, portable across teammates. Migration result: 2 source rows (LCSH + TGN — the only two schemes in the production dump) → 2 D11 terms (tid=1, tid=2), all 5 custom fields + name + description (with format preserved) populated; entity-reference from the external_classification paragraph's field_external_class_scheme finds both terms, so 1a.7 image migrations will be able to reference them. Surprise: production has only 2 schemes, not the 5–10 we'd assumed. New deferred note: migrate_drupal noise on drush ms (Low — workaround is --group=mandala_images).
2026-06-23 1a.7 ✅ — The pattern-setting migration shipped: D7 shanti_image nodes → D11 nodes with the satellite graph transformed to owned paragraphs, run end-to-end against the full production dump. Two custom source plugins: D7ImageSatellite (sources satellites per reference so shared agents fan out one-paragraph-per-image and orphan satellites are skipped; mapJoinable() disabled for computed ids) and D7ShantiImage (joins the shanti_images sidecar for i3fid/dims and emits [{nid,delta}] lookup keys per satellite). Four migrations (d7_images_image_agent / _image_descriptions / _external_classification / _shanti_image) in mandala_migrations; node refs wire to fanned-out paragraphs via sub_process + migration_lookup (@_para/0,@_para/1) with a skip_on_empty guard for dangling refs. Full-run reconciliation (0 failures): 111,340/111,340 images; 111,194 agent paragraphs (= 111,345 refs − 151 dangling); 55,038 description paragraphs (3 oversized summaries — see below); 9 classifications; node-level KMaps exact to baseline (subjects 79,337 · places 68,755 · terms 61,668 · collections 83,494); description field_language 7,303 = exactly the referenced-description count (orphans correctly skipped). Tibetan title + transliteration diacritics verified NFC-preserved. Four data-model decisions landed as committed config: added field_description_title (89% of referenced descriptions carry a meaningful caption) + field_external_class_label carrier fields; empty agent names → "Unknown"; field_iiif_id (36 imageless records) and field_image_agents (151 dangling-only images) relaxed to optional. Found + fixed: the D7 KMaps raw column is angle-bracket ancestors, not the pipe composite the D11 field needs — copying it silently dropped all KMaps; migration now rebuilds raw via concat (deferred note). Model fix: field_summary was string(255) but D7 summaries reach 750 chars → widened to 1024 (applies on fresh install; the 3 affected rows migrate cleanly in the staging run). File binaries deferred (99.5% display via IIIF; note). The full prod run remains a staging activity per the acceptance criteria (1a.9).
2026-06-22 1a.5 ✅ — IIIF wiring proven and shipped as new shanti_iiif module (IiifUrlBuilder service + IiifImageFormatter field formatter + admin settings at /admin/config/media/shanti-iiif). Four storage fields added to shanti_image for the i3fid linkage: field_iiif_id (required), field_iiif_mms_id, field_iiif_width, field_iiif_height. field_image view display swapped from stock image formatter to iiif_image. Reachability gate passed — DDEV web → iiif.lib.virginia.edu/mandala/{i3fid}/info.json returns HTTP 200 in ~150ms warm. URL contract verified byte-identical to D7 for any siid (/mandala/{i3fid}/full/!W,H/0/default.jpg). End-to-end smoke test: a shanti_image node with field_iiif_id = shanti-image-680687 renders the same IIIF derivative URL the D7 site uses. Upload/delete path explicitly out of scope (covered by 1a.7 migration which carries i3fid forward verbatim, and by post-MVP user-upload). Two deferred notes filed: Cantaloupe 404 info disclosure (Medium); /mandala/ vs canonical /iiif/2/ prefix alignment (Low).
2026-07-01 1a.8 ✅ — mandala_kmassets_sync module: kmassets doc-builder (fixture-validated, 114-match, 0 defects) + direct-to-master Solr sink (KmassetDirectSink — synchronous POST, IP-gated, no creds) + node lifecycle hooks (hook_node_insert/update/delete, errors logged/never crash) + Drush commands (kmassets:index, kmassets:index-all, kmassets:delete). uid format: {service}-11-{nid} (frozen generation marker — not the Drupal version, survives D12/D13 upgrades). __BASE_URL__ token in URL templates with per-environment $config[] override in settings.php (DDEV → https://mandala.ddev.site). Direct writes to Solr master confirmed always acceptable (IP-gated from VPC, no HTTP credentials needed). File/S3 sink explicitly deferred — direct sink covers Sprint 1 scope. Deferred notes filed: kmassets uid identity across migration, kmassets uid consumer analysis, kmassets sync error-management. First task of 1a.9: build kmassets:audit command (missing/orphaned/stale detection + --fix flag) before anything else.
2026-07-07 1a.9 ◐ started (PR #19, branch feat/1a9-rollback-story) — first task done: kmassets:audit drift detector, the validation tool the rollback cycle needs. New KmassetAuditor service + kmassets:audit [bundle] [--check-stale] [--fix] [--batch-size]. Detects three drift classes: missing (published node, no doc), stale (doc node_changed lags node — opt-in --check-stale, the expensive pass), orphaned (doc for a deleted/unpublished node). Two passes: Drupal→Solr batched uid:(…) lookups (missing/stale) + Solr→Drupal cursorMark sweep (orphaned); --fix reuses the 1a.8 index/delete primitives. KmassetDirectSink gained a read path (select()). Verified end-to-end against the staging Solr master (111,339 published images, rigged-then-cleaned scenario): detection reconciled exactly (missing 111,336 + present 3), stale/orphan repair primitives corrected node_changed and removed a ghost doc. Found + fixed: cursorMark must sort on uid — the schema API confirms uid (not id) is the kmassets uniqueKey; Solr 400s otherwise. Follow-ups flagged in PR: confirmation guard for large --fix runs; Pass A speed-up (loads full node entities for id+changed, ~1.5 min/111k). Second task also landed: scripts/migration-cycle.sh + runbook — the repeatable migrate → validate → rollback cycle (phases: validate/import/rollback/audit/cycle). validate reconciles 9 counts against the 2026-06-11 baseline and exits non-zero on drift (CI/gate-friendly); proven locally (all 9 PASS). Written for stock macOS bash 3.2; DRUSH env override for staging/CI. Remaining 1a.9: execute the full cycle against the prod-DB copy in staging (the acceptance run), plus the non-count criteria (NFC fidelity, KMaps round-trip, IIIF, security) — several already passed in 1a.5/1a.7. That closes Step 1a.
2026-07-07 1a.9 ◐ local rehearsal executed (PR #22, branch feat/1a9-legacy-nid-baseline) — ran the full migrate → validate → rollback cycle end-to-end on DDEV/MySQL 8.4 (ADR 012) against the 2026-07-07 staging D7 source dump. Migration verified 1:1 vs D7 source for every count (KMaps fields exact) → faithful; the validate FAILs vs the old baseline were pure newer-dump drift, incl. field_kmap_terms 61,668 → 55,553 — investigated and confirmed a real source-data change between dumps, not a defect (D7src == D11 migrated). Added field_legacy_nid — records the D7 source nid on shanti_image nodes (indexed; durable old→new identity that survives rollback/reimport; upstream feed for the planned uid_legacy_s Solr field). Populated on all 111,343 nodes, 0 mismatches. Found + fixed: migration-cycle.sh phase_rollback used a double-quoted SQL literal that Drupal's ANSI_QUOTES MySQL connection parsed as an identifier (Unknown column 'shanti_image') — switched to a bound placeholder; only surfaced now because prior runs never exercised rollback/cycle locally. Recalibrated the validate baseline to the July 7 dump + added a baseline subcommand that emits EXPECT_LIST for future dumps (cycle now all-9-PASS). New deferred notes: sync hook fires during migration, load-staging-baseline false-clean.
2026-07-08 1a.9 staging run deferred to end of Sprint 1 (post-1b). The security acceptance criterion is 1b-gated, so a complete run cannot happen until after 1b.3 regardless. The local MySQL 8.4 rehearsal has de-risked migration quality (ADR 012); what staging uniquely adds is infrastructure validation, which is better done once comprehensively after 1b. The DevOps prerequisites (staging-migration-execution-prerequisites.md) are needed for 1b staging work in any case. The combined end-of-Sprint-1 run remains the gate before Images goes to production.
2026-07-09 1b.2 ◐ started (branch feat/1b2-group-collections) — Group module enabled (3.3.5 + gnode); collection and subcollection Group types created as CMI config with 6 roles (anonymous/outsider/member on each), group_node:shanti_image relation plugin on both types. Four fields committed to CMI: field_group_access (int: 0=public/1=private/2=subscribable, on both types — migrated from D7 group_access), field_parent_collection (entity-ref → collection, on subcollection — enforces one-level constraint via type system), field_visibility_overridden (bool, on subcollection), field_legacy_nid (integer, preserves D7 nid on both group types — mirrors the pattern on shanti_image). Also recovered field_legacy_nid node CMI gap — field was created during 1a.9 but never exported to config/sync/; recreated and exported this session. New custom module mandala_group_inheritance implements ADR 011 hooks: hook_entity_access (deny view on private group content for non-members; respects bypass group access), hook_group_presave + hook_group_update (visibility inheritance), hook_group_insert + hook_group_relationship_insert/delete (membership cascade with sub-only retention via _inherited flag). Four migrations added to mandala_migrations: d7_images_collections ✅ 55/55, d7_images_subcollections ✅ 119/119 (parent refs resolved via D7ImageSubcollection source plugin joining og_membership), d7_images_image_collection_membership ✅ 111,307/111,307, d7_images_collection_memberships ⚠️ 38 created / 211 skipped (users not yet migrated; to be re-run post user migration). 4-layer verification passed: counts exact, 16 private collections correct, 1 orphan subcollection (faithfully reflects D7 data), access control correct (anon denied private / allowed public; admin allowed all), 0 failed map rows. D7 data profile confirmed: 55 collections (39 public, 16 private), 119 subcollections (102 public, 16 private, 1 subscribable), 111,307 image→group and 249 user→group OG memberships. Bugs found + fixed this session: (1) Group auto-created uid=0 memberships on group insert (creator uid defaults to 0 before user migration) — fixed by forcing uid: 1 in collection/subcollection migrations and deleting 174 stale uid=0 membership records; (2) hook_entity_access didn't check bypass group access before returning FORBIDDEN — admin was incorrectly denied.
2026-07-10 1b.2 closed out; 1b.1 parts 1–3 of 4 done (branch feat/1b1-hybrid-solr-proxy). Session opened by reconciling a cross-machine field_legacy_nid config conflict (this driver's locally-populated field vs. Than's independently-recreated one — resolved by realigning UUIDs via config:set, never a destructive cim, so no data loss on either side; PR merged as 9863c42). PR #28 landed mid-session from another driver — merged cleanly into this branch — fixing the same field_legacy_nid gap for collections/subcollections plus a uid=0 correction; re-ran both migrations locally to backfill (174/174). 1b.2 fully closed: d7_images_image_collection_membership finished its full 111,307/111,307 run (interrupted once by an errant ddev restart mid-flight; migrate:reset-status + resume completed it cleanly). 1b.1: (1) forked solr-proxy/ from D7, replaced the D7 proxy's circular Solr-membership-query with a Redis read of mandala_solr_fq:{uid} (ADR 014), corrected $OAUTH_ROOT to D11's /oauth/* paths (Spike 10), dropped 5 dead D7 per-site Apache configs; (2) installed simple_oauth, registered a real solrproxy OAuth2 client, verified the full authorization-code flow live (sub:"2" = Drupal uid, reproducing Spike 10's 4 criteria against a real client); (3) new mandala_solr_visibility module writes/deletes the Redis token on login/Group-membership-change/logout, verified end-to-end including cascade interaction with mandala_group_inheritance. Along the way, unstubbed CollectionFieldContributor (kmassets doc-builder) to populate real visibility_i/collection_uid_s on image docs — the field ADR 014's fq depends on — scoped narrowly to what Sprint 1's security criterion needs (collection-doc indexing itself deferred). Bugs found: (1) mine — hook_user_logout() needs AccountInterface, not UserInterface like hook_user_login(); fixed. (2) pre-existing, more serious — mandala_group_inheritance's cascade-remove logic reads a data field that doesn't exist anywhere on group_relationship, so Group::removeMember() throws on any collection with subcollections; this breaks already-merged 1b.2 functionality, root-caused (confirmed by disabling the module) but not fixed here — deferred note. Also planned the D7→D11 user migration: D7's five sites share one user base via a MySQL cross-database table-prefix kludge (mandala_shared, not any per-site dump) — deferred note with the verified config and what's already gated on it (1b.2's 211 skipped memberships, SAML/NetBadge account mapping for 1b.1 part 4). Remaining 1b.1: part 4 (confirm SAML+OAuth2 coexistence outside DDEV — Spike 10 only proved it there); branch not yet pushed/PR'd.
2026-07-13 1b.1 parts 1–3 merged; part 4 (SAML/NetBadge) app-repo half done (PR #32, feat/1b1-part4-netbadge-saml). Session opened by reconciling state with origin/main — parts 1–3 had merged as PR #30 (2026-07-10), Spike 4 split into 4a (Tibetan Unicode round-trip) / 4b (CKEditor footnotes — decisive Fail: footnotes 4.x can't represent D7's cross-page pattern, PR #31), and Spike 3 corrected to ● Proven. Scoped part 4 against the UVA Library NetBadge house pattern (drupal-dsf, drupal-netbadge, terraform-infrastructure) and the D7 legacy SimpleSAMLphp config: the architectural SAML+OAuth2 coexistence question is already proven (Spike 10) — part 4 is environment/real-IDP integration, not re-proving it. App-repo half (merged): enabled simplesamlphp_auth (+externalauth); committed simplesamlphp_auth.settings.yml matching the fleet mapping — auth_source: default-sp, user_name/unique_id = urn:oid:0.9.2342.19200300.100.1.1 (UVA computing id), mail_attr = …100.1.3, register_users: false baseline; /saml_login verified live in DDEV; config round-trips clean. Verified terraform bug (probed live prod mandala.library.virginia.edu): the mandala ALB routes mod_shib paths (/user/netbadge, /Shibboleth.sso/* → separate authproxy) but the SP is SimpleSAMLphp (/saml_login, /simplesaml/*, served by the Drupal container) — /Shibboleth.sso/* 404s yet NetBadge login works, so those rules are already dead. Decision: delete the 5 obsolete public-0-auth-* rules (don't retarget); the authproxy component stays for the Solr proxies; fold cleanup into the later SimpleSAMLphp env-var/Ansible pass — deferred note (with a match-existing-vs-auto-provision test matrix; register_users stays false committed, provisioning proven via dev toggle — mandala will likely lift the restriction, unlike DSF). Env target for outside-DDEV work is the dev instance (a second env inside the staging configs). Remaining part 4: terraform ALB rule deletion + SimpleSAMLphp env-var/Ansible config + outside-DDEV validation.
2026-08-13 D11 kmassets population executed; 1b.3 proven at the data/token level. Not a Sprint-1-task-numbered item — a standalone gap-closing action flagged after 2026-08-12's solr-proxy CI/CD ship (the intervening 2026-07-14→08-12 weeks, incl. the user migration and ADR 015, aren't reflected in this table — see docs/deferred/, docs/session-logs/, and docs/adr/ for that period): the D11 kmassets write path had never actually run against the shared staging Solr index, so VisibilityTokenBuilder's real D11-format tokens matched zero documents (deferred note). Closed the config gap first — solr_master_url/base_url had never actually been in committed config/sync (the export predated those keys); added them (PR #113) and applied on dev-0 via a targeted config:set (the deploy still doesn't run a full cim). Piloted at small scale (0.066s/doc, in-process timing — a naive per-invocation drush kmassets:index loop measures bootstrap overhead, not real throughput), then ran the full kmassets:index-all shanti_image — hit the same 128MB CLI memory_limit OOM the 2026-07-17/18 migration hit (same CacheTagsChecksumTrait crash, ~48,900/111,340 in), same fix (php -d memory_limit=1024M vendor/bin/drush.php), restarted clean (idempotent — same uid overwrites): 111,340 indexed, 0 skipped, 0 errors. kmassets:audit --check-stale: 0 missing/stale/orphaned. Confirmed additive-only against the shared staging index (572,150 → 683,490, exactly +111,340; no D7-era doc touched). Proved ADR 014's authenticated path end-to-end with a real user, not a synthetic token: built the actual token via \Drupal::service('mandala_solr_visibility.token_builder')->build($user) for uid 600 (real migrated member of 4 real private D11 collections) — 25 docs exist across those collections, the anonymous filter matches 0, the real token matches all 25. PR #114 closed out the deferred notes and a stale checklist assumption. What this does NOT cover: only shanti_image/Images is indexed (the only bundle configured); AV/Sources/Texts have no migrations built yet at all (Spikes 5/7 both still fully Pending — see their spike docs); the live browser/NetBadge/OAuth2 session path for 1b.3 is still untested.
2026-08-18 1b.4 ✅ proven — no code change needed. Investigated whether a private image's satellite paragraphs (image_agent/image_descriptions/external_classification) are independently retrievable, bypassing the node-level private-collection check. Two findings: (1) no retrieval surface exists that could bypass it — jsonapi/rest are not enabled, no View exposes the paragraphs_item base table, no Search API index touches paragraphs, and the paragraph entity type declares no canonical route (confirmed by reading the contrib Paragraph entity annotation); the one custom JSON endpoint that embeds paragraph field values (mandala_node_api's NodeJsonController) is already gated by the same _entity_access: node.view requirement its route declares, so it can't leak anything the node access check wouldn't already allow. (2) the mechanism itself already works, and was never custom-built — contrib ParagraphAccessControlHandler::checkAccess() unconditionally ANDs a 'view' check with $paragraph->getParentEntity()->access($operation, $account, TRUE), so mandala_group_inheritance's existing node-access hook applies to paragraphs for free. Verified live in DDEV against real migrated data, not just read from source: a flat private collection (group 5, node 1 "Crab Nebula", agent paragraph 10) and a private subcollection nested under a public parent (group 173 under collection 8, node 111342, agent paragraph 111207) — anonymous and a plain non-member authenticated account get FALSE/FALSE (node/paragraph) in both cases, a real added member gets TRUE/TRUE. Confirms ADR 011's explicit expectation that 1b.4 "composes correctly... under nesting." Test users created and removed within the same script; no residual state. Closes the last open item in Step 1b except the still-separately-tracked 1b.3 live-session gap and the deferred staging acceptance run.
2026-08-18 (cont.) First live 1b.3 NetBadge login attempt — blocked, then unblocked at the mechanism level. Attempting the live-session proof that 1b.3 still needs surfaced a real bug: /saml_login on dev-0 returned a hard HTTP 500 for everyone. Root-caused live via SSH (not guessed): watchdog's MetadataNotFound was a red herring (the IdP metadata file was present, valid, and loaded fine when queried directly) — the actual failure only appeared by simulating simplesamlphp_auth's own bootstrap (new SimpleSAML\Auth\Simple('default-sp')), which threw CriticalConfigurationError: predis/predis is not available before any SAML logic ran. dev-0's SIMPLESAML_STORE_TYPE=redis session store requires the predis/predis Composer package specifically (Predis\Client) — distinct from the native ext-redis extension already present, which SimpleSAMLphp's RedisStore doesn't use. predis/predis was only ever a require-dev/suggest of simplesamlphp/simplesamlphp, never actually installed. Fixed and deployed same session: PR #119 added predis/predis: ^3.3 to drupal/composer.json (ddev composer require, v3.6.0 resolved, no conflicts/advisories); merged, pipeline ran, new image deployed to dev-0. Verified live post-deploy: predis/predis present in the deployed vendor/, Predis\Client resolves, Auth\Simple::getLoginURL() succeeds, and /saml_login now returns a proper 303 redirecting to UVA's real production IdP (shibidp.its.virginia.edu) instead of a 500. Full incident writeup: simplesamlphp-redis-store-missing-predis.md (Resolved). Still blocking the actual 1b.3 proof: completing a real login needs a real UVA NetBadge credential for a specific migrated private-collection user (the one confirmed account, uid 4/dfg9w, maps to a real person) — not practical for routine testing. dev-0 already has SimpleSAMLphp's built-in example-userpass test auth source enabled but not wired as the active one, and closing that gap needs both an attribute-mapping fix and a matching authmap row; there's also an untested cookie-domain mismatch between the SP's configured hostname (mandala-dev) and the one the team actually uses (mandala-images-dev). Full scope: dev-0-needs-test-idp-for-saml-login-testing.md — this is the next-session starting point, to be picked up by Yuji.
2026-08-18 (second session) Test IdP built and proven; 1b.3's live session gap closed, two new OAuth2 defects found right at the finish line. Picked up exactly where the prior session that day left off. Wired dev's default-sp to a self-hosted SimpleSAMLphp test IdP (gated on SIMPLESAML_ENABLE_EXAMPLE_AUTH, so staging/production untouched) — new hosted-IdP + SP-remote + IdP-remote metadata in terraform-infrastructure, a core:AttributeMap/name2oid authproc filter fixing the attribute mismatch generally (not per-identity), default-sp's idp switching targets on the flag. Two more real bugs found by actually running the deploy: a bogus MYSQL_* requirement in deploy_netbadge.yml (copy-pasted from the dsf reference, unused by SimpleSAMLphp) and a missing enable.saml20-idp config key (no default in SimpleSAMLphp core, silently 500s the whole saml module IdP frontend without it). Linked the staff test identity to a real non-admin migrated private-collection user via ExternalAuth::linkExistingAccount() (identity kept in mandala-navina-docs, private, not here). Proven live, with hard evidence: SAML login through the test IdP → real Drupal session (confirmed via the sessions table) → full OAuth2 authorization-code exchange against solrproxy, including state CSRF round-trip and automatic_authorization. That walkthrough surfaced two further, independent, previously-uncaught OAuth2 defects right at the last step — neither is a SAML/test-IdP problem: (1) simple_oauth's signing keypair lives outside any persistent bind mount in deploy_backend.yml, so it gets silently wiped on every normal deploy (regenerated by hand tonight; not fixed at the infra level — deferred note); (2) the solr-proxy's OAuth2 client (league/oauth2-client's GenericProvider) never overrides getAuthorizationHeaders(), so it sends no Authorization: Bearer token on the /oauth/userinfo call — confirmed empirically (a garbage Bearer token gets a clean 401 from the same route, proving the route itself is fine) — deferred note, open, different repo (uvalib/mandala-solr-proxy). Full writeup: session log. Next-session starting point: fix the UserInfo Bearer-header bug in mandala-solr-proxy, then the last unproven link is the proxy's own Redis visibility-token read for a real OAuth2-authenticated session (previously only proven with a hand-written Redis key). Also flagged, unrelated to 1b.3: contact Bob about updating the real IdP-side config for mandala-dev/mandala-staging — see saml-sp-cert-expired-renewal.md.
2026-08-19 Both 2026-08-18 OAuth2 defects fixed, deployed, and verified live — then re-verification surfaced a third, independent defect. Correction first: last night's notes wrongly said the Bearer-header bug lived in a separate uvalib/mandala-solr-proxy repo — that repo doesn't exist (confused with the ECR image name); ADR 014 forked the D11 proxy into this monorepo's solr-proxy/ back in 1b.1. Fixed both defects where they actually live: solr-proxy/proxy/src/BearerGenericProvider.php (new, PR #121) overrides getAuthorizationHeaders() so auth.php sends a real Authorization: Bearer header; terraform-infrastructure's deploy_backend.yml (29bdb6cc9) now bind-mounts drupal/keys/ and idempotently generates the keypair only if missing (won't invalidate tokens on every deploy). Both merged/committed, then deployed live via a manually-triggered uva-mandala-drupal-codepipeline run (source→build→deploy all green, ~8 min). Re-ran the full SAML→OAuth2→UserInfo chain live (SSH+curl reconstruction of the SAML POST bindings, mirroring a real browser since the Chrome extension wasn't connected this session) with the same staff/uid 600 test identity: SAML login → real Drupal session → /oauth/authorize→/oauth/token exchange all still work. Both fixes confirmed working — added temporary debug logging (removed after) proving auth.php now sends a correct, well-formed Bearer JWT, and Drupal's response carries X-Consumer-ID: solrproxy, proving the resource server authenticates it. But /oauth/userinfo still doesn't return JSON — a third, previously-hidden gap, precisely root-caused by reading simple_oauth 6.1.1's source: Oauth2AccessPolicy::alterPermissions() (added by security advisory SA-CONTRIB-2025-114) intersects the user's real permissions with only what the token's granted OAuth2 scopes explicitly confer via a ScopeGranularity plugin; the openid scope is umbrella: true with none configured, so it grants zero permissions by design, regardless of the user's actual roles (TokenAuthUser::getRoles() correctly resolves ["authenticated"] — confirmed by direct drush php:eval comparison against a real User entity, which correctly gets access content). Not a Drupal/simple_oauth bug — a scope-configuration gap left over from Spike 10, never exercised against a permission-gated route until now. See the corrected deferred note. Yuji decided: reuse the openid scope (the established pattern), not a second scope. First attempt (PR #124: Permission granularity granting access content, umbrella left true) tested correct in DDEV but failed identically live — root cause: Oauth2AccessPolicy calls Oauth2ScopeProvider::getPermissions(), which checks isUmbrella() first and ignores the scope's own granularity when true (unions child-scope permissions instead, of which openid has none) — the module's own admin form enforces this by force-nulling granularity fields whenever umbrella is checked, so the hand-edited YAML had bypassed a real functional constraint, not just a UI one. Tested the correction directly on dev-0 first (drush config:set --input-format=yaml ... umbrella false, per Yuji's request, to avoid another ~6-8 min pipeline cycle per attempt — the plain string form silently no-ops on booleans, cost one extra attempt) before committing PR #125. Confirmed correct against a real token from a live walkthrough: TokenAuthUser->hasPermission('access content') now YES (was NO), access_manager->checkNamedRoute('simple_oauth.userinfo', ...) now isAllowed: YES. The scope-permission fix is genuinely done. But a fourth, distinct, not-yet-root-caused issue surfaced right behind it: with access now correctly allowed, /oauth/userinfo still doesn't return JSON — watchdog shows repeated "Session closed for [uid 600]" instead of "access denied", and the response is a redirect loop between /oauth/userinfo and /, consistent with Drupal's session-handling layer reacting badly to a stateless Bearer-authenticated request resolving to a real identity with no session cookie. Deliberately left for a fresh session rather than continuing to dig at the end of a long one. Next-session starting point: root-cause the session-handling redirect loop in Drupal core's session subsystem; the Redis visibility-token read remains the step after that.
2026-08-20 1b.3 ✅ closed — the OAuth2 authenticated path works end-to-end and ADR 014's visibility filtering is proven, not merely functional. PRs #128–#132 (session log). The fourth defect was not in Drupal core's session subsystem as the previous session predicted: simplesamlphp_auth's checkAuthStatus() force-logs-out any request lacking a live SAML session, which a stateless Bearer request by definition lacks. New mandala_saml_oauth module (PR #128) swaps simplesamlphp_auth_event_subscriber for a subclass exempting OAuth2 requests via SimpleOauthRequestPolicyInterface::isOauth2Request(); verified by replaying the identical request against both images — 302 → / before, HTTP 200 application/json after (PR #129). Visibility verified with a negative discriminator, which is what makes it a proof rather than a smoke test: uid 600 sees images-11-95599 (private, their own collection) and does not see images-11-3 (private, someone else's); anonymous sees neither; *:* differs 751,032 vs 751,566. Client fq injection is stripped; a forged sid degrades to anonymous. reverse_proxy + trusted_host_patterns configured (PR #131) — never set before, so Request::getUri() returned http:// and the SAML RelayState carried the wrong scheme. Two findings there: each VPC has two CIDR blocks (staging 10.130.109.0/24 + 10.130.112.0/24, production .110 + .113) and health checks arrive from both — listing only the primaries half-works, because Drupal falls back silently rather than erroring; and trusted_host_patterns fails closed, so a wrong pattern set takes ALB targets unhealthy (patterns tested against 27 real and 5 hostile hostnames before deploying). Two regression tests + two how-tos landed: scripts/verify-oauth2-userinfo.sh (PR #130) and scripts/verify-solr-proxy-visibility.py (PR #132), documented in docs/dev-notes/. Three false alarms, all self-inflicted, all worth carrying forward: (1) the previously-designed fix tested $this->account instanceof TokenAuthUserInterface, but @current_user is an AccountProxy that wraps the real account, so it could never match — a no-op caught by reading core before shipping; (2) a post-deploy test gated on the module directory appearing raced config:import by 12 seconds and reported a confident FAIL on a working fix; (3) id is not unique in kmassets — four docs share id 1821 (the ADR 006 taxonomy shadows plus an audio-video asset), so using it as a fixture produced got 4, expected 0, which reads exactly like a visibility breach. A fourth correction from earlier the same day: the watchdog evidence blamed on checkAuthStatus() for the reported logout problem was our own test traffic. Still open, none blocking 1b.3: the NetBadge IdP advertises no SingleLogoutService and logout_goto_url is unset, so a Drupal logout never ends the NetBadge session (needs the canonical URL from UVA ITS and a trusted.url.domains entry); SimpleSAMLphp sessions expire at 8h against Drupal's 23d (upstream MR !48 is RTBC on our 4.x, but whether 8h is right at all is our call and unmade); and a dev-0 cron re-run loop, 26 entries/48h, uninvestigated.

Next-session blocker: ~~Group collections inheritance decision~~ — resolved 2026-06-18 by ADR 011 (Option D: entity-reference + custom hooks). Inheritance hooks fold into task 1b.2 below; no collections work lands in Step 1a.

Step 1a — public plumbing ✅ complete (2026-07-08)

# Task Depends on Status
1a.1 Build shanti_image content type: 50-field inventory, types, cardinalities, required flags (per audit) audit ✅
1a.2 Build agents / descriptions / classifications as Paragraph types embedded on shanti_image (owned, per-image, cascade-deleted) 1a.1 ✅
1a.3 Build external_classification_scheme as a taxonomy vocabulary referenced from the classification paragraph 1a.2 ✅
1a.4 Productionize the KMaps field (Spike 1 tail): write path, ancestor-path resolution, migration mapping; wire field_subjects / field_places / field_kmap_terms / field_kmap_collections to kmassets Spike 1 ✅
1a.5 Wire image display to the existing IIIF server: confirm endpoints/credentials reachable from D11; port the upload/display path (shanti_images_*_url, shanti_image_formatter); preserve i3fid / mmsid / field_other_ids linkage 1a.1 ✅
1a.6 Migrate scheme nodes → taxonomy terms (no deps, migrate first) 1a.3 ✅
1a.7 Migrate shanti_image with node→paragraph transform for satellites; source satellites via the image reference field, not the raw node table (skips ~12k orphan agents, ~17k orphan descriptions); expect shared-agent fan-out (~111k agent paragraphs) 1a.6 ✅
1a.8 Solr write/sync via reindeer_x (Spike 8 parts A/B), informed by the Phase 0 cost/architecture conversation Spike 8 ✅
1a.9 Rollback story: repeatable test-run → validate → rollback cycle; local rehearsal ✅ passed (111,343/111,343; 0 failures); staging acceptance run deferred to end of Sprint 1 (post-1b) — see staging prerequisites 1a.7 ✅

All 1a code merged to main; migration pattern (Migrate API → KMaps → Solr sync → rollback cycle) established for replication in per-site tracks.

Step 1b — auth increment ✅ complete (2026-08-20)

# Task Depends on Owner
1b.1 Hybrid Solr proxy for D11: (1) ✅ fork proxy into solr-proxy/; wire $OAUTH_ROOT to D11 simple_oauth; replace setCollections() with Redis read of mandala_solr_fq:{uid}; (2) ✅ install + configure simple_oauth in D11; register proxy OAuth2 client; (3) ✅ Drupal event hooks — write/invalidate Redis token on login, Group membership change, logout; (4) ✅ SAML+OAuth2 coexistence outside DDEV — app-repo half merged (PR #32); terraform CI/CD half done 2026-08-12 (ECR repo + build-only pipeline + deploy_solrproxy.yml, session log) — deployed on dev-0 as mandala-solr-proxy-0:8765, ALB idx target healthy, anonymous search serving 562,952 docs with the correct filter. Design: ADR 013, ADR 014 Spike 10 — unblocked 2026-07-09 ✅ all 4 parts done (PR #30 + PR #32 + solr-proxy CI/CD 2026-08-12)
1b.2 Build Group collection/subcollection bundles + group_node:shanti_image relation as CMI config; implement custom inheritance module (ADR 011 Option D — visibility + membership hooks, visibility_overridden flag, sub-only retention); migrate OG → Group: group_content_access (Visibility) + field_og_collection_ref → Group membership Spike 3, ADR 011 ✅ PR #27 + #28 merged; full data migration 111,307/111,307 (2026-07-10)
1b.3 Solr-proxy visibility filtering; prove access-control coherence (search results agree with node/Group access) 1b.1, 1b.2 ✅ proven at the data/token level 2026-08-13: real user (uid 600, real migrated member of 4 private D11 collections), real VisibilityTokenBuilder token, real D11 kmassets docs — anonymous matches 0/25 private docs, the real token matches all 25. Enforcement mechanism itself (proxy reads Redis token, injects it, Solr honours it) separately proven 2026-08-11/12 with a synthetic token (3,112 private docs vs 0 anon). ✅ Live SAML+OAuth2 session proven 2026-08-18 (second session that day, resolving dev-0-needs-test-idp-for-saml-login-testing.md): a new dev test IdP + real non-admin migrated user → real Drupal session → full OAuth2 authorization-code exchange against solrproxy, all with hard evidence not assumption. ✅ Both follow-on defects fixed and verified live 2026-08-19: oauth2-signing-keys-not-persisted-across-deploy.md and solr-proxy-genericprovider-no-bearer-header-on-userinfo.md — deployed via a manually-triggered pipeline run, then confirmed live: auth.php sends a correct Bearer header, and Drupal authenticates it (X-Consumer-ID: solrproxy). ✅ Third gap (scope-permission) fixed and confirmed correct 2026-08-19 (PRs #124, #125): openid now grants access content via Permission ScopeGranularity with umbrella: false (the first attempt left umbrella: true, which silently made the granularity config dead code on the path Oauth2AccessPolicy actually uses — Oauth2ScopeProvider::getPermissions(), not the entity's own method). Tested directly on dev-0 via drush config:set before committing; confirmed against a real token: hasPermission('access content') YES, access_manager allows the route. ✅ Fourth and last defect fixed and verified live 2026-08-20 (PR #128, verification #129): the session-handling redirect loop was simplesamlphp_auth's checkAuthStatus() force-logging-out any request without a live SAML session — including a stateless Bearer one. The new mandala_saml_oauth module's ServiceProvider swaps simplesamlphp_auth_event_subscriber for a subclass that exempts OAuth2 requests via SimpleOauthRequestPolicyInterface::isOauth2Request(). Same replay: 302 → / on the pre-fix image, HTTP 200 application/json on the post-fix one. See the deferred note. ✅ The full chain is now proven with a negative discriminator, not just a positive one: uid 600 sees images-11-95599 (private, their collection) but not images-11-3 (private, someone else's); anonymous sees neither; both see the public doc. Client-supplied fq injection is stripped and a forged sid falls back to anonymous. Two regression tests landed — scripts/verify-oauth2-userinfo.sh (PR #130) and scripts/verify-solr-proxy-visibility.py (PR #132) — each documented in docs/dev-notes/. 1b.3 is complete. Three residual SAML items are tracked separately and do not block it: NetBadge logout, 8h vs 23d session expiry, and a dev-0 cron re-run loop
1b.4 Confirm paragraph access inheritance: a private image's satellite paragraphs are not independently retrievable 1b.3 ✅ proven 2026-08-18 — see progress log

Acceptance criteria

Sprint 1 closes when, against a copy of the production Images DB on dev-0:

Scope decision (2026-08-25, Yuji): the acceptance run targets dev-0. These criteria were written saying "in staging", when a D11 staging environment was assumed to exist. It does not. The terraform-infrastructure workspace named staging holds two hosts, and only one runs D11: uva-mandala-drupal-staging-0 → mandala-drupal-dev-0. The other, …-staging-1 → mandala-drupal-dev-1, still runs legacy D7 and is the migration source (decided 2026-07-13: "replace dev-0 in place; dev-1 stays as D7 staging source"). For now we are working only on dev-0.

This is a reasonable substitution, not a weakening: dev-0 carries the full production Images dump on real RDS and real containers, which is what the criteria were reaching for. What it does not prove, and should not be claimed: a promotion rehearsal between two D11 environments, and the per-environment config-override mechanism — which stays untested while only one D11 environment exists (the residual open in spike-solr-demo-enabled-with-anonymous-route.md). Revisit before production rollout, not before Sprint 1 closes.

  • [x] shanti_image + the three paragraph types + the scheme vocabulary install via CMI config. Verified 2026-08-27 on the from-scratch dev-0 rebuild: drush config:status reports "No differences between DB and sync directory"; node.type.shanti_image, taxonomy.vocabulary.external_classification_scheme, and all 3 paragraph types (image_agent, image_descriptions, external_classification) confirmed present. See 2026-08-27 session log.
  • [x] A full migration run completes; per-type counts reconcile against the data profile (111,340 images; orphan satellites excluded; shared-agent fan-out as expected). Verified 2026-08-27 — EXPECT_FILE=scripts/baselines/dev-0.txt ./scripts/migration-cycle.sh validate run as a real gate against dev-0's from-scratch rebuild: all 12 counts PASS, integrity:legacy_nid_dupes = 0. See 2026-08-27 session log.
  • [x] Transliteration diacritic normalization is preserved (NFC/NFD fidelity) through Migrate API → MySQL collation → Solr — verified, not assumed. Fully verified 2026-08-28. DB leg (2026-08-27): 100/100 randomly-sampled diacritic-bearing shanti_image titles byte-exact (===) vs D7 source via the migrate DB connection. Solr leg (2026-08-28, once the reindex below actually completed): of 2,044 diacritic/Tibetan-script-bearing published titles, 2,040 matched byte-exact against the Solr search reader (mandala-index-dev); the remaining 4 (all pure Tibetan-script titles, nids 95394–95397) simply don't exist in that index yet — see the retrievability gap below, same root cause, not a separate defect.
  • [x] The 4 KMaps fields round-trip (save → reload → correct display) and term IDs match the live KMaps API. Verified 2026-08-27 against the kmterms Solr shadow index (ADR 006) rather than the raw external KMaps API directly (which rate-limited after a few probe requests — treated as a live production service not to be hammered). 6/6 sampled field_kmap_terms term IDs resolved to real, current kmterms docs; 5/6 headers matched the shadow index's header field exactly (Wylie), and the 1 stored as Tibetan Uchen script matched the shadow index's name_tibt field exactly — same term, different field, not a mismatch.
  • [x] Content indexes and is retrievable via existing query patterns (not search quality). Reindex completed 2026-08-27 (kmassets:index-all shanti_image: 111,339 indexed, 0 errors) and kmassets:audit --check-stale reports clean (0 missing/0 stale/0 orphaned) 2026-08-28. Team decision 2026-08-28 (meeting: Than, Yuji, Xiaoming): a separate 70-document master/reader sync gap was found the same day — that audit only validates the write master, not the search reader real queries use, and checking the reader directly found docs present on the master missing from it. Ruled a distinct infrastructure-replication issue, not a migration/content defect, and explicitly does not block this criterion or Sprint 1 — tracked separately for Yuji in kmassets-audit-checks-master-not-search-reader.md.
  • [x] Images render through the existing IIIF server with i3fid linkage intact. Verified 2026-08-27 — 3 randomly-sampled shanti_image nodes' field_iiif_id values all returned valid IIIF Image API 2.0 info.json (level2 profile, tiles present) from https://iiif.lib.virginia.edu/mandala/{id}/info.json. Side note, not blocking: stored field_iiif_width/field_iiif_height don't always match the server's reported dimensions (one sample had them swapped) — not chased further, flag for whoever next touches IIIF display sizing.
  • [x] Security: a restricted Images item is non-retrievable by an unauthorized user via the D11 search path, and retrievable by an authorized one. Proven twice, independently. (1) Via the actual D11 search path: 1b.3 (2026-08-13/18/20) proved this at the Solr-proxy/token level with a positive and negative discriminator (uid 600 sees their own private doc, not someone else's; anonymous sees neither). (2) Via direct node/collection access, 2026-08-27: full URL smoke-test matrix (public/private/bogus × anonymous/authenticated) on the from-scratch dev-0 rebuild — see 2026-08-27 session log and the local-login fix, PR #165 (the authenticated leg initially failed for an unrelated reason — a checkAuthStatus() session bug affecting local logins for every migrated user, now fixed).
  • [x] The test-run → validate → rollback cycle is documented and repeatable. Documented in scripts/migration-cycle.sh (rollback/import/validate/cycle phases). Repeatability proven in the local DDEV rehearsal (1a.9: 111,343/111,343, 0 failures). dev-0 deliberately substitutes a from-scratch rebuild for rollback→reimport — not an unaddressed gap, a considered exception: migrate:rollback doesn't reset AUTO_INCREMENT, so a real rollback→reimport on dev-0 would silently re-key every nid, breaking the ADR 016/017 nid-determinism the from-scratch approach exists to guarantee (see the rebuild runbook). The validate phase of the cycle was re-proven directly against dev-0 today (see above), so this criterion rests on the DDEV proof for rollback specifically plus a live dev-0 proof for validate — not a full untouched rollback rehearsal on dev-0's real data.

On close: kick off the Jira issue-tracking integration (sequenced to start after this sprint; backfill open deferred notes as tickets).

References