Session Log: Kmasset Solr Doc Contract

Date: 2026-06-24
Participants: Yuji Shinozaki, Claude (Sonnet 4.6 → Opus 4.8)
Outcome: Re-oriented onto current main (Step 1a 1a.4–1a.7 landed; ADR 011 accepted). Opened up Sprint 1 task 1a.8 and traced the full kmasset Solr document contract end-to-end — deployed topology (Solr 7.7.3 master/replica + read-only proxy), the generative doc grammar (kmaps_engine), the type layer (kmassets schema.xml), and both consumer generations (D7 AjaxSolr clients + mandala-om React). Captured as the foundation doc KMasset Solr Doc Contract. Write transport (A: S3→ECS / B: reindeer_x / C: direct Drupal→master) left as an open decision gated on the Dave Goldstein cost conversation.

⚠ Correction (2026-06-25): Two findings recorded below were later corrected. (1) This session concluded "kmasset docs are nested block-join documents" — that is wrong. The block-join structure belongs to the kmterms index; kmassets docs are FLAT (verified in Spike 2 against live Solr). (2) mediabase was listed as an Images producer — it is A/V only; shanti_images is the Images producer. Both are fixed in the corrected contract doc, §5 + §4 (commit 75f0a33). The erroneous reasoning is left intact below as the historical record of this session; see the 2026-06-25 session log for the correction discussion.

---

This is the raw conversation transcript. Tool calls and code output are omitted; only the text exchanges are recorded.

---

User — 2026-06-24 14:15

So where are we now?

---

Claude — 2026-06-24 14:15

Let me read the orientation docs as the project instructs.

---

Claude — 2026-06-24 14:16

Let me also check the latest session log and the sprint 1 branch status.

---

Claude — 2026-06-24 14:16

Let me check the sprint doc and PR status quickly.

---

User — 2026-06-24 14:16

[Request interrupted by user for tool use]

---

User — 2026-06-24 14:16

Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.

---

User — 2026-06-24 14:16

/model model

---

User — 2026-06-24 14:16

Set model to Opus 4.8 and saved as your default for new sessions

---

User — 2026-06-24 14:16

So where are we now?

---

Claude — 2026-06-24 14:17

Here's where things stand, based on the orientation docs and the last session log (2026-06-17).

Current branch: sprint-01/images-content-model-d11

You're mid-flight on Sprint 1: Mandala Images, with three commits sitting on this branch (not yet merged to main):

• 0f5d8d8 — session wrap-up + session log
• a158625 — Collections inheritance decision brief (teed up as next session's agenda)
• ae92c46 — Sprint 1 tasks 1a.1–1a.3: Images content model on D11

What got done last session (1a.1–1a.3 ✅)

• D10.6 → D11.3.11 bump (composer, DDEV PHP 8.1→8.3, Drush 12→13) — reconciled the scaffold with ADR 002. Paragraphs + entity_reference_revisions added.
• Images content model built as a 382-file CMI baseline in drupal/config/sync: shanti_image (48 fields), 3 paragraph satellites, external_classification_scheme vocab. Round-trips with zero diff.
• DSF settings convention adopted (monorepo-adapted): committed env-driven settings.php, config_sync_directory = '../config/sync'.
• Dockerfile → D11 with monorepo paths fixed (but not built — deferred to deploy track).
• Two deferred notes written: prod-packaging pass (High), field_agent_name migration mapping (Medium).

What's open in Step 1a

• 1a.4 — KMaps search-root productionization (currently placeholders for field_kmap_collections root 2823, field_language root 301)
• 1a.5 — IIIF wiring (verification task)
• 1a.6/1a.7 — migrations (D7 dump at mandala-legacy/mandala-prod-images-db_2026-06-11.sql.gz is the source)
• 1a.8 — Solr sync (Spike 8 partial)
• 1a.9 — rollback

The next thing on deck

The session ended teeing up the Group collections inheritance decision as the first agenda item for the next meeting — David... rather, Than was reading along and the brief at docs/planning/collections-inheritance-decision-brief.md frames it as ratify/amend Option D (entity-ref + custom hooks). That's the gate before site/Group collections can be modeled as config.

---

One thing I noticed: there's an uncommitted drupal/config/sync/.htaccess in your working tree (untracked, from the git status snapshot).

What would you like to pick up — the collections inheritance decision, or push forward on the open 1a tasks? (Also worth knowing: has a PR been opened for this branch yet, or is it still local only?)

---

User — 2026-06-24 14:17

please pull the latest for git

---

Claude — 2026-06-24 14:17

Fetched. A few things came down from the remote. Let me show you what's new.

---

Claude — 2026-06-24 14:17

The fetch is done, but here's the important picture: a lot has landed on main in parallel, and your current branch is stale.

Your local sprint-01/images-content-model-d11 is up to date with its own remote — but that branch was already merged to main (PR #6, commit 7a6c0ca). Since then, the team has merged a substantial amount of work that you don't have locally. The --left-right shows zero commits unique to your branch — everything here is already on main.

What's new on main since you branched

The rest of Step 1a got built out by the team:

• 1a.4 — KMaps search-root productionization (KmapsPathResolver service, D7→D11 mapping doc, domain fix on field_kmap_collections/field_language)
• 1a.5 — IIIF wiring (shanti_iiif module, i3fid storage + formatter)
• 1a.6 — first migration: scheme nodes → taxonomy terms (mandala_migrations module)
• 1a.7 — pattern-setting image migration (shanti_image + paragraphs) — just merged via PR #11
• ADR 011 — Group collections inheritance: the decision got made → Option D (entity-ref + custom hooks). That's the question that was teed up for "tomorrow's meeting" in your last session — it's resolved now.
• Plus D11 compatibility fixes, Solr field-type updates for the API, and a Developer Notes docs section.

Recommendation

You're sitting on a fully-merged, stale branch. To get current, switch to main and pull:

git checkout main && git pull

There's also a new remote branch sprint-01/image-migration (already merged via PR #11, so likely safe to ignore).

One snag: you have an untracked drupal/config/sync/.htaccess in the working tree — it won't block a checkout, but want me to look at where it came from first? And shall I go ahead and switch you to main and pull?

---

User — 2026-06-24 14:17

yes let's switch to main

---

Claude — 2026-06-24 14:18

The pull is blocked: that untracked .htaccess is now a tracked file on main (Drupal auto-generates it in config/sync as a deny-all security guard). Let me confirm your local copy matches the incoming one before removing it.

---

User — 2026-06-24 14:18

and ignore the .htaccess file

---

Claude — 2026-06-24 14:18

Identical content — safe to remove. Since it's now a tracked file on main, I'll just delete the local untracked copy and let the pull restore it.

---

Claude — 2026-06-24 14:19

Done. You're on main, fast-forwarded 22 commits to 2194cec, working tree clean.

---

Claude — 2026-06-24 14:19

You're now on main at 2194cec, fully synced with origin, clean tree. The duplicate .htaccess is resolved — it's tracked in the repo now (as the standard Drupal deny-all guard), so nothing to gitignore.

What you just pulled in

The team's parallel work on the rest of Step 1a, plus the docs that came with it:

• ADR 011 — Group collections inheritance, Option D (the decision that was teed up for your meeting is now accepted)
• mandala_migrations module — scheme migration (1a.6) and the pattern-setting image migration (1a.7)
• shanti_iiif module — IIIF wiring (1a.5)
• KMaps search-root productionization (1a.4): KmapsPathResolver, mapping doc
• A new docs/dev-notes/ section (local-dev howtos, session-workflow rituals)
• Several new deferred notes (IIIF Cantaloupe 404 disclosure, IIIF prefix alignment, image binary migration, kmaps raw-format rebuild, migrate-drupal noise, description summary length)
• Session logs for the 1a.4/1a.5/1a.6 sessions

Per the CLAUDE.md startup ritual, the deferred/spike/ADR indexes have all moved since I last read them. Want me to re-orient on the current state (re-read the three indexes + check which Step 1a tasks remain), or are you heading toward something specific — like Step 1b and the now-decided Group collections model?

---

User — 2026-06-24 14:20

yes please reorient

---

Claude — 2026-06-24 14:20

I'll re-read the three index docs plus check what's changed in the deferred/ADR/spike landscape and the sprint status.

---

Claude — 2026-06-24 14:20

Now the sprint doc and ADR 011 (the new decision), plus the most recent session log to see where 1a.7 left off.

---

Claude — 2026-06-24 14:20

Re-oriented. Here's the current state of play on main.

Sprint 1 (Images pilot) — Step 1a is essentially complete

Everything in the public-plumbing half has landed since you were last here:

Task	Status	What shipped
1a.1–1a.3	✅	shanti_image + 3 paragraphs + scheme vocab (CMI baseline)
1a.4	✅	KMaps field productionized — search_root_kmapid setting, KmapsPathResolver service, collections→root 2823, language→root 301; live-Solr fix: both roots are subjects domain, not terms
1a.5	✅	shanti_iiif module — URL builder + formatter, byte-identical to D7, reachability gate passed
1a.6	✅	First migration: scheme nodes → taxonomy terms (mandala_migrations module). Surprise: prod has only 2 schemes (LCSH + TGN), not 5–10
1a.7	✅	The pattern-setting migration — full prod run, 0 failures: 111,340/111,340 images, 111,194 agent paragraphs, 55,038 descriptions; KMaps counts exact to baseline; Tibetan NFC preserved

Still open in Step 1a: - 1a.8 — Solr write/sync via reindeer_x (gated on Spike 8, still ◐ Partial) - 1a.9 — rollback story (repeatable test-run → validate → rollback against staging prod-DB copy)

Step 1b (auth increment) — now unblocked

ADR 011 resolved the collections inheritance question — Option D: entity-reference nesting + a custom inheritance module (working name mandala_group_inheritance). The two High-priority deferred notes are marked resolved by it. Step 1b tasks: - 1b.1 — wire into existing proxy-auth contract (gated on Spike 2 deferred item, blocking — still not done) - 1b.2 — Group collection/subcollection config + the inheritance module (5 hooks, visibility_overridden flag, ~2–3 days) + OG→Group migration - 1b.3 / 1b.4 — Solr-proxy visibility coherence + paragraph access inheritance

Things worth your attention

1. New High-priority deferred note: kmaps-raw-format-rebuild-on-migration.md — the D7 KMaps raw column is angle-bracket ancestors, not the pipe composite D11 needs; silent KMaps drop, now fixed in the migration but flagged as a pattern others must follow.

2. Doc inconsistency: ADR 011 declares Spike 3 → ● Proven for MVP, but the spike README table still shows ◐ Partial (line 7). A commit claimed to fix the nav glyph, but the table itself wasn't updated. Quick fix if you want it consistent.

3. The acceptance criteria are all about a staging prod-DB run — 1a.7's full run was done locally against the DDEV d7_images DB. The formal staging validation (and rollback, 1a.9) is still pending.

Where do you want to point next — close out 1a.8/1a.9, start 1b, or something else?

---

User — 2026-06-24 14:28

Let's talk about 1a.,8

---

Claude — 2026-06-24 14:28

Let me pull the relevant context for 1a.8 — Spike 8, the Solr-sync architecture note, and the two ADRs that govern reindeer_x.

---

Claude — 2026-06-24 14:29

Let me ground two more things — the cost/architecture note that 1a.8 depends on, and whether the current D11 shanti_kmaps_fields module even has the Solr-doc write path.

---

Claude — 2026-06-24 14:29

Good one to dig into — 1a.8 is the least-settled task in Step 1a, and looking at the code grounds why. Here's the honest picture.

What 1a.8 has to deliver for Sprint 1

Just one acceptance criterion: "Content indexes and is retrievable via existing query patterns." For Images, that means image nodes have to land in the kmassets Solr index and be findable. Note the scope discipline — it's retrievable, not search quality (Tibetan quality is explicitly deferred).

Critically, this is Population 2 sync (Drupal content → kmassets), which is a different pipeline from reindeer_x's original job (Population 1: kmterms → kmassets). The task name "via reindeer_x" somewhat conflates them.

The finding that reframes the task

The D11 write path doesn't exist yet. I checked the current shanti_kmaps_fields module — it's entirely read-side: widget, autocomplete controller, formatter, KmapsPathResolver. There's no on-save JSON-doc generation, no solrdocs write, no /post, no S3 upload. In D7 that write path was:

node save → shanti_kmaps_fields writes JSON to private://solrdocs/
          → clsync → synchandler(Perl) → S3 → SQS → ECS transform → kmassets Solr

In D11, Aegir is gone and there's no shared filesystem between Drupal and reindeer_x. So 1a.8 isn't "wire up reindeer_x" — it's build the D11 content-asset write path from scratch, then decide what it feeds into.

Two problems with leaning on Spike 8 here

1. Spike 8 Part A proved the wrong half for D11. Part A folded clsync/synchandler into reindeer_x as a chokidar file watcher — but that's the shared-filesystem model. With no shared FS in D11, the relevant path is Drupal → HTTP POST → reindeer_x, which Spike 8 explicitly lists under "does NOT establish."
2. Parts B/C (SQS/SNS) are orthogonal to Images. They fix the kmterms trigger race condition — Population 1. They don't touch content indexing and can stay deferred for MVP.

The unresolved fork (the real blocker)

The cost-discussion note frames an architecture decision that's still pending a Dave Goldstein conversation:

Option	Path	Trade-off
A — keep ECS	Drupal queue worker → HTTP POST → reindeer_x → S3 → ECS transform → Solr	Reuses existing pipeline; 4+ async hops, fire-and-forget, the "what happened to my node?" visibility problem
B — direct write	Drupal queue worker → direct Solr write (bypass ECS)	Simpler, cheaper for low-frequency content saves, debuggable; diverges from the UVA ECS pattern

Dave's team sized the ECS pipeline for high-throughput indexes; Drupal content saves are low-frequency, so B may be both cheaper and more debuggable. But that's his call to weigh in on.

My read on scope

There's also a bigger productionization tail — reindeer_x still runs on the legacy dev server, not Terraform/ECS, with the AWS credential strategy unresolved (ADR 007 + the credential note). That's arguably beyond Sprint 1; the Images acceptance criterion only needs image docs to reach kmassets and be retrievable, which Option B could satisfy with much less infra.

---

So — what angle do you want to take this? A few directions: 1. Decide the A-vs-B fork (needs the Dave conversation — I can draft the specific question set / a decision brief). 2. Spec the D11 write path in shanti_kmaps_fields (queue worker + doc builder) independent of which sink it targets. 3. Re-scope 1a.8 for the sprint — what's the minimum to pass the retrieval criterion vs. what's reindeer_x-productionization that belongs later.

Where's your head at?

---

User — 2026-06-24 14:29

Also look at mandala-reindeer_x in a parallel directory.

---

Claude — 2026-06-24 14:30

Local checkout is on the spike/08-reindeer-x-consolidation branch. Let me read the actual source — the server endpoints, the sync logic, and the package manifest.

---

Claude — 2026-06-24 14:31

Let me confirm one architecturally decisive thing — whether reindeer_x writes to kmassets Solr directly (via solr-client) for the kmterms path, and check what the /post endpoint actually triggers.

---

Claude — 2026-06-24 14:32

Looked through it. The code changes the picture in three concrete ways — the existing primitives don't close the D11 content-sync gap the way the architecture note assumed.

What's actually in reindeer_x (on the spike branch)

1. reindeer_x already writes directly to kmassets Solr. queueConfigs.js:61 builds a kmassets_write_client = solr.createClient(KMASSETS_AUTH), and kmassetSync.js:825 does target_write_client.add(doc, cb). So "direct-to-Solr write" is not a new capability to invent — it's the proven path reindeer_x already uses for the kmterms→kmassets sync (Population 1). That materially de-risks Option B from our earlier table.

2. The /post endpoint is the wrong shape for content. It looks like the answer to "Drupal POSTs a doc," but it isn't — it accepts {query, force, limit, source} and calls processRequest, which fires a kmterms query-and-sync job (Population 1). It does not accept a content document to index. The architecture note's line — "the file-watch path becomes an HTTP POST to reindeer_x's existing /post endpoint" — doesn't hold against the actual code. That endpoint would need a new handler.

3. The Part-A file watcher is hardwired to the D7/Aegir model. fileWatcher.js:24:

const APP_PATH_RE = /(mandala-[\w-]+\.lib\.virginia\.edu)\/private\/files\/solrdocs/;

It extracts a per-site host from an Aegir multisite path and assumes a shared filesystem it can watch. Under D11 single-site (ADR 005) there's no multisite path (it falls back to DEFAULT_APP), and reindeer_x as a separate ECS container has no shared FS with Drupal to watch. So Part A — the "proven" half of Spike 8 — is the half that doesn't transfer to D11 without an EFS mount or a different ingestion door.

Also confirmed: Parts B/C truly absent (only @aws-sdk/client-s3; no SQS/SNS, no consumer). And the package still self-identifies as kmaps-solr-sync / shanti-uva — cosmetic post-transfer cleanup.

A fourth path the code surfaces

Your spike_solr_demo module ships search_api.server.kmassets.yml + search_api.index.kmassets.yml — Drupal already has a search_api connection to the kmassets index (Spike 2, read-side). search_api_solr can also write Drupal entities into a Solr index natively. So there's a path that needs no reindeer_x, no S3, no ECS for content: let search_api index shanti_image straight into kmassets.

The catch — and it's the real one: the kmassets index has a specific document contract (asset_type, the field structure the React app and existing queries depend on). search_api_solr writes its own doc shape. Writing Drupal content into the same index via search_api risks producing docs that aren't byte-compatible with the D7 format — which collides with the acceptance criterion ("retrievable via existing query patterns") and ADR 004 (match the existing Solr contract, don't refactor).

So the decision for 1a.8 sharpens to:

Path	Reuses	The risk
A — S3 → ECS (D7 model)	existing ECS transform produces correct doc format	needs shared EFS or a new ingest door; fileWatcher regex is legacy multisite; 4-hop fire-and-forget visibility problem
B — Drupal POST → reindeer_x → direct Solr write	reindeer_x's existing kmassets_write_client	new /post content handler + a content-doc transform (the kmterms transform ≠ content transform); reindeer_x productionization (still on legacy dev server)
C — Drupal search_api → direct Solr	Spike 2 connection already configured	doc-format fidelity vs. the existing kmassets contract — the React app / existing queries could break

All three still pivot on the Dave Goldstein cost/architecture conversation (is ECS the right home for low-frequency Drupal writes?), and B/C both depend on whoever owns the kmassets doc-format contract confirming the doc shape is faithful.

My instinct: B is the most aligned — it reuses reindeer_x's proven direct-writer and keeps the doc-transform logic in one place (where the kmterms transform already lives), which is the natural home to guarantee format fidelity. But it makes 1a.8 inseparable from reindeer_x productionization (ECS/Terraform + the credential strategy), which is a bigger lift than the Images retrieval criterion strictly needs.

Want me to (a) draft the concrete decision brief / question set for Dave that pins down A/B/C, (b) prototype the doc-format comparison (what a D7 content kmasset doc looks like vs. what search_api would emit), or (c) something else?

---

User — 2026-06-24 14:57

reindeer_x was originally written as mechanism to rewrite kmaps -> kmassets, where we controlled the solr doc format. It is admittedly a kludge. Refactoring it out of the D11 mandala is a worthy goal. However we DO need to retain the solr doc format (informal contract) that it writes, and that the various kmasset media managers write (for av, images, sources, texts). Note that there is an additional use of solr (which has a whole different ancestry): transcript indexing in the audio-video app.

---

Claude — 2026-06-24 14:58

That reframing matters — it separates the transport (which is up for grabs) from the doc-format contract (which is fixed). Let me play back the corrected model to make sure I've got it, because it re-scopes 1a.8.

The corrected mental model: three Solr usages, one shared contract

Index / usage	Ancestry	Writer(s)	D11 relevance
kmterms	KMaps Rails (Andres/Than)	external	read-only from Drupal
kmassets — Pop. 1 (subjects/places/terms shadow docs)	reindeer_x — the kludge	reindeer_x	the "refactor out" target
kmassets — Pop. 2 (content: av/images/sources/texts)	the per-site kmasset media managers	Drupal (5 managers)	this is what 1a.8 Images needs
AV transcript index	separate ancestry entirely	the audio-video app	new concern — not Images, not kmassets

The key thing you're telling me: the kmasset doc format is an informal contract that both reindeer_x (Pop. 1) and every media manager (Pop. 2) must emit identically, so the single index stays uniformly queryable by the React/KMaps app. The transport can be rewritten; the doc shape cannot drift. That's fully consistent with ADR 004 (Solr is source of truth; match the existing contract, don't refactor the index).

How this re-scopes 1a.8

1a.8 (Images) is not "wire up / productionize reindeer_x." It's: build the D11 Images kmasset media manager so it writes content docs in the established format. reindeer_x's Pop. 1 sync is orthogonal — Images doesn't touch it. So the A/B/C transport table from before collapses in priority: the contract is the deliverable; the transport is an implementation choice we can make late and even change later (it's reversible as long as the bytes on the Solr side match).

That also means my earlier worry about Option C (search_api native indexing) is precisely the real risk and the real gating question — not "can Drupal write to Solr" but "can a D11 writer emit the exact kmasset content-doc format." Whatever transport we pick, it has to be validated against the golden doc.

Two concerns that belong outside Sprint 1

• Refactoring reindeer_x out of D11 Mandala — a worthy goal, but it's the Pop. 1 shadow-sync, which Images doesn't depend on. It lightly evolves ADR 007 (which currently keeps reindeer_x as an independent service), so when you pursue it, it deserves its own successor ADR. For MVP, reindeer_x stays as-is externally; we just preserve the format it writes.
• AV transcript index — different ancestry, AV-app-specific, not kmassets. It's a genuinely new item that isn't tracked anywhere yet. Worth a deferred note + likely its own spike in the AV track (Spike 7 / Phase 3 territory), so it doesn't get silently folded into "Solr sync."

The concrete de-risking move

Before choosing any transport, capture the kmasset content-doc format as a golden contract — the one artifact that makes every downstream choice safe and testable:

• Pull the authoritative doc shape from two sources: the D7 Images media-manager builder code (the legacy shanti_kmaps/media-manager that wrote JSON to solrdocs/, in mandala-scripts/mandala-drupal) and a real Images doc from the live kmassets index (query it directly).
• Write it down as the spec — field-by-field — so any D11 writer (search_api mapping, or a custom doc-builder) can be validated byte-for-byte against it.

That single artifact turns 1a.8 from "pick an architecture and hope it's faithful" into "build a writer; diff its output against the golden doc." It also becomes the shared reference the other media managers (Sources/Texts/AV) reuse in their sprints.

---

So my read is: 1a.8 = Images kmasset media manager emitting the established doc format; defer reindeer_x refactor (own ADR later) and the AV transcript index (own note/spike). Does that match your intent? If so, the highest-value next step I can take is to go extract that golden doc-format contract — I can pull the D7 builder code and a live kmassets sample and write up the field-level spec. Want me to do that now, or capture these scoping notes (transcript index + reindeer-x-refactor goal) to docs/deferred/ first?

---

User — 2026-06-24 15:06

yes, I do think establishing a "golden doc" format is the right thing to do. What is your plan for that?

---

Claude — 2026-06-24 15:07

Good — let me ground the plan in what's actually reachable before I lay it out, so it names real sources rather than hand-waving. Two quick checks: where the D7 doc-builder lives, and how we'd reach the live kmassets index.

---

Claude — 2026-06-24 15:07

Quick find for the legacy D7 source, since the memory'd path didn't resolve:

---

Claude — 2026-06-24 15:08

Found the legacy source at /Users/ys2n/Code/mandala-legacy/mandala-drupal (the old solr-proxy and s3_synch are there too — both useful). Note: that's a different path than my memory had (/Users/ys2n/Code/mandala/…), so the pointer was stale — I'll fix that.

Here's the plan.

Deliverable

A committed kmasset Solr doc contract: a field-by-field spec plus real annotated JSON fixtures, for the Images content-asset doc — structured so the shared core generalizes to the other media managers (Sources/Texts/AV) in their sprints. The fixtures double as the acceptance test for the D11 writer.

Triangulate three authorities + two cross-checks

The format is an informal contract, so no single source is canonical — I reconcile across:

#	Source	Role	Where
1	D7 doc-builder code — the Images media-manager function that assembles the JSON written to solrdocs/	primary — defines every field and how it derives from the D7 node	mandala-legacy/mandala-drupal
2	kmassets Solr schema — field names + types	settles the string-vs-text / language_field quirks	schema in mandala-legacy + live /solr/kmassets/schema
3	Live kmassets sample docs	ground-truth of what's actually indexed	…/solr/kmassets (reachability caveat ↓)
4	reindeer_x kmassetSync.js builder	cross-check — reveals the common core Pop.1 & Pop.2 share	mandala-reindeer_x
5	spike_solr_demo search_api kmassets config	cross-check — the field set Spike 2 already mapped (read side)	this repo

Phases

1. Acquire — locate the exact D7 image doc-builder (module + function); pull the schema; capture a representative set of live docs, not just one: public, restricted, multi-agent, all-four-KMaps-domains, and a Tibetan/diacritic doc (so we lock the NFC-fidelity acceptance criterion into the contract).
2. Reconcile — build the table: field · Solr type · cardinality · D7 source/derivation · transform notes · example. Flag the known landmines from memory (language_field drops KMaps taxonomy; title/names_txt are string not text). Split common-core vs. image-specific fields explicitly.
3. Write up — commit docs/planning/kmasset-solr-doc-contract.md + the annotated JSON fixtures (as test fixtures under the module that'll own the writer). Record open questions for any field whose provenance is unclear.
4. Wire as validation — designate the fixtures as the golden check for the D11 Images writer: build writer → diff output vs. golden. That's the payoff and what makes the transport choice (A/B/C) low-stakes.

The one thing I need from you: live-index reachability

mandala-solr-proxy.internal.lib.virginia.edu is internal DNS + auth-gated — I almost certainly can't reach it from here. Options:

• (a) You're on VPN and I attempt read queries directly;
• (b) You dump a few sample docs via ! curl … so they land in the session;
• (c) We build the contract from the D7 builder + schema (which are the defining authority anyway) and treat live-sampling as a verification pass.

I'd lean (c) to start — the D7 builder code is the contract; live docs just confirm it — then fold in real samples via (a)/(b) when convenient.

---

Want me to kick off Phase 1 — dig into the D7 media-manager in mandala-legacy/mandala-drupal and find the doc-builder — on approach (c)? And shall I fix that stale memory path while I'm at it?

---

User — 2026-06-24 15:19

Also, please look at: /Users/ys2n/Code/mandala-legacy/mandala-om This is a react-app that is employed for several public facing sites. (I have checked the most recent prod branch)

---

Claude — 2026-06-24 15:20

This is a key find — mandala-om is a consumer of the kmasset format, which makes it arguably the binding authority (the contract exists to serve these queries). I'm already seeing exact field names the app depends on (asset_type, asset_subtype, names_txt, related_uid_ss). Let me pull the field lists it actually requests and reads.

---

Claude — 2026-06-24 15:21

That changes the shape of the contract — literally. mandala-om is the consumer side, and it reveals things the producer-side sources alone wouldn't.

What mandala-om tells us

It's the production React app (release branch release/v1.1.0-rc) querying the kmassets index across the public sites. Because it reads the docs, it defines what cannot change without breaking production. Concrete findings:

• It queries the full asset_type vocabulary in one index — audio-video images texts visuals sources subjects places terms collections (useInfiniteSearch.js:380). One unified index, faceted by asset_type — exactly the single-contract picture you described.
• Field-level dependencies, confirmed: asset_type, asset_subtype, id, uid, names_txt (searched with wildcard + boosting — names_txt:${query}*^70, which confirms the memory pitfall that names_txt is a string, not a tokenized text field), names, related_uid_ss, related_subjects_id_s.
• The big structural finding: kmasset docs are nested block-join documents, not flat. The app retrieves children via Solr child transformers — fl: '*,[child parentFilter=block_type:parent childFilter=-block_child_type:places_shape limit=1000]' (useKmap.js:24), and [child limit=5000] in SearchResults/ResultItem. So the doc has block_type:parent + typed block_child_type children. Any D11 writer has to reproduce that parent/child structure, not just a flat field set.

What this does to the plan

mandala-om becomes a third defining authority — and the binding one, since it's the live consumer:

Authority	Side	Defines
D7 media-manager builder	producer	every field + derivation
kmassets Solr schema	types	string-vs-text, multivalue suffixes
mandala-om (new)	consumer	which fields/structure are load-bearing for the public sites; the nested block-join shape
reindeer_x / spike_solr_demo	cross-check	shared core

And it adds an explicit task: document the parent/child block structure (which block_child_types an Images doc carries) as a first-class part of the golden doc — not an afterthought.

One downstream flag (not now): nested block-join docs are awkward for search_api_solr's native indexing, which leans toward flat docs. That nudges the eventual transport toward a custom doc-builder (Option B-ish) over Option C — but we don't decide that until the golden doc is in hand. It's just one more reason to capture the contract first.

---

So Phase 1 now reads from three producer/consumer authorities. I'd start by (a) finding the D7 Images doc-builder in mandala-legacy/mandala-drupal, and (b) extracting mandala-om's exact image-doc field reads (the render components for an image hit) so the consumer contract is pinned. Want me to go do that Phase-1 extraction now and report back with a first-cut field inventory?

---

User — 2026-06-24 15:22

Also scan the D7 mandala code, there are javascript clients there too.

---

Claude — 2026-06-24 15:22

The custom modules are now visible — including the producer (shanti_kmaps_solr, shanti_images), the AV transcripts module, and several search UIs likely holding JS clients. Let me locate the JS directly in those.

---

Claude — 2026-06-24 15:23

Found them — there's a whole AjaxSolr-based client layer in D7, predating the React app. Let me extract the actual field vocabulary they query so the finding is concrete, then I'll synthesize the full picture.

---

Claude — 2026-06-24 15:24

Scanned them. There's a whole legacy JS client layer in D7 — an ajaxsolr library plus shanti widgets — that predates the React app and queries the same kmassets contract. Concrete field vocabulary they depend on (counts = occurrences):

asset_type (94)   kmapid (27)        block_child_type (20)   block_type (15)
id_s (11)         header_s (9)       names_txt (8)           relation_label_s (7)
related_subjects_t/s   related_definitions_{path,source,level,author}   feature_type_s
caption / caption_eng / caption_key   node_id   kmaps_id   position_i   level_i

Two structural facts this nails down:

1. The block-join nested structure is original, not a React-era artifact — block_type/block_child_type show up 35× in the D7 clients too. So nested parent/child docs are core to the contract, full stop.
2. Language-suffixed fields — caption_eng (vs. bare caption) signals per-language field variants. That's directly relevant to the Tibetan/diacritic fidelity acceptance criterion — the contract carries language variants, and the D11 writer must reproduce them.

I also located the AV transcript index you mentioned: D7 transcripts module + kmaps_explorer/js/transcripts_ui.js. Confirmed as the separate-ancestry Solr usage — noted for its own deferred item, out of the Images contract.

The authority set is now comprehensive — producer + two consumer generations

Authority	Side	Notes
shanti_kmaps_solr / shanti_images / mediabase (D7)	producer	the doc-builder — defines fields + derivation
kmassets Solr schema	types	string-vs-text, multivalue suffixes, language variants
D7 AjaxSolr clients (views_solr, faceted_search, integrated_search, explorer, gallery)	consumer — legacy	original field vocabulary; may still be live on un-migrated sites
mandala-om React	consumer — current	the modern binding consumer
reindeer_x kmassetSync + spike_solr_demo	cross-check	shared core (Pop.1) + read-side mapping

This is the part that makes the golden doc trustworthy: the contract is pinned where producer output and both consumer generations agree. Any field only one side touches gets flagged as optional/uncertain rather than guessed.

---

Before I invest in the full extraction + write-up: is this the complete source set, or are there more consumers (other React apps, KMaps Rails, the Andres/Casa Tibet app, API endpoints) that read kmassets? Once you confirm the boundary, I'll run Phase 1 for real — find the D7 image doc-builder, reconcile it against these consumers + the schema, and produce a first-cut field inventory for the Images golden doc. Want me to proceed on that basis?

---

User — 2026-06-24 15:31

Look at /Users/ys2n/Code/mandala-legacy/kmaps_engine That is a the base engine for Andres' KMaps

---

Claude — 2026-06-24 15:31

This is the upstream — Andres' KMaps Rails engine, the producer of the kmterms index that reindeer_x shadows into kmassets (Pop.1). Its Solr-doc builder defines the KMaps-side field vocabulary that flows into the contract. Let me find that indexing code.

---

Claude — 2026-06-24 15:31

rsolr direct. The Feature model (= the KMaps subject/place/term node) is where the doc gets built. Let me extract its field vocabulary.

---

Claude — 2026-06-24 15:32

This is the keystone — and it reveals the contract is richer than a field list. kmaps_engine is the upstream producer (Andres' Rails engine) that writes the kmterms docs; reindeer_x shadows those into kmassets Pop.1, and the Drupal media managers had to mirror their shape to share the index. Looking at Feature#document_for_rsolr / nested_documents_for_rsolr (the doc builders):

The contract is a generative document grammar, not a flat schema

Three structural facts, straight from feature.rb:

1. The nested block-join structure originates here. block_type: ['parent'] with a _childDocuments_ array of block_type: ['child'] docs, each tagged by block_child_type (e.g. related_names, per-relation prefixes). This is the source of the block_type/block_child_type nesting every consumer (D7 JS and React) depends on — line 385 even queries it with {!child of=block_type:parent}.

2. Field names are generated, not fixed. Fields are built by interpolation:

   "caption_#{c.language.code}_#{c.id}"     # → caption_eng, caption_tib, …
   "summary_#{s.language.code}_#{s.id}"
   "code_#{c.geo_code_type.code}"
   

   That's why my literal-field grep mostly found only the structural fields (block_type, block_child_type, child_node_id, parent_node_id, context_id) — the rest are a naming grammar ({prefix}_{langcode}_{id}, _suffix for type), which is exactly why the consumers query with wildcards (names_txt:…*).

3. The tagging logic is spread across models — rsolr_document_tags / …_for_notes on Note/Citation attach fields to a doc under a given prefix. So the grammar is assembled from several models, not one method.

What this does to the golden doc

The golden doc can't just be "here are 40 fields." It has to capture the grammar: the parent/child block structure, the {prefix}_{langcode}_{id} naming convention, the block_child_type taxonomy, and the _suffix type encoding — then the concrete Images instance of it. That's what makes a D11 writer faithful AND lets the other media managers reuse it.

The authority landscape is now complete — and it's a chain

kmaps_engine (Rails)         → defines the doc GRAMMAR + writes kmterms
   → reindeer_x              → shadows kmterms → kmassets Pop.1 (same grammar)
   → Drupal media managers   → write kmassets Pop.2 (av/images/sources/texts), mirroring the grammar
        consumed by ↓
   → D7 AjaxSolr clients (legacy)  +  mandala-om React (current)  +  the kmassets schema (types)

The Images golden doc sits at the Pop.2 node, but its shape is dictated upstream by kmaps_engine and downstream by both consumer generations.

---

I think we've now traced the full producer→consumer chain. Two things, your call:

1. Is this the source boundary, or is there more (the Drupal-side mediabase/shanti_kmaps_solr builder I haven't opened yet is the one remaining producer I'd still read to pin the Images-specific fields)?
2. We've accumulated a lot of high-value discovery in conversation that'll be lost. Want me to consolidate it now into a docs/planning/kmasset-solr-doc-contract.md foundation — the source map + the grammar findings above — as the scaffold I then fill in during Phase 1? That captures the architecture before we go deep on the Images field-by-field extraction.

---

User — 2026-06-24 15:36

Here's more data for you: /Users/ys2n/Code/mandala-legacy/solr-shanti-configsets

---

Claude — 2026-06-24 15:36

The schema configsets — this is the type authority that grounds the grammar's _suffix conventions (the dynamic field rules). Let me look.

---

Claude — 2026-06-24 15:36

Multiple Solr versions/cores. Production uses solr_version: '7', core kmassets. Let me pull the kmassets schema for 7.3.x (and check 9.x) — the field types and dynamic-field rules that encode the grammar.

---

Claude — 2026-06-24 15:37

That's the capstone — the type layer that makes the grammar concrete. This is the production kmassets schema (Solr 7.3.x, matching the solr_version: '7' in your search_api config). Two halves:

Static core (the fixed contract every kmasset doc shares): - Identity: id, ids, uid (all required), parent_uid, _root_ - Faceting spine: asset_type, asset_subtype — both string, single-valued (so one type per doc) - KMaps linkage: kmapid, kmapid_strict (mv string) - Names/titles: name, names, title(s), names_txt — type text_kw (confirms the memory pitfall: keyword-analyzed, behaves string-like → wildcard queries), name_autocomplete - Description: caption, summary, description(s), subject, author, creator(s), keywords, content - Access projection: node_user, collection_uid_s, collection_uid_path_ss (the collection hierarchy path), collection_nid/title — this is the Step 1b auth contract (Solr-proxy visibility filtering reads these) - Media: url, thumbnail_url, content_type, resourcename, links

Dynamic grammar (the _suffix decoder ring): - Types: _i/_is int · _s/_ss string · _t/_txt text · _b _f _d _l _dt … (s = multivalued) - Language analyzers — the Tibetan-fidelity key: *_tibt/*_bo (Tibetan), *_sa (Sanskrit/Hindi), *_zh (CJK), *_en, *_latin (+ _sort variants). So kmaps_engine's {prefix}_{langcode} grammar maps onto language-specific Solr analyzers — this is precisely where NFC/diacritic fidelity lives in the contract. - Geo/special: *_idfacet, *_rptgeom, url_*, name_*

Three things this settles

1. The grammar is fully typed now — fixed core + {prefix}_{lang}_{suffix} dynamic rules. A D11 writer can be validated against this, field by field.
2. The Solr-proxy access contract is in here (node_user, collection_uid_path_ss) — so this schema feeds both 1a.8 (the write format) and 1b.3 (visibility filtering). Worth noting they share a source.
3. There's also a Solr 9 staging configset (solr9.10.11/) — if a version bump is in flight, the contract should be diffed across 7→9.

One open question it surfaces: the D7 caption_eng-style fields don't cleanly match a dynamic rule (_eng ≠ _en); how those resolve is a concrete Phase-1 reconciliation item — exactly the kind of thing the golden doc must pin down.

---

We now have the complete chain: grammar (kmaps_engine) → shadow (reindeer_x) → types (this schema) → Pop.2 producer (Drupal media managers, still to open) → consumers (D7 JS + React). I'm confident that's enough to write the foundation.

My strong recommendation: let me consolidate all of this into docs/planning/kmasset-solr-doc-contract.md now — the source map, the chain, the grammar, and this schema's static-core + dynamic-grammar tables — as the scaffold, before the discovery context is lost. Then Phase 1 fills in the Images-specific field derivations from the shanti_images/mediabase builder. Is the source set complete, or is there more to fold in first?

---

User — 2026-06-24 15:39

Please examine /Users/ys2n/Code/uvalib/terraform-infrastructure/mandala/solr to see what is actually deployed (solr version, etc)

---

Claude — 2026-06-24 15:40

Master/replica Solr on ECS. Let me find the actual deployed version (Docker image tag) and core config.

---

Claude — 2026-06-24 15:41

Now I have the deployed reality, and it's decisive for both the golden doc and the transport choice.

What's actually deployed

• Solr 7.7.3-slim (Docker docker/library/solr:7.7.3-slim) — both production and staging. So the authoritative schema is the solr7.3.x/production/kmassets configset (7.x family, classic schema.xml). The solr9.10.11/ configset in the repo is not deployed — aspirational/prep, not live. We build the golden doc against 7.x, full stop.
• Master–slave (leader/follower) replication, not SolrCloud. Replicas run -Denable.slave=true -Dmaster.url=http://mandala-solr-master-production-private…:8080/solr. Writes go to the master; replicas are read-only followers.
• Port 8080, SOLR_HOME=/solr/home, heap -Xmx8g. Infra last changed Oct 2022 — stable.

Why this matters

1. Golden doc → lock to Solr 7.x. Confirms solr7.3.x/production/kmassets/conf/schema.xml is the type authority. (Minor: configset is labeled 7.3 but the binary is 7.7.3 — same major, schema-compatible. The Solr 9 diff becomes a future migration concern, not part of the MVP contract.)

2. Transport (1a.8) gains hard constraints. Any D11 writer — whichever of A/B/C — must:

3. Write to the master endpoint, not the proxy-read path or a replica (Option B/C "direct write" has to target master);
4. Treat schema changes as file-deploys through this terraform repo + the configsets repo (no SolrCloud managed-schema API). So if the Images doc needs any field the current kmassets schema doesn't already cover, that's a coordinated schema.xml deploy with Dave's infra, not a Drupal-side change. (Good news: the static core + dynamic grammar we just read is broad enough that most Images fields already have a home.)

One thing to verify with infra

Drupal only knows the proxy endpoint (mandala-solr-proxy.internal…). The open question: does the proxy route writes to the master and reads to replicas, or is it read-only? That determines whether a D11 writer can go through the proxy at all or needs a separate master write-path. That's a Dave Goldstein question — and it pairs naturally with the cost/architecture conversation already queued.

---

This is the last structural piece — we now have the full chain end to end: grammar (kmaps_engine) → shadow (reindeer_x) → deployed types (Solr 7.7.3 / kmassets schema.xml) → Pop.2 producer (Drupal media managers) → consumers (D7 JS + React) → and the write-path topology (master/replica).

I really do think now's the moment to consolidate before this context is lost — let me write docs/planning/kmasset-solr-doc-contract.md capturing the source map, the chain, the grammar, the 7.x schema's static-core + dynamic-grammar tables, and the deployment/write-path constraints. That becomes the durable foundation; Phase 1 then fills in the Images-specific field derivations from shanti_images/mediabase. Shall I write it?

---

User — 2026-06-24 15:47

So. okay this is the way it works (as I understand it). There is a solr master and replicas. in the end we write to the master and read from the replicas. We do not read from the master as it is set for replication and not for reading. On top of that (and this is the subject of sprint 1.b) there is a proxy which asserts visibility rules to the queries that are coming in from clients. The clients ONLY see the proxy. The proxy reads from the replicas and not master. That is READ-ONLY. This is totally by design, to isolate the actual solr index from direct queries and attacks (and to provide the public/private read controls).

---

Claude — 2026-06-24 15:48

That's the missing architectural invariant — and it corrects something I floated. Let me play it back to make sure I've got it exactly:

   WRITE path  (1a.8)                          READ path  (1b)
   ─────────────────                           ──────────────
   D11 Images media manager ┐
   reindeer_x (Pop.1)       ├─ write ─▶  MASTER ──replication──▶ REPLICAS
   ECS ingest pipeline      ┘          (write-only,              (read-only
                                        NOT readable)             followers)
                                                                      ▲
                                                                      │ read-only
                                                                   PROXY  ◀── asserts public/private
                                                                      ▲        visibility on every query
                                                                      │
                                                            CLIENTS see ONLY this
                                                          (mandala-om React, D7 JS)

So, corrected on my end:

• The proxy is read-only by design — it does not route writes. My earlier "does the proxy route writes to master?" was the wrong question; the answer is "never, deliberately." Writers reach the master directly over the internal network; the proxy is purely the client read boundary.
• Master is write-only, replicas are read-only followers, and clients only ever touch the proxy — which isolates the index from direct query/attack and is where the public/private read controls live.
• That's why spike_solr_demo's search_api connection points at the proxy: correct for reading, but a D11 writer cannot use it — it needs a separate master write-connection (exactly the AUTH vs. UNAUTH split reindeer_x already has: kmassets_write_client = master, kmassets_read_client = proxy/replica).

The insight this crystallizes: 1a.8 and 1b meet at the schema's access fields

The write path and read path are separate by design — but they have a contract handoff in the document itself:

• 1a.8 (writer) must populate the access-control fields — node_user, collection_uid_s, collection_uid_path_ss.
• 1b (proxy) reads and enforces those exact fields to assert public/private visibility on client queries.

So the golden doc isn't just "what fields render in the UI" — it's also the security contract. If the D11 Images writer gets node_user / collection_uid_path_ss wrong, the proxy's visibility rules silently fail (over- or under-exposing content) — which is precisely the Sprint 1b security acceptance criterion. The two sprints are coupled through these fields.

---

This topology is durable, non-obvious infra design that isn't written down in the codebase — it belongs in the foundation doc (and probably a project memory + a dev-note, since it governs every future media manager).

When I write docs/planning/kmasset-solr-doc-contract.md, I'll lead with this write/read topology and flag the access-field handoff as the 1a.8↔1b seam. Is the topology as I've drawn it correct, and shall I now write the foundation doc?

---

User — 2026-06-24 15:53

Yes, please write this out. Both to record this, and to let us verify the understanding. To clarify what you drew into your diagram above: reindeer_x and the ECS ingest pipeline aren't the decided final implementation. We are also investigating a direct Drupal-write to solr... correct?>

---

Claude — 2026-06-24 15:54

Yes — exactly right, and that's an important correction to bake in: the topology (write→master / read→replicas / proxy read-only client boundary) is settled infra fact, but which writer feeds the master is explicitly undecided. reindeer_x and the ECS ingest pipeline are the inherited D7 path, not a chosen D11 design — and direct Drupal→Solr write is a live candidate we're investigating (the A/B/C options from earlier). I'll represent the writer side as an open decision, not settled.

Let me check the planning conventions so the doc fits, then write it.

---

Claude — 2026-06-24 15:54

Now I'll write the foundation doc.

---

Claude — 2026-06-24 16:03

Now register it in the planning nav so it surfaces in mkdocs.

---

Claude — 2026-06-24 16:03

Done. Written and registered in the planning nav.

docs/planning/kmasset-solr-doc-contract.md — the foundation doc, structured so you can verify the understanding section by section:

• §2 Topology (SETTLED) — the write→master / read→replicas / proxy-read-only diagram, with the four invariants and the design rationale (isolation + visibility control). Grounded in terraform-infrastructure/mandala/solr: Solr 7.7.3-slim, master-slave, not SolrCloud.
• §3 Write transport (OPEN) — your correction is baked in: reindeer_x + ECS are labeled the inherited D7 path, not a chosen D11 design, and direct Drupal→master write is a live candidate (Options A/B/C, all writing to master, gated on the Dave conversation). Also flags that schema changes are file-deploys, not API.
• §4 Source chain — the producer→shadow→types→consumers table with exact file locations.
• §5 Generative grammar — nested block-join + {prefix}_{lang}_{id} naming, from kmaps_engine.
• §6 Schema type layer — the static-core and dynamic-grammar tables (incl. the Tibetan/CJK/Sanskrit language analyzers).
• §7 Images inventory — explicitly marked Phase 1 TODO (the shanti_images/mediabase builder I haven't opened yet).
• §8 Pitfalls, §9 the 1a.8↔1b security seam, §10 Open questions (incl. the master write endpoint/auth and the caption_eng vs _en reconciliation).

Nothing committed yet — it's an untracked file on main.

A few things worth your eye when you verify: - §2 invariant 4 — my claim that spike_solr_demo's search_api proxy connection can't be a write path (so a writer needs separate master creds). That's the operational crux of §10 Q2. - §8 language-suffix — I flagged caption_eng (D7) vs *_en (schema) as unresolved rather than papering over it.

Two suggestions for where to go next: 1. Run Phase 1 — open the D7 shanti_images/mediabase/shanti_kmaps_solr doc-builder and fill in §7 (the Images field inventory + a first golden fixture). 2. This topology is durable, non-obvious infra knowledge — worth a project memory so future sessions start with it. Want me to save that, and/or commit this doc (on a branch, per the repo convention)?

Which direction — Phase 1 extraction now, or pause to commit + capture first?

---

User — 2026-06-24 16:04

Thanks! Let's close this session, and that will give us a chance to review the documents and proposed changes.

---

Claude — 2026-06-24 16:04

Let me run the session-end ritual. First, check the session-log script and current working-tree state.

---

Claude — 2026-06-24 16:04

Clean working tree except the two intended changes. Let me find this session's transcript, save the log, and update my memory in parallel.

---