Skip to content

Solr Sync Architecture for D11

Area: solr / kmassets / kmterms Raised during: Session 2026-06-12 (kmterms/kmassets architecture investigation) Jira: (add when available) Priority: High

Background

The D7 Mandala platform maintains two Solr indices:

  • kmterms — The KMaps ontology: subjects, places, and terms. Owned and written by the external KMaps Rails applications (Andres Montano / Casa Tibet Guatemala). Drupal only reads from this index.
  • kmassets — All asset descriptions across the Mandala platform. Contains two distinct populations.

Two populations in kmassets

Population 1 — KMaps-derived assets (asset_type: subjects, places, terms): One kmasset entry per kmterm. These are generated 1:1 from kmterms and kept in sync by the kmaps-solr-sync service (Node.js, runs as the reindeer_x Docker container). They make KMaps taxonomy terms themselves discoverable as assets in search — subjects/places/terms are managed conceptually like assets in Drupal, and the shadow entries are what makes that possible with a single index.

Population 2 — Drupal content assets (asset_type: audio-video, images, texts, visuals, sources, collections): One kmasset entry per Drupal node. These are generated by the shanti_kmaps_fields module on node save.

Full D7 sync pipeline

Content assets (Drupal → kmassets): 1. Node saved in Drupal → shanti_kmaps_fields writes JSON doc to private://solrdocs/ 2. clsync (watching that directory inside the Aegir/hostmaster container) detects the file 3. synchandler (Perl) uploads the JSON file to S3 (mandala-ingest-{env}-inbound/kmassets-inbound/) via rclone 4. S3 notification → SQS → ECS pipeline (transform → Solr update)

The synch (shell) and synchandler (Perl) scripts live in the reindeer_x container and are committed at kmaps-solr-sync/docker/reindeer_x/files/usr/local/bin/.

KMaps-derived assets (kmterms → kmassets): 1. KMaps Rails posts updated kmterms data to S3 (kmterms-inbound/) → SQS → ECS tasks update kmterms Solr 2. KMaps Rails simultaneously sends a UDP ping to reindeer_x 3. reindeer_x queries kmterms Solr for changed entries (!schema_version_i:40) and writes corresponding kmassets directly to kmassets Solr

The kmaps-solr-sync source is now in the monorepo at kmaps-solr-sync/.

Known bug: race condition in kmterms → kmassets sync

KMaps Rails fires the UDP ping at the same time it posts to S3. reindeer_x races ahead of the ECS pipeline — it queries kmterms Solr before the ECS update task has finished writing, reads stale data, and silently writes stale kmassets entries. This has caused "update didn't take" behavior in production.

Fix: Move the trigger to the completion of the ECS kmterms Solr update task, not the start of the KMaps Rails upload. See proposed architecture below.

Problems for D11

1. Content asset sync (Drupal → kmassets)

The D7 mechanism relies on: - shanti_kmaps_fields writing JSON to private://solrdocs/ (method 1 in the D7 module) - clsync watching that directory inside the Aegir container - synchandler uploading to S3

In D11, Aegir is gone. The equivalent pipeline needs to be designed. See solr-pipeline-cost-discussion.md for the full options and the Dave Goldstein conversation needed before deciding.

Update (2026-06-26) — working model, discussion ongoing. An initial conversation with Dave clarified that his mechanism is a small-batch (~32-doc) S3 poster with no transform — the producer writes the complete, final add-doc and reprocessing is forced object-level (rewrite/rename a file, or a "regen directory" for larger sets). The emerging shape: a single doc-builder feeding two sinks — a file sink (→ S3 → Dave's batch; authoritative) and a direct-to-master sink (incremental day-to-day updates + a fast synchronous diagnostic loop for batch failures). The D11 file-sink mechanics (Drupal → S3 directly, or relayed via reindeer_x HTTP /post per Spike 8 Part A) are an implementation detail, not a transport fork. Not yet closed with Dave — notably whether a second (direct) writer to the master is acceptable. Full write-up: kmasset-solr-doc-contract.md §3.

2. KMaps-derived asset sync (kmterms → kmassets)

The kmaps-solr-sync service needs to be modernized: - Race condition bug (see above) causes silent stale updates - UDP trigger is fragile and fired at the wrong point in the pipeline - Code notes from 2021 flag it as "not production ready" - No reporting — failures are invisible

3. Operational ownership

kmaps-solr-sync was previously in the shanti-uva GitHub org and outside the UVA Library deployment pipeline. It is now in mandala-navina but its deployment is still on the legacy dev server (mandala-drupal-dev-0). It needs to be brought into the Terraform/ECS infrastructure with proper environment config.

Proposed architecture: reindeer_x as a fully managed event-driven subsystem

The proposal is to evolve reindeer_x into a unified, self-contained sync service that monitors AWS events directly rather than waiting for external triggers.

Consolidation: fold synchandler into reindeer_x

Replace the shell/Perl synch + synchandler scripts with Node.js equivalents inside reindeer_x: - Replace clsync file watching with chokidar (Node.js standard) - Replace rclone S3 uploads with @aws-sdk/client-s3 - One container, one codebase, one log stream for all sync operations - In D11 (no shared filesystem with Drupal), the file-watch path becomes an HTTP POST from Drupal's shanti_kmaps_fields to reindeer_x's existing /post endpoint

Event-driven triggering via SQS

Replace the UDP ping with SQS subscription:

KMaps Rails → S3 (kmterms-inbound)
  → SQS → ECS transform → ECS kmterms Solr update
    → SNS/SQS completion notification  ← reindeer_x subscribes here
      → reindeer_x queries (now up-to-date) kmterms Solr
        → writes kmassets
          → publishes completion/failure to SNS topic
  • Race condition eliminated by definition — trigger fires after Solr is updated
  • HTTP /post endpoint and UDP port remain for manual/debug triggering
  • reindeer_x polls SQS (long-polling, cost-efficient in Node.js) rather than waiting to be called

Reporting

After sync, reindeer_x publishes to an SNS topic: - "kmassets sync complete" with count of written/skipped documents - "kmassets sync failed" with error detail - Drupal (or an alerting service) subscribes and surfaces failures visibly - Solves the "what happened to this update?" visibility problem

What reindeer_x becomes

A unified sync hub with: - Inbound channels: SQS event (kmterms update complete), HTTP POST (Drupal document push), UDP (manual/legacy) - Processing: kmterms→kmassets transform (existing), Drupal doc→S3 upload (folded in from synchandler) - Outbound: SNS completion/failure events, Arena queue UI for real-time state - Observability: full AWS event trail + Arena UI, no silent failures

  1. Consolidate: Fold synchandler logic into reindeer_x (Node.js + AWS SDK). This is independent of D11 decisions and can be done now.
  2. Fix the race: Coordinate with Dave Goldstein to add an SNS/SQS completion notification to the ECS kmterms Solr update task. Wire reindeer_x to subscribe.
  3. Discuss with Dave: Cost and architecture options for D11 content asset sync (see solr-pipeline-cost-discussion.md).
  4. Clarify with Andres/Than: Will KMaps Rails apps continue to send a UDP/HTTP trigger, or will the ECS pipeline completion event be the sole trigger?
  5. Spike: Prove D11 Drupal → reindeer_x HTTP POST path as replacement for clsync/synchandler.