Skip to content

Spike 8: reindeer_x Consolidation as Managed Sync Subsystem

Status: Partial — Part A (file watcher) proven; Parts B (SQS) and C (SNS) deferred — 2026-08-17: recommend not building these at all — see the addendum below, which argues the trigger should be pull/scheduled rather than push, making the SQS race-fix (Part B) and its SNS reporting (Part C) unnecessary rather than merely undone. Date: 2026-06 Branch/commit: uvalib/mandala-reindeer_x branch spike/08-reindeer-x-consolidation

Theory

The synch/synchandler shell+Perl pipeline (clsync + rclone) can be replaced with Node.js equivalents inside the existing reindeer_x process, and reindeer_x can subscribe to AWS SQS events directly to eliminate the UDP trigger and the kmterms race condition — making it a fully self-managed sync subsystem.

Background

Currently the kmassets sync pipeline has three separate runtime components: 1. reindeer_x (Node.js) — kmterms → kmassets transform and write 2. synch (shell) + synchandler (Perl) — watches Drupal's private/files/solrdocs/ directories and uploads JSON to S3 via rclone 3. A UDP ping from KMaps Rails triggers reindeer_x — but fires before the ECS kmterms Solr update completes, causing a race condition and silent stale writes

This spike proves that all three responsibilities can be unified inside reindeer_x with no shell or Perl dependencies, and that the race condition can be eliminated by subscribing to an SQS queue rather than waiting for a UDP ping.

See docs/deferred/solr-sync-architecture-d11.md for the full architectural context.

Work

Part A — Fold synchandler into reindeer_x

  1. Add chokidar to kmaps-solr-sync/package.json
  2. Add @aws-sdk/client-s3 to kmaps-solr-sync/package.json
  3. Implement a sync/fileWatcher.js module that:
  4. Accepts a list of watch directories (equivalent to the six site paths in synch)
  5. Uses chokidar to watch for new/changed .json and .ids files
  6. Uploads .json files to s3://mandala-ingest-{env}-inbound/kmassets-inbound/
  7. Uploads .ids files to s3://mandala-ingest-{env}-inbound/kmassets-delete/
  8. Uses AWS SDK (not rclone) for all S3 operations
  9. Wire fileWatcher.js into server/index.js startup
  10. Verify that synch and synchandler can be retired from the Docker image

Part B — SQS subscription for kmterms completion event

  1. Add @aws-sdk/client-sqs to kmaps-solr-sync/package.json
  2. Implement a queue/sqsConsumer.js module that:
  3. Long-polls a configurable SQS queue (KMTERMS_COMPLETE_QUEUE env var)
  4. On receipt of a completion message, calls the same processRequest path as the UDP handler (so the trigger logic is shared)
  5. Acknowledges (deletes) the SQS message on successful job creation
  6. Reports errors without crashing the consumer loop
  7. Wire sqsConsumer.js into server/index.js startup (behind an env var flag so it can be disabled if the SQS queue doesn't exist yet)
  8. Coordinate with Dave Goldstein to add a completion notification to the ECS kmterms Solr update task (out of scope for this spike — just verify reindeer_x's side works against a manually published test message)

Part C — SNS completion reporting (stretch)

  1. Add @aws-sdk/client-sns to kmaps-solr-sync/package.json
  2. After each sync batch completes, publish a summary to a configurable SNS topic (SYNC_COMPLETE_TOPIC env var): count written, count skipped, count failed, duration
  3. On sync failure, publish to a separate error topic (SYNC_ERROR_TOPIC env var)

Demo

# Start reindeer_x locally with file watcher enabled
cd kmaps-solr-sync
cp .env.dist .env   # configure S3 bucket, watch dirs
npm run reindeer_x:dev

# Part A: drop a test JSON file in a watched directory
echo '{"uid":"subjects-99999"}' > /tmp/test-solrdocs/test.json
# → reindeer_x logs: "Uploaded test.json to s3://..."
# → verify file appears in S3 bucket

# Part B: publish a test SQS message
aws sqs send-message \
  --queue-url $KMTERMS_COMPLETE_QUEUE_URL \
  --message-body '{"event":"kmterms-update-complete"}'
# → reindeer_x logs: "SQS message received, triggering sync"
# → sync job appears in Arena UI at http://localhost:4567

Pass Criteria

Part A: - chokidar detects new JSON files in watched directories reliably - AWS SDK uploads files to S3 with correct path structure - No rclone or Perl dependency needed for file upload - synch and synchandler scripts are no longer needed at runtime

Part B: - reindeer_x polls SQS and receives a test message - Receipt of the message triggers a sync job (visible in Arena UI) - Message is deleted from SQS after successful job creation - SQS consumer loop recovers from errors without crashing

Part C (stretch): - Sync completion publishes a summary message to SNS - Message contains count fields and duration

Fail Criteria and Response

Finding Response
chokidar misses file events in Docker volume mounts Evaluate polling mode (usePolling: true) or switch to a dedicated S3 upload endpoint instead of file watching
AWS SDK auth doesn't work in container context Verify IAM role / instance profile; fall back to explicit credential env vars
SQS queue doesn't exist yet for testing Use LocalStack or mock SQS for Part B; defer real integration to post-spike
ECS kmterms task can't be modified to publish completion event Keep UDP trigger as interim; document as remaining gap

Findings (Part A — proven)

Part A is proven. A new sync/fileWatcher.js module folds the synch (clsync) + synchandler (Perl/rclone) pipeline into reindeer_x as native Node.js, wired into server/index.js startup behind the ENABLE_FILE_WATCHER flag.

Verified end-to-end against a throwaway S3 bucket (no production buckets touched) — see the demo script run on branch spike/08-reindeer-x-consolidation:

  • chokidar reliably detects new files in a watched directory (add/change events), with awaitWriteFinish so partially-written docs aren't uploaded and usePolling available for Docker bind mounts.
  • AWS SDK v3 (@aws-sdk/client-s3) uploads with the correct path structure — s3://{bucket}/kmassets-inbound/test/{app}/{file}. The per-site {app} segment is derived from the solrdocs path with the same regex synchandler.prod used.
  • Behaviour parity with the Perl handler: empty files are skipped (the -s test); *.ids deletion files route to a separate kmassets-delete/ prefix.
  • No rclone or Perl needed. The Node module fully replaces both scripts.

Retiring the legacy pipeline: once ENABLE_FILE_WATCHER=true is the default, Dockerfile.reindeer_x can drop the clsync, rclone, and s3fs apt installs, the rclone.conf copies, and the synch/synchandler COPY+chmod lines (lines ~16–33). Left in place for now so the spike branch only adds the new path; the Dockerfile cleanup is a follow-up when Part A is promoted to default.

Parts B (SQS subscription) and C (SNS reporting) were not attempted — see scope note below.

Credential strategy

The watcher creates its S3 client with no hardcoded credentials (new S3Client({ region })), so it resolves credentials via the AWS SDK default provider chain (env vars → shared ini → … → ECS task role / IMDS). The application code needs no change across environments — only which identity the chain finds differs. Full design and infra hand-off in reindeer-x-aws-credential-strategy.md.

  • Local / testing — defer to the developer's own AWS identity. No task role or IAM setup needed. The Node SDK resolves the login_session profile, whose token can expire independently of the AWS CLI; eval "$(aws configure export-credentials --format env)" bridges an active CLI session to the SDK (env vars sit first in the chain). Part A was demonstrated this way against a throwaway bucket.
  • Deployed (dev / staging / production) — per-environment ECS task role granting s3:PutObject on mandala-ingest-{env}-inbound/*. Blocking gotcha: the legacy image bakes a static ~/.aws/credentials file, which is resolved before the task role in the chain and silently shadows it. So removing the baked credential files (app repo) and adding the task role (Terraform infra repo) must land together.
  • Manual / operator runs against a real environment (diagnosing a downstream failure, testing a downstream fix) — operators assume the same task role rather than using static keys or a personal identity, so the run has exactly the deployed service's permissions (no drift), with temporary creds and CloudTrail attribution. This requires the task role's trust policy to allow both ecs-tasks.amazonaws.com and a scoped operator group (treat prod as break-glass). "Run as , manually" is then pure configuration (INGEST_BUCKET, etc.) — no code path. Caution: writing to a real …-inbound bucket feeds the live downstream pipeline; for pure diagnosis use the …/test/ prefix or a scratch bucket.

What this does NOT establish

  • That the D11 Drupal → reindeer_x HTTP POST path works (separate spike needed)
  • That the ECS kmterms Solr update task publishes completion events (requires Dave Goldstein coordination — out of scope)
  • That reindeer_x is production-ready for D11 deployment (this spike proves the Node.js consolidation, not the full ECS integration)
  • Cost impact of SQS polling vs. current UDP model

Addendum (2026-08-17): the open question is the trigger, not the transform

Following the drift measurement in kmassets-production-index-frozen.md (53% of kmterms terms records touched since the kmassets shadow froze, in bursty bulk-shaped activity, not a steady trickle), the natural next question was whether reindeer_x — as currently built — is actually the right way to close that gap, or whether the project should step back before investing further in it. Read the live main branch of uvalib/mandala-reindeer_x (server/index.js, queue/jobCreationQueue.js, queue/queueConfigs.js) to check, rather than assuming from the spike write-up above.

Finding: the sync/transform logic already fits a scheduled-delta model — it doesn't need to be built, it's already the default behavior.

// server/index.js
const DEFAULT_QUERY = process.env.DEFAULT_QUERY || "_timestamp_:[NOW-1WEEK TO NOW]";

generateJobspecs (queue/jobCreationQueue.js) takes that query, chunks the matching kmterms result set into paginated jobs, and dispatches them through a Redis-backed Bee-Queue (jobCreationQueue → indexerQueue) to transform and write each chunk to kmassets. This is already a time-windowed incremental sync, not a full reindex — exactly the shape a scheduled delta job against updated_at/_timestamp_ would need.

Finding: the racy UDP trigger is not load-bearing — a plain HTTP alternative already exists and calls the identical code path:

app.post('/post', (req, res) => {
  ...
  processRequest(order)  // same function the UDP handler calls
  ...
});

Both the UDP datagram handler and POST /post funnel into the same processRequest(). So Spike 8 Part B (SQS subscription) exists to fix a timing race that is only a problem because the trigger is a fire-and-forget UDP ping racing against an ECS Solr commit. Nothing about the actual sync logic requires that trigger mechanism — anything capable of an HTTP POST or a direct processRequest() call (a cron job, an ECS Scheduled Task, a GitHub Action, a person) can drive a correct, non-racy sync run today, through code that already exists.

Conclusion: keep the transform/sync implementation as-is. The thing that's actually oversized for the job is the deployment shape wrapped around it — an Express server binding an ALB-exposed port 24/7, a UDP socket listening for a trigger that (per the frozen-index finding) nothing has reliably sent in over a year, a setInterval health-check loop running forever, and an Arena dashboard. That shape is why the remaining spike scope (Parts B/C) and the open deferred notes — no ECR repo or pipeline, the 9000/9001 ALB port mismatch — exist: they are problems that only arise because reindeer_x is modeled as a standing, push-triggered service. Given the bursty, infrequent, previously-unnoticed-for-15-months usage pattern the drift measurement established, that model doesn't match the job.

Recommendation: don't build Parts B/C, and don't build the always-on pipeline/ALB work in reindeer-x-has-no-ecr-repo-or-pipeline.md as currently scoped. Decide the trigger cadence instead (scheduled pull vs. on-demand manual — the actual push-vs-pull question, still Than/Andres's call) and let that decision determine deployment shape. A scheduled/on-demand task needs no ALB target, no UDP listener, and no fix for the port-mismatch defect — it only needs to run to completion when invoked, which is a smaller build than what's currently scoped.

What this does NOT establish: - Whether anyone actually needs near-real-time propagation (edits visible in Mandala search within minutes). Nothing found argues for it — 15 months of staleness went unnoticed — but this is a fact question for Andres/Than, not something the code can answer. - Which specific scheduling mechanism (ECS Scheduled Task, cron, GitHub Action) — not decided, and lower-stakes than the pull-vs-push decision itself. - Whether the Bee-Queue consumer (jobCreationQueue.process(8, processor)) can cleanly drain and exit for a one-shot invocation, or needs modification to do so — not verified against a real run, just read from the source.

Deferred notes