CMR-11492: Validate reindexer at production scale in WL

[daniel-zamora]

Overview

What is the objective?

Introduces the cmr reindexer, a FastAPI service that drives bulk reindexing of cmr metadata by dispatching concept-update messages to the cmr indexer's SQS queue. Replaces the reindex functionality previously handled ad-hoc via the bootstrap service.

What are the changes?

• new service (reindexer/) a fastAPI app with endpoints to trigger reindex by concept type, single concept, granules (all providers / by provider / by collection), with date filtering and job cancellation support
• Oracle client that uses keyset pagination for granule streaming.
• throttler worker thread that dequeues collection work items, streams granules from Oracle, and dispatches to the indexer queue via a token bucket rate limiter
• ES health gate pauses dispatch when clusters are not green
• Job tracking with dynamoDB tracked job state with heartbeat, progress counters, cancellation, stalled-job resume on startup, and mid-collection checkpoint for task recovery
• SQS batch publishing
• test suite with 450 unit tests

What areas of the application does this impact?

New standalone service.

Required Checklist

• New and existing unit and int tests pass locally and remotely
• clj-kondo has been run locally and all errors in changed files are corrected
• I have commented my code, particularly in hard-to-understand areas
• I have made changes to the documentation (if necessary)
• My changes generate no new warnings

Additional Checklist

• I have removed unnecessary/dead code and imports in files I have changed
• I have cleaned up integration tests by doing one or more of the following:
  • migrated any are2 tests to are3 in files I have changed
  • de-duped, consolidated, removed dead int tests
  • transformed applicable int tests into unit tests
  • reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

[jmaeng72]

would be nice to see the original url that triggered this job -- not having it be required though if it's too much work to add now

[daniel-zamora]

oh you meant in the job status response? I thought you meant in the readme.

[daniel-zamora]

I can add that it shouldnt be that big of a lift, if you mean the request url carried in the status response.

[jmaeng72]

yes, I mean the request url