A full reindex killed mid-run (pod restart, OOM, deploy), or two overlapping ones, could leave tables REBUILDING indefinitely. This adds liveness, a single-run guard and automatic recovery, with no schema change.
Liveness (heartbeat). A full reindex is alive while one of its RUNNINGFULL_REINDEX run records has a startTime or modifyDate within fullReindexStaleMinutes. The heartbeat refreshes modifyDate:
after every indexed page;
right before the pre-swap ownership check;
on every post-swap table update;
before every batch of the captured-delete drain.
It is best effort (a failure is logged, never aborts a page) and written only for FULL_REINDEX runs, single-table ones included, because nothing reads it for basepull or reconcile. The all-tables rebuild creates every table's run record before it marks any table REBUILDING, and keeps them RUNNING until its captured deletes are applied.
Threshold. New additive config fullReindexStaleMinutes, default 30 (null means the default; zero or negative fails validation). One page is one source query plus sourceBatchSize / bulkBatchSize bulk requests, each bounded by the transport timeouts: a few minutes at most with the defaults. 30 minutes leaves a wide margin, and a value set too low costs a wasted run, never a lost delete. With basepull every 5 minutes, a dead run is recovered within about 35 minutes.
Guard. Starting a full reindex (all tables or one) while another is alive fails with a QException before anything is created. The all-tables rebuild checks again right after creating its run records, so two runs started at the same moment cannot both proceed (at worst both refuse). A stale run never blocks.
Recovery (FullReindexStep.recoverDeadRun). When no run is alive, in this order:
REBUILDING tables go back to ACTIVE, with lastBasepullTime moved back to the dead run's start so updates a post-swap death left out are re-read. The prior status is not stored anywhere, so this follows the convention markRebuilding already uses; drift detection re-marks NEEDS_REINDEX in the same basepull.
AWAITING_REINDEX rows up to the newest id read before the run records become PENDING.
Last, the stale RUNNING records become FAILED with a clear message. A recovery that fails partway therefore leaves them RUNNING, and the next one starts over with the run's start still available for the rewind.
Recovery runs:
at the start of every full reindex (an all-tables run leaves the captured rows for its own drain, which checks for re-created keys);
at the start of every basepull, before failed-event replay. Basepull is the only scheduled path that consumes PENDING rows, runs every few minutes, and the check costs three small queries. Reconcile is optional (cron) and expensive, so it is not used.
Replaying handed-over deletes. Replay is harmless whether or not the dead run swapped. Before a swap the alias points at the old index, which the listener already deleted from (deleteDocuments counts a missing document as success). After a swap it removes the copy the dead run read before the delete; the integration test covers this case. Basepull's DELETE replay now skips, and resolves, an event whose record exists again (re-created with the same key), as the drain already did. Basepull still replays one sourceBatchSize batch of PENDING rows per run (unchanged), so a large handed-over backlog drains over several runs.
Captured-delete drain. The drain reads only rows up to the newest AWAITING_REINDEX id at its start (taken right after the tables leave REBUILDING); a later row belongs to a newer run. It heartbeats before every batch and stops once a recovery has marked the run's records FAILED. On failure, a run that was not taken over restores the tables and drains before it marks its records FAILED. A run that was taken over, or cannot tell, leaves the tables, the rows and its run records to recovery.
Races. Recovery reads the REBUILDING rows and the newest AWAITING_REINDEX id before it reads the run records. A run creates its records before it marks a table, and the listener captures only after it sees REBUILDING. So anything recovery reads that belongs to a live run comes with a live record, and nothing of a live run is touched. Before its swap, and right after a heartbeat, the rebuild checks that its records are still RUNNING and every table is still REBUILDING; otherwise it fails without swapping. That covers two cases:
a run that starts between a recovery's liveness read and its writes;
a live run stalled past the threshold.
A run judged stopped after its swap (for example during a long drain) stops draining and never touches a newer run's rows.
Residual window: a run must stall longer than the whole threshold between the heartbeat before its ownership check and the swap call, which are milliseconds apart. Only then could it swap after a recovery handed its deletes to basepull.
Other limits:
A single-table full reindex has only the up-front check; two started at the same instant both run, which is harmless (in-place reconcile, no REBUILDING, no captured rows).
Clock skew between nodes shifts the threshold by the skew.
A run killed before its swap still leaves its unaliased physical index on the cluster (unchanged, not in scope).
Testing
./mvnw test: 374 unit tests pass (21 new). They cover:
a live run blocks a second run (all tables and single table);
a stale run does not block and is recovered;
recovery restores the status, rewinds the watermark, flips AWAITING_REINDEX to PENDING and fails the run;
recovery hands over only rows up to the id it read;
a recovery that fails partway leaves the run RUNNING, and the next one still rewinds;
a live run's table, record and rows are untouched;
recovery without a run record, and recovery with nothing left behind;
one heartbeat per page, only for FULL_REINDEX, and best effort on failure;
basepull recovery replays the captured delete;
basepull skips a replayed delete of a re-created record;
a run recovered while still going does not swap;
a table no longer REBUILDING before the swap stops the swap;
a simultaneous start refuses;
records stay RUNNING while captured deletes are applied;
a drain taken over midway leaves the new run's rows alone;
the drain stops at the newest row read at its start;
config default and validation.
Two existing tests that simulate "during a full reindex" now insert a live run record.
CI=true ./mvnw verify: 20 integration tests pass, including the new FullReindexRecoveryIntegrationTest. It kills a real rebuild right after its alias swap, which leaves the deleted record back in search, the table REBUILDING, a RUNNING run record and a captured delete. It checks that a new full reindex is refused while the record is fresh, backdates the record and runs basepull. It then asserts the record is gone from search, the table is ACTIVE, the run is FAILED and no failed events are left. JaCoCo gate met.
## Summary
A full reindex killed mid-run (pod restart, OOM, deploy), or two overlapping ones, could leave tables `REBUILDING` indefinitely. This adds liveness, a single-run guard and automatic recovery, with no schema change.
**Liveness (heartbeat).** A full reindex is alive while one of its `RUNNING` `FULL_REINDEX` run records has a `startTime` or `modifyDate` within `fullReindexStaleMinutes`. The heartbeat refreshes `modifyDate`:
- after every indexed page;
- right before the pre-swap ownership check;
- on every post-swap table update;
- before every batch of the captured-delete drain.
It is best effort (a failure is logged, never aborts a page) and written only for `FULL_REINDEX` runs, single-table ones included, because nothing reads it for basepull or reconcile. The all-tables rebuild creates every table's run record before it marks any table `REBUILDING`, and keeps them `RUNNING` until its captured deletes are applied.
**Threshold.** New additive config `fullReindexStaleMinutes`, default 30 (null means the default; zero or negative fails validation). One page is one source query plus `sourceBatchSize / bulkBatchSize` bulk requests, each bounded by the transport timeouts: a few minutes at most with the defaults. 30 minutes leaves a wide margin, and a value set too low costs a wasted run, never a lost delete. With basepull every 5 minutes, a dead run is recovered within about 35 minutes.
**Guard.** Starting a full reindex (all tables or one) while another is alive fails with a `QException` before anything is created. The all-tables rebuild checks again right after creating its run records, so two runs started at the same moment cannot both proceed (at worst both refuse). A stale run never blocks.
**Recovery** (`FullReindexStep.recoverDeadRun`). When no run is alive, in this order:
1. `REBUILDING` tables go back to `ACTIVE`, with `lastBasepullTime` moved back to the dead run's start so updates a post-swap death left out are re-read. The prior status is not stored anywhere, so this follows the convention `markRebuilding` already uses; drift detection re-marks `NEEDS_REINDEX` in the same basepull.
2. `AWAITING_REINDEX` rows up to the newest id read before the run records become `PENDING`.
3. Last, the stale `RUNNING` records become `FAILED` with a clear message. A recovery that fails partway therefore leaves them `RUNNING`, and the next one starts over with the run's start still available for the rewind.
Recovery runs:
- at the start of every full reindex (an all-tables run leaves the captured rows for its own drain, which checks for re-created keys);
- at the start of every basepull, before failed-event replay. Basepull is the only scheduled path that consumes `PENDING` rows, runs every few minutes, and the check costs three small queries. Reconcile is optional (cron) and expensive, so it is not used.
**Replaying handed-over deletes.** Replay is harmless whether or not the dead run swapped. Before a swap the alias points at the old index, which the listener already deleted from (`deleteDocuments` counts a missing document as success). After a swap it removes the copy the dead run read before the delete; the integration test covers this case. Basepull's DELETE replay now skips, and resolves, an event whose record exists again (re-created with the same key), as the drain already did. Basepull still replays one `sourceBatchSize` batch of `PENDING` rows per run (unchanged), so a large handed-over backlog drains over several runs.
**Captured-delete drain.** The drain reads only rows up to the newest `AWAITING_REINDEX` id at its start (taken right after the tables leave `REBUILDING`); a later row belongs to a newer run. It heartbeats before every batch and stops once a recovery has marked the run's records `FAILED`. On failure, a run that was not taken over restores the tables and drains before it marks its records `FAILED`. A run that was taken over, or cannot tell, leaves the tables, the rows and its run records to recovery.
**Races.** Recovery reads the `REBUILDING` rows and the newest `AWAITING_REINDEX` id before it reads the run records. A run creates its records before it marks a table, and the listener captures only after it sees `REBUILDING`. So anything recovery reads that belongs to a live run comes with a live record, and nothing of a live run is touched. Before its swap, and right after a heartbeat, the rebuild checks that its records are still `RUNNING` and every table is still `REBUILDING`; otherwise it fails without swapping. That covers two cases:
- a run that starts between a recovery's liveness read and its writes;
- a live run stalled past the threshold.
A run judged stopped after its swap (for example during a long drain) stops draining and never touches a newer run's rows.
Residual window: a run must stall longer than the whole threshold between the heartbeat before its ownership check and the swap call, which are milliseconds apart. Only then could it swap after a recovery handed its deletes to basepull.
Other limits:
- A single-table full reindex has only the up-front check; two started at the same instant both run, which is harmless (in-place reconcile, no `REBUILDING`, no captured rows).
- Clock skew between nodes shifts the threshold by the skew.
- A run killed before its swap still leaves its unaliased physical index on the cluster (unchanged, not in scope).
## Testing
- `./mvnw test`: 374 unit tests pass (21 new). They cover:
- a live run blocks a second run (all tables and single table);
- a stale run does not block and is recovered;
- recovery restores the status, rewinds the watermark, flips `AWAITING_REINDEX` to `PENDING` and fails the run;
- recovery hands over only rows up to the id it read;
- a recovery that fails partway leaves the run `RUNNING`, and the next one still rewinds;
- a live run's table, record and rows are untouched;
- recovery without a run record, and recovery with nothing left behind;
- one heartbeat per page, only for `FULL_REINDEX`, and best effort on failure;
- basepull recovery replays the captured delete;
- basepull skips a replayed delete of a re-created record;
- a run recovered while still going does not swap;
- a table no longer `REBUILDING` before the swap stops the swap;
- a simultaneous start refuses;
- records stay `RUNNING` while captured deletes are applied;
- a drain taken over midway leaves the new run's rows alone;
- the drain stops at the newest row read at its start;
- config default and validation.
Two existing tests that simulate "during a full reindex" now insert a live run record.
- `CI=true ./mvnw verify`: 20 integration tests pass, including the new `FullReindexRecoveryIntegrationTest`. It kills a real rebuild right after its alias swap, which leaves the deleted record back in search, the table `REBUILDING`, a `RUNNING` run record and a captured delete. It checks that a new full reindex is refused while the record is fresh, backdates the record and runs basepull. It then asserts the record is gone from search, the table is `ACTIVE`, the run is `FAILED` and no failed events are left. JaCoCo gate met.
Closes #24
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
A full reindex killed mid-run (pod restart, OOM, deploy), or two overlapping ones, could leave tables
REBUILDINGindefinitely. This adds liveness, a single-run guard and automatic recovery, with no schema change.Liveness (heartbeat). A full reindex is alive while one of its
RUNNINGFULL_REINDEXrun records has astartTimeormodifyDatewithinfullReindexStaleMinutes. The heartbeat refreshesmodifyDate:It is best effort (a failure is logged, never aborts a page) and written only for
FULL_REINDEXruns, single-table ones included, because nothing reads it for basepull or reconcile. The all-tables rebuild creates every table's run record before it marks any tableREBUILDING, and keeps themRUNNINGuntil its captured deletes are applied.Threshold. New additive config
fullReindexStaleMinutes, default 30 (null means the default; zero or negative fails validation). One page is one source query plussourceBatchSize / bulkBatchSizebulk requests, each bounded by the transport timeouts: a few minutes at most with the defaults. 30 minutes leaves a wide margin, and a value set too low costs a wasted run, never a lost delete. With basepull every 5 minutes, a dead run is recovered within about 35 minutes.Guard. Starting a full reindex (all tables or one) while another is alive fails with a
QExceptionbefore anything is created. The all-tables rebuild checks again right after creating its run records, so two runs started at the same moment cannot both proceed (at worst both refuse). A stale run never blocks.Recovery (
FullReindexStep.recoverDeadRun). When no run is alive, in this order:REBUILDINGtables go back toACTIVE, withlastBasepullTimemoved back to the dead run's start so updates a post-swap death left out are re-read. The prior status is not stored anywhere, so this follows the conventionmarkRebuildingalready uses; drift detection re-marksNEEDS_REINDEXin the same basepull.AWAITING_REINDEXrows up to the newest id read before the run records becomePENDING.RUNNINGrecords becomeFAILEDwith a clear message. A recovery that fails partway therefore leaves themRUNNING, and the next one starts over with the run's start still available for the rewind.Recovery runs:
PENDINGrows, runs every few minutes, and the check costs three small queries. Reconcile is optional (cron) and expensive, so it is not used.Replaying handed-over deletes. Replay is harmless whether or not the dead run swapped. Before a swap the alias points at the old index, which the listener already deleted from (
deleteDocumentscounts a missing document as success). After a swap it removes the copy the dead run read before the delete; the integration test covers this case. Basepull's DELETE replay now skips, and resolves, an event whose record exists again (re-created with the same key), as the drain already did. Basepull still replays onesourceBatchSizebatch ofPENDINGrows per run (unchanged), so a large handed-over backlog drains over several runs.Captured-delete drain. The drain reads only rows up to the newest
AWAITING_REINDEXid at its start (taken right after the tables leaveREBUILDING); a later row belongs to a newer run. It heartbeats before every batch and stops once a recovery has marked the run's recordsFAILED. On failure, a run that was not taken over restores the tables and drains before it marks its recordsFAILED. A run that was taken over, or cannot tell, leaves the tables, the rows and its run records to recovery.Races. Recovery reads the
REBUILDINGrows and the newestAWAITING_REINDEXid before it reads the run records. A run creates its records before it marks a table, and the listener captures only after it seesREBUILDING. So anything recovery reads that belongs to a live run comes with a live record, and nothing of a live run is touched. Before its swap, and right after a heartbeat, the rebuild checks that its records are stillRUNNINGand every table is stillREBUILDING; otherwise it fails without swapping. That covers two cases:A run judged stopped after its swap (for example during a long drain) stops draining and never touches a newer run's rows.
Residual window: a run must stall longer than the whole threshold between the heartbeat before its ownership check and the swap call, which are milliseconds apart. Only then could it swap after a recovery handed its deletes to basepull.
Other limits:
REBUILDING, no captured rows).Testing
./mvnw test: 374 unit tests pass (21 new). They cover:AWAITING_REINDEXtoPENDINGand fails the run;RUNNING, and the next one still rewinds;FULL_REINDEX, and best effort on failure;REBUILDINGbefore the swap stops the swap;RUNNINGwhile captured deletes are applied;Two existing tests that simulate "during a full reindex" now insert a live run record.
CI=true ./mvnw verify: 20 integration tests pass, including the newFullReindexRecoveryIntegrationTest. It kills a real rebuild right after its alias swap, which leaves the deleted record back in search, the tableREBUILDING, aRUNNINGrun record and a captured delete. It checks that a new full reindex is refused while the record is fresh, backdates the record and runs basepull. It then asserts the record is gone from search, the table isACTIVE, the run isFAILEDand no failed events are left. JaCoCo gate met.Closes #24