During a full reindex of all tables, real-time deletes go through the alias to the old physical index, so a record deleted after the rebuild read it stayed in the new index after the swap (and was still returned by search with applyRecordSecurityLocks=false).
Mechanism, with no schema change (new values on existing status fields only):
The reindex marks each table's quickSearchIndex row REBUILDING before reading any table, and puts it back to ACTIVE (or the prior status on failure) when it ends.
After a delete commits and before it is published, the listener reads that status from the database (not the one-minute enabled cache, so a reindex on another node is seen). Reading before the publish matters: if it sees no REBUILDING, either the reindex has not started reading the table or the alias already points at the new index. When REBUILDING, it also writes each delete to quickSearchFailedEvent with status AWAITING_REINDEX.
After its own writes and the alias swap, and after clearing REBUILDING, the reindex applies the AWAITING_REINDEX deletes through the alias (idempotent) and removes the rows; rows the cluster rejects become PENDING for basepull. A record re-created with the same key since its delete keeps its document (its row is just removed). The drain pages by id, so rows QQQ reports as not removed (per-row errors) are not re-read; they stay for the next full reindex. On a failed rebuild the deletes are applied to the old index, which is harmless.
basepull only replays PENDING, so it cannot consume a captured delete against the old index before the swap.
A listener that races the end of the rebuild re-reads the status after inserting; if the table is no longer REBUILDING it turns its rows PENDING so basepull replays them.
Drift detection (re-reading the row before it writes) and a concurrent reconcile no longer overwrite REBUILDING.
README documents the new status values, that drift detection is suspended while REBUILDING, and that a killed full reindex is recovered by running a full reindex of all tables again.
Testing
Unit tests for the listener capture, the status read happening before the publish (status flips during publish), and the hand-over; the reindex drain (applied after the swap, failure to PENDING, status restore on failure, re-created records skipped, rows with per-row delete errors not re-read); basepull ignoring AWAITING_REINDEX; drift (including a stale snapshot) and reconcile keeping REBUILDING.
FullReindexDeleteReplayIntegrationTest (Testcontainers) deletes a record mid-rebuild after its document is in the new physical index and asserts it is absent from OpenSearch and from search after the swap. It fails with the post-swap drain disabled.
CI=true ./mvnw verify: 321 unit tests and 19 integration tests pass, coverage checks met, BUILD SUCCESS.
## Summary
During a full reindex of all tables, real-time deletes go through the alias to the old physical index, so a record deleted after the rebuild read it stayed in the new index after the swap (and was still returned by search with `applyRecordSecurityLocks=false`).
Mechanism, with no schema change (new values on existing `status` fields only):
- The reindex marks each table's `quickSearchIndex` row `REBUILDING` before reading any table, and puts it back to `ACTIVE` (or the prior status on failure) when it ends.
- After a delete commits and before it is published, the listener reads that status from the database (not the one-minute enabled cache, so a reindex on another node is seen). Reading before the publish matters: if it sees no `REBUILDING`, either the reindex has not started reading the table or the alias already points at the new index. When `REBUILDING`, it also writes each delete to `quickSearchFailedEvent` with status `AWAITING_REINDEX`.
- After its own writes and the alias swap, and after clearing `REBUILDING`, the reindex applies the `AWAITING_REINDEX` deletes through the alias (idempotent) and removes the rows; rows the cluster rejects become `PENDING` for basepull. A record re-created with the same key since its delete keeps its document (its row is just removed). The drain pages by id, so rows QQQ reports as not removed (per-row errors) are not re-read; they stay for the next full reindex. On a failed rebuild the deletes are applied to the old index, which is harmless.
- basepull only replays `PENDING`, so it cannot consume a captured delete against the old index before the swap.
- A listener that races the end of the rebuild re-reads the status after inserting; if the table is no longer `REBUILDING` it turns its rows `PENDING` so basepull replays them.
- Drift detection (re-reading the row before it writes) and a concurrent reconcile no longer overwrite `REBUILDING`.
README documents the new status values, that drift detection is suspended while `REBUILDING`, and that a killed full reindex is recovered by running a full reindex of all tables again.
## Testing
- Unit tests for the listener capture, the status read happening before the publish (status flips during publish), and the hand-over; the reindex drain (applied after the swap, failure to `PENDING`, status restore on failure, re-created records skipped, rows with per-row delete errors not re-read); basepull ignoring `AWAITING_REINDEX`; drift (including a stale snapshot) and reconcile keeping `REBUILDING`.
- `FullReindexDeleteReplayIntegrationTest` (Testcontainers) deletes a record mid-rebuild after its document is in the new physical index and asserts it is absent from OpenSearch and from search after the swap. It fails with the post-swap drain disabled.
- `CI=true ./mvnw verify`: 321 unit tests and 19 integration tests pass, coverage checks met, BUILD SUCCESS.
Closes #14
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
During a full reindex of all tables, real-time deletes go through the alias to the old physical index, so a record deleted after the rebuild read it stayed in the new index after the swap (and was still returned by search with
applyRecordSecurityLocks=false).Mechanism, with no schema change (new values on existing
statusfields only):quickSearchIndexrowREBUILDINGbefore reading any table, and puts it back toACTIVE(or the prior status on failure) when it ends.REBUILDING, either the reindex has not started reading the table or the alias already points at the new index. WhenREBUILDING, it also writes each delete toquickSearchFailedEventwith statusAWAITING_REINDEX.REBUILDING, the reindex applies theAWAITING_REINDEXdeletes through the alias (idempotent) and removes the rows; rows the cluster rejects becomePENDINGfor basepull. A record re-created with the same key since its delete keeps its document (its row is just removed). The drain pages by id, so rows QQQ reports as not removed (per-row errors) are not re-read; they stay for the next full reindex. On a failed rebuild the deletes are applied to the old index, which is harmless.PENDING, so it cannot consume a captured delete against the old index before the swap.REBUILDINGit turns its rowsPENDINGso basepull replays them.REBUILDING.README documents the new status values, that drift detection is suspended while
REBUILDING, and that a killed full reindex is recovered by running a full reindex of all tables again.Testing
PENDING, status restore on failure, re-created records skipped, rows with per-row delete errors not re-read); basepull ignoringAWAITING_REINDEX; drift (including a stale snapshot) and reconcile keepingREBUILDING.FullReindexDeleteReplayIntegrationTest(Testcontainers) deletes a record mid-rebuild after its document is in the new physical index and asserts it is absent from OpenSearch and from search after the swap. It fails with the post-swap drain disabled.CI=true ./mvnw verify: 321 unit tests and 19 integration tests pass, coverage checks met, BUILD SUCCESS.Closes #14