QRun-IO/qbit-quick-search
3
0 Fork 0
Code Issues 2 Pull Requests 4 Actions Packages Projects Releases 14 Wiki Activity

Recover from a stuck REBUILDING status after a killed full reindex #24

New Issue
Closed
opened 2026-10-08 12:10:41 +00:00 by KofTwentyTwo · 0 comments
KofTwentyTwo commented 2026-10-08 12:10:41 +00:00 (Migrated from github.com)
Copy link
Copy source

If a full reindex of all tables is killed partway through (pod restart, OOM, deploy), or two full reindexes overlap, the tables it marked REBUILDING stay that way. Only another all-tables full reindex clears that status. Until then:

  • the listener keeps writing every delete as an AWAITING_REINDEX row, so those rows pile up without limit;
  • drift detection is off for those tables;
  • two overlapping runs drain each other's captured rows.

No data is lost: search stays correct, and the next full reindex applies the captured deletes. Found in the PR #20 review (issue #14).

Fix: before starting, detect whether a full reindex is still running, using the FULL_REINDEX run record. Refuse a second concurrent run. Recover a stale REBUILDING (a dead run) by restoring the table status and turning its AWAITING_REINDEX rows PENDING so basepull replays them. No schema change.

Part of #12.

If a full reindex of all tables is killed partway through (pod restart, OOM, deploy), or two full reindexes overlap, the tables it marked `REBUILDING` stay that way. Only another all-tables full reindex clears that status. Until then: - the listener keeps writing every delete as an `AWAITING_REINDEX` row, so those rows pile up without limit; - drift detection is off for those tables; - two overlapping runs drain each other's captured rows. No data is lost: search stays correct, and the next full reindex applies the captured deletes. Found in the PR #20 review (issue #14). Fix: before starting, detect whether a full reindex is still running, using the `FULL_REINDEX` run record. Refuse a second concurrent run. Recover a stale `REBUILDING` (a dead run) by restoring the table status and turning its `AWAITING_REINDEX` rows `PENDING` so basepull replays them. No schema change. Part of #12.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
architecture
Architecture review: coupling, cohesion, UI API
 bug
Something isn't working
 dependencies
documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 github_actions
Pull requests that update GitHub Actions code
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 java
Pull requests that update java code
 question
Further information is requested
 release:rc
Merging this develop PR cuts the next release candidate
 secuurity
wontfix
This will not be worked on
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
forge-recovery (Forge Recovery) james.maes (James Maes)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: QRun-IO/qbit-quick-search#24