fix(status): count real-time indexing failures

realTimeErrorCount was only ever written as 0, so the status screen,
diagnostics and the status API always said there were no real-time
errors. When the change listener records failed events it now adds
their number to the table's count (after commit, serialized in the JVM,
never failing the user's write). A successful basepull or full reindex
resets it; the field's help text says so.

Fixes #53

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -6,6 +6,9 @@ All notable changes to this project are documented here. The format follows Keep
 
 Requires QQQ 4.1.0-RC.6 or later (the spec-managed UI API extension contract).
 
+### Fixed
+- Real-time indexing failures now increase the table's error count by the number of failed events. Successful basepulls and full reindexes reset the count. Counter update failures are logged without failing the source write.
+
 ### Added
 - Supported host status API (`api.QuickSearchStatusService.get()`), immutable status values, and `QuickSearchApi` home navigation constants (issue #50/F10). Permission-checked, bounded diagnostics refreshes share a short per-runtime cache and reuse saved validation counts. New spec-managed `GET /qqq/v1/qbit/quick-search/status` (`getQuickSearchStatus`) with committed OpenAPI fragment and strict HTTP/contract tests. RC.6 requires the default `/qbit/quick-search` prefix; `/quick-search` is not permitted by core. Existing routes and wire shapes are unchanged.
 - Record developer panel: `QuickSearchDeveloperPanelProvider` (id `quickSearch`) adds a "Quick Search" panel to the developer view of every record of an indexed table (`/app/{table}/{id}/dev` in the Next UI), for users who may read Quick Search Index. It shows the table's index row (enabled, status, record count, last basepull and full reindex, searchable fields), the latest index run, the record's indexed document compared field by field with the database (Inspect Search Document's comparison; needs `debugPermissionRules`), the record's failed index events, the Inspect Search Document and Reindex in Search actions the user may run, and links to the admin app and the index row. An unreachable OpenSearch shows a warning in the panel. Registered by `QuickSearchDeveloperPanelsMetaDataProducer`.

README.md

@@ -257,6 +257,8 @@ The panel's status is Indexed, Stale, Not indexed, Disabled (the table is turned
 - `quick_search_validation_run` (with `enableDiagnosticTools`): `status` (`PASS`, `WARN`, `FAIL`), `triggerType` (`MANUAL`, `SCHEDULED`), `userName`, `tableNames`, `sampleSize`, `tablesChecked`, `failCount`, `warnCount`, `summary`, `reportHtml` (HTML), `reportJson` (text), `startTime`, `endTime`, `createDate`. Purged after `runHistoryRetentionDays`.
 - `quick_search_failed_event`: real-time events that could not be applied; `status` `PENDING` or `EXHAUSTED` after 10 attempts. Deletes made while a full reindex of all tables runs are also recorded here with status `AWAITING_REINDEX`: the reindex applies them to the new index after the alias swap (basepull does not replay them). While a table is `REBUILDING`, drift detection is suspended for it. Only one full reindex runs at a time: starting one while another is running fails. A full reindex shows it is running through its `FULL_REINDEX` run records, which stay `RUNNING` with `modifyDate` refreshed after every page. If it is killed mid-run (pod restart, OOM, deploy), recovery is automatic once its run records have shown no sign of life for `fullReindexStaleMinutes` (30 by default): the next basepull, or the next full reindex, marks those run records `FAILED`, returns the tables to `ACTIVE` with `lastBasepullTime` moved back to the run's start, and turns the `AWAITING_REINDEX` rows `PENDING` so basepull replays the deletes. No data is lost. A run that was only slow and is judged stopped notices before its alias swap and fails without swapping, or, after its swap, stops applying its captured deletes; basepull skips a replayed delete whose record exists again.
 
+Real-time error counts on Quick Search Index show failures since the last successful full reindex or basepull. Each failed event increases the count; deferred changes while indexing is paused do not. Counter updates are best effort and serialized across listeners in one JVM; concurrent application nodes can still race because QQQ has no arithmetic update API.
+
 ## Extending
 
 `IndexEventPublisher` lets a host replace the synchronous publisher with a queue. In 1.0 the SPI is experimental: `IndexEvent` carries the table, record id, action and the record snapshot, but no sequence number, so an asynchronous consumer must order INDEX and DELETE for the same record itself. `OpenSearchTransportCustomizer` customizes the HTTP transport.

docs/SESSION-STATE.md

@@ -1,6 +1,15 @@
 # Session State
 
-Updated: 2026-10-13
+Updated: 2026-10-14
+
+## Real-time error counter fix (2026-10-14)
+
+- Failed-event recording now increments the table's `realTimeErrorCount` after commit, using QQQ reads and system writes. Null counts start at zero; update failures are logged and cannot fail the source write or prevent replay.
+- QQQ has no arithmetic update API. The read-modify-write fallback is serialized across listeners in this JVM; separate application nodes can still race.
+- Successful basepull and full-reindex status updates reset the count, including the single-table reindex path. Investigation found the previous zero assignments only initialized new rows. Field help, README and Unreleased changelog now describe the reset window.
+- Regression coverage includes event counts, concurrent listeners, counter-update failure, after-commit timing, basepull reset and both full-reindex reset paths.
+- Final gate: `./mvnw.cmd -o -B verify` passed against released QQQ 4.1.0-RC.6: 476 unit tests, zero failures/errors; 41 integration tests skipped because Docker is unavailable; architecture guardrails and JaCoCo coverage gates passed. No dependency changes or offline artifact gaps. Checkstyle, PMD and SpotBugs are not configured in this Maven build.
+- No commit, push or branch creation. Second-brain save is blocked by the vault being outside writable roots; this section retains the durable handoff for later vault synchronization.
 
 ## Status API worktree (2026-10-13, issue #50 / F10)
 

docs/TODO.md

@@ -1,5 +1,9 @@
 # TODO
 
+## Real-time error counter
+
+- [x] Count failed real-time events after commit, serialize concurrent listeners within a JVM, tolerate counter-write failures, and reset after successful basepull/full reindex; add regression tests and field/documentation help.
+
 ## Architecture status API (#50 / F10)
 
 - [x] Supported Java status API, bounded cache/rate limiting, administrator authorization, home navigation, spec endpoint and committed fragment.

src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java

@@ -19,7 +19,9 @@ package com.kingsrook.qbits.quicksearch.listeners;
 
 import java.io.Serializable;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
@@ -39,6 +41,7 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigWrites;
 import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
 import com.kingsrook.qbits.quicksearch.processes.AbstractIndexingStep;
@@ -283,6 +286,7 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
          }
 
          insertFailedEventRows(runtime, events, message, QuickSearchFailedEvent.STATUS_PENDING);
+         incrementRealTimeErrorCounts(runtime, events);
       }
       catch(Exception e)
       {
@@ -292,6 +296,54 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
 
 
 
+   /*******************************************************************************
+    ** QQQ UpdateInput has no arithmetic update. Serialize this read-modify-write
+    ** across listeners in this JVM, after the source transaction has committed.
+    ** Counter failures must not prevent replay or fail the source write.
+    *******************************************************************************/
+   private static synchronized void incrementRealTimeErrorCounts(QuickSearchRuntime runtime, List<IndexEvent> events)
+   {
+      Map<String, Integer> counts = new HashMap<>();
+      for(IndexEvent event : events)
+      {
+         counts.merge(event.getTableName(), 1, Integer::sum);
+      }
+
+      for(Map.Entry<String, Integer> entry : counts.entrySet())
+      {
+         try
+         {
+            String     indexTable = runtime.getConfig().getQuickSearchIndexTableName();
+            QueryInput queryInput = new QueryInput();
+            queryInput.setTableName(indexTable);
+            queryInput.setFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, entry.getKey())).withLimit(1));
+            List<QRecord> rows = new QueryAction().execute(queryInput).getRecords();
+            if(CollectionUtils.nullSafeIsEmpty(rows))
+            {
+               continue;
+            }
+
+            QRecord row      = rows.get(0);
+            Integer previous = row.getValueInteger("realTimeErrorCount");
+            UpdateInput updateInput = new UpdateInput();
+            updateInput.setTableName(indexTable);
+            updateInput.setRecords(List.of(new QRecord().withValue("id", row.getValue("id"))
+               .withValue("realTimeErrorCount", Math.addExact(previous == null ? 0 : previous, entry.getValue()))));
+            List<QRecord> updated = QuickSearchConfigWrites.asSystem(() -> new UpdateAction().execute(updateInput)).getRecords();
+            if(CollectionUtils.nullSafeIsEmpty(updated) || CollectionUtils.nullSafeHasContents(updated.get(0).getErrors()))
+            {
+               throw (new QException("Counter update returned errors"));
+            }
+         }
+         catch(Exception | LinkageError e)
+         {
+            LOG.warn("Could not increment real-time indexing error count", e, logPair("tableName", entry.getKey()));
+         }
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Keep changes made while indexing is paused, for the basepull to replay
     ** when it resumes. Never throws.

src/main/java/com/kingsrook/qbits/quicksearch/metadata/QuickSearchTableMetaDataHelper.java

@@ -26,6 +26,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
@@ -99,7 +100,8 @@ public class QuickSearchTableMetaDataHelper
          .withField(new QFieldMetaData("lastErrorMessage", QFieldType.TEXT).withIsEditable(false))
          .withField(new QFieldMetaData("recordCount", QFieldType.INTEGER).withIsEditable(false))
          .withField(new QFieldMetaData("documentCount", QFieldType.INTEGER).withIsEditable(false))
-         .withField(new QFieldMetaData("realTimeErrorCount", QFieldType.INTEGER).withIsEditable(false))
+         .withField(new QFieldMetaData("realTimeErrorCount", QFieldType.INTEGER).withIsEditable(false)
+            .withHelpContent(new QHelpContent("Real-time indexing failures since the last full reindex or basepull")))
          .withField(new QFieldMetaData("status", QFieldType.STRING).withIsEditable(false))
          .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))

src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java

@@ -423,6 +423,7 @@ public class BasepullIndexStep extends AbstractIndexingStep
          {
             Map<String, Serializable> values = new HashMap<>();
             values.put("lastBasepullTime", runStart);
+            values.put("realTimeErrorCount", 0);
             Integer documentCount = ReconcileIndexStep.safeCount(client, tableName);
             if(documentCount != null)
             {

src/main/java/com/kingsrook/qbits/quicksearch/processes/FullReindexStep.java

@@ -239,6 +239,7 @@ public class FullReindexStep extends AbstractIndexingStep
 
             Map<String, Serializable> values = new HashMap<>();
             values.put("lastFullReindexTime", Instant.now());
+            values.put("realTimeErrorCount", 0);
             values.put("lastBasepullTime", startTimes.get(tableName));
             values.put("recordCount", tableCounts.processed());
             Integer documentCount = ReconcileIndexStep.safeCount(client, tableName);

src/main/java/com/kingsrook/qbits/quicksearch/processes/ReconcileIndexStep.java

@@ -169,6 +169,7 @@ public class ReconcileIndexStep extends AbstractIndexingStep
             if(ReconcileIndexStep.RUN_TYPE.equals(runType) == false)
             {
                values.put("lastFullReindexTime", Instant.now());
+               values.put("realTimeErrorCount", 0);
             }
             updateIndexRow(indexId, values);
          }

src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java

@@ -19,6 +19,10 @@ package com.kingsrook.qbits.quicksearch.listeners;
 
 import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
@@ -36,6 +40,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
@@ -203,7 +208,9 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
       }
 
       assertThat(queryFailedEvents()).isEmpty();
+      assertThat(realTimeErrorCount()).isZero();
       transaction.commit();
+      assertThat(realTimeErrorCount()).isEqualTo(2);
 
       List<QRecord> failed = queryFailedEvents();
       assertThat(failed).extracting(r -> r.getValueString("recordId")).containsExactlyInAnyOrder("7", "8");
@@ -328,6 +335,102 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
 
 
 
+   /*******************************************************************************
+    ** Count failed events, including repeated failures, rather than batches.
+    *******************************************************************************/
+   @Test
+   void testPublishFailure_incrementsErrorCount() throws QException
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishDeleteEvents(anyList());
+      List<IndexEvent> events = List.of(
+         new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("7").withAction(IndexEventAction.DELETE),
+         new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("8").withAction(IndexEventAction.DELETE));
+
+      QuickSearchRecordChangeListener.publish(runtime, List.of(), events);
+      assertThat(realTimeErrorCount()).isEqualTo(2);
+      QuickSearchRecordChangeListener.publish(runtime, List.of(), events);
+      assertThat(realTimeErrorCount()).isEqualTo(4);
+      assertThat(queryFailedEvents()).hasSize(4);
+   }
+
+
+
+   /*******************************************************************************
+    ** Independent listeners must not overwrite each other's increments.
+    *******************************************************************************/
+   @Test
+   void testConcurrentFailures_preserveErrorCount() throws Exception
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishDeleteEvents(anyList());
+      QInstance instance = QContext.getQInstance();
+      CountDownLatch start = new CountDownLatch(1);
+      try(var executor = Executors.newFixedThreadPool(4))
+      {
+         List<Future<?>> futures = new ArrayList<>();
+         for(Integer worker = 0; worker < 4; worker++)
+         {
+            futures.add(executor.submit(() ->
+            {
+               QContext.init(instance, new QSession());
+               try
+               {
+                  assertThat(start.await(10, TimeUnit.SECONDS)).isTrue();
+                  for(Integer i = 0; i < 5; i++)
+                  {
+                     QuickSearchRecordChangeListener.publish(runtime, List.of(), List.of(
+                        new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("7").withAction(IndexEventAction.DELETE)));
+                  }
+               }
+               finally
+               {
+                  QContext.clear();
+               }
+               return (null);
+            }));
+         }
+         start.countDown();
+         for(Future<?> future : futures)
+         {
+            future.get(20, TimeUnit.SECONDS);
+         }
+      }
+      assertThat(realTimeErrorCount()).isEqualTo(20);
+   }
+
+
+
+   /*******************************************************************************
+    ** Counter storage failure must leave the source write and replay intact.
+    *******************************************************************************/
+   @Test
+   void testCounterUpdateFails_doesNotFailWrite() throws QException
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishIndexEvents(anyList());
+      try(MockedConstruction<UpdateAction> ignored = mockConstruction(UpdateAction.class, (mock, context) ->
+         when(mock.execute(any(UpdateInput.class))).thenThrow(new QException("counter unavailable"))))
+      {
+         assertThat(insert("widget", null)).isNotNull();
+      }
+      assertThat(queryFailedEvents()).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Read the persisted counter from the memory backend.
+    *******************************************************************************/
+   private Integer realTimeErrorCount() throws QException
+   {
+      QueryInput input = new QueryInput();
+      input.setTableName(QuickSearchIndex.TABLE_NAME);
+      return (new QueryAction().execute(input).getRecords().get(0).getValueInteger("realTimeErrorCount"));
+   }
+
+
+
    @Test
    void testDisabledTable_isSkipped() throws QException
    {
@@ -410,6 +513,7 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
    @Test
    void testUpdate_reReadFails_recordsFailedEventsAfterCommit() throws QException
    {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
       QBackendTransaction transaction = new QBackendTransaction();
       try(MockedConstruction<QueryAction> ignored = failingQueries(new QException("possible-value backend unavailable")))
       {
@@ -417,7 +521,9 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
       }
 
       assertThat(queryFailedEvents()).isEmpty();
+      assertThat(realTimeErrorCount()).isNull();
       transaction.commit();
+      assertThat(realTimeErrorCount()).isEqualTo(2);
 
       List<QRecord> failed = queryFailedEvents();
       assertThat(failed).extracting(r -> r.getValueString("recordId")).containsExactlyInAnyOrder("7", "8");

src/test/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStepTest.java

@@ -304,6 +304,26 @@ class BasepullIndexStepTest extends BaseQuickSearchTest
 
 
 
+   /*******************************************************************************
+    ** A successful basepull starts a new real-time failure counting window.
+    *******************************************************************************/
+   @Test
+   void testBasepull_resetsRealTimeErrorCount() throws QException
+   {
+      QRecord row = insertIndexRow(TEST_ENTITY_TABLE, true, null, 5);
+      new UpdateAction().execute(new UpdateInput().withTableName(QuickSearchIndex.TABLE_NAME)
+         .withRecord(new QRecord().withValue("id", row.getValue("id")).withValue("realTimeErrorCount", 7)));
+      insertTestEntities(2, Instant.now());
+
+      new BasepullIndexStep().run(new RunBackendStepInput(), new RunBackendStepOutput());
+
+      QueryInput input = new QueryInput();
+      input.setTableName(QuickSearchIndex.TABLE_NAME);
+      assertThat(new QueryAction().execute(input).getRecords().get(0).getValueInteger("realTimeErrorCount")).isZero();
+   }
+
+
+
    /*******************************************************************************
     ** Test: no discovered tables, no quickSearchIndex rows - step completes
     ** without calling the OpenSearch client.

src/test/java/com/kingsrook/qbits/quicksearch/processes/FullReindexStepTest.java

@@ -162,6 +162,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
          .withValue("enabled", enabled)
          .withValue("basepullIntervalMinutes", 5)
          .withValue("documentCount", documentCount)
+         .withValue("realTimeErrorCount", 7)
          .withValue("status", "ACTIVE")));
       new InsertAction().execute(insertInput);
    }
@@ -268,6 +269,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
    @Test
    void testAllTables_updatesIndexRowAfterSwap() throws QException
    {
+      insertIndexRow(true);
       insertTestEntities(2);
       when(mockClient.countDocumentsForTable(TEST_ENTITY_TABLE)).thenReturn(2L);
 
@@ -276,6 +278,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
       QRecord row = queryIndexRow();
       assertThat(row).isNotNull();
       assertThat(row.getValue("lastFullReindexTime")).isNotNull();
+      assertThat(row.getValueInteger("realTimeErrorCount")).isZero();
       assertThat(row.getValue("lastBasepullTime")).isNotNull();
       assertThat(row.getValueInteger("recordCount")).isEqualTo(2);
       assertThat(row.getValueInteger("documentCount")).isEqualTo(2);
@@ -449,6 +452,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
    @Test
    void testSingleTable_reconcilesInPlace_neverWipes() throws QException
    {
+      insertIndexRow(true);
       insertTestEntities(2);
 
       RunBackendStepOutput output = new RunBackendStepOutput();
@@ -471,6 +475,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
       assertThat(runs.get(0).getValueInteger("recordsProcessed")).isEqualTo(2);
 
       assertThat(queryIndexRow().getValue("lastFullReindexTime")).isNotNull();
+      assertThat(queryIndexRow().getValueInteger("realTimeErrorCount")).isZero();
       assertThat(output.getValueInteger("recordsIndexed")).isEqualTo(2);
       assertThat(output.getValueBoolean("aliasSwapped")).isFalse();
    }