fix(opensearch): size index replicas to the cluster so a single node stays green #67

2026-10-16T16:43:30Z

KofTwentyTwo commented

2026-10-16 16:43:30 +00:00

(Migrated from github.com)

Problem

kof22 prod: quick_search.opensearch DEGRADED, "OpenSearch health is yellow or incomplete". The cluster is a single-node OpenSearch 2.19.6 (discovery.type: single-node). The one physical index behind the kof22-content alias (kof22-content-v2-1792028226289) was created without number_of_replicas, so it got OpenSearch's default of 1. The replica can never be placed (same_shard decider), so the index stays yellow. All other indexes on that cluster are green.

Fix

Config (zero config needed): indexReplicas (default null = automatic: min(maxAutoReplicas, data nodes - 1)) and maxAutoReplicas (default 1). The result is 0 replicas on one node and 1 on larger clusters. An explicit indexReplicas overrides it. Both are validated as non-negative.
Index creation: createPhysicalIndex sets the target, which covers both a new install and the full-reindex alias swap.
Startup reconcile: ensureIndexExists checks the existing index or indexes behind the alias and sends PUT /<index>/_settings {"index":{"number_of_replicas":N}} only when the value differs. It is idempotent and logged. A failure is logged and recorded but never thrown, so it cannot block startup. Indexes with auto_expand_replicas are left to OpenSearch, and OpenSearch Serverless (aoss) is not touched.
Health/status: the snapshot gains unassignedReplicaShards (computed from index-level health: no initializing shards and every primary active), targetReplicas and replicaSettingsFailed, and they appear in the quick_search.opensearch details. A yellow index on a 1-node cluster where every unassigned shard is a replica is reported UP, with the note "replicas cannot be assigned on a single-node cluster". Yellow with an inactive primary, or yellow on a multi-node cluster, stays DEGRADED. Red stays DOWN.
The OpenAPI fragment is regenerated (additive, nullable fields). CHANGELOG and README are updated.

Tests

QuickSearchReplicasTest: covers the sizing table and the stub-server behaviour (single node drops the replica only once, multi-node keeps 1, explicit override skips the health call, auto_expand is respected, 403 and 500 failures and an unreachable cluster are recorded rather than thrown, Serverless is unmanaged).
QuickSearchHealthChecksTest.singleNodeReplicas and QuickSearchStatusTest.singleNodeReplicaYellow cover the health and status decisions.
OpenSearchIntegrationTest (Testcontainers, single node) checks that a new index gets 0 replicas and is green. It then forces 1 replica, confirms the index is yellow, runs the reconcile, and checks it is back to 0 and green, with a second reconcile changing nothing.
Local CI=true ./mvnw verify: 566 unit and 46 integration tests pass, and the JaCoCo check passes.

🤖 Generated with Claude Code

## Problem kof22 prod: `quick_search.opensearch` DEGRADED, "OpenSearch health is yellow or incomplete". The cluster is a single-node OpenSearch 2.19.6 (`discovery.type: single-node`). The one physical index behind the `kof22-content` alias (`kof22-content-v2-1792028226289`) was created without `number_of_replicas`, so it got OpenSearch's default of 1. The replica can never be placed (`same_shard` decider), so the index stays yellow. All other indexes on that cluster are green. ## Fix - **Config** (zero config needed): `indexReplicas` (default null = automatic: `min(maxAutoReplicas, data nodes - 1)`) and `maxAutoReplicas` (default 1). The result is 0 replicas on one node and 1 on larger clusters. An explicit `indexReplicas` overrides it. Both are validated as non-negative. - **Index creation:** `createPhysicalIndex` sets the target, which covers both a new install and the full-reindex alias swap. - **Startup reconcile:** `ensureIndexExists` checks the existing index or indexes behind the alias and sends `PUT /<index>/_settings {"index":{"number_of_replicas":N}}` only when the value differs. It is idempotent and logged. A failure is logged and recorded but never thrown, so it cannot block startup. Indexes with `auto_expand_replicas` are left to OpenSearch, and OpenSearch Serverless (`aoss`) is not touched. - **Health/status:** the snapshot gains `unassignedReplicaShards` (computed from index-level health: no initializing shards and every primary active), `targetReplicas` and `replicaSettingsFailed`, and they appear in the `quick_search.opensearch` details. A yellow index on a 1-node cluster where every unassigned shard is a replica is reported **UP**, with the note "replicas cannot be assigned on a single-node cluster". Yellow with an inactive primary, or yellow on a multi-node cluster, stays DEGRADED. Red stays DOWN. - The OpenAPI fragment is regenerated (additive, nullable fields). CHANGELOG and README are updated. ## Tests - `QuickSearchReplicasTest`: covers the sizing table and the stub-server behaviour (single node drops the replica only once, multi-node keeps 1, explicit override skips the health call, `auto_expand` is respected, 403 and 500 failures and an unreachable cluster are recorded rather than thrown, Serverless is unmanaged). - `QuickSearchHealthChecksTest.singleNodeReplicas` and `QuickSearchStatusTest.singleNodeReplicaYellow` cover the health and status decisions. - `OpenSearchIntegrationTest` (Testcontainers, single node) checks that a new index gets 0 replicas and is green. It then forces 1 replica, confirms the index is yellow, runs the reconcile, and checks it is back to 0 and green, with a second reconcile changing nothing. - Local `CI=true ./mvnw verify`: 566 unit and 46 integration tests pass, and the JaCoCo check passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-quick-search#67