ClickHouseKeeperInstallation stuck at status: InProgress despite healthy 3-node quorum (operator 0.27.0)

[KyriosGN0]

Description

With altinity-clickhouse-operator 0.27.0, a ClickHouseKeeperInstallation (CHK) never transitions to status: Completed. It stays InProgress indefinitely even though the Keeper cluster is fully healthy, and the operator reconcile-loops on the CHK every ~10s without converging. A sibling ClickHouseInstallation (CHI) managed by the same operator reconciles to Completed normally.

Environment

• clickhouse-operator: 0.27.0 (chart altinity-clickhouse-operator-0.27.0)
• Kubernetes: 1.35 (GKE)
• ClickHouse Keeper image: 26.5.x, 1 shard / 3 replicas

What's healthy

The Keeper has a full quorum — from mntr on a node:

zk_server_state        leader
zk_followers           2
zk_synced_followers    2

All 3 keeper pods are Running/Ready, the StatefulSets report rolling-update complete, and the ClickHouseInstallation (server) that uses this Keeper reaches status: Completed.

Observed behavior

The CHK stays status: InProgress; taskIDsStarted keeps growing while taskIDsCompleted stays at 0. The operator repeats this cycle continuously:

worker-reconciler-chk.go:57  reconcileCR():start
object-status.go:47          GetObjectStatusFromMetas(): cur and new objects are equal ... Update of the object is not required
worker-reconciler-chk.go:795 prepareStsReconcileOptsWaitSection(): Setting option SetWaitUntilReady   (once per host)
worker-reconciler-chk.go:91  reconcileCR(): CR has reconcile work - continue reconcile
worker-reconciler-chk.go:132 reconcileCR():end

It reproduces on a freshly created cluster, so it is not tied to any pre-existing or corrupted state.

Expected

Once the Keeper StatefulSets are ready and the quorum is healthy, the CHK should reach status: Completed (as the CHI does).

Questions

• Is there a readiness/coordination gate (e.g. reconcile.coordination.keeper.readyTimeout) that the CHK controller waits on and never satisfies in this scenario?
• Any known issue with CHK status finalization in 0.27.0?

Happy to add a minimal reproducer manifest and full operator logs.

[alex-zaitsev]

Could you check with 0.27.1?

[KyriosGN0]

Could you check with 0.27.1?

hey @alex-zaitsev i did, still the same result, even with a new cluster, deployed by a new operator in a new k8s cluster, this behavior persists

[atsarevskiy]

We hit this same failure on 0.26.3 (also observed unchanged in the 0.27.x code path). The root cause is in reconcile status finalization, not a readiness/coordination gate.

Symptom fingerprint: CHK stuck status: InProgress while fully healthy — completed = N/N hosts, action empty, error empty, quorum intact — and taskIDsStarted keeps incrementing while taskIDsCompleted stays 0. A sibling CHI managed by the same operator reconciles to Completed normally.

Root cause (file/line refs from release-0.26.3):

• markReconcileStart writes status: InProgress via ReconcileStart() — pkg/controller/chk/worker.go:229.
• status: Completed is written only by ReconcileComplete(), reached solely through finalizeReconcileAndMarkCompleted — pkg/controller/chk/worker.go:247-269.
• That finalize bails out early on if util.IsContextDone(ctx) { return } — worker.go:248, and there is a second identical guard at pkg/controller/chk/worker-reconciler-chk.go:118. If the reconcile's context is cancelled anytime between start and finalize (a newer CR generation supersedes the in-flight reconcile, or the operator pod restarts mid-reconcile), the CR is left InProgress. This matches taskIDsStarted incrementing while taskIDsCompleted stays 0.
• Once the object is converged, the next reconcile takes the default: "No reconcile work - abort reconcile" → return nil path — worker-reconciler-chk.go:94-97 — which never calls finalize. So the status never self-heals.

Reproduction trigger: anything that supersedes an in-flight CHK reconcile before it finalizes — rapid successive spec changes on the CHK (or on its owned StatefulSets), or an operator restart mid-reconcile. After that, the CHK is converged so no later reconcile ever routes back through finalize, and it stays InProgress indefinitely.

Impact: cosmetic — the Keeper is healthy and serving. But .status.status is not a trustworthy completion signal, which breaks automation/monitoring that gates on it. It clears only on a reconcile that has real work (e.g. a spec change / redeploy) or a manual kubectl patch --subresource=status.

[alex-zaitsev]

Fixed by #2070 in 0.27.4

[atsarevskiy]

@alex-zaitsev I think this is still the issue, reproduced this in https://github.com/atsarevskiy/altinity-ch-operator-integration-tests/tree/main/cases/2035-chk-stuck-in-progress