Skip to content

feat(indexers): add a scalable NATS JetStream source - #6792

Draft
Totodore wants to merge 45 commits into
quickwit-oss:mainfrom
CentreonLabs:feat-durable-nats-source
Draft

Totodore wants to merge 45 commits into
quickwit-oss:mainfrom
CentreonLabs:feat-durable-nats-source

Conversation

@Totodore

@Totodore Totodore commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Current PR relies on #6768 which must be merged before. It will be rebased once done.

Description

Adds a NATS JetStream source consuming through a pre-provisioned durable consumer (fetch-only: never created, updated, nor deleted; must be AckExplicit). Stacked on #3 (pipeline drain), which it depends on.

Delivery semantics

  • Each message is acknowledged — with a server-confirmed ack (double ack) — once the split containing it is published (suggest_truncate). The consumer's ack floor is the resume point; the metastore checkpoint only carries synthetic positions correlating published batches with their acks.
  • Exactly-once on planned teardowns: the pipeline drain publishes and acks in-flight messages before the source exits; prefetched-but-unprocessed messages are NAK'd (with a short delay so they don't bounce back into the dying connection) for immediate redelivery to surviving pipelines.
  • At-least-once on crashes: unacknowledged messages are redelivered after ack_wait and indexed again.
  • Transient stream errors (missed idle heartbeat — which also fires under downstream backpressure — pull request failures, no responders) are retried with a 1s backoff instead of failing the pipeline; only terminal errors (consumer deleted, push-based consumer) kill it.

Also ships

  • num_pipelines > 1 work-queue load balancing across pipelines sharing the consumer.
  • TLS (custom CA + mutual TLS), user/password and token auth.
  • W3C traceparent propagation into per-message spans.
  • Config validation, reference docs, Stack Overflow tutorial, docker-compose nats service, coverage CI wiring, broker tests (nats-broker-tests feature).

Tests

cargo nextest run -p quickwit-indexing --features nats-broker-tests against the docker-compose NATS service: ingestion + ack floor, end-to-end graceful drain (pipeline exits on its own with acks confirmed flushed), work-queue load balancing across two pipelines, missing-consumer failure.

Tested on a dev k8s cluster with continuous ingestion.

Benchmarks

Benchmarks were conducted on a 16 vCPU VM with Quickwit + nats-server + Kafka co-located on tmpfs with kafka acting as a baseline.

  • Scaling: at 512 KiB/message, 52 → 147 MiB/s from 1 to 4 pipelines and with similar number for kafka:
image
  • Small messages are ack-bound: one server-confirmed ack per message means that at 1 KiB the source caps at 72–80 MiB/s while Kafka reaches 150, the broker spending 5.1 cores on acks against Kafka's 0.85 (52 vs 5 µcores/document). The ceiling is a message rate not a byte rate, and it is gone by 256 KiB. Batching records into one message (otlp_* input formats) is what moves a workload off it.
image

Totodore and others added 30 commits September 4, 2026 17:48
Drain indexing pipelines on planned teardowns instead of killing them:
the source stops emitting, flushes the in-flight batches with a forced
commit, and the pipeline exits on its own once everything is published
and settled. Sources opt in via source_needs_drain; others keep the
plain kill semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NATS JetStream source binding to a pre-provisioned durable consumer:
messages are acknowledged (server-confirmed) once the split containing
them is published, and the pipeline drain settles in-flight messages on
planned teardowns. Ships config, docs, tutorial, and broker tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drain every pipeline on teardown (sources without acknowledgment state
exit as soon as their flush batch is pushed), share the drain state
machine between both pipeline flavors, and fix the drain accounting:
DrainAllPipelines now also waits for pipelines a plan change already
detached, merge pipelines survive their draining indexing pipelines,
and a paused draining pipeline is no longer reaped as exited.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	quickwit/quickwit-indexing/src/source/mod.rs
Bound the node-shutdown drain wait with a configurable indexer
shutdown_drain_timeout (QW_INDEXER_SHUTDOWN_DRAIN_TIMEOUT, 300s by
default) so a drain can no longer outlive the deployment's grace period.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lled synchronously to match previous semantics
…lled synchronously to match previous semantics
@Totodore Totodore changed the title Feat durable nats source feat(indexers): add a scalable NATS JetStream source Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant