Skip to content

Configuration Parameters

Jump to bottom
github-actions[bot] edited this page Oct 29, 2026 · 18 revisions

🎛️ Engine & Algorithmic Tuning

Deep tuning guide for Spector's programmatic Java engine, HNSW graph parameters, quantization profiles, and hybrid search weights. Learn how to optimize for recall, latency, throughput, or memory footprint using the SpectorConfig builder API.

  • :material-layers-outline: Configuration Architecture

    6-layer resolution hierarchy, precedence order, and profile mechanism.

    View Architecture Guide ↗{ .md-button }

  • :material-file-document-outline: spector.yml Master Reference

    Exhaustive dictionary of all 32 configuration domains and 150+ YAML keys.

    Explore spector.yml ↗{ .md-button }

  • :material-variable: Environment Variables & Secrets

    Canonical screaming snake_case mappings, short aliases, and Docker secrets.

    Browse Environment Guide ↗{ .md-button }

  • :material-cloud-outline: Deployment & Cloud Config

    Docker container matrices, Helm values.yaml, and multi-cloud Terraform modules.

    View Cloud Config ↗{ .md-button }

  • :material-api: REST API & Runtime Parameters

    Complete request schemas, headers, query parameters, and cognitive modifiers.

    Explore API Parameters ↗{ .md-button }

🎯 Programmatic Core Parameters

When initializing Spector via the Java API, SpectorConfig provides the foundational builder for vector memory:

Parameter Default Range Description
dimensions 384 1–4096 Vector dimensionality (must match your embedding model)
capacity 100,000 1–10,000,000 Maximum document count
similarityFunction COSINE COSINE, DOT_PRODUCT, EUCLIDEAN Distance metric

Tip

Quick model reference:

Model Dimensions
all-MiniLM-L6-v2 384
e5-base-v2 768
text-embedding-ada-002 1536
nomic-embed-text 768

Choosing a similarity function:

  • COSINE — Normalized embeddings (most models)

  • DOT_PRODUCT — Unnormalized embeddings where magnitude matters

  • EUCLIDEAN — Spatial/geometric data

🗜️ Quantization Parameters

Parameter Default Range Description
quantization NONE NONE, SCALAR_INT8, SCALAR_INT4, SCALAR_INT2, IVF_PQ Quantization type
oversamplingFactor auto 1–20 Rescore oversampling (auto: INT8→1, INT4→3, INT2→5)

🎛️ Quantization Profiles

Priority Type Oversampling Compression Recall Use Case
🎯 Max recall INT8 1 (none) 4× 95–99% Quality-critical search
⚖️ Balanced INT4 3 8× 85–95% Best compression/recall ratio
💾 Memory-first INT2 5 16× 75–90% Fit large datasets in RAM
🚀 Billion-scale IVF_PQ — 32× 75–90% Massive datasets

Tip

Start with INT4 for most workloads. It gives 8× compression with excellent recall when paired with the default 3× rescore. Only go to INT2 if memory is the binding constraint, or IVF-PQ if you're at billion scale.

Oversampling Tuning

The oversamplingFactor controls how many extra candidates are retrieved before rescoring with exact distances:

  • 1 — No rescore (fastest, quantized scores returned directly)

  • 3 — Good balance for INT4 (retrieves 3×K candidates, rescores to top-K)

  • 5 — Recommended for INT2 (compensates for aggressive quantization)

  • 10+ — Diminishing returns; use only if recall is still insufficient

// INT4 with custom oversampling
var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(50_000_000)
    .withQuantization(QuantizationType.SCALAR_INT4)
    .withRescore(5);  // Higher oversampling = better recall, slightly slower

🌐 HNSW Index Parameters

Parameter Default Range Description
M 16 4–64 Max connections per node per layer
efConstruction 200 16–800 Construction beam width
efSearch 50 10–500 Search beam width

🎛️ Tuning Profiles

Priority M efConstruction efSearch Trade-off
🎯 High recall 32–64 400–800 200–500 More memory, slower build/search
⚖️ Balanced 16 200 50 Good recall with fast performance
⚡ Low latency 8–12 100 20–30 Faster search, lower recall
💾 Memory-constrained 4–8 100 20 Minimal memory, lower recall

Important

efSearch should be ≥ topK for meaningful results. Setting efSearch < topK means you're asking for more results than the algorithm explores.

📝 BM25 Parameters

Parameter Default Range Description
k1 1.2 0.0–3.0 Term frequency saturation
b 0.75 0.0–1.0 Document length normalization
Corpus Type Recommended k1 Recommended b
Short docs (tweets, titles) 1.2 0.3
Medium docs (articles) 1.2 0.75
Long docs (books, papers) 1.5–2.0 0.75
Mixed lengths 1.2 0.5

🧬 Hybrid Search (RRF)

Parameter Default Range Description
RRF k 60 1–1000 Reciprocal Rank Fusion constant

  • k = 60 — Original paper recommendation, works well generally

  • Lower k (10–30) — Emphasizes top-ranked results more strongly

  • Higher k (100+) — Flattens rank importance

🎮 GPU Configuration

Parameter Default Range Description
gpuEnabled false true/false Enable CUDA GPU acceleration
gpuMemoryBudget 256 MB 256 MB – GPU max Maximum GPU memory allocation
gpuBatchWindow 10 ms 1–100 ms Batching window for query collection
gpuMaxBatchSize 1024 1–1024 Maximum queries per GPU batch

Note

Enable GPU for batch workloads with >10K vectors. Single queries are often faster on CPU SIMD due to zero kernel launch overhead. For INT4/INT2 quantization, GPU acceleration requires dimensions to be a multiple of 32. Non-aligned dimensions automatically fall back to CPU/SIMD.

🚀 ColBERT v2 Reranker Configuration

Parameter Default Range Description
enableReranker false true/false Toggle token-level late interaction reranking
rerankerDepth 20 5–100 Number of first-stage candidates to rerank

Tip

ColBERT reranking executes in off-heap memory using Panama SIMD vector kernels. It typically adds less than 1ms of latency, providing cross-encoder precision with minimal overhead.

🖥️ Server Configuration

Parameter Default Description
port 7070 HTTP server port
apiKey — Optional API key (empty = no auth)
corsOrigins * Allowed CORS origins 
# Start Spector Synapse server on port 7070 with custom API key
SPECTOR_API_KEY=my-secret-key mvn -Psynapse -pl synapse/spector-synapse spring-boot:run

🌐 Cluster Configuration

Parameter Default Range Description
shardCount 2 2–256 Number of data shards
replicaCount 1 1–5 Replicas per shard
heartbeatInterval 2s 500ms–30s Cluster heartbeat interval
heartbeatTimeout 10s 3s–120s Node unavailability timeout
queryTimeout 10s 1s–60s Per-shard query timeout

Tip

Rule of thumb: 100K–500K docs per shard for optimal balance. Set heartbeatTimeout to at least 5× heartbeatInterval.

🧠 Memory Configuration

Operating Mode

Parameter Default Options Description
mode MEMORY MEMORY Unified memory operating mode (incorporates search and cognitive retrieval)

Memory Tier Parameters

Parameter Default Range Description
nodesPerPartition 10,000 1,000–1,000,000 Records per semantic partition file
workingCapacity 100 10–10,000 Working memory slots (volatile circular buffer)
episodicPartitionCapacity 10,000 1,000–100,000 Records per episodic partition
semanticCapacity 5,000 100–1,000,000 Single-file semantic capacity (in-memory mode)
proceduralCapacity 500 10–100,000 Procedural memory slots

Strength & Recall Telemetry Parameters

Parameter Default Range Description
spector.memory.strength.enabled true true/false Enable the dedicated off-heap strength memory region (RegionId.STRENGTH)
spector.memory.strength.stride-bytes 96 64–256 Record stride in bytes for each strength entry (DEFAULT_MEMORY_STRENGTH_STRIDE_BYTES)
spector.memory.strength.auto-ltp-cooldown-ms 300000 (5 min) ≥ 0 Cooldown window in ms between passive Auto-LTP reinforcement passes
spector.memory.strength.auto-ltp-storage-increment 0.05 0.0–1.0 Storage strength $S(t)$ increment applied during Auto-LTP passes
spector.memory.strength.reserved-bytes 16 0–64 Trailing reserved bytes per strength record for forward compatibility

Note

These configuration keys configure the dedicated strength region (RegionId.STRENGTH). Default values are supplied via SpectorPropertyConstants.DEFAULT_MEMORY_STRENGTH_*.

🗂️ Namespace Storage Layout & Routing

Parameter Default Range Description
spector.namespace.tenant-rooted.enabled true true/false Enables tenant-rooted namespace sharding layout (tenants/XX/YY/tenantId/namespaces/ZZ/WW/namespaceId) for tenanted accounts. Untenanted accounts (tenantId == null) always resolve to legacy flat sharded path (namespaces/XX/YY/namespaceId).
spector.namespace.dual-read.enabled true true/false Enables dual-read fallback from layout B (tenant-rooted) to layout A (flat) during migration window (Req R5.3). Never dual-writes (I6). Fallbacks increment spector.namespace.layout.fallback.

Note

Data Plane vs. Identity Plane Root Distinction:

  • spector.data-dir: The node-level data directory holding db/synapse.mv.db (catalog) and identity/ (character-prefix sharded soul & salience bundles).
  • spector.memory.persistence-path (or persistence-path): The rememberer persistence root (defaults to ${spector.data-dir}/cognitive in Synapse, or .spector/memory in embedded kernel). All data plane rememberers (NamespacePathResolver) resolve relative to this directory.

Retrieval Stack Parameters

Parameter Default Options Description
text-search-mode HYBRID HYBRID, KEYWORD_ONLY, VECTOR_ONLY, SPLADE, SPLADE_HYBRID, LI_LSR, COLBERT_RERANK, FULL_STACK Active retrieval layers and paths
bm25-enabled true true/false Enable SIMD-accelerated BM25 keyword matching
splade-enabled true true/false Enable SPLADE learned sparse retrieval
colbert-enabled true true/false Enable ColBERT v2 late-interaction reranking

Partitioned Semantic Storage

When using DISK persistence mode, semantic memories are stored in rolling partition files:

.spector/memory/semantic/
  semantic-000.mem     ← partition 0 (oldest, immutable)
  semantic-001.mem     ← partition 1 (immutable)
  semantic-002.mem     ← partition 2 (active, accepts writes)

Tuning nodesPerPartition:

  • Smaller partitions (1K–5K) → faster compaction, more parallel search threads, more files
  • Larger partitions (10K–50K) → fewer files, slightly lower overhead per partition
  • Default (10K) → good balance for most workloads

Tip

Existing single-file semantic.mem stores are automatically migrated to the partitioned format on first startup. No manual migration needed.

Cluster Replication for Partitions

Parameter Default Description
partitionReplicationEnabled false Enable file-level partition snapshot shipping
replicaCount 1 Replicas per shard (1–5)

When enabled, immutable semantic partitions are shipped as snapshots to replica nodes. Only the active (mutable) partition requires WAL-based delta replication.

📥 Ingestion & Chunking Configuration

Parameter Default Range Description
maxTokens 512 1–8192 Max tokens per chunk
overlapTokens 50 0–maxTokens-1 Overlap between chunks
embeddingBatchSize 32 1–256 Batch size for embedding generation
embeddingRetries 3 0–10 Retry count for failed batches

🎯 Configuration Examples

🎯 High-Recall Setup

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(500_000)
    .withQuantization(QuantizationType.SCALAR_INT8)
    .withM(32)
    .withEfConstruction(400)
    .withEfSearch(200);

🗜️ Balanced Compression (INT4)

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(50_000_000)
    .withQuantization(QuantizationType.SCALAR_INT4)
    .withRescore(3);  // default for INT4

💾 Maximum Compression (INT2)

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withCapacity(200_000_000)
    .withQuantization(QuantizationType.SCALAR_INT2)
    .withRescore(5);  // default for INT2

⚡ Low-Latency Setup

var config = SpectorConfig.DEFAULT
    .withDimensions(128)
    .withCapacity(100_000)
    .withM(12)
    .withEfConstruction(100)
    .withEfSearch(30);

🎮 GPU-Accelerated Batch Processing

var config = SpectorConfig.DEFAULT
    .withDimensions(768)
    .withCapacity(1_000_000)
    .withGpu(true)
    .withGpuMemoryBudget(2048);  // 2 GB

🤖 RAG Pipeline

var config = SpectorConfig.DEFAULT
    .withDimensions(384)
    .withMaxTokens(1024)
    .withOverlapTokens(100)
    .withEmbeddingBatchSize(64);

🌐 Cell High Availability & Ownership Ring (spector.cell.*)

Spector Synapse supports multi-node Cell High Availability via a Ketama consistent hash ring.

Property Environment Variable Default Allowed Values Description
spector.cell.id SPECTOR_CELL_ID null String (e.g. us-east-1) Cell identifier. Required when role != standalone.
spector.cell.role SPECTOR_CELL_ROLE standalone standalone, owner, replica, gateway Operational role. standalone short-circuits the ring and owns all namespaces locally.
spector.cell.node-id SPECTOR_CELL_NODE_ID Hostname String (e.g. pod-0) Node identity in the cell. Defaults to pod hostname for stable StatefulSet ordinals.
spector.cell.ring.version SPECTOR_CELL_RING_VERSION 1 Integer $\ge 1$ Generation of the Ketama hash ring.
spector.cell.ring.members SPECTOR_CELL_RING_MEMBERS [] List of Strings Explicit canonical member list of nodes participating in the hash ring.
spector.cell.ring.members-file SPECTOR_CELL_RING_MEMBERS_FILE null Path string Path to newline-delimited member list file (used in Docker Compose).

Important

Cell Topology Semantics & Single-Writer Invariants:

  • Dynamic Reload: Membership and ring configuration can be loaded at startup or reloaded dynamically without restart via control store updates.
  • Ownership vs. Placement: Namespace ownership (which server process is the authoritative single writer) is decided purely in-memory by the Ketama ring prior to namespace open. Placement (which filesystem directory holds the partition bundle) is determined by storage tenant paths (StoragePaths) and remains completely invariant to cluster routing changes (Invariant J6).
  • Fail-Closed: Non-owner nodes refuse both writes and recall with HTTP 421 Misdirected Request (NamespaceNotOwnedException) identifying the authoritative owner node and ring epoch. Non-owner nodes never invoke runtime.attach or map files into memory.

⚡ Cell Routing Cache & Gateway Resilience (spector.routing.*)

Spector Synapse accelerates routing resolution via a three-tier waterfall (L1 Caffeine $\to$ L2 Redis $\to$ L3 Ketama Hash Ring fallback), provides pub/sub invalidation, and implements gateway forwarding with bounded retry.

Property Environment Variable Default Allowed Values Description
spector.routing.redis.enabled SPECTOR_ROUTING_REDIS_ENABLED false Boolean Enables distributed Redis L2 routing cache and pub/sub invalidation bus. When false, cell operates seamlessly in degraded L1+L3 mode.
spector.routing.redis.uri SPECTOR_ROUTING_REDIS_URI redis://localhost:6379 URI string Redis connection URI with cluster hash-tag support. Credentials should be supplied via environment variable placeholders only.
spector.routing.redis.timeout-ms SPECTOR_ROUTING_REDIS_TIMEOUT_MS 100 Integer $\ge 10$ Short lookup timeout with fail-fast. A lookup slower than the recall it precedes defeats its purpose.
spector.routing.redis.ttl-seconds SPECTOR_ROUTING_REDIS_TTL_SECONDS 30 Integer $\ge 1$ Load-bearing staleness bound. Missed invalidations self-heal within one TTL. Not merely a hit-rate tuning knob.
spector.routing.caffeine.ttl-seconds SPECTOR_ROUTING_CAFFEINE_TTL_SECONDS 5 Integer $\ge 1$ Node-local and gateway L1 in-process cache TTL.
spector.routing.caffeine.max-size SPECTOR_ROUTING_CAFFEINE_MAX_SIZE 200000 Integer $\ge 1$ Maximum entries retained in the L1 Caffeine cache.
spector.routing.gateway.retry-max SPECTOR_ROUTING_GATEWAY_RETRY_MAX 2 Integer $\ge 0$ Maximum retry attempts on STALE_ROUTE (HTTP 421). Prevents retry storms under rebalances.
spector.routing.gateway.scheme SPECTOR_ROUTING_GATEWAY_SCHEME http http | https Default URL scheme applied when resolving owner node addresses that do not specify an explicit scheme prefix. Automatically upgrades to https when spector.routing.gateway.tls.enabled=true or TLS material is configured.
spector.routing.gateway.default-port SPECTOR_ROUTING_GATEWAY_DEFAULT_PORT 7070 TCP Port (1–65535) Default port appended when a ring member node identifier does not specify an explicit :port.
spector.routing.gateway.tls.enabled SPECTOR_ROUTING_GATEWAY_TLS_ENABLED false Boolean Enables HTTPS/TLS for internal gateway-to-owner forwarding hops. Fails closed at startup if certificate or key paths are unreadable or malformed.
spector.routing.gateway.tls.ca-cert-path SPECTOR_ROUTING_GATEWAY_TLS_CA_CERT_PATH "" File path Path to a PEM-encoded X.509 CA certificate bundle used to verify owner node server certificates.
spector.routing.gateway.tls.client-cert-path SPECTOR_ROUTING_GATEWAY_TLS_CLIENT_CERT_PATH "" File path Path to a PEM-encoded X.509 client certificate chain for mutual TLS (mTLS) authentication to owner nodes. Must be paired with client-key-path.
spector.routing.gateway.tls.client-key-path SPECTOR_ROUTING_GATEWAY_TLS_CLIENT_KEY_PATH "" File path Path to a PEM-encoded PKCS#8 private key corresponding to client-cert-path.
spector.routing.gateway.tls.trust-store-path SPECTOR_ROUTING_GATEWAY_TLS_TRUST_STORE_PATH "" File path Path to a PKCS#12 truststore for verifying owner node certificates (alternative to ca-cert-path).
spector.routing.gateway.tls.trust-store-password SPECTOR_ROUTING_GATEWAY_TLS_TRUST_STORE_PASSWORD "" String Password for trust-store-path.
spector.routing.gateway.tls.key-store-path SPECTOR_ROUTING_GATEWAY_TLS_KEY_STORE_PATH "" File path Path to a PKCS#12 client keystore for mutual TLS authentication (alternative to client-cert-path / client-key-path).
spector.routing.gateway.tls.key-store-password SPECTOR_ROUTING_GATEWAY_TLS_KEY_STORE_PASSWORD "" String Password for key-store-path.
spector.routing.gateway.tls.hostname-verification-enabled SPECTOR_ROUTING_GATEWAY_TLS_HOSTNAME_VERIFICATION_ENABLED true Boolean Enforces HTTPS endpoint hostname/SAN verification on owner node certificates during TLS handshakes.

Note

Operational Note — Redis Outage Behavior (Req R9.2):

  • What breaks when Redis is down: Latency increases slightly as lookups fall back to local pure-computation Ketama ring evaluation. Failover overrides are temporarily masked until the control store republishes.
  • What does NOT break: Single-writer correctness and write availability never break. The cell functions continuously as a single-writer system even if Redis crashes entirely (Invariant K1). Zero dual-writers, zero data loss. Operators should not page on transient Redis hiccups.
  • Production Topology (Req R9.3, R9.4): Managed cloud offerings (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis) are recommended for production; containerized Redis is for local testing. Redis instances must be provisioned per cell, never shared across multi-region cells.

🔄 Cell Snapshot Replication & Replica Recall (spector.replication.*)

Spector Synapse enables high-throughput asynchronous namespace replication from authoritative cell owners to clean replica nodes over a dedicated mutual TLS 1.3 transport, bounded-staleness replica recall, and sealed-partition deduplication.

Property Environment Variable Default Allowed Values Description
spector.replication.enabled SPECTOR_REPLICATION_ENABLED false Boolean Master switch for replication plane listener, coordinators, and shippers.
spector.replication.port SPECTOR_REPLICATION_PORT 9090 TCP Port (1024–65535) Dedicated replication network port (B2, Req R6.1). Must be distinct from public REST/gRPC API port.
spector.replication.bind-host SPECTOR_REPLICATION_BIND_HOST 127.0.0.1 IP address Local bind host. Defaults to loopback to prevent inadvertent public network exposure (Req R6.4).
spector.replication.snapshot-interval-seconds SPECTOR_REPLICATION_SNAPSHOT_INTERVAL_SECONDS 60 Integer $\ge 5$ Periodic replication debounce interval (Req R7.3).
spector.replication.snapshot-min-changes SPECTOR_REPLICATION_SNAPSHOT_MIN_CHANGES 100 Integer $\ge 1$ Minimum WAL modifications required to trigger debounced snapshot shipping.
spector.replication.max-replica-lag-seconds SPECTOR_REPLICATION_MAX_REPLICA_LAG_SECONDS 30 Integer $\ge 1$ Freshness bound for bounded-staleness replica recall (Req R10.2). Read requests exceeding this lag are rejected with HTTP 412 (Invariant N7).
spector.replication.full-resync-lag-threshold-seconds SPECTOR_REPLICATION_FULL_RESYNC_LAG_THRESHOLD_SECONDS 300 Integer $\ge 10$ Lag threshold triggering full snapshot resynchronization instead of incremental tail chase (Req R8.3).
spector.replication.replica-reads-enabled SPECTOR_REPLICATION_REPLICA_READS_ENABLED false Boolean Safety Pin: Gateway replica read fan-out defaults strictly false (Req R10.3). Clients must explicitly opt-in with X-Spector-Allow-Replica: true.
spector.replication.replica-hot-cap SPECTOR_REPLICATION_REPLICA_HOT_CAP 100 Integer $\ge 1$ Maximum number of namespaces concurrently kept hot in-memory on a replica node before unmapping cold replicas.

Important

Replication Plane Invariants & Consistency Contracts:

  • Default OFF & Opt-In Reads (Req R10.3): Replica reads default OFF. A client that does not explicitly pass X-Spector-Allow-Replica: true will never receive stale reads, even if replica infrastructure is healthy.
  • Three Simultaneous Conditions (Req R10.1, N7): A replica serves recall if and only if: (1) namespace is locally mapped, (2) $(\text{now} - \text{snapshotTime}) \le \text{maxReplicaLagSeconds}$, and (3) incoming request permits replica reads. If staleness bound is violated, the replica refuses rather than serves stale data (ReplicaStalenessExceededException).
  • Unconditional Write Refusal (Req R10.6, Invariant N2): A replica never writes. Any write operation dispatched to a replica is rejected immediately with HTTP 405 (ReplicaWriteRefusedException). The X-Spector-Allow-Replica header widens read consistency, never write routing.
  • HWM-Advance-Last Crash Safety (Req R5.2, N5): The replica apply engine extracts bundles to temporary staging, fsyncs, cryptographically verifies magic 0x534D4B4D, layout 0x42554E44, and SHA-256 digests against the manifest, atomically renames files into place, replays WAL records, and advances the High-Water Mark last. A crash during transfer or apply leaves existing data intact and corrupt bytes unpublishable.

🛡️ Cell Failover, Fencing & Coordination (spector.failover.*, spector.coordinator.*, spector.control-store.*)

Spector Synapse provides lease coordinator election, monotonic write fence tokens, automated monitored failover, dynamic ring reloads, and maintenance draining across multi-node cells.

Property Environment Variable Default Allowed Values Description
spector.failover.enabled SPECTOR_FAILOVER_ENABLED false Boolean Safety Pin: Master switch for automated failover orchestration. Defaults strictly false.
spector.failover.mode SPECTOR_FAILOVER_MODE observe_only observe_only, active Staged rollout mode. observe_only evaluates and logs structured audit records without mutating cluster topology.
spector.failover.fail-after-seconds SPECTOR_FAILOVER_FAIL_AFTER_SECONDS 15 Integer $\ge 1$ Continuous readiness probe failure duration before declaring owner node down. Prevents false positives from single missed probes.
spector.failover.cooldown-seconds SPECTOR_FAILOVER_COOLDOWN_SECONDS 60 Integer $\ge 1$ Hysteresis bound suppressing repeated promotions on flapping or oscillating nodes.
spector.coordinator.lease-duration-seconds SPECTOR_COORDINATOR_LEASE_DURATION_SECONDS 15 Integer $\ge 5$ Duration of renewable cell coordinator lease held in the control store.
spector.coordinator.renew-interval-seconds SPECTOR_COORDINATOR_RENEW_INTERVAL_SECONDS 10 Integer $\ge 1$ Background heartbeat interval for coordinator lease renewal.
spector.override.ttl-seconds SPECTOR_OVERRIDE_TTL_SECONDS 300 Integer $\ge 10$ Time-to-live for explicit namespace override leases pinning namespaces to survivor nodes.
spector.control-store.type SPECTOR_CONTROL_STORE_TYPE in-memory in-memory, file, k8s Single-source-of-truth store holding membership, coordinator lease, and override records.
spector.control-store.file-path SPECTOR_CONTROL_STORE_FILE_PATH null Path string Filesystem path for durable JSON control store when control-store.type=file.

Important

Fencing, Single-Writer Invariance & Recovery Guarantees:

  • Monotonic Fence Tokens: Every write request carries the current namespace fence token (X-Spector-Fence). Writing to a partitioned, stale, or superseded owner node is rejected with HTTP 409 Conflict (FENCED). Validation is in-memory, allocation-free, and I/O-free.
  • Precedence Over Hash: Active override leases in the control store strictly supersede consistent hash ring resolution. Hash ring resolution resumes only after safe lease expiration or explicit release.
  • Verify Before Promote: Survivor candidate data is cryptographically and structurally validated before promotion. If validation fails, promotion is refused to prevent defect propagation into data loss.
  • In-Flight Request Expectations: In-flight requests holding region leases on a failed node die with the process. Clients receive connection drops or refusals and retry transparently against the newly promoted survivor.
  • Volatile Memory Handling: In-memory working state is lost during ungraceful node crash. Durable committed records are preserved with zero data loss (RPO = 0). Clients detect failover transitions via the incremented namespace epoch.

☸️ Kubernetes Cell Topology & Resource Isolation (spector.cell.*, spector.pager.*)

Spector cells separate cluster responsibilities into three distinct roles: owners (authoritative single-writers managing live namespaces and local NVMe storage), replicas (read/warm followers maintaining pre-warmed namespace subsets), and gateways (stateless API routers).

Property Environment Variable Default Allowed Values Description
spector.node-role SPECTOR_NODE_ROLE owner owner, replica, gateway, standalone Cluster execution role defining write authority and data ownership boundaries.
spector.cell-id SPECTOR_CELL_ID cell-1 String identifier Unique cell cluster identifier preventing cross-cell collision in multi-tenant environments.
spector.node-id SPECTOR_NODE_ID Hostname String identifier Stable pod node identity used for consistent hash ring mapping.
spector.pager.hot-cap SPECTOR_PAGER_HOT_CAP 2000 Integer $\ge 1$ Maximum number of active memory namespaces mapped concurrently per owner node.
spector.pager.warm-cap SPECTOR_PAGER_WARM_CAP 20000 Integer $\ge 1$ Secondary tiered namespace capacity backed by local NVMe storage.
spector.data-dir SPECTOR_DATA_DIR /data Filesystem path Storage root for persistent bundles, write-ahead logs, and identity records.

Important

Host Kernel & Operating System Prerequisites:

  • Filesystem Selection: Local storage directories must be formatted with XFS or ext4. Off-heap memory-mapped (mmap) page cache flushing and durability guarantees are strictly validated on these filesystems. Non-standard filesystems trigger startup warnings.
  • Map Count Arithmetic: Required memory mappings are calculated using layout arithmetic: $\approx (\text{hotCap} \times 2) + \text{WAL segments} + \text{overhead}$. Each open namespace utilizes ~2 maps (the runtime bundle and the active partition bundle).
  • Kernel Tuning Parameters:
    • vm.max_map_count: Set to at least 262144 (or 1048576 for dense multi-tenant deployments). An under-provisioned host fails readiness probes at startup.
    • vm.swappiness: Set to 1 to prevent memory-mapped working sets from paging to swap disk.
    • ulimit -n: Set to at least 262144 to prevent descriptor exhaustion under high connection and bundle volume.
    • memlock: Configured as unlimited for memory-locking preambles.
  • Memory Parity Invariant: StatefulSet owner configurations must enforce requests.memory == limits.memory == 16Gi (with 2 GiB heap). This prevents Linux cgroups v2 from evicting clean page-cache pages prematurely under transient memory pressure.
  • Inter-Pod Anti-Affinity: Owner pods require strict host anti-affinity (topologyKey: kubernetes.io/hostname) to prevent multiple owner replicas from co-locating on the same physical compute node.

🛡️ Cell Disaster Recovery & Compliance (spector.dr.*)

Disaster recovery provides cross-cell resilience and snapshot exports to S3-compatible object stores for mutable active namespaces. Remote uploads pace background bandwidth to avoid competing with foreground client traffic, enforce geographic residency boundaries, and preserve verifiable bundle integrity.

Property Environment Variable Default Allowed Values Description
spector.dr.export-enabled SPECTOR_DR_EXPORT_ENABLED false true, false Master toggle enabling periodic snapshot exports of active mutable namespaces to object storage. Defaults to disabled.
spector.dr.export-interval-seconds SPECTOR_DR_EXPORT_INTERVAL_SECONDS 900 Integer $\ge 60$ Export cadence defining the mutable snapshot Recovery Point Objective (RPO). Default is 900 seconds (15 minutes).
spector.dr.bandwidth-limit-bytes-per-sec SPECTOR_DR_BANDWIDTH_LIMIT_BYTES_PER_SEC 52428800 Long $\ge 0$ Token-bucket rate-limiter bandwidth cap in bytes per second (default 50 MB/s; 0 disables throttling).
spector.dr.alert-lag-multiplier SPECTOR_DR_ALERT_LAG_MULTIPLIER 2 Integer $\ge 1$ SLA lag multiplier before emitting high-severity export lag alerts (e.g., $2 \times 900\text{s} = 1800\text{s}$).
spector.dr.object-store.endpoint SPECTOR_DR_OBJECT_STORE_ENDPOINT null URL string REST endpoint for S3-compatible cloud object store (AWS S3, MinIO, Cloudflare R2).
spector.dr.object-store.bucket SPECTOR_DR_OBJECT_STORE_BUCKET spector-dr String identifier Destination bucket name holding exported snapshot manifests and bundles.
spector.dr.object-store.region SPECTOR_DR_OBJECT_STORE_REGION us-east-1 Cloud region string Geographic region identifier validated against organization data residency pins.
spector.dr.object-store.access-key SPECTOR_DR_OBJECT_STORE_ACCESS_KEY null Secret string AWS SigV4 HMAC access key identifier.
spector.dr.object-store.secret-key SPECTOR_DR_OBJECT_STORE_SECRET_KEY null Secret string AWS SigV4 HMAC secret access key.

Important

Disaster Recovery, Integrity & Compliance Invariants:

  • Atomic Manifest Publication: Partition bundles and payload files are transferred to object storage first. The manifest.json descriptor is uploaded strictly last to guarantee atomic visibility. Restorations ignore partial or unmanifested epochs.
  • Strict Verification Before Serving: Restoring nodes validate bundle preamble magic (0x534D4B4D), layout identifiers (0x42554E44), and SHA-256 payload checksums. Corrupted or mismatched snapshots are strictly rejected and refuse to serve.
  • Explicit Standby Promotion: Standby cell activation requires human decision-making with mandatory --force acknowledgment and audit trail logging. Automatic failover across cell boundaries is forbidden to eliminate split-brain write corruption.
  • Compliance Erasure & Legal Hold: Tenant deletion executes physical recursive wiping across local NVMe storage and prefix purging across cloud object storage. Namespaces tagged with legal hold strictly refuse deletion.
  • Data Residency Enforcement: Org-to-cell residency bindings are enforced during both export and restore. Any transfer violating regional jurisdiction triggers hard refusal.

📊 Memory Analytics (spector.memory.analytics.*, spector.memory.stats.*)

Spector Synapse supports namespace-isolated memory analytics snapshots, periodic telemetry persistence, and stats caching (ADR-0083).

Property Env Var Default Constraints Description
spector.memory.analytics.history.enabled SPECTOR_MEMORY_ANALYTICS_HISTORY_ENABLED true Boolean Enables the background scheduler that periodically captures per-namespace memory analytics snapshots to the H2 database. When disabled, the MemoryAnalyticsScheduler bean is not created and no snapshot rows are written. Prometheus scraping of live Micrometer metrics is unaffected. (ADR-0083)
spector.memory.analytics.history.interval SPECTOR_MEMORY_ANALYTICS_HISTORY_INTERVAL 10000 Integer $\ge 1000$ Flush interval in milliseconds for the analytics snapshot scheduler. Controls how often per-namespace census and activity deltas are persisted. (ADR-0083)
spector.memory.analytics.instance-id SPECTOR_MEMORY_ANALYTICS_INSTANCE_ID local Non-empty string Instance identifier written to the instance_id column in snapshot rows. Used for multi-pod disambiguation in Kubernetes deployments. (ADR-0083)
spector.memory.stats.cache-ttl SPECTOR_MEMORY_STATS_CACHE_TTL 5s Duration (Spring format) TTL for the namespace-scoped getStats() and getScoringStats() cache. Lower values increase engine scan frequency; higher values reduce load but serve staler data. (ADR-0083) 
# ── Memory Analytics (ADR-0083) ────────────────────────
spector:
  memory:
    analytics:
      history:
        enabled: true          # set false to disable H2 snapshot writes
        interval: 10000        # ms between snapshot flushes
      instance-id: local       # pod identity for composite PK
    stats:
      cache-ttl: 5s            # namespace stats cache TTL

🔗 See Also

🏠 Home

Clone this wiki locally