Skip to content

DOCUMENTATION.md

Latest commit

 

History

History
2355 lines (1848 loc) · 120 KB

DOCUMENTATION.md

File metadata and controls

2355 lines (1848 loc) · 120 KB
Raw

📘 GreenBoost v3.4 , integration guide for inference tools

GreenBoost is a CUDA Memory & Compute Orchestrator for NVIDIA GPUs: it tiers system RAM/NVMe into VRAM, quantizes models to fit (gb-quant), and pools LAN machines into one virtual device. This guide shows how to wire each inference tool into it.

Check GREENBOOST_COMMANDS.md for all the available commands. Check CHANGELOG.md for what changed in each version.

📑 Table of contents (click to expand)

Getting oriented

Framework integration guides

Direct / advanced control

Debugging & telemetry

Performance & features

Other

Reading this document

Two audiences read this file:

  • Newcomers who installed GreenBoost and want to make a specific inference tool work. Jump straight to your tool's section (Ollama, vLLM, PyTorch, Hugging Face, …). The first paragraph of each section is a copy-pasteable recipe; everything after that is optional tuning.
  • Engineers integrating GreenBoost into infrastructure who want the architectural model - phase detector, KV-cache pinning, cluster fabric, security. Read § How GreenBoost hooks in first, then the § Cluster and § Debug Vitals sections.

If you only want to use GreenBoost with one inference tool, you can skip the architecture sections entirely. The shim is transparent - your existing Python script does not change.

What's new in v3.5

The release where the 12 GB RTX 5070 was measured end to end. Every figure below was measured on that box; full detail in CHANGELOG.md.

  • GPU expert cache for MoE models. gb_moe_cache.py sizes llama.cpp's --moe-cache-mib from the free VRAM and serve() keeps all experts on the host with the hot ones cached on the GPU: Qwen3.8-35B-A3B decode 54 -> 104-115 tok/s, using less VRAM. It needs an engine with upstream PR 29887; the installer builds one (synapse build-moecache-engine, synapse use-engine moecache|vendored). MTP stays on with the cache at draft depth 2 (181 tok/s against 128 for the cache alone, measured; GB_MOE_CACHE_MTP=0 drops MTP for a bigger cache). Engine builds pin the CUDA toolkit nvcc belongs to and refuse an engine whose libggml-cuda links a different libcudart major (a mixed toolkit reads cudaDeviceProp wrongly and aborts with CUDA error: invalid argument); see workflow/overnight-improvements.md.
  • fp8 for convolutions. gb_quant_conv.py: conv weights as fp8 with per-channel scales (first/last conv kept, 6% error guard, 256 MiB gate), and quant_advisor returns fp8_paths, the advice per model kind for the GPU it runs on.
  • Telemetry and advice (round 2). The sizing decisions are dataflux events: moe_cache_plan (inputs, chosen cache MiB, --n-cpu-moe, MTP on/off with the reason, engine), engine_runtime_check (CUDA runtime vs nvcc major), sdpa_backend, compile_decision, quant_conv_result; dataflux_critic names the fix for a cache under 1 GB or a MoE served on the engine without --moe-cache-mib. synapse_status carries an engine block (active engine, version, flag support, last cache decision) and the new read-only gpu_advice tool lists the 5070-class recommendations for the detected GPU. greenboost doctor adds the toolchain checks (workflow/round2-improvements.md).
  • Speed levers. gb_sdpa.py (cuDNN attention first on cc 12.x), gb_compile.py (regional torch.compile, automatic from 8 images) and gb_cnn_tune.py (channels_last).
  • Serving. First-class prefill batch/ubatch, the shim on by default with a safety probe, vision serving priced before launch, GPU phase leases, MoE placement fixes, session persistence.
  • Measured and left alone. The GPU power limit never binds for decode.

Workflow notes: workflow/gb-moe-cache.md, workflow/gb-quant-conv.md, workflow/missing-5070-features.md.

What's new in v3.4

The release where GreenBoost started being able to explain itself. Full detail in CHANGELOG.md.

Inference speed. Decode on the reference workload roughly doubled (4.6 → 9.4 tok/s) by raising MTP speculative-draft depth , free throughput on a bandwidth-bound forward pass rather than a quality tradeoff, since the pass costs the same wall time regardless of how many tokens it emits. A second round chasing another 2x found the hard ceiling and landed ~9% instead, which is reported as ~9% rather than dressed up. The larger win was outside decode entirely: GB-CLI's system prompt interleaved volatile content (date, git status, live tier state) immediately after the static block, capping the reusable prefix at ~12%; reordering it took real conversations to 100% cached prompt tokens. MoE expert offload via --n-cpu-moe reached 45.72 tok/s on a model whose experts fit that pattern.

PCIe 5 was investigated and ruled out, with the evidence recorded so it is not re-litigated: the root port's LnkCap2 is a read-only capability register reporting a 16 GT/s ceiling, and the GPU's LnkCtl2 already requests 32 GT/s and still negotiates down. There is no runtime register left to write.

GB-Semantics. Segment verdicts became reactive (semantic_transition events on transition only, riding the existing snapshot tick rather than adding a poller) and three-valued , matched / clear / unknown , because a telemetry failure that reads as a clean bill of health is the failure mode the layer exists to prevent. never_use traps now carry the incident that proves each one, checks/check_semantics_coverage.py blocks work that adds a failure mode without the metric that would reveal it, and an evaluator-contract test covers every segment through the public API.

GB-CLI. Pause and resume a served model mid-session, returning the VRAM (slot KV save/restore: 740 ms to save 54,377 tokens, 537 ms to restore). Plan mode implements the plan it just wrote. The running-action line reports the action, a live elapsed timer and cached hardware vitals. Terminal rendering was rebuilt around a single serialized writer, a display-width measurement that counts columns rather than codepoints, and a live-region suspension so a multi-row block cannot be painted through.

The boot image stopped hiding a stale kernel module. initramfs.conf carries MODULES=most, so initramfs-tools baked greenboost.ko into the initrd, whose own systemd-modules-load inserted it before switch-root. A reinstall rebuilds /lib/modules and never touches that image, so the machine kept loading the frozen copy , v3.2 across several reboots while v3.4 was the installed build, with rebooting the one action guaranteed not to help. Loading that early also meant /var was unmounted, so Tier 3 failed open on every boot (T3 backing file unavailable: -2 - T3 disabled) while the status page kept printing the configured 73 GB cap. Three fixes, because one was not enough: initramfs/zz-greenboost-exclude keeps the module out of the boot image and the installer regenerates it (with the matching removal in the purge path); greenboost_boot_guard.sh now compares the loaded module's srcversion against the installed .ko's instead of gating on lsmod | grep greenboost , presence is not identity, and gating on presence is why the self-heal reported success throughout , and it refuses to unload while refcnt > 0, emitting the kmod_version_drift kind rather than killing a live CUDA process over a version number. Governed as kmod_load_stage / t3_enabled / t3_capacity_configured_mb, the kmod_stale_from_initramfs and t3_configured_but_disabled segments, and a never_use trap on GbSnapshot.loaded and t3_total_mb: both read healthy while the subsystem they describe is wrong.

The supervisor stopped breaking AppArmor. gb_supervisor._patch_apparmor() appended two libgreenboost_audit.so mr, rules to every snapd-generated snap-confine.snapd.* profile. Those are complete profiles; an append lands past the final closing brace, where apparmor_parser expects profile NAME {, so it grants nothing and makes the whole profile unparseable , two dead profiles, snapd.apparmor.service failed, systemctl is-system-running stuck at degraded. The installer learned this on 2026-07-27 (when it took down Firefox, Chromium, snap-store and cups system-wide) and was fixed then; the supervisor was not, and reintroduced it on 2026-08-20. It now writes only the local/ override , which snapd never rewrites and apparmor includes INSIDE the profile block , creating the file when absent rather than skipping, and emits the apparmor_remediation kind (files_written, parser_ok, snapd_profiles_skipped, reason) because a function that edits system security policy and reports nothing is how this went unnoticed twice. The trap worth carrying: grep libgreenboost_audit <profile> returning a hit is the FAILURE signal, not confirmation the grant is active.

Host memory became observable. The OOM killer took the terminal twice on 2026-08-18 while both greenboost.ko's T2 guard and the supervisor's _RamMonitor had already computed critical ten minutes earlier , to the journal, where nothing consumed it. Host pressure is now the host_mem_pressure dataflux kind, the host_oom_imminent governed segment, and a status-line segment that appears only when it means something. gb_memwatch.py samples top RSS consumers with cmdline and smaps_rollup so the next incident names the process rather than "python3".

Serving and security. GB-Synapse's proxy moved off :11435, gained bearer-token auth and a bind guard that refuses a non-loopback bind without one. The NemoClaw audit landed an engine adapter registry, a declarative per-agent tool policy, GreenBoost-native serving recipes with digest pinning, typed pre-serve probes, preset-driven dispatch, and a readiness contract behind greenboost doctor --json.

Known and unfixed, stated rather than left to discover: the process behind those OOM kills is still unidentified; a 15.85 GB model on an 11.9 GB card still streams ~4 GB per token across PCIe and sits near 5 tok/s, which is arithmetic rather than a defect; and --cache-reuse remains broken upstream for this hybrid architecture.

What's new in v3.3

  • Selective-SSM (Mamba/Mamba-2) support, both phases. GreenBoost's memory model, telemetry, and semantics layer now understand recurrent (fixed-size, non-KV-growing) state as a first-class thing across BOTH the GGUF/llama.cpp path (gguf_summary()) and the safetensors/torch-core path (safetensors_summary()), for hybrid architectures (Qwen3.5/3.6 Gated DeltaNet) and pure-recurrent ones alike:
    • gb_synapse.ModelEntry gained n_recurrent_layers, ssm_d_conv, ssm_conv_width, ssm_state_elems, is_recurrent_only - n_kv_layers == 0 is now distinguishable as "genuinely no attention" vs. the old silent "unknown, assume every layer is attention" ~4x KV-cache overestimate.
    • estimate_ssm_state_gb() sizes the recurrent state itself (previously charged as zero bytes everywhere), wired as a fixed cost (never scaled by ctx) into _solve_ctx_and_layers/_clamp_ctx_to_budget/ _fit_gpu_layers/recommend().
    • The shim gained GREENBOOST_SSM_STATE_MB - a size-independent classification signal that pins small recurrent-state allocations in T1 (GB_ALLOC_KV_CACHE|GB_ALLOC_T1_PRIORITY) instead of leaving them GB_ALLOC_ACTIVATIONS-classified and T2/T3-spill-eligible.
    • New ssm_state dataflux kind + ssm_state_gb/recurrent_layer_fraction governed GB-Semantics metrics (with never_use traps for the exact misreads this fix closes).
    • Native Mamba-2 model support in gb-synapse's own torch-core engine (synapse_engine/gllm/models/mamba.py, new) - not just accounting for recurrent state, but actually serving a pure-recurrent checkpoint end to end. The SSD chunked-scan and decode kernels are vendored verbatim from vLLM's own Triton port (itself from the Mamba-2 reference implementation) rather than hand-written, on the same reuse-tested-kernel principle as the existing GPTQ/AWQ Triton dequant path. Four pre-existing gLLM engine assumptions (every one some variant of "a real KV page pool always exists") were found and fixed along the way - the first time any purely-recurrent model has run through this engine at all.
  • gb_synapse.pull()'s GGUF quant matching is a plain substring match against an unordered file listing - confirmed live to be ambiguous for more quant-name variants than previously documented (:MTP-IQ4_XS matched three different files in one real repo). Pass the exact, full quant token for any repo shipping multiple -XXX-<quant> suffixed files of the same base quant.
  • Fixed a wrong CUDA attribute constant that was silently degrading serving to partial CPU offload. greenboost_cuda_shim.c and greenboost_vmm_override.c hardcoded CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED as 193 instead of the real value, 102 (GB_ATTR_VMM_SUPPORTED in greenboost_vmm_override.c). A new _gate_cpu_offload() check (gb_synapse_backends.py) now refuses any capacity-driven GPU-layer reduction by default, emits a cpu_spillover dataflux event either way, and only serves degraded under an explicit GB_SYNAPSE_ALLOW_CPU_OFFLOAD=1 override.
  • KV cache VRAM reservation is now measurement-backed, not formula-only. estimate_kv_gb() ran ~2.9x too generous for this release's reference hybrid architecture even with the n_kv_layers fix applied. _persist_kv_measurement()/_load_kv_measurement() (gb_synapse.py) cache the real shim-observed KV size per (model, ctx, kv_type) in /var/lib/greenboost/synapse/kv_measurements.json and feed it into the next serve's reservation. Live-verified at ctx=65536: VRAM fill 67-73% to 85.1%, decode 2.6-4.3 to 5.27 tok/s, T2 overflow ~12.1-12.3GB to 10.3GB.
  • greenboost clear memory-pool / gb clear-memory-pool now identify leftover inference processes from GB-Dataflux's own record of what GreenBoost started, instead of matching process names against a fixed pattern list. Shared implementation: gb_reclaim.py (classify_processes, plan_reclaim, run_reclaim, _recent_dataflux_pids).

What's new in v3.2

The 3.2 cycle's headline is greenboost cluster: the first release where it works well enough to actually recommend, not just experiment with. See CHANGELOG.md for the full writeup and a real measured result (a diffusion pipeline going ~7.8× faster by putting a second GPU to work). Everything else this cycle builds on the v3.1 telemetry/orchestrator foundation:

Cluster (greenboost_netd.c, greenboost_netc.c, gb_cluster.py, new)

  • Feeder link reliability overhaul: dedicated heartbeat/reconnect thread, chunked large transfers, zombie-daemon cleanup, node-parity checks so a host and feeder running mismatched builds are caught at connect time.
  • gb_cluster.py - a stable Python API (shim_env, feeders, run_stage_on_feeder, offload_tail_blocks, parallel_map_with_feeder) any PyTorch pipeline can import to use a feeder's GPU for a whole stage or the tail of a model.
  • New key-management and daemon commands: feeders genkey, feeders export-key / import-key, feeders redeploy-netd.
  • Running a CUDA kernel directly on a feeder against data that already lives there has been demonstrated for a couple of basic operations, but it isn't reliable enough for real inference workloads yet , that stays research-in-progress and is not part of what this release ships.

gb-synapse , HuggingFace-native, cluster-distributed GGUF serving (gb_synapse.py, gb_synapse_api.py, new)

  • Pulls GGUFs directly from any HuggingFace repo (gated or public, given a token), indexes GGUFs Ollama already downloaded, and serves either through a matched llama.cpp build on every cluster node.
  • Cross-node split rides llama.cpp's own --rpc backend (layer-granular, only activations cross the wire) while the GreenBoost shim keeps extending each node's own share into that node's local RAM/disk underneath it.
  • One port (:11369) speaks Ollama's API, OpenAI's API, and HuggingFace TGI at once, so existing tooling doesn't need to change. See § gb-synapse below and GREENBOOST_COMMANDS.md.
  • Binds 127.0.0.1 with no auth by default (every local consumer keeps working unchanged); GB_SYNAPSE_BIND/GB_SYNAPSE_TOKEN (or /etc/greenboost/synapse_token, 0600) opt into a LAN/container-reachable bind, and a non-loopback bind with no token refuses to start (gb_synapse_api.py's main()). synapse_ps/ServerState report bind/auth_enabled per running server; a rejected request emits a synapse_auth dataflux event.

greenboost-dataflux MCP server (gb_dataflux_mcp.py), grown from 3 tools to 8

  • Read-only queries over the dataflux log for any MCP-compatible AI assistant: dataflux_summary, dataflux_events (now filterable by event kind), dataflux_errors, plus four new ones , dataflux_kinds, dataflux_tier_moves, dataflux_quantization, dataflux_tok_s. See § dataflux below.
  • recipe_override (2026-08-18, group synapse) fires when a serving recipe's MEASURED value is discarded because the caller passed an explicit one. The precedence is deliberate; the silence was not. Live the same day: greenboost-cli passed llamacpp_n_ctx=65536 from its config, the recipe's measured 16384 was dropped without a word, and a model that FITS this card at 16384 served with VRAM at 52.3% and 11.5 GB overflowing to T2 , rule1_underfilled caught the result while nothing named the cause. Pass ctx=0 to let a recipe govern.
  • driver_reset (2026-10-10, group health) records a supervisor driver reload decision: the kernel NVRM: Xid codes that triggered it (xid_codes) and the modules reloaded (modules). A reload now needs a kernel Xid line with code 95, 119 or 120, a 6-hour cooldown and no process holding /dev/nvidia*.
  • host_mem_pressure (2026-08-18, group health) records every host RAM/swap pressure TRANSITION from gb_supervisor._RamMonitor , distinct from GreenBoost's own T2 accounting, because CPU-offload streaming and any non-GreenBoost consumer take ordinary system RAM the shim never sees. It exists because the monitor already computed that state and wrote it only to the journal and a status file: on 2026-08-18 it went critical twice (MemAvailable 4044 MB = 6.6% of 61 GB, past its own 8% floor) about ten minutes before the OOM killer killed a 27 GB python3 and took the user's terminal with it, and nothing downstream could see either warning. Pairs with the governed segment host_oom_imminent (severity violation , it does not slow a session down, it ends it) and the metrics host_mem_available_gb / t2_oom_guard_state.

Reactive orchestration (gb_orchestrator.py, gb_control.py, new gb_reactive.py, gb_topology.py)

  • Expanded from the 3 feedback loops introduced in v3.1 to cover thermal stress, memory-bandwidth stress, clock throttling, and a predictive KV-cache grower gated behind six independent safety checks.
  • Opt-in continuous OS tuning for the root daemon (CPU governor, GPU clock/power limits, kernel memory tunables), reversible with the new greenboost tune-revert. Off by default (GB_ORCH_ACTUATE=1 to enable) - GreenBoost only observes and reports until you turn it on.

dataflux , a flight recorder for the whole system (gb_dataflux.py, new)

  • Continuous local event log covering VRAM/GPU/CPU/temp/power, KV-cache pressure, memory-tier movement, quantization decisions, and per-machine cluster throughput once a feeder is connected.
  • greenboost dataflux-ui (or python3 gb_dataflux.py serve) opens a live, auto-refreshing web page at :8799; works standalone, no feeder required. See § dataflux below.

Telemetry (gb_telemetry.py), polished through daily use

  • PCIe link-health tracking, host-level pressure metrics (CPU/memory/I/O via the kernel's pressure-stall accounting), feeder-aware telemetry so a connected feeder's GPU health rolls into the same picture as the local one, a no-pip-dependency GPU-metrics fallback, and a measured-tokens-per-second signal that closes the loop between orchestration decisions and the actual speed a model delivered.

Hot/cold residency + observability

  • The kernel evictor now prefers genuinely cold buffers over pure LRU order (see full writeup in CHANGELOG.md), with a new eBPF-based tracer and three new commands: greenboost top, greenboost residency, greenboost faults.

Two experimental, honestly-reported performance layers

  • gb_prefetch.py - overlaps loading a transformer's next layer with computing the current one. Hasn't beaten the no-prefetch baseline on a gb-quant-placed model in testing so far; useful for models gb-quant hasn't already placed.
  • gb_diffcache.py - a TeaCache/DeepCache-style activation cache for diffusion models that skips recomputing a step when little has changed.

Fixes

  • Full Install no longer reinstalls Ollama on every run - it checks the latest release first and only updates when actually behind.

AI memory OS tranche + polish roundup

  • gb_monitor.py - canonical read-only telemetry/capability client unifying the shim_stats parsers and ioctl mirrors that had grown independently. Backs greenboost capabilities (§ below), which prints the installed/ running shim's feature manifest.
  • gb_pilot.py - a read-only "instrument panel" over the dataflux log; turns recorded trends into evidence-backed advice naming the exact gb_control lever to pull, never actuates anything itself. New greenboost pilot command and MCP tool.
  • gb_dataflux_mcp.py gained greenboost_status, greenboost_capabilities, and greenboost_pilot tools alongside the existing dataflux-log tools.
  • gb_synapse_tools.py - text-based tool-call injection for GGUFs that don't emit native OpenAI tool_calls, shared by gb-synapse serving.
  • gb_synapse_fallback.py - minimal single-request OpenAI-compatible server (transformers + gb-quant quantize-to-fit); gb-synapse's fallback engine when its torch-core engine (vendored gLLM) can't take a checkpoint or isn't installed.
  • gb_placement.py - fp8-floor cluster-fit planner (GB_PLACEMENT=1); prefers a connected feeder over dropping below fp8, and arbitrates between GGUF RPC tensor-split and PyTorch offload_tail_blocks.
  • gb_kernel_backends.py - pluggable low-bit GEMM backend selector for gb-quant (GB_KERNEL_BACKEND: GemLite default, scaled_mm fp8, bf16 passthrough, CUTLASS reserved).
  • gb_nvml_ctypes.py - pynvml-compatible NVML binding over ctypes, so telemetry works without nvidia-ml-py pip-installed.
  • Cluster fabric zstd compression on host-to-device payloads, negotiated at handshake (protocol v3, backward compatible with v2 feeders).
  • Phase-aware KV prefetch stats mode (GREENBOOST_KV_PREFETCH), exposing kv_prefetch_* telemetry counters.
  • CUTLASS sm_120a NVFP4 scaffold (third_party/gb_cutlass, GB_CUTLASS_ENABLE) - groundwork only, not in the default kernel path.
  • TurboQuant Triton autotune persistence + turboquant_attention_from_env.
  • Proxy-owned tok/s measurement on gb-synapse's Ollama-surface streaming.
  • Four default-safe efficiency wins: version-scoped gb-quant autotune cache (opt-out GB_QUANT_NO_AUTOTUNE_CACHE=1), transparent zstd for T3 checkpoints (opt-out GB_T3_COMPRESS=0), phase-aware T1 workspace reserve (GREENBOOST_T1_WORKSPACE_MB, default off), opt-in tiered precision (GB_QUANT_TIERED_PRECISION=1 - fp8 stays the default).

821 tests green.

What's new in v3.1

The 3.1 cycle stitches a reactive telemetry layer into the CUDA shim and replaces four installed-but-inert systemd daemons with a single unified Python runtime:

Runtime stack (new modules)

  • gb_supervisor , one long-running daemon that subsumes greenboost-recovery, greenboost-sentinel, greenboost-vram-watchdog, and greenboost-idle-reclaim. One process, one log stream, zero redundant pynvml connections.
  • gb_orchestrator , signal-driven actuator that closes three feedback loops previously left open: ECC responder, VRAM pressure ratchet, idle-reclaim gate.
  • gb_control , the single place that mutates GreenBoost runtime state, backed by sysfs → ioctl → /run/greenboost/control in that priority order.
  • gb_nvml , unified pynvml singleton shared by all of the above (was duplicated across four modules with divergent query sets).
  • gb_reactive , stdlib-only Signal/Computed primitives (debounce, EWMA, hysteresis) used by the orchestrator; no external dependencies.
  • gb_vitals_helper , in-process pynvml for the vitals/cluster commands, removing the nvidia-smi subprocess fork that previously ran on every refresh tick.

Shim

  • greenboost_cuda_shim.c extended to publish live ECC, power, and PCIe health signals at 500 ms resolution , previously these were logged only, never acted on.

Fixes

  • vmm_override PLT symbol resolution fixed for Blackwell (cc 12.x) targets.
  • greenboost_setup.sh refactored: duplicate code consolidated, TUI refresh loop de-spammed, feeders upgrade-greenboost atomic install path hardened.

What's new in v3.0

The 3.0 cycle brought a multi-month hardening campaign on the cluster fabric and the LD_PRELOAD shim, a full Python quantization stack (gb-quant), and embedded DCGM telemetry. No new user-visible APIs to learn for existing workflows, but plenty of fixes you'll notice in production:

Security

  • Cluster fabric upgraded from proto v3 to proto v4 with mutual authentication: both the host and each feeder prove knowledge of the PSK. A man-in-the-middle who only replays a prior handshake can no longer impersonate either side.
  • Every message after the handshake carries a truncated HMAC-SHA256 computed over (header ‖ payload) keyed by an HKDF-derived session key. Tampering is detected at receive time, not after parsing.
  • A v4 host still talks to a v3 feeder if GREENBOOST_PSK_V4=0 is set (so you can roll feeders one at a time); default is v4-only.

Robustness

  • 117-test pytest suite covering wire-protocol, PSK loading, LAN filter, HKDF, exported symbols, and end-to-end handshake.
  • Kernel-module DMA path holds a dma_resv reservation across every device mapping, with active_mappings reference counting so the eviction worker can never free memory the GPU is reading.
  • cuFuncGetParamInfo (CUDA 12.3+) probes kernels for argument arity, so cuLaunchKernel argument scanning has correct bounds instead of guessing.
  • The hot allocation path is fully big-endian-safe - every wire field is read via GB_LE_U16/U32/U64 accessors. The full BE port still requires opt-in (-DGREENBOOST_BE_INCOMPLETE=1).
  • CUDA 13 / cu130 compatibility: the shim no longer fails with cudaErrorDeviceUninitialized (998) when frameworks call cuMemGetInfo_v2 or cuDeviceTotalMem_v2 before the first cudaSetDevice. On INVALID_CONTEXT (201) the shim falls back to the runtime API which retains the primary context transparently.
  • Clang-built kernel support: the Makefile auto-detects CONFIG_CC_IS_CLANG=y and passes LLVM=1 to the kernel sub-make, fixing build failures on CachyOS, Arch/clang, and Gentoo/clang kernels. dkms.conf now sets explicit MAKE/CLEAN so DKMS auto-rebuilds on kernel updates also use the correct toolchain.

New tools

  • greenboost stability - a Python long-running monitor that watches /run/greenboost/shim_stats for monotone-counter regressions, tier-gauge leaks (memory still allocated after idle), and fragmentation drift over hours of inference.
  • greenboost feeders diag - host-side T1/T2/T3 + compute test against a remote feeder.
  • A worker-pool scaffold inside greenboost-netd (opt in with GREENBOOST_WORKERS=N) so a slow CUDA op on one client no longer blocks the whole reactor.

Decoupled

  • The experimental Vulkan layer that let games use the GreenBoost pool has been moved into a sister project: GreenBoost Gaming Suite. The CUDA-inference path stays small and auditable; gamers get a GTK4 GUI instead of CLI. See § Sister project: GreenBoost Gaming Suite.

MCP & A2A surface (current , supersedes tool counts quoted in per-version notes above)

GreenBoost exposes five MCP servers (each registered in .mcp.json for a fresh clone, or via greenboost register-mcp for the installed path) plus one A2A gateway. Per-version bullets above (e.g. "grown from 3 tools to 8") describe that release's delta only , this section is the live count.

Server Entry file Tools
greenboost-dataflux gb_dataflux_mcp.py 23 , event-log queries (dataflux_*), plus mirrored greenboost_status/greenboost_capabilities/greenboost_pilot/synapse_status/tiering_status. All 21 tools declare MCP readOnlyHint (2026-08-19) , this server only ever answers questions, and saying so in the protocol is what lets a client overlap a batch of lookups instead of running them serially. GB-6's spec_decode (what the model's own MTP draft head is paying at the depth being served , depth, acceptance rate, the decode rate it produced; depth is non-monotonic, so it is chosen by sweeping gb_bench_spec.py, never by taking the largest) and GB-1's cache_index are queryable here too (which slot a conversation was routed to, and the first content-addressed chunk that changed , changed_chunk: null means the turn only appended and must cost nothing). tuner_decision (one control-loop tick , the lever, the action, and the reason it was or was not taken) is queryable here too. Three agent-harness kinds are queryable here: agent_tool_schema_miss (the model called a tool by a name it was not shown , unknown_instrument is the hallucinated-tool signal, and the before/after any schema rename is judged on), agent_context_edit (a history compaction, with how many leading messages stayed pinned , correlate against prompt_cache to catch a compaction that moved the reusable prefix), agent_eval_run (greenboost-cli's own agent benchmark score), and agent_memory_recall (project memory pulled into a turn , rules counts standing corrections recalled unconditionally, scoped counts memories surfaced because the turn touched their subsystem, and chars is what recall spent of the window), and agent_prefix_shift (GB-1 , the assembled prompt changed somewhere other than the trailing user turn; first_changed names the earliest chunk that moved, which is the one that invalidated everything after it. Reuse is all-or-nothing here because --cache-reuse is rejected for this hybrid architecture, and the 14-day baseline puts a lost prefix at ~166s time-to-first-token against ~5.5s for a kept one). The 14-day baseline that motivated all three: a turn reusing >=99% of its prefix reached first token in ~5.5s, one in the 50-90% band took ~140s Two session-scoped tools close the gap that per-kind queries structurally cannot: dataflux_sessions lists the serving/agent sessions on record (discovered from activity gaps in the session-defining kinds and widened to the bracketing shim phase transitions, so model-load time belongs to the session , snapshots deliberately never extend one, or the always-on SnapshotRecorder would fuse every run since boot into a single session), and dataflux_session_audit reconstructs ONE session end to end: decode against this box's own historical median for the same model/ctx/kv key, the prompt-cache and prefill curve including what the cold first turn cost, VRAM/tier trajectory, segment transitions, gate runs, tool calls, errors by kind, and graded findings that each carry evidence and one action. It exists because the findings that matter are cross-kind and invisible in any single-kind view: on 2026-08-20 a 41.9-minute run read as flat "slow decode" everywhere until the timeline showed a 283-second cold prefill (11% of the session, ~51 tok/s on 14,507 tokens at 0% cache) and a decode median 2.6x under the same box's baseline for a byte-identical serve config. Each run emits the session_audit kind, so "was that run audited, and what did it conclude" is a query rather than a re-run
greenboost-orchestrator gb_mcp.py 27 , gpu_advice (per-GPU serving advice with evidence), gpu_phase_status/gpu_phase_acquire/gpu_phase_release (shared GPU phase lease, acquire/release double-gated), greenboost_overview(compact=True), optimize_inference, quant_advisor, tuner_tick (one tick of the control loop , settle cycles, latching hysteresis, an EMA baseline with a minimum population, frozen domains with re-probe, and harvest: give the power limit back when decode is bandwidth-bound and holding it buys no throughput. Read-only advice unless double-gated with apply=True, confirm=True under GB_ORCH_ACTUATE=1; every tick emits a tuner_decision kind, including the ones that decided to hold, because "why did nothing happen" is what a control loop is usually asked. The decision half lives in gb_tuner.py and imports no hardware, so its edge cases are unit tests rather than things found live), advisories (NemoClaw audit, Phase 4 — unified gb_advisories.py contract over gb_pilot/gb_dataflux/gb_mcp producers; a blocking/fatal result emits an advisory dataflux kind), gb_plan (CB-3 tier-plan), gated actuation (tier_actuate, set_quant_policy, run_under_greenboost, a2a_gateway), support_bundle (NemoClaw audit round 2, item 11 — redacted diagnostic tarball via gb_debug_bundle.py, dry-run by default; a support_bundle dataflux kind on every real attempt), shim_env, the GB-Semantics governed layer (semantic_metrics/semantic_resolve/semantic_segments/semantic_answer, backed by gb_semantics.py), plus mirrors of the status/capabilities/pilot/synapse_status/cluster_status/dataflux_summary tools (shared impl in gb_mcp_common.py) for "one server suffices"
greenboost-cluster gb_cluster_mcp.py 7 , live feeder/VRAM/T2/T3 state, gated cluster_dispatch/cluster_ensure_feeder_ready
greenboost-synapse gb_synapse_mcp.py 15 , serving control (synapse_serve takes vision/mmproj/image_min_tokens/image_max_tokens; the serve emits dataflux kinds vision_fit, serve_recovery, proxy_limits), synapse_recommend, gated serve_and_repoint, CLI bridge, synapse_recover_cache (2026-09-01 — re-registers HF-cached models missing from the manifest, no download; adopts a matching serving recipe's pinned name over the raw filename stem, and names any duplicate-snapshot-revision file it deliberately skipped rather than picking one silently; each decision emits the model_recovery dataflux kind, status ok or skipped); synapse_doctor reports a nemoclaw block (CLI-on-PATH + foreign-:11369-owner detection, see the maintainer notes) AND a recipes block (serving-recipe schema/digest coverage, NemoClaw audit Phase 5e — see serving/recipe.schema.json; each row also carries model_resolves (2026-09-01, tri-state — null when the manifest can't be read, never a bare false standing in for "don't know") since valid only means the recipe file itself passed schema+digest and said nothing about whether the model it pins can actually be served — this box had two valid: true recipes pinning models absent from the manifest, one of them the reference workload); synapse_serve gains recipe= (explicit serving-recipe path, wins over ctx/mtp_draft_n), use_preset= (NemoClaw audit Phase 7 follow-up — hands use_cluster to the resolver's live-facts resolution outright; dry-run preview shows the winning preset before anything commits), and kv_type= (2026-08-10 — per-call --cache-type-k/-v override, wins over both the live budget-driven choice and a recipe's kvCache; closes the gap where forcing a KV type required a raw extra_args flag that llama.cpp's own parser silently dropped in favor of the backend's own emitted flag — gb_synapse_backends._dedupe_extra_args_overrides() fixes that path too); synapse_recommend gains preview_recipe= (attach each model's matching recipe, if any, without serving); new synapse_resolve_preset (NemoClaw audit Phase 7) — read-only preview of which serving/presets/*.yaml preset current live facts (kmod/topology/cluster) resolve to, with a hard error (not a silent pick) on a priority tie between two eligible presets A route_decision kind records gb_router's per-turn model choice (turn class, model, and WHY , a routing heuristic nobody can audit is worse than none). The spread it routes across is measured: MoE with expert offload 45.72 tok/s vs dense Qwen3.8-27B 2.7-3.5 on the same box. gb_router.check_pool_is_local() refuses a non-local endpoint outright per CLAUDE.md's Local-First Must-Rule , the mechanism is copied from NemoClaw's router/pool-config.yaml, the targets deliberately are not. A kv_quality kind records gb_bench_kv.py runs , what quantizing the KV cache actually costs, scored with NIAH recall rather than a smoke gate (quantized WEIGHTS fail by collapsing into repetition, which a smoke gate catches; quantized KV stays fluent and quietly loses long-range retrieval, which only a recall gate sees). Running it required fixing the proxy to forward llama.cpp's read-only native routes (/tokenize, /detokenize, /props) , niah_certify() sizes its haystack with /tokenize and had been getting a 404, so quality certification could not run at all. A turn_bench kind records gb_bench_turn.py runs , the before/after every inference-speed change is measured against, splitting a turn into real prefill and estimated recurrent replay (the hybrid architecture re-runs its 48 Gated DeltaNet layers over the whole prompt every turn even at a 99.9% KV hit rate; measured 2026-08-18, median 6,937 ms prefill for 42 new tokens). Governed as recurrent_replay_ms + the prefill_dominated_by_replay segment. Governed segment verdicts are also reactive now: gb_semantics_watch.SegmentWatcher gives every segment in semantics/segments.yaml a gb_reactive.Signal (the same Signal + confirm= hysteresis + pipe(pairwise()) idiom gb_orchestrator.py already uses) and emits the semantic_transition dataflux kind ON TRANSITION ONLY, riding the SnapshotRecorder tick rather than adding a second poller. evaluate_segment() is a pull API and nothing polled it, so a matched segment could sit unreported indefinitely , on 2026-08-18 rule1_underfilled was matched while Rule #1's own text requires a tripwire to fire on exactly that condition. Verdicts are three-valued (matched/clear/unknown): unknown is tracked separately because a telemetry failure must never read as a clean bill of health.; new synapse_pause / synapse_resume / synapse_paused (2026-08-18) , pause saves every populated slot's KV state via llama.cpp's /slots?action=save, stops the engine and returns the VRAM; resume re-serves with the same parameters and restores the state. Both DRY-RUN unless confirm=True, same protection as synapse_stop. Exists because an idle session holds most of the card for nothing , measured on the reference box, 10.4 GiB of 12 GB at 0% GPU utilization, 787 MHz, 23 W. Slot save/restore makes that reclaimable rather than destructive: 740 ms to save 54,377 tokens, 537 ms to restore. pause refuses while a request is generating unless force=True, because llama.cpp cannot checkpoint a half-finished generation. Emits the serve_pause dataflux kind (action pause
greenboost (CLI) greenboost-cli/greenboost_cli/mcp/server.py 16 , RAG/goals/history/factory

A2A gateway (gb_a2a.py, installed + enabled by Full Install as the greenboost-a2a.service systemd unit , loopback-only, actuation gate off, until an operator opts in): JSON-RPC 2.0 over HTTP, GET /.well-known/agent.json (legacy AgentCard shape) AND GET /.well-known/agent-card.json (A2A protocol v0.3 shape , interop with ai-forge's studio/server/a2a_gateway.py, which serves the same path). Loopback-bound by default (GB_A2A_TOKEN required for a LAN bind). Verbs map 1:1 to gb_actuation.VERBS (now including run_under_greenboost) so MCP and A2A share the same double-gated (confirm=True + GB_ORCH_ACTUATE=1) dispatch , A2A can never actuate anything an MCP tool couldn't. Liveness + recent requests queryable via the a2a_status tool on greenboost-dataflux; unit-level status/restart via a2a_gateway on greenboost-orchestrator.

Keep this table current when a server's tool count changes , see checks/check_mcp_parity.py (once landed) for the mechanical version of this check.

Build compatibility

GreenBoost builds without manual flags on all supported configurations. The build system auto-detects the environment:

Environment Detection Behaviour
CUDA 12 and 13 side-by-side `ls /usr/local/cuda-[0-9]* sort -V`
Clang-built kernels (CachyOS, Arch/clang, Gentoo/clang) CONFIG_CC_IS_CLANG=y in $(KDIR)/.config Appends LLVM=1 to all kernel sub-make targets (Makefile + DKMS)
GCC-built kernels no CONFIG_CC_IS_CLANG Standard GCC path, unchanged
No versioned CUDA dirs fallback /usr/local/cuda → /usr/cuda → /opt/cuda

If you have both CUDA 12 and CUDA 13 installed and want to force a specific version, set CUDA_DIR=/usr/local/cuda-12 make before building.

How GreenBoost hooks in

GreenBoost works by intercepting cudaMalloc and related CUDA symbols via LD_PRELOAD, plus intercepting dlsym so that the virtual VRAM total (T1 VRAM + T2 DDR pool, computed from detected hardware) is returned to any app that queries VRAM size at runtime through dlopen+dlsym.

The shim is injected per-process , never system-wide via /etc/ld.so.preload. Using /etc/ld.so.preload forces the CUDA interposer into every process on the system including systemd PID 1, which freezes early boot. Injection happens via two safe mechanisms instead:

  • systemd service drop-ins , Environment="LD_PRELOAD=…" in /etc/systemd/system/<service>.d/99-greenboost.conf
  • Wrapper scripts , greenboost-run, greenboost-run-tgi, etc. export LD_PRELOAD only for the target command.

GREENBOOST_ACTIVE=1 still gates whether the shim does any work. In interactive login shells (terminal, SSH), /etc/profile.d/greenboost.sh exports GREENBOOST_ACTIVE=1 automatically , no wrapper needed. Just run your script directly:

python your_script.py               # shim already active in login shells
python -m vllm.entrypoints.openai.api_server ...

In non-login contexts (cron jobs, Docker entrypoints, sudo scripts, spawned subshells) profile.d is not sourced. Use the wrapper or set the variable explicitly:

# Wrapper - sets GREENBOOST_ACTIVE=1 for one command
greenboost run python your_script.py

# Or inline
GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so python your_script.py

# Or in a systemd unit / Docker environment
Environment="GREENBOOST_ACTIVE=1"
Environment="LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so"

The five GreenBoost layers

GreenBoost is not a single library , it is five composable layers, each solving one specific problem in the memory-limited GPU inference stack. A beginner AI engineer can adopt them one at a time:

┌─────────────────────────────────────────────────────────────────┐
│  Layer 5 · gb_diffusion_orch.py , diffusion pipeline orchestrator │
│  (multi-component management: VAE, CLIP, UNet/DiT)               │
├─────────────────────────────────────────────────────────────────┤
│  Layer 4 · gb_synapse_fallback.py , LLM inference quantization  │
│  (HuggingFace Transformers, apply gb-quant at load time)          │
├─────────────────────────────────────────────────────────────────┤
│  Layer 3 · gb_quant.py , weight quantize-to-fit                  │
│  (shrink weights to fit VRAM: bf16 → int8 → int4 → tq3 → tq2)   │
├─────────────────────────────────────────────────────────────────┤
│  Layer 2 · gb_init.py + gb_telemetry.py , bootstrap + telemetry  │
│  (single import wires all layers; ECC guard; GPU metrics)         │
├─────────────────────────────────────────────────────────────────┤
│  Layer 1 · CUDA shim + kernel module , memory tier extension      │
│  (transparent cudaMalloc overflow to DDR/NVMe/cluster)            │
└─────────────────────────────────────────────────────────────────┘

Layer 1 , CUDA shim + kernel module

What it solves: Your model is 33 GB, your VRAM is 12 GB. cudaMalloc would fail and the framework falls back to slow CPU math.

How it works: libgreenboost_cuda.so is loaded via LD_PRELOAD before your Python process starts. It intercepts cudaMalloc and, when the allocation would exceed VRAM, routes it to one of three hardware paths (in order of speed):

  • Path A , DMA-BUF pinned DDR (requires greenboost.ko): the kernel module exports hugepages as a DMA-BUF, which CUDA imports as a device pointer via cuImportExternalMemory. The GPU reads DDR at PCIe speeds (~25 GB/s). The CPU executes zero tensor math.
  • Path B , cuMemHostRegister (no kernel module): works inside Docker, VMs, WSL2. Slightly higher latency than Path A but portable.
  • Path C was removed. UVM/managed memory caused GPU page-fault stalls.

The framework (Ollama, vLLM, PyTorch) sees a single GPU with inflated total memory: VRAM + T2_DDR_pool + T3_NVMe_pool. It schedules the full model onto that virtual device. No code changes required in your Python script.

Under the hood: The shim uses .symver trampolines (greenboost_cuda_v12.c) to export both the @@libcudart.so.12 default and @libcudart.so.12 non-default symbols, so frameworks linked against CUDA 12 or CUDA 13 both resolve correctly. See greenboost_documentation_extension_official_nvidia.md for the full technical explanation of where GreenBoost departs from NVIDIA's documented behaviour (inflated cuDeviceTotalMem, DMA-BUF interop details, data-driven feeder dispatch).

Layer 2 , gb_init.py + gb_telemetry.py (bootstrap + telemetry)

What it solves: Starting a Python inference session requires wiring together GPU metrics, an ECC error guard, and the memory-tier singletons. Without a single entry point each project re-invents this boilerplate.

gb_init.py is a one-import bootstrap:

import gb_init  # that's it

# What happens automatically:
# • torch.cuda.empty_cache → no-op (prevents conflict with the DynamicVRAM allocator)
# • TelemetryManager started (500 ms poll)
# • ECC DBE callback installed (stderr warning on uncorrectable memory error,
#   before inference silently corrupts outputs)
# • gb_stream_sched / gb_model_tier / gb_mem_pool singletons exposed

gb_telemetry.py uses a provider chain:

Provider Requires What it reports
NVMLProvider pynvml power, temp, VRAM free/total, GPU utilisation
DCGMProvider dcgm Python bindings ECC single/double-bit errors (FI 310/311/313), NVLink BW (FI 449), health check (PCIe / SM / Memory / Thermal / …)
GreenBoostProvider /dev/greenboost T1/T2/T3 usage via ioctl
TorchFallbackProvider torch.cuda last-resort VRAM numbers

The embedded DCGM mode (dcgmStartEmbedded) means no dcgmd daemon is needed on bare-metal workstations or cluster nodes.

gb_init.pre_inference_check() blocks inference if an ECC double-bit error has occurred since last boot , GPU memory is corrupted, output is wrong, better to know before wasting 10 minutes of generation.

Layer 3 , gb_quant.py (weight quantize-to-fit)

What it solves: Even with T2 DDR overflow, streaming large weights over PCIe each token costs bandwidth. If the weights fit entirely in VRAM at a lower precision, every token is fast.

How it works: quantize_module shrinks model weights using a quality-first planner , each component gets the highest precision that still fits the VRAM budget:

bf16  (16 bits) → highest quality, largest
int8  ( 8 bits)
int4  ( 4 bits) ← default sweet spot, HQQ backend
tq3   ( 3 bits) ← TurboQuant: Triton LUT-GEMM, rotated activations
tq2   ( 2 bits) ← TurboQuant: most compact, useful for huge models

The low-bit GEMM backend (GemLite + HQQ) ships inside GreenBoost (third_party/, Apache-2.0). Your venv needs nothing extra.

from gb_quant import quantize_module

model = load_your_model()
model = quantize_module(model)   # planner picks the right precision for each layer

GB_QUANT_BUDGET_GB overrides the VRAM budget. A warm-kernel autotune cache (~/.cache/greenboost/) means Triton compilation only happens once.

Layer 4 , gb_synapse_fallback.py (LLM inference)

What it solves: Wiring gb-quant into HuggingFace Transformers with the right hooks and profiler annotations. gb-synapse's own torch-core engine (vendored gLLM) is the primary LLM serving path for continuous-batching throughput; this layer is what it falls back to for a checkpoint the torch core can't yet take, and what any script can import directly for a quick single-request load.

from gb_synapse_fallback import load_causal_lm

# Loads a HF model, auto-selects quantization, pins KV cache to the right tier
model, tokenizer = load_causal_lm("mistralai/Mistral-7B-v0.3")

NVTX range gb:llm_load:<model> is emitted around the load phase so Nsight Systems traces show exactly how long quantization + tier placement took.

Layer 5 , gb_diffusion_orch.py (diffusion pipeline orchestrator)

What it solves: Diffusion models (Stable Diffusion, FLUX, LTX-Video) have three or four sub-models that each allocate GPU memory: a VAE, a text encoder (CLIP / T5), and the main denoiser (UNet or DiT). Managing them as separate quantize_module calls leaves memory fragmentation gaps and misses the chance to tune per-component budgets.

gb_diffusion_orch.py treats the whole pipeline as one budget:

from gb_diffusion_orch import DiffusionOrchestrator

orch = DiffusionOrchestrator(pipeline)
orch.fit()      # quantizes each component to the right precision
orch.run(prompt="a cat", steps=20)

Uses the gb_init singletons for telemetry and only stops the telemetry thread if it was the one that created it (_tel_owned flag), so stacking multiple orchestrators in one session works correctly.

The official CUDA extension document

greenboost_documentation_extension_official_nvidia.md is GreenBoost's technical companion to the CUDA Programming Guide. It is written in the same register as NVIDIA's Programming Guide chapters 3 and 4, and explicitly calls out every place where GreenBoost combines documented NVIDIA primitives in ways NVIDIA never described:

  • Inflating cuDeviceTotalMem / nvmlDeviceGetMemoryInfo to include DDR + NVMe pools
  • cuImportExternalMemory(OpaqueFd) + DMA-BUF zero-copy pinned DDR (Path A)
  • cuMemHostRegister(DEVICEMAP) over GB_IOCTL_PIN_USER_PTR (Path A pinned sub-method)
  • Virtual device aggregation across TCP-connected feeder machines
  • cuLaunchKernel data-driven remote dispatch via fake pointer scanning
  • CUDA 13 invalid-context runtime fallback in cuMemGetInfo_v2 / cuDeviceTotalMem_v2

Read it if you are integrating GreenBoost into a new framework and need to understand precisely what the shim reports vs. what the NVIDIA driver would report.

Ollama

Handled automatically by install-sys-configs, which injects the shim and GREENBOOST_ACTIVE=1 into the systemd unit. No manual wrapper needed.

ollama run glm-4.7-flash:q8_0   # GreenBoost is transparent

If running Ollama outside systemd:

greenboost run ollama serve

vLLM

vLLM loads libcuda lazily through PyTorch (torch.cuda → ctypes.CDLL).

greenboost run python -m vllm.entrypoints.openai.api_server \
    --model /opt/models/glm-4.7-flash-hf \
    --dtype float16 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.95

As a systemd service, add to the unit:

[Service]
Environment="LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so"
Environment="GREENBOOST_ACTIVE=1"

PyTorch scripts

Any import torch; torch.cuda.* call triggers lazy loading of libcuda via ctypes.

greenboost run python your_inference_script.py

Or inline without the wrapper:

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python your_script.py

text-generation-inference (TGI)

TGI uses a PyTorch backend - same lazy-CUDA pattern:

greenboost run text-generation-launcher \
    --model-id /opt/models/glm-4.7-flash-hf \
    --num-shard 1 \
    --max-total-tokens 131072

As a systemd service:

[Service]
Environment="LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so"
Environment="GREENBOOST_ACTIVE=1"

CTranslate2

CTranslate2 loads libcudart via ctypes in its Python bindings:

greenboost run python your_ctranslate2_script.py

Hugging Face Transformers

Transformers loads CUDA through PyTorch - same lazy-CUDA pattern as vLLM and TGI.

greenboost run python your_transformers_script.py

With device_map="auto", Transformers queries available VRAM before placing layers. GreenBoost reports the detected T1+T2 total, so the full model is placed on the "GPU" (T1+T2) instead of being split to CPU:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "/opt/models/glm-4.7-flash-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto",          # sees T1+T2 total - loads entire model onto T1+T2
)
tokenizer = AutoTokenizer.from_pretrained("/opt/models/glm-4.7-flash-hf")

messages = [{"role": "user", "content": "Hello!"}]
input_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, return_tensors="pt", add_generation_prompt=True
).to(model.device)

output = model.generate(input_ids, max_new_tokens=300)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Pipeline API works the same way:

from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="/opt/models/glm-4.7-flash-hf",
    torch_dtype="bfloat16",
    device_map="auto",
)
print(pipe("Hello!", max_new_tokens=200)[0]["generated_text"])

Downloading a model then running it in one shot:

greenboost run python - <<'EOF'
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

snapshot_download("THUDM/glm-4.7-flash-hf", local_dir="/opt/models/glm-4.7-flash-hf")

model = AutoModelForCausalLM.from_pretrained(
    "/opt/models/glm-4.7-flash-hf", torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("/opt/models/glm-4.7-flash-hf")
ids = tokenizer("Hello!", return_tensors="pt").input_ids.to(model.device)
print(tokenizer.decode(model.generate(ids, max_new_tokens=100)[0]))
EOF

When using the GreenBoost venv:

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    /opt/greenboost/venv/bin/python your_transformers_script.py

ExLlamaV3

ExLlamaV3 loads CUDA through PyTorch:

greenboost run python your_exllama_script.py

When using the GreenBoost venv:

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    /opt/greenboost/venv/bin/python your_exllama_script.py

The GreenBoost-patched KV cache layer (CacheLayer_greenboost) allocates directly from /dev/greenboost and does not go through the shim's cudaMalloc path for cache tensors - GREENBOOST_ACTIVE=1 is still needed for virtual VRAM reporting.

PyTorch - direct tensor allocation control

PyTorch loads libcuda lazily via ctypes.CDLL when torch.cuda is first accessed. GreenBoost intercepts from that point forward.

greenboost run python your_script.py
# or inline:
GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so python your_script.py

Explicit KV cache flagging from Python

For custom inference loops where you know exactly which tensors are KV cache, use GB_ALLOC_KV_CACHE | GB_ALLOC_T1_PRIORITY via the MADVISE IOCTL so the kernel freezes them in T2's LRU and refuses T3 spill:

import fcntl, struct, os
from pathlib import Path

# GB_IOCTL_MADVISE = _IOW('G', 4, struct gb_madvise_req)  - 8 bytes
_IOC_MADVISE     = (1 << 30) | (ord('G') << 8) | 4 | (8 << 16)
GB_MADVISE_T1_PREFER = 3   # mark as T1-priority: freeze in T2 LRU, refuse T3 spill

def gb_mark_kv(buf_fd: int) -> None:
    """
    Tell GreenBoost that buf_fd (DMA-BUF fd from GB_IOCTL_ALLOC) is a KV cache buffer.
    Kernel will: freeze in T2 LRU, refuse T3 spill, set t1_priority.
    buf_fd is the fd returned by GB_IOCTL_ALLOC (stored in gb_alloc_req.fd).
    """
    dev = Path("/dev/greenboost")
    if not dev.exists():
        return
    # struct gb_madvise_req { s32 buf_id; u32 advise; }
    buf = struct.pack("iI", buf_fd, GB_MADVISE_T1_PREFER)
    fd  = os.open(str(dev), os.O_RDWR)
    try:
        fcntl.ioctl(fd, _IOC_MADVISE, buf)
    finally:
        os.close(fd)

# Usage with PyTorch (via ExLlamaV3 native IOCTL path)
# buf_fd is obtained from GB_IOCTL_ALLOC, not from cudaMalloc.
# For pure-PyTorch workflows, use GREENBOOST_KV_OVERFLOW=1 or the
# phase detector (automatic) instead of calling this directly.

Optimal loading order

GreenBoost's phase detector expects: weights first, KV cache second. Respect this order to get maximum T1 utilization:

import torch

# 1. Load model weights (phase = MODEL_LOAD - reserve inactive, weights fill T1)
model = MyModel().cuda()
model.load_state_dict(torch.load("model.pt", map_location="cuda"))

# 2. Allocate KV cache (phase = INFERENCE - reserve activates, KV lands in T1)
kv_cache = torch.zeros(batch, n_layers, seq_len, d_model, device="cuda")
# ↑ GreenBoost auto-classifies this as KV via quiet-gap heuristic
#   OR call gb_mark_kv() above for certainty

# 3. Run inference
output = model.generate(input_ids, past_key_values=kv_cache)

If your script loads KV cache before or interleaved with weights, set GREENBOOST_KV_OVERFLOW=1 to disable the heuristic.

Hugging Face Transformers - direct KV cache control

Transformers uses PyTorch's CUDA allocator; GreenBoost intercepts all cudaMalloc calls. device_map="auto" queries VRAM size - GreenBoost reports the detected T1+T2 total, so the full model is placed on T1+T2 instead of being split to CPU.

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python inference.py

Pinning the KV cache to T1

Transformers' DynamicCache allocates keys/values lazily during generation. To ensure they land in T1 rather than T2, set a generous KV reserve before generation starts and tell GreenBoost the phase is INFERENCE:

import os
os.environ["GREENBOOST_KV_RESERVE_MB"] = "4096"   # 4 GB reserved for KV
os.environ["GREENBOOST_KV_OVERFLOW"]   = "0"       # use phase detector (default)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "/opt/models/glm-4.7-flash-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto",          # sees T1+T2 total - all layers on GPU
    attn_implementation="flash_attention_2",   # requires OLLAMA_FLASH_ATTENTION=1 equivalent
)
tokenizer = AutoTokenizer.from_pretrained("/opt/models/glm-4.7-flash-hf")

# All tokens below will have KV cache allocated in T1 (phase detector: INFERENCE)
out = model.generate(
    tokenizer("Hello!", return_tensors="pt").input_ids.cuda(),
    max_new_tokens=512,
    use_cache=True,   # ensure KV cache is used
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Mark Transformers' KV cache explicitly (advanced)

If the phase detector misclassifies activations as KV, disable it and mark manually:

import os
os.environ["GREENBOOST_PHASE_DETECT"] = "0"    # disable auto-classification

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(..., device_map="auto")

# Monkey-patch _update_causal_mask to mark KV after each alloc
_orig_forward = model.forward

def _patched_forward(*args, **kwargs):
    out = _orig_forward(*args, **kwargs)
    # Walk past_key_values and mark each tensor as KV
    if out.past_key_values:
        for layer_kv in out.past_key_values:
            for tensor in layer_kv:
                if tensor.is_cuda:
                    gb_mark_kv(tensor.data_ptr(), tensor.nbytes)
    return out

model.forward = _patched_forward

Context-window KV reserve sizing

OLLAMA_NUM_CTX / max_new_tokens Recommended GREENBOOST_KV_RESERVE_MB
≤ 8 K tokens 1024 MB
8 K – 32 K tokens 2048 MB (default)
32 K – 64 K tokens 4096 MB
64 K – 128 K tokens 6144 MB
> 128 K tokens 8192 MB

vLLM - direct KV cache placement

vLLM manages its own block-based KV cache allocator (PagedAttention). It pre-allocates the entire KV cache up front from GPU memory before inference begins.

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python -m vllm.entrypoints.openai.api_server \
        --model /opt/models/glm-4.7-flash-hf \
        --max-model-len 131072 \
        --gpu-memory-utilization 0.95 \
        --enforce-eager

Why --gpu-memory-utilization 0.95 with GreenBoost

vLLM queries torch.cuda.mem_get_info() to calculate how many KV blocks it can allocate. GreenBoost intercepts this and returns T1 free space. At 0.95 utilization, vLLM will try to allocate ~11.4 GB of KV cache from "GPU" memory - GreenBoost routes any T1 overflow to T2 via DMA-BUF.

Increase KV reserve before launching vLLM so that vLLM's pre-allocated KV cache gets priority over leftover weight fragments in T1:

# Set 8 GB reserve before starting vLLM (131K context needs ~7–9 GB KV)
GREENBOOST_KV_RESERVE_MB=8192 \
GREENBOOST_KV_OVERFLOW=1 \
GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python -m vllm.entrypoints.openai.api_server \
        --model /opt/models/glm-4.7-flash-hf \
        --max-model-len 131072

GREENBOOST_KV_OVERFLOW=1 is recommended for vLLM because PagedAttention allocs KV blocks in a dedicated phase that doesn't follow the weight-then-KV timing the phase detector expects.

vLLM systemd service

[Service]
Environment="GREENBOOST_ACTIVE=1"
Environment="GREENBOOST_KV_OVERFLOW=1"
Environment="GREENBOOST_KV_RESERVE_MB=8192"
Environment="LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so"
ExecStart=python -m vllm.entrypoints.openai.api_server --model ...

Ollama 0.18+ and the CUDA VMM path

Ollama 0.18+ switched its ggml CUDA backend from cudaMalloc to the CUDA Virtual Memory Management API (cuMemCreate → cuMemMap → cuMemSetAccess). GreenBoost intercepts cuMemCreate transparently - no configuration change is needed. When a cuMemCreate call fails due to T1 being full, the shim retries with CU_MEM_LOCATION_TYPE_HOST, which creates a pinned host-memory (system DDR) allocation the GPU accesses over PCIe - the same effective result as Path B, but initiated from within the VMM code path.

TensorFlow / Keras

TensorFlow loads libcuda.so via ctypes when tf.config.list_physical_devices("GPU") is called. GreenBoost intercepts from that moment.

GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python your_tf_script.py

Memory growth vs pre-allocation

TensorFlow defaults to pre-allocating all GPU memory on startup. GreenBoost reports the detected T1+T2 total, which TF would try to allocate entirely - set memory growth instead:

import os
os.environ["GREENBOOST_ACTIVE"] = "1"
# LD_PRELOAD must be set before Python starts, not here

import tensorflow as tf

# Prevent TF from trying to allocate the entire T1+T2 virtual pool at once
gpus = tf.config.list_physical_devices("GPU")
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

# From this point forward cudaMalloc calls are GreenBoost-managed:
# weights → T1, KV-like buffers → T1-priority (if phase heuristic fires)
model = tf.keras.models.load_model("/opt/models/my_model.keras")

Manual KV cache marking for TF custom training

import ctypes, struct, os

def gb_mark_tensor_kv(tf_tensor) -> None:
    """Mark a TensorFlow GPU tensor as KV cache in GreenBoost."""
    # Get raw device pointer via experimental C API
    from tensorflow.python.framework import ops
    ptr = ctypes.cast(
        tf_tensor._handle(),  # internal: raw device pointer
        ctypes.c_void_p
    ).value
    if ptr:
        gb_mark_kv(ptr, tf_tensor.nbytes)

TF with XLA and GreenBoost

XLA compilations may pre-allocate scratch buffers that confuse the phase detector. Disable the heuristic for XLA workloads:

GREENBOOST_PHASE_DETECT=0 GREENBOOST_KV_OVERFLOW=0 \
GREENBOOST_ACTIVE=1 LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
    python xla_training.py

KV Cache Fine-Tuning - Bypassing the Phase Heuristic

GreenBoost auto-classifies CUDA allocations via a temporal phase detector (INIT → MODEL_LOAD → INFERENCE → STEADY). This works transparently for Ollama and llama.cpp, but some engines or custom workloads need explicit control.

When the heuristic needs help

Scenario Symptom Solution
ExLlamaV3 (native IOCTL path) KV allocs don't follow weight-then-KV timing GREENBOOST_KV_OVERFLOW=1
Batched inference servers Multiple parallel KV allocs confuse phase transitions GREENBOOST_KV_OVERFLOW=1
Very small models (<8K ctx) 2 GB default reserve wastes T1 for tiny KV GREENBOOST_KV_RESERVE_MB=512
Long-context (128K ctx) Default 2 GB reserve undersized; KV spills to T2 GREENBOOST_KV_RESERVE_MB=6144
Custom training loops Activation buffers wrongly classified as KV GREENBOOST_PHASE_DETECT=0
Verify KV location Want to confirm KV is in T1, not T2 Check kv_t2_mb in sysfs

Environment variable reference

All variables are read by the shim at startup and refreshed every 64 allocations from the kernel module. They can be set in the Ollama service drop-in, in shell exports, or prepended to any command.

Variable Type Default Description
GREENBOOST_KV_RESERVE_MB integer from kernel (2048) MB of T1 VRAM reserved exclusively for KV cache. Weights overflow to T2 sooner; KV lands in T1 at 336 GB/s instead of T2 at 32 GB/s. Auto-scaled from OLLAMA_NUM_CTX if unset: 8K→1 GB · 32K→2 GB · 64K→4 GB · 128K→6 GB · >128K→8 GB. Capped at T1−2 GB.
GREENBOOST_KV_OVERFLOW 0/1 0 Master bypass: when 1, every overflow allocation is flagged GB_ALLOC_KV_CACHE | GB_ALLOC_T1_PRIORITY - the phase detector is skipped entirely. Use for ExLlamaV3 or any engine whose overflow allocs are predominantly KV.
GREENBOOST_PHASE_DETECT 0/1 1 Disable the temporal phase classifier. All overflow allocs use generic flags. Combine with GREENBOOST_KV_OVERFLOW=1 for full manual control.
GREENBOOST_KV_SIZE_THRESHOLD_MB integer 64 Minimum alloc size (MB) to classify as KV during the INFERENCE phase. Raise to 256 if activation buffers are being wrongly pinned as KV.
GREENBOOST_VRAM_HEADROOM_MB integer 512 MB kept free in T1 as a safety buffer. Reduce to 256 to squeeze more weights into T1.
GREENBOOST_VIRTUAL_VRAM_MB integer computed Override the virtual VRAM total reported to CUDA apps (default: T1 + T2 in MB).
GREENBOOST_ACTIVE 0/1 1 if shim loaded Must be 1 for the shim to intercept allocations. Set to 0 to disable without unloading.
GREENBOOST_DEBUG 0/1 0 Enable verbose shim logging to stderr. Each alloc decision is logged: phase, flags, tier placement.

Quick recipes

# Ollama with a 128K context - increase KV reserve to 6 GB
GREENBOOST_KV_RESERVE_MB=6144 ollama run nemotron-3-super:120b

# ExLlamaV3 - bypass phase detector, all overflow is KV
GREENBOOST_KV_OVERFLOW=1 GREENBOOST_ACTIVE=1 \
  LD_PRELOAD=/usr/local/lib/libgreenboost_cuda.so \
  python your_exllama_script.py

# Debug KV placement (stderr shows each allocation decision)
GREENBOOST_DEBUG=1 ollama run glm-4.7-flash:q8_0 2>&1 | grep -E "KV|Phase|VRAM"

# Disable KV heuristic entirely for a custom training script
GREENBOOST_PHASE_DETECT=0 greenboost run python train.py

# Small model, shrink reserve to reclaim 1.5 GB of T1
GREENBOOST_KV_RESERVE_MB=512 ollama run llama3.2:3b

Runtime KV reserve update via IOCTL

The KV reserve can be changed at runtime without restarting Ollama. The shim polls GB_IOCTL_GET_INFO every 64 allocations and picks up the new value:

# Via Synapse CLI slash command (recommended)
/kv-reserve 4096

# Via greenboost_setup.sh (reads current then patches)
sudo ./greenboost_setup.sh tune-kv-reserve 4096

# Via Python (see synapse/greenboost.py)
from synapse.greenboost import set_kv_reserve
set_kv_reserve(4096)   # returns True on success

GB_ALLOC_* flags - programmatic allocation control

These flags are used by ExLlamaV3's CacheLayer_greenboost when calling GB_IOCTL_ALLOC directly on /dev/greenboost. They bypass the shim entirely and give the inference engine precise control over tier placement.

/* from greenboost_ioctl.h */
#define GB_ALLOC_WEIGHTS       (1u << 0)  /* model weight tensor             */
#define GB_ALLOC_KV_CACHE      (1u << 1)  /* KV cache - never spills to T3;
                                           * auto-frozen in T2 LRU           */
#define GB_ALLOC_ACTIVATIONS   (1u << 2)  /* ephemeral activation buffer     */
#define GB_ALLOC_FROZEN        (1u << 3)  /* never evict from T2             */
#define GB_ALLOC_NO_HUGEPAGE   (1u << 4)  /* force 4K pages (T3-spillable)   */
#define GB_ALLOC_T1_PRIORITY   (1u << 5)  /* KV-like; moved to LRU head,
                                           * weight bufs evicted first        */
#define GB_ALLOC_KV_COMPRESSED (1u << 6)  /* TurboQuant-compressed KV;
                                           * 3-7× more KV history in T2/T3  */

Flag semantics:

  • GB_ALLOC_KV_CACHE - marks the buffer as KV cache. The kernel auto-freezes it in the T2 LRU (so it is never evicted to T3) and refuses T3 spill with ENOSPC. KV bandwidth at T3 (~1.8 GB/s) would collapse generation to unusable speeds.

  • GB_ALLOC_T1_PRIORITY - combined with GB_ALLOC_KV_CACHE it sets t1_priority=1 on the buffer, making weight buffers preferred for eviction over this one. Always use together: GB_ALLOC_KV_CACHE | GB_ALLOC_T1_PRIORITY.

  • GB_ALLOC_KV_COMPRESSED - marks the buffer as TurboQuant-compressed KV. The kernel tracks compression savings separately (kv_compressed_mb, kv_compression_bits in pool_info). Set via GB_IOCTL_SET_TURBOQUANT first.

  • GB_ALLOC_FROZEN - prevents eviction from T2 regardless of type. Use for buffers that must never be paged out (e.g. LoRA adapter weights during inference).

Usage pattern (ExLlamaV3 native):

struct gb_alloc_req req = {
    .size_bytes = kv_size,
    .flags      = GB_ALLOC_KV_CACHE | GB_ALLOC_T1_PRIORITY,
    .tier_hint  = 1,   /* prefer T1 */
};
if (ioctl(gb_fd, GB_IOCTL_ALLOC, &req) == 0)
    kv_handle = req.handle;

Diagnosing KV spill

If KV cache spills from T1 to T2, the shim logs a warning to stderr and kv_t2_mb in sysfs becomes non-zero:

# Live monitoring
watch -n1 'grep kv /sys/class/greenboost/greenboost/status'

# One-liner: show KV location
python3 -c "
data = {k.strip(): int(v.strip())
        for line in open('/sys/class/greenboost/greenboost/status')
        if '=' in line for k, v in [line.split('=', 1)]}
kv_t2 = data.get('kv_t2_mb', 0)
kv_used = data.get('kv_used_mb', 0)
print(f'KV used: {kv_used} MB  |  KV in T2: {kv_t2} MB  |  KV in T1: {kv_used - kv_t2} MB')
print('✓ KV fully in T1' if kv_t2 == 0 else f'⚠ KV spilling to T2 - increase GREENBOOST_KV_RESERVE_MB')
"

Session Priority Management

For multi-model deployments where several inference processes share T2 DDR, GreenBoost provides two IOCTLs to control which process's buffers get evicted first when T2 runs low:

IOCTL Cmd Effect
GB_IOCTL_SESSION_IDLE 18 Move all caller-PID T2 buffers to LRU tail - first to be evicted under pressure
GB_IOCTL_SESSION_ACTIVE 19 Move all caller-PID T2 buffers to LRU head - last to be evicted

Call SESSION_IDLE when a model goes idle (no pending requests) and SESSION_ACTIVE when a new request arrives. This lets the active model keep its weights in T2 while the idle model's weights become eviction candidates.

import fcntl, struct, os

# GB_IOCTL_SESSION_IDLE   = _IOW('G', 18, struct gb_session_req)  - 8 bytes
# GB_IOCTL_SESSION_ACTIVE = _IOW('G', 19, struct gb_session_req)  - 8 bytes
_IOC_SESSION_IDLE   = (1 << 30) | (ord('G') << 8) | 18 | (8 << 16)
_IOC_SESSION_ACTIVE = (1 << 30) | (ord('G') << 8) | 19 | (8 << 16)

def gb_session_idle() -> None:
    """Mark this process's T2 buffers as low priority (idle model)."""
    _gb_session_ioctl(_IOC_SESSION_IDLE)

def gb_session_active() -> None:
    """Mark this process's T2 buffers as high priority (active model)."""
    _gb_session_ioctl(_IOC_SESSION_ACTIVE)

def _gb_session_ioctl(cmd: int) -> None:
    dev = "/dev/greenboost"
    if not os.path.exists(dev):
        return
    # struct gb_session_req { u32 pid; u32 reserved; }  - pid=0 means caller's PID
    buf = struct.pack("II", 0, 0)
    fd  = os.open(dev, os.O_RDWR)
    try:
        fcntl.ioctl(fd, cmd, buf)
    finally:
        os.close(fd)

Verify GreenBoost is active

# Should show T2 pool in use (non-zero) after loading a model larger than 12 GB:
cat /sys/class/greenboost/greenboost/status

# Confirm virtual VRAM is visible (should report T1+T2 total, not just physical VRAM):
# From an interactive terminal - GREENBOOST_ACTIVE=1 is already set by profile.d
python -c "
import torch
print('VRAM reported:', round(torch.cuda.get_device_properties(0).total_memory / 1e9, 1), 'GB')
"

# From a non-login shell (cron, Docker, sudo), use the wrapper:
# greenboost run python -c "import torch; print(round(torch.cuda.get_device_properties(0).total_memory/1e9,1), 'GB')"

# Check shim is loaded into a running process (e.g. Ollama):
grep greenboost /proc/$(pgrep ollama)/maps | head -3

Capabilities, pilot & health-check

Three read-only commands for "what's actually running right now, and what should I do about it":

greenboost capabilities   # installed/running shim feature manifest
greenboost pilot           # dataflux-driven advice on the next tuning step
greenboost health-check    # one-shot PASS/FAIL/WARN across module/VRAM/T2/T3/cluster
  • greenboost capabilities is backed by gb_monitor.py, the canonical telemetry/capability client that replaced four independently-grown shim_stats parsers. It reads the shim's feature manifest, written at install time and refreshed at runtime to /run/greenboost/capabilities.json.
  • greenboost pilot is backed by gb_pilot.py. It reads the dataflux log (§ dataflux below) and turns recorded trends into concrete, evidence-backed suggestions - each one names the exact gb_control lever to pull. It never actuates anything itself; you decide whether to act.
  • Both are also exposed as MCP tools (greenboost_capabilities, greenboost_pilot) by gb_dataflux_mcp.py, alongside greenboost_status, so an MCP-connected AI assistant can query the same state directly.

Live debug signals - real-time telemetry during inference

GreenBoost exposes three layers of live telemetry readable at any time, even mid-inference.

Layer 1: sysfs kernel interface

The kernel module exports a sysfs tree at /sys/class/greenboost/greenboost/. These files are always readable while the module is loaded, regardless of whether any app is running.

# One-liner pool summary - ideal for continuous monitoring
watch -n2 'cat /sys/class/greenboost/greenboost/pool_brief'
# → T1:11GB T2:23/42GB(56%) T3:0/73GB PRESSURE:ok KV_RSV:0MB KV_T2:0MB

# Full tier status with KV cache breakdown and T3 pressure
watch -n2 'cat /sys/class/greenboost/greenboost/status'

# Hardware topology (CPU/GPU/RAM/NVMe, NUMA, kthread affinity) - static
cat /sys/class/greenboost/greenboost/hw_info

# Count of live DMA-BUF tensor handles
watch -n1 'cat /sys/class/greenboost/greenboost/active_buffers'

# KV cache reserve in MB (0 = no reserve, all T2 free for model weights)
cat /sys/class/greenboost/greenboost/kv_reserve_mb
echo 0 | sudo tee /sys/module/greenboost/parameters/kv_reserve_mb   # set live

Layer 2: /run/greenboost/ runtime files (LD_PRELOAD shim)

These files are written by any process running with LD_PRELOAD=libgreenboost_cuda.so. They reflect the most recent process that ran with the shim active.

Ownership note: If a previous process (e.g. Ollama running as ollama) wrote these files, they will be owned by that user and a new process cannot overwrite them. Fix: rm -f /run/greenboost/shim_stats /run/greenboost/metrics.json before starting your process. The /run/greenboost/ directory is world-writable so any user can unlink these files.

File Format Content
shim_stats key=value Full shim counters - see field table below
metrics.json JSON Same data in JSON with a feeders[] array per connected feeder
nvtx_events.log TSV Chronological event trace: allocations, phase transitions, OOM events
phase plain text Current phase: MODEL_LOAD / INFERENCE / STEADY / OOM_RECOVERY

Key shim_stats fields and what they mean:

# Watch the most diagnostic fields live during inference
watch -n2 'grep -E "^phase=|^tier_t1_feeder_cur|^tier_t2_feeder_cur|^kernel_dispatch|^remote_alloc_count|^t2_pool_frag|^t2_above_warn|^vram_headroom|^path_[abc]_count" /run/greenboost/shim_stats'
Field Healthy value Problem if...
phase STEADY or INFERENCE OOM_RECOVERY = memory crisis
vram_headroom_mb > 200 < 100 = T1 pressure
tier_t1_feeder_cur_mb > 0 when feeder pooled 0 = feeder not contributing memory
kernel_dispatch_count > 0 when feeder pooled 0 = feeder compute idle despite pooling
remote_alloc_count > 0 when feeder pooled 0 = all allocations staying local
path_a_count increases during inference - (normal T2 DDR path)
path_c_count 0 > 0 = T3 NVMe spill → 10× slowdown
t2_pool_frag_pct < 30 > 80 = fragmentation pressure
t2_above_warn 0 1 = T2 pool > 75% full
kv_dedup_hits increases over time 0 = KV dedup inactive

Inspect feeder connectivity from metrics.json:

python3 -c "
import json, sys
d = json.load(open('/run/greenboost/metrics.json'))
print(f'Phase:             {d[\"phase\"]}')
print(f'Kernel dispatches: {d[\"kernel_dispatch_count\"]}')
print(f'T1 local cur:      {d[\"tiers\"][\"t1_local\"][\"cur_mb\"]} MB')
print(f'T1 feeder cur:     {d[\"tiers\"][\"t1_feeder\"][\"cur_mb\"]} MB')
print(f'T2 feeder cur:     {d[\"tiers\"][\"t2_feeder\"][\"cur_mb\"]} MB')
for f in d.get('feeders', []):
    print(f'Feeder: {f[\"feeder\"]}  health={f[\"health_state\"]}  bw={f[\"bw_measured_mbs\"]} MiB/s  dispatched={d[\"kernel_dispatch_count\"]}')
"

Layer 3: NVTX event log

The NVTX log is a chronological trace of every significant event the shim handles. It persists across process restarts (appended, not overwritten).

# Live tail - see allocations and phase transitions as they happen
tail -f /run/greenboost/nvtx_events.log

# Same via greenboost CLI (interactive pager, auto-scrolling)
greenboost nvtx-logs

# Clear log for a fresh diagnostic session
sudo greenboost clear nvtx-logs

Log format: <epoch_ns> <EVENT_TYPE> <TIER> <MB> ptr=<hex> <message>

Key event types to look for:

Event type Meaning
SHIM_INIT Process attached to GreenBoost shim
PHASE_MODEL_LOAD Model weights loading phase started
PHASE_INFERENCE Model loaded, inference phase started
ALLOC_T1_VRAM Allocation placed in GPU VRAM (fast path)
ALLOC_T2_POOL Allocation placed in T2 DDR pool (path A)
ALLOC_T3_UVM Allocation spilled to T3 NVMe (path C) - investigate if frequent
FEEDER_ALLOC Allocation placed on a remote feeder GPU
KERNEL_DISPATCH CUDA kernel dispatched to feeder for remote execution
OOM_EVICT OOM pressure - tensor evicted from T1 to T2/T3
FREE_T2_POOL T2 tensor freed (deallocation)

Layer 4: Web dashboard

# Start greenboost-cli dashboard server (served at http://localhost:7821)
greenboost-cli
# or: python -m greenboost_cli

# Navigate to: http://localhost:7821/greenboost

The /greenboost page provides:

  • SHIM STATISTICS - auto-polls shim_stats every 2 s: path A/B/C counters, per-tier allocations (local + feeder peaks), kernel dispatch count, T2 fragmentation, T1 headroom
  • FEEDER VITALS - per-feeder T1/T2/T3 current and peak MB, remote alloc count, connection status badge
  • NVTX EVENT LOG - scrollable live log of the last N NVTX events with timestamps

dataflux , the flight recorder (port 8799)

This is a different, complementary page from the Layer-4 dashboard above. Where the :7821 dashboard shows live shim counters right now, dataflux (gb_dataflux.py) keeps a continuous local log of the whole system over time , VRAM used/free, GPU/CPU load, temperature/power, KV-cache pressure, memory-tier movement, quantization decisions, and (once a feeder is connected) per-machine cluster throughput , plus real, client-measured tokens-per-second, not just what GreenBoost predicted.

greenboost dataflux-ui                            # opens the web UI, port 8799
python3 gb_dataflux.py serve [--port 8799] [--days 5]

# headless snapshot instead of the browser UI
greenboost dataflux-ui --llm
python3 gb_dataflux.py summary

Works standalone on a single machine, no feeder required. The web page auto-refreshes every 5 seconds and breaks down activity by script/label, one row per script run, and rolling VRAM/GPU-util/CPU-util/KV-pressure sparklines plus a raw snapshot table:

GreenBoost dataflux web UI

The same log is also exposed read-only over MCP (gb_dataflux_mcp.py, registered as greenboost-dataflux), so any MCP-compatible client can query cluster and inference history directly instead of opening the web UI. Seven tools, each answering a specific question:

Tool Answers
dataflux_summary Cheap overview , totals, per-node/label/run rollups, tok/s
dataflux_events Raw events, filterable by node, label, kind, and status
dataflux_errors What broke recently, and where
dataflux_kinds What kinds of activity actually happened (tier moves, quantization, tok/s, cluster chunks, ...) before drilling in
dataflux_tier_moves T1/T2/T3 promote/demote/evict events from gb_model_tier.py (each carries thrash_events / thrash_active from gb_thrash.py)
dataflux_events(kind="damon_cold") Host cold-page evidence from the kernel's DAMON_STAT, sampled by gb_supervisor (root) and republished to /var/lib/greenboost/damon_cold; read it unprivileged via gb_tiering.host_cold_status(). Observational , nothing places on it yet. Negative idle values mean actively accessed, not idle.
dataflux_quantization Quantization decisions from gb_quant.py , component, bits, budget vs. actual GiB
dataflux_events(kind="quant_calib_guard") Per-layer sensitivity calibration found gb_attn's global SDPA patch active and removed it before measuring. Calibration exists to measure what quantization costs; under that patch it measures quantized attention against quantized attention, so the sensitivities plan_quality() spends bits on are wrong , silently, because the run still completes and returns plausible errors. k_bits/v_bits say what the patch was configured with, so a reader can tell which calibration was at risk. The patch is deliberately NOT restored: the caller that applied it knows when to apply it again, and this event is how that caller learns it happened. Nothing checked this before 2026-08-22.
dataflux_tok_s Measured, real tokens/sec per model, with the full time series when you ask for one model by name
dataflux_events(kind="agent_context_edit") greenboost-cli context edits. Four ops share this kind , compact, microcompact, calibrate, hard_trim , so anything that means "a compaction" must filter op=="compact". A compact carries measured prefix accounting from gb_prefix_guard, computed before the rewrite lands: prefix_kept_pct (100 = pure append, the cheap case), kept_chunks/prev_chunks, and tokens_lost, the tokens that must be re-prefilled that the engine's cache already held. Absent fields mean not measured, never zero.
dataflux_critic compact=True by default (counts by kind/hint, last max_incidents, top recommendations; the full report is ~57k characters). Incident diagnosis, plus the governed layer: since 2026-08-22 it also evaluates every gb_semantics segment and folds matching ones in as [semantics:<severity>] ..., worst first. segments_evaluated: 0 means the semantic layer did not load , its silence is then not a clean bill of health.

The op trap on agent_context_edit

Worth stating on its own, because it cost a day. agent_compaction_prefix_kept_pct read every agent_context_edit event as a compaction. On 2026-08-21 the log held exactly one such event in its entire history and it was an op=calibrate, which edits nothing and so carries no head_kept. The metric scored it 0%, the agent_compaction_broke_prefix segment fired VIOLATION, and casebook C-09 recorded a destroyed prefix that had never happened. Fixed 2026-08-22 , the resolver filters op=="compact" and returns None when there are no compactions, because "nothing has been compacted" and "a compaction went badly" are different answers.

Tier-placement modules

Module What it does Consumed by
gb_evict_policy.py Prioritized-LRU victim selection per tier (PerTierEvictionController). With default priorities it degenerates to the plain LRU it replaced, which is asserted in tests. gb_model_tier.py, gb_tiering.py
gb_thrash.py Tier-thrashing detection, ported from UVM's uvm_perf_thrashing.c. Observation only , it reports, it does not actuate. lapse_s is provisional and reported as uncalibrated. gb_model_tier.py::_df_emit_tier
gb_damon.py Host cold-page evidence from the kernel's DAMON_STAT. Needs root to read (/sys/module/damon_stat is 0400), so gb_supervisor samples it and republishes to /var/lib/greenboost/damon_cold. gb_supervisor.py, gb_tiering.host_cold_status()
gb_prefix_guard.py Prefix-reuse accounting over content-addressed conversation chunks , "how much of the reusable prefix does this rewrite destroy?", answered before the rewrite is applied. Accounting only: it reports a number and a verdict, it never edits a request or chooses a slot. greenboost-cli _compress_context

Diagnosing "feeder shows zero activity" (pooled mode)

If tier_t1_feeder_cur_mb=0 and kernel_dispatch_count=0 despite greenboost cluster showing the feeder connected:

# 1. Check shim_stats is writable by the current user
ls -la /run/greenboost/shim_stats
# If owned by another user → delete and re-run:
rm -f /run/greenboost/shim_stats /run/greenboost/metrics.json

# 2. Check feeder daemon is actually running on the feeder
ssh <user>@<feeder-ip> 'ss -tlnp | grep 9740'
# If not listening → start it:
ssh <user>@<feeder-ip> 'sudo systemctl start greenboost-netd || sudo nohup /usr/local/bin/greenboost-netd -d -p 9740 &'
# Then re-register:
sudo greenboost connect <feeder-ip>

# 3. Check T3 spillover (NVMe path C) - causes dramatic slowdown
cat /sys/class/greenboost/greenboost/pool_brief
# If T3 > 0 during active inference:
echo 0 | sudo tee /sys/module/greenboost/parameters/kv_reserve_mb

# 4. One-command feeder redeploy if daemon is stale
sudo greenboost feeders upgrade-greenboost

TurboQuant - global toggle (v2.9+)

TurboQuant compresses the K/V cache during attention computation, reducing the bandwidth needed to move tensors between T2 DDR and T1 VRAM.

Enable / disable system-wide

# Enable for Ollama (q4_0 KV cache) + Python inference apps
sudo greenboost turboquant on

# Disable and revert to q8_0 KV cache
sudo greenboost turboquant off

# Check current state
greenboost turboquant status

turboquant on does two things simultaneously:

  1. Ollama - updates /etc/systemd/system/ollama.service.d/99-greenboost.conf to set OLLAMA_KV_CACHE_TYPE=q4_0 and OLLAMA_FLASH_ATTENTION=1, then restarts the Ollama service.
  2. Python apps - writes /etc/greenboost/turboquant.enabled and creates /etc/profile.d/greenboost-turboquant.sh which exports GREENBOOST_TURBOQUANT=1 for all future login shells.

Using TurboQuant in Python code

When GREENBOOST_TURBOQUANT=1 is set (or the flag file exists), Python inference scripts can opt in:

import sys, os
sys.path.insert(0, "/path/to/greenboost_all/greenboost")
from gb_attn import turboquant_attention

# Recommended: asymmetric K=4bit / V=3bit (~4× bandwidth reduction, +0.23% PPL)
with turboquant_attention(k_bits=4, v_bits=3, sparse_v=True):
    output = model(input)

Or check the flag and apply conditionally:

from pathlib import Path
if Path("/etc/greenboost/turboquant.enabled").exists() \
        or os.environ.get("GREENBOOST_TURBOQUANT") == "1":
    from gb_attn import turboquant_attention
    ctx = turboquant_attention(k_bits=4, v_bits=3)
else:
    from contextlib import nullcontext
    ctx = nullcontext()

with ctx:
    output = model(input)

gb-quant - weight quantize-to-fit (v3.0+)

The fastest tier is the one you fit in. gb_quant.py shrinks model weights so the working set lives in T1 VRAM at full GPU bandwidth instead of streaming over PCIe from T2. The planner is quality-first: every component gets the HIGHEST precision that still fits the budget (bf16 > int8 > int4 > tq3 > tq2); T3 is never used for weights. The low-bit Triton GEMM backend ships inside GreenBoost (third_party/, Apache-2.0) - nothing to install in your venv.

Sub-4-bit modes "tq3" / "tq2" (3.1 / 2.1 bits per weight effective) are TurboQuant weight quantization (gb_quant_tq.py: random rotation + Lloyd-Max codebook + a Triton LUT-GEMM kernel). Below 4 bits TurboQuant beats HQQ on quality, so the sub-int4 ladder rungs are TQ; int4-HQQ stays the default floor. Select per call (quantize_module(m, bits="tq3"), quantize_to_fit(pipe, prefer_bits="tq3")) or via env (GB_QUANT_BITS=tq3 next to GB_QUANT_BUDGET_GB) - the env value is a precision FLOOR, the planner still gives every component the highest precision that fits.

import sys; sys.path.insert(0, "/path/to/greenboost")
import gb_quant

# any diffusers pipeline or nn.Module:
report = gb_quant.quantize_to_fit(pipe, budget_gb=11.0)

# text-conditioned pipelines, two-phase (encoder freed before denoiser):
embeds = gb_quant.encode_then_quantize(pipe, prompts)

# pipelines you don't own - env hook, zero code:
#   GB_QUANT_BUDGET_GB=11   (or "fit")   then call:
gb_quant.maybe_quantize_from_env(model)

LLMs go through gb_synapse_fallback.py (load_causal_lm for transformers) or, for continuous-batching throughput, gb-synapse's own torch-core engine (vendored gLLM , native bf16/GPTQ/AWQ/FP8 quantization at load time, see workflow/gb-synapse.md). Ollama runs pre-quantized GGUF - there the lever is the shim plus the quant level you pull.

gb-quant and the tiers are complementary: quantize to fit FIRST, then let T2 absorb only what genuinely exceeds the quantized footprint. Measured on an RTX 5070 12 GB: FLUX.2-klein-9B went from ~7 min/image (BF16 via T2 overflow) to ~5 s/image steady state (int4 in T1, peak 8.2 GiB, quality preserved); gemma-3-12b (22.7 GiB bf16) fits in 6.2 GiB int4.

Note: gb-quant currently runs WITHOUT the LD_PRELOAD shim (its backend cannot initialise under it yet); for fits-after-quantization models the shim is unnecessary anyway.

New tuning knobs (v3.2 AI memory OS tranche)

All default-safe (off, or the previous behavior, unless set):

Variable Default Description
GB_PLACEMENT 0 Enable the gb_placement.py fp8-floor cluster-fit planner: prefer a connected feeder over dropping model precision below fp8.
GB_KERNEL_BACKEND gemlite Low-bit GEMM backend gb-quant uses: gemlite, scaled_mm (fp8), bf16 (passthrough). CUTLASS reserved for future use.
GB_QUANT_NO_AUTOTUNE_CACHE 0 Disable the version-scoped (GPU+CUDA+gemlite) persisted Triton autotune cache.
GB_T3_COMPRESS 1 Transparent zstd compression for models evicted to T3 NVMe. Set 0 to store plain.
GB_QUANT_TIERED_PRECISION 0 Hot (fitting) weights stay fp8; cold overflow tail drops to nvfp4 (Blackwell) or int4. fp8 remains the default when unset.
GREENBOOST_T1_WORKSPACE_MB unset (off) Reserve T1 VRAM for per-step compute workspace during model load, released at inference.
GREENBOOST_KV_PREFETCH 0 Enable phase-aware KV prefetch stats mode, exposing kv_prefetch_* telemetry counters.
GB_CUTLASS_ENABLE 0 Build/enable the CUTLASS sm_120a NVFP4 GEMM scaffold (third_party/gb_cutlass). Groundwork only.
GB_QUANT_DP_PLAN 0 Opt-in DP budget-optimal per-layer planner (gb_quant_dp.py) inside plan_quality, replacing the default greedy ladder walk. Skipped for target="compact".
GB_QUANT_DP_MAX_STATES auto (scales with layer count) Beam-width cap for the DP planner above.
GB_QUANT_CALIB frobenius plan_quality's lazy-calibration source when sensitivity= is unset: frobenius (zero-data weight proxy, default) or activation (real-forward-pass, needs run_calibration=).
GB_TIER_KV_COMPRESS 1 Kill switch for KV-cache tier-serde compression (gb_tier_kv.py) on T3 spill. Mirrors GB_T3_COMPRESS; only affects components registered with kv_bits=.
GB_TIER_KV_PRESET polar4 KV tier-serde codec family: polar4/polar8 → PolarQuant, turbo_k8v4 → TurboQuant (K/V asymmetric split needs matching kv_keys=), off → disabled. Bit width always comes from the caller's explicit kv_bits.
GB_TIER_KV_MIN_ELEMS 65536 Minimum tensor element count for KV tier-serde eligibility (below this, overhead exceeds the saving).
GB_STATS_DIR unset (uses /run/greenboost, falling back to /tmp) Override the shim's stats directory (shim_stats/metrics.json + their per-PID variants). Test-only — lets a locally-rebuilt shim be exercised under LD_PRELOAD without writing into a live system's /run/greenboost.

Optimized inference configuration generator (v2.9+)

greenboost gen-inference-config emits a configuration block tuned for the current machine and environment. Useful when setting up Ollama or a Python inference service on a machine that may be a VM, container, or WSL2 instance.

# Print Ollama env block + Python snippet to stdout
greenboost gen-inference-config --format both

# Save to a file, with TurboQuant explicitly enabled
greenboost gen-inference-config --format ollama \
    --turboquant \
    --output /etc/greenboost/inference.env

# Then source it before starting Ollama manually
export $(grep -v '^#' /etc/greenboost/inference.env | xargs)
ollama serve

Flags:

Flag Default Effect
--format ollama|hf|both ollama Which config blocks to emit
--turboquant (reads flag file) Force TurboQuant settings on
--no-turboquant - Force TurboQuant settings off
--output <path> stdout Write to file instead of printing
--model <name> - Include a model name hint in the output

Cluster - connect remote machines as GPU memory and compute feeders

GreenBoost can use the VRAM, RAM, and GPU compute of other machines on your network. The host machine (running Ollama) aggregates all feeder resources into a single virtual GPU. Model layers that overflow into a feeder's VRAM are not just stored there - they are also computed there. GreenBoost automatically dispatches CUDA kernels to whichever machine owns the data, with no manual configuration.

Set up a feeder machine

# On the feeder machine - start the daemon
sudo greenboost feed start

# The daemon listens on TCP port 9740

Connect from the host

# Connect to a feeder (saves to /etc/greenboost/cluster.conf)
sudo greenboost connect 192.168.1.50

# Connect to multiple feeders
sudo greenboost connect 192.168.1.51
sudo greenboost connect 192.168.1.52

# View live cluster status - updates every 5 seconds
greenboost cluster

# Disconnect a feeder
sudo greenboost disconnect 192.168.1.50

What greenboost cluster shows

greenboost cluster is the live interactive view of all connected feeders. It displays:

  • Each feeder's IP, VRAM available, DDR available, connection latency
  • Total memory contributed by the cluster
  • Per-tier breakdown (T1 feeder / T2 feeder / T3 feeder)

Press Ctrl+S to refresh immediately, Ctrl+C to exit.

Upgrading GreenBoost on feeders

After building or updating GreenBoost on the host, push the new binaries to all feeders in one command:

# Push greenboost-netd + CUDA shim + setup script + build stamp to all feeders,
# then restart their daemon automatically.
sudo greenboost feeders upgrade-greenboost

# Check that all machines are on the same build:
greenboost built-stamp --feeders

built-stamp shows the BUILD_ID, version, git hash, and build date for the local machine. With --feeders it also SSHes to every configured feeder and prints their stamp side-by-side. Mismatched stamps are highlighted in amber as a reminder to upgrade.

# Check only the local stamp:
greenboost built-stamp

How the cluster works with Ollama

GreenBoost presents one single virtual GPU to Ollama, regardless of how many feeders are connected. Ollama sees one large GPU with total memory = local T1+T2+T3 + all feeder T1+T2+T3. All memory is pooled into device 0 - there is no multi-GPU scheduling.

Memory allocation order when VRAM overflows:

Local T1 VRAM → Feeder T1 VRAM → Local T2 DDR → Feeder T2 DDR → Local T3 NVMe → Feeder T3 NVMe

Compute dispatch (data-driven, automatic): When a CUDA kernel is launched, GreenBoost inspects the kernel's arguments. If any tensor lives on a feeder (identified by its fake pointer in the 0xAA00… range), the entire kernel is sent to that feeder's GPU over TCP to be executed there. The result is returned to the host. This means feeder GPUs are not idle storage - they actively run inference for the layers stored in their VRAM. No configuration is needed; dispatch follows the data automatically.

gb-synapse , HuggingFace-native, cluster-distributed GGUF serving (v3.2)

Not to be confused with the "Synapse CLI" terminal assistant below , that's a different, older, largely dormant project. gb-synapse is GreenBoost's own model-serving layer, new in v3.2.

Ollama pulls models from its own registry only. gb-synapse (gb_synapse.py) goes straight to the source instead: it pulls GGUF models directly from any HuggingFace repository, gated or public, given a token, and it also indexes whatever GGUFs Ollama already downloaded, so one command sees both.

For clustering, gb-synapse doesn't reinvent the wheel , it hands the cross-node split to llama.cpp's own --rpc backend: real layer-granular tensor split, only activations cross the network, not the weights. The GreenBoost shim still runs on every node underneath it (GREENBOOST_CLUSTER=0 on each), extending that node's own layer share into its local RAM/disk if it doesn't fit VRAM. In short: llama.cpp's RPC owns the split between machines, GreenBoost owns the tiers within each one.

greenboost doctor                          # cluster hardware view: host + feeders
greenboost recommend                       # which pulled models fit the live cluster budget
sudo greenboost synapse login [TOKEN]      # store a HuggingFace token
sudo greenboost pull <repo>[:quant] [name] # download a GGUF
sudo greenboost synapse build-engine       # build llama-server/llama-cli/rpc-server (run on every node)
greenboost synapse run <model> [port]      # serve it, cluster-aware, default :11369
greenboost synapse ps                      # what's running
greenboost synapse stop <model>

gb_synapse_api.py is the thin proxy synapse run launches in front of the llama-server it starts: one port speaks Ollama's API (/api/generate, /api/chat, /api/tags, /api/show, /api/ps), OpenAI's API (/v1/*, relayed byte-for-byte including streaming), and HuggingFace TGI (/generate, /generate_stream) at once, so existing tooling doesn't need to know anything changed.

Full command reference: GREENBOOST_COMMANDS.md § gb-synapse.

Synapse CLI - terminal AI assistant with GreenBoost integration (v2.9+)

Note: this is the older terminal-assistant project, distinct from gb-synapse above (the current, cluster-distributed model server, new in v3.2). Kept here for reference.

Synapse CLI is the companion terminal assistant at greenboost_synapse_cli_new/. Install it once, then run synapse-cli or python synapse_cli.py.

GreenBoost features inside Synapse

Startup banner shows live GreenBoost state:

🐙 Synapse CLI
──────────────────────────────────────────────────
  Model        nutboy02/Qwen3.6-35B-...  (ollama)
  Permissions  auto
  GreenBoost   GB  TQ  +1 feeder(s)
  /backend · /model · /turboquant · /help

GB = GreenBoost installed · TQ = TurboQuant on · +N feeder(s) = cluster connected

/turboquant slash command:

/turboquant status    show current TurboQuant state
/turboquant on        sudo greenboost turboquant on
/turboquant off       sudo greenboost turboquant off

System prompt context - every AI request includes a short GreenBoost paragraph so the model knows it has extended memory available and can be asked to work with large contexts.

Running the Qwen3.6-35B model

cd greenboost_synapse_cli_new
python synapse_cli.py --model "qwen3.6:latest"

This routes through the Ollama backend (localhost:11369). GreenBoost provides the extended memory (T2+T3+cluster) so the 35B model fits even when local VRAM is smaller than the model size. Enable TurboQuant before loading for best throughput:

sudo greenboost turboquant on
python synapse_cli.py --model "qwen3.6:latest"

Diffusion Models - Best Practices

FLUX.1-dev and FLUX.2-klein (image generation) have different memory and performance characteristics from LLM models. Key recommendations for running diffusion pipelines through GreenBoost:

Model selection and speed

Model Mode VRAM Est. speed GreenBoost required
FLUX.2-klein gb-quant int4 ~8 GB T1 only ~5 s/image ★ No (shimless)
FLUX.2-klein FP8 + TurboQuant ~16 GB T1+T2 4–6 s/image Optional
FLUX.2-klein BF16 + TurboQuant ~22 GB T1+T2 5–8 s/image Required
FLUX.1-dev NF4 + TurboQuant ~8 GB T1 ~2 min/image Optional
FLUX.1-dev BF16 + TurboQuant ~23 GB T1+T2 ~4 min/image Required

★ Recommended default (gb-quant) , zero overflow, fits a 12 GB card alone; first image adds ~28 s Triton autotune.

Environment variables for diffusion pipelines

# Required for GreenBoost aggregation
export GREENBOOST_ACTIVE=1
export LD_PRELOAD='/usr/local/lib/libgreenboost_cuda.so'

# Diffusion models have no KV cache - set to 0 to give all T2 to model weights
export GREENBOOST_KV_RESERVE_MB=0

# Enable T1↔T2 bandwidth compression (reduces PCIe congestion during denoising steps)
export GREENBOOST_KV_COMPRESS=1

# Keep Path A0 (DMA-BUF zero-copy) disabled - cudaImportExternalMemory not
# supported on RTX 5070 Laptop + current driver combination
export GREENBOOST_A0_DISABLE=1

Why KV reserve = 0 for diffusion

Unlike LLMs, diffusion transformers perform attention over spatial latent tokens - there is no autoregressive KV cache to accumulate across denoising steps. Each step is a full forward pass with no persistent KV state. Setting GREENBOOST_KV_RESERVE_MB=0 dedicates all T2 DDR to model weight storage, which is where diffusion models need the memory.

TurboQuant with diffusion models

TurboQuant compresses attention tensors during the forward pass. For diffusion models:

  • Effective for BF16 modes: large per-step attention tensors benefit from 4-bit K / 3-bit V compression
  • Less critical for FP8 modes: FP8 weights are already compact; TurboQuant still helps on T1↔T2 transfers
  • Enable with a --turboquant flag in your generation pipeline's script, wired to call gb_attn.turboquant_attention() / patch_sdpa() (this repo only provides the layer; personalized pipelines are the ones that reference it at their call sites)

T3 spillover detection

T3 (system RAM via PCIe) causes dramatic slowdown (~10× slower than T2). Monitor during generation:

watch -n2 'cat /sys/class/greenboost/greenboost/status'
# T3 used_mb should stay at 0 during normal klein/flux generation

If T3 > 0 appears, the model did not fit in T1+T2. Switch to a smaller quantization mode or reduce batch size.

Debug Vitals

GreenBoost provides a comprehensive, toggleable debug vitals system that surfaces 9 data sources in a single command. Essential for diagnosing issues, measuring performance, and guiding enhancement work.

Quick start

# Show full vitals dump immediately (no flag needed)
greenboost debug vitals

# Enable persistent vitals mode - activates verbose instrumentation
sudo greenboost debug vitals on

# Disable persistent vitals - restores minimal-overhead settings
sudo greenboost debug vitals off

What vitals on/off does

sudo greenboost debug vitals on:

  1. Sets kernel debug_mode=1 via sysfs (live, no restart required)
  2. Writes /etc/profile.d/greenboost-vitals.sh → exports GREENBOOST_NVTX_VERBOSE=1
  3. Creates /etc/greenboost/debug_vitals.enabled flag file
  4. Restarts greenboost-idle-reclaim daemon
  5. Lists other GB-aware services that need manual restart
  6. Warns if a reboot is needed for full shim injection

sudo greenboost debug vitals off:

  1. Resets debug_mode=0
  2. Removes profile.d file
  3. Removes flag file and restarts services

Data sources table

Source Path What it provides
IOCTL /dev/greenboost T1/T2/T3/KV/pressure, 24 fields real-time
Sysfs brief /sys/class/greenboost/greenboost/pool_brief One-liner T1/T2/T3 for polling
Sysfs status /sys/class/greenboost/greenboost/status Full text status, KV placement
Shim stats /run/greenboost/shim_stats Path counters, H2D/D2H, dispatch, frag, dedup
Phase file /run/greenboost/phase Current CUDA phase string
NVTX log /run/greenboost/nvtx_events.log Per-allocation events with timestamps
Metrics JSON /run/greenboost/metrics.json Structured JSON + feeder vitals
nvidia-smi ext nvidia-smi extended query Temp, power, util%, SM/mem clocks
Kernel journal journalctl -k Module events (when debug_mode=1)

From greenboost-cli

/gb-vitals          # Full vitals in the Python CLI (Rich rendering)
/gb-vitals on       # Enable (calls sudo greenboost debug vitals on)
/gb-vitals off      # Disable

In scripts

Your own generation scripts and wizards can show the extended vitals panel when the flag file exists , check for /etc/greenboost/debug_vitals.enabled and, if present, render an extra panel (e.g. after your normal T1/T2/T3 progress bars) by reading /run/greenboost/shim_stats or /run/greenboost/metrics.json at your own checkpoints (before/after each generation stage is a common choice). This is a convention personalized pipelines can opt into , GreenBoost itself doesn't require any particular script structure.

Typical use cases

Scenario What to look for
T3 spillover t3_used_mb > 0 or Path C count increasing
KV cache pressure kv_t2_mb > 0 means KV spilling into DDR
Memory fragmentation shim_frag_pct > 30 indicates T2 fragmentation
Feeder not being used shim_remote_alloc_count == 0 despite feeder connected
Phase stuck shim_phase stays MODEL_LOAD after model should be ready
OOM guard triggered oom_active = true in IOCTL data
Kernel events journalctl -k --grep=greenboost when debug_mode=1

Long-running stability monitor (v3.0+)

Once you have a workload running for hours or days, you want a sanity check that nothing is drifting. That's what greenboost stability does.

What it watches

The script (gb_stability_monitor.py) polls /run/greenboost/shim_stats at a configurable interval and flags four invariant violations:

  1. Counter regressed. Fields like kernel_dispatch_count, h2d_mb, d2h_mb, remote_alloc_count, tier_*_lifetime_mb should only ever go up. If one decreases without an accompanying PID change, that's either a wraparound bug or a reset that wasn't declared.
  2. Tier-gauge leak. Gauges like kv_t1_tracked_mb, tier_t1_local_cur_mb, tier_t2_local_cur_mb should drop close to zero once the shim's reported phase leaves INFERENCE or STEADY and a cool-down period passes. If the gauge stays above 10 % of its peak after 2 minutes of idle, that's a memory leak somewhere.
  3. Fragmentation drift. t2_pool_frag_pct and kv_internal_frag_mb climbing monotonically across 30 consecutive samples (≈15 min at the default 30-second interval) without ever retreating is reported as drift.
  4. Shim wedged. If timestamp in shim_stats doesn't move for 90 seconds while the file still exists, the shim has stopped writing - usually a sign of a hang inside an LD_PRELOAD hook.

A shim PID change resets the monotone baselines automatically - it's not a leak, just a fresh process.

Running it

# Short check - useful in CI:
greenboost stability --interval 5 --duration 60 --strict

# Day-long observation, JSON output for log shippers:
greenboost stability --interval 30 --duration 86400 --json \
    --log /var/log/greenboost-stability.log

# Until you Ctrl-C, plain text to stdout:
greenboost stability

Exit codes:

  • 0 clean run, or --strict not used.
  • 1 at least one violation observed (only when --strict).
  • 2 invalid arguments / setup error.

Use with Prometheus

If you already scrape greenboost_exporter.py, the stability monitor covers a different angle: the exporter tells you what the system currently looks like, the stability monitor tells you whether that view has been internally consistent over time.

A workflow that has worked in practice: scrape metrics into Prometheus for short-term graphs and alerts, run the stability monitor with --json --log /var/log/greenboost-stability.log for the long-running invariant trail.

Worker pool (v3.0+, opt-in)

greenboost-netd is single-threaded by default. A slow CUDA op on one client (a big cudaMalloc, a large cudaMemcpy) blocks the reactor and every other client waits behind it. For most setups that's fine - clusters are usually 1–4 feeders serving one host.

If you're running a larger cluster or a multi-tenant feeder, set:

GREENBOOST_WORKERS=8 sudo greenboost feed start

That spawns up to 32 worker threads behind a FIFO queue of 256 jobs. When the queue is full, gb_workpool_submit() returns -1 and the caller falls back to inline execution - no work is dropped silently.

At the time of v3.0 the scaffold is in place but only diagnostic handlers use it. Migration of handle_cuda_memcpy_h2d/d2h and handle_cuda_exec onto the pool is gated on a lock-discipline audit (several globals - g_alloc_lock, g_inflight_ops, ra_count - currently assume single-threaded callers).

You can verify the pool is alive by watching netd logs for PR-VV: worker pool active (N threads) at startup and the drain line at shutdown reporting submitted/completed/rejected counts.

Sister project: GreenBoost Gaming Suite

Through v2.9 the main GreenBoost repository shipped an experimental Vulkan layer (libVkLayer_greenboost.so) that let Steam games via Proton see the GreenBoost memory pool. That layer touched a different graphics API stack (Vulkan, not CUDA), required different testing, and attracted a different audience (gamers vs. ML engineers).

In v3.0 the Vulkan layer was extracted into its own repository:

GreenBoost Gaming Suite

What you'll find there:

  • The GVM Vulkan implicit layer that inflates each game's reported device-local heap by routing overflow allocations through the GreenBoost CUDA pool. Vulkan apps see VRAM that doesn't fit on the card.
  • A GTK4 desktop application (shows up in the GNOME app grid) that scans your Steam library, lists installed games with their current Proton/DLSS versions, and applies per-game optimal settings.
  • A DLSS / FSR / XeSS updater that finds the relevant DLLs inside each Steam Proton prefix and bumps them to the latest version.
  • A GPU profile editor (clocks, power limit, fan curve) for sustained gaming sessions.
  • The GreenBoost Proton wrapper that injects the environment variables games need to activate the layer.

Installation pre-req

The Gaming Suite requires GreenBoost (this project) to be installed first. The Vulkan layer is just a frontend onto the same CUDA pool that your inference workflow already uses, so the kernel module and shim must be in place. The Gaming Suite installer detects whether GreenBoost is present and refuses with a clear instruction if not.

Do I need it?

  • Doing AI inference only → no. You can ignore the Gaming Suite entirely.
  • Mostly gaming, no AI work → install GreenBoost (this repo) and the Gaming Suite. The CUDA inference parts of GreenBoost stay idle; it's the kernel module + memory pool that the Vulkan layer needs.
  • Both → install both. GreenBoost provides the pool, the Gaming Suite exposes it to Vulkan games, and your inference workflow continues to work unchanged.

Agent-harness and greenboost-cli dataflux kinds

Kind Emitted by What it tells you
agent_task, agent_route, agent_tool, agent_retrieval, agent_eval, agent_shunt optimal-claude (consumer repo, planned) Claude Code harness activity next to GPU activity: task outcome and mode, tool cost, retrieval hits, paired-trial eval results, worker spend.
quant_decision consumer pipelines (planned) A quantization choice and why.
web_fetch greenboost-cli instruments/safe_fetch.py Every WebFetch: ok, blocked (with the refusal reason) or error. Query: dataflux_events(kind="web_fetch", status="blocked").
factory_task greenboost-cli workflow/factory.py A Factory task that needs a person: interrupted (its factory died; it is not re-run), resumed, expired.