LCORE-1675: Documentation for conversation compaction

docs/README.md

@@ -29,6 +29,8 @@ See the full documentation at [`../README.md`](../README.md) or browse sub-pages
 
 [OKP guide](https://lightspeed-core.github.io/lightspeed-stack/user_doc/okp_guide.html)
 
+[Conversation compaction](https://lightspeed-core.github.io/lightspeed-stack/user_doc/conversation_compaction.html)
+
 [Authentication and Authorization](https://lightspeed-core.github.io/lightspeed-stack/user_doc/auth.html)
 
 [User data collection](https://lightspeed-core.github.io/lightspeed-stack/user_doc/user_data_collection.html)

docs/devel_doc/ARCHITECTURE.md

@@ -35,7 +35,7 @@ To keep requests on-topic and protect sensitive data, LCore applies **safety shi
 - **Multi-Provider Support**: Works with multiple LLM providers (Ollama, OpenAI, Watsonx, etc.)
 - **Enterprise Security**: Authentication, authorization (RBAC), and secure credential management
 - **Resource Management**: Token-based quota limits and usage tracking
-- **Conversation Management**: Multi-turn conversations with history and caching
+- **Conversation Management**: Multi-turn conversations with history, caching, and automatic compaction
 - **RAG Integration**: Retrieval-Augmented Generation for context-aware responses
 - **Tool Orchestration**: Model Context Protocol (MCP) server integration
 - **Observability**: Prometheus metrics, structured logging, and health checks
@@ -370,6 +370,83 @@ External A2A requests go through LCore's standard authentication system (K8s, RH
 
 ---
 
+### 2.11 Conversation Compaction (`utils/compaction.py`, `utils/conversation_compaction.py`)
+
+**Purpose:** Automatically summarize older conversation turns when the conversation history approaches the LLM's context window limit, preventing HTTP 413 failures and enabling arbitrarily long conversations.
+
+**Design Philosophy (Option A):** Once compaction triggers, LCore takes ownership of the context sent to the LLM. The `conversation` parameter is dropped from the OGX call (`omit_conversation=True`), and LCore constructs the input explicitly from summaries + recent turns + new query. The full original history remains in OGX for auditing.
+
+**Architecture:**
+
+The compaction system is split into two layers:
+
+1. **Pure Logic Layer** (`utils/compaction.py`) — Side-effect-free functions:
+   - `partition_conversation()` — Splits conversation items into old and recent chunks using a *degrading guard*: starts with the configured `buffer_turns` and shrinks one pair at a time until the recent chunk fits the token budget
+   - `summarize_chunk()` — Single LLM call to produce a `ConversationSummary` from older turns
+   - `recursively_resummarize()` — Folds multiple accumulated summaries into one when they approach the context limit
+
+2. **Runtime Integration Layer** (`utils/conversation_compaction.py`) — Manages side effects:
+   - Per-conversation locking (serializes concurrent requests on the same conversation)
+   - Compaction state loading (cache-preferred with marker fallback)
+   - Marker persistence (`[lightspeed:compaction-summary]` sentinel in conversation items)
+   - `CompactionStartedEvent` emission for streaming progress indicators
+   - `apply_compaction()` (async generator) — Main entry point used by all endpoints
+   - `store_compacted_turn()` — Appends user query + LLM output when in compacted mode
+
+**Data Flow:**
+
+```
+User Query → Estimate Tokens → Exceeds Threshold?
+                                    │
+                              No    │    Yes
+                              ↓     │     ↓
+                          Pass-through  Acquire Lock
+                                        ↓
+                                    Fetch Conversation Items
+                                        ↓
+                                    Load Compaction State
+                                    (cache → marker fallback)
+                                        ↓
+                                    Partition (old | recent)
+                                        ↓
+                                    Summarize Old Chunk (LLM call)
+                                        ↓
+                                    Write Marker + Cache Summary
+                                        ↓
+                                    Recursive Fold (if needed)
+                                        ↓
+                                    Build Explicit Input:
+                                    [summaries + recent + query]
+                                        ↓
+                                    Set omit_conversation=True
+                                        ↓
+                                    Release Lock → Continue to LLM
+```
+
+**Endpoint Integration:**
+
+| Endpoint | Mode | Cache | `context_status` |
+|---|---|---|---|
+| `/v1/query` | Blocking (`apply_compaction_blocking()`) | Yes | Yes (`"full"` / `"summarized"`) |
+| `/v1/streaming_query` | Streaming (`apply_compaction()` generator) | Yes | Yes (in `end` event) |
+| `/v1/responses` | Blocking | Yes | No (OpenAI-compatible, silent) |
+| `/a2a` | Blocking, marker-only (no cache) | No | No (A2A protocol scope) |
+
+**Configuration:**
+
+Compaction is controlled by `CompactionConfiguration` in `lightspeed-stack.yaml`:
+- `enabled` (default: `false`) — Master switch
+- `threshold_ratio` (default: `0.7`) — Fraction of context window that triggers compaction
+- `token_floor` (default: `4096`) — Minimum token count before compaction can fire
+- `buffer_turns` (default: `4`) — Recent turns kept verbatim
+- `buffer_max_ratio` (default: `0.3`) — Max fraction of window for the buffer
+
+Models must have context windows registered via `inference.context_windows` (a map of model ID to token count).
+
+**Concurrency:** A per-conversation lock dictionary serializes concurrent compaction requests on the same conversation. Lock entries are reference-counted and cleaned up when the last waiter exits.
+
+---
+
 ## 3. Request Processing Pipeline
 
 This section illustrates how requests flow through LCore from initial receipt to final response.
@@ -400,11 +477,12 @@ Here's how a real query flows through the system:
 5. **Model Selection** - Use configured default model (e.g., `meta-llama/Llama-3.1-8B-Instruct`)
 6. **Context Building** - Retrieve conversation history, query RAG vector stores for relevant docs, determine available MCP tools
 7. **Shield moderation** - LCore-owned direct-run moderation (and agent capabilities where applicable) using shields configured in LCORE config
-8. **Llama Stack / agent call** - Send request with system prompt, RAG context, and MCP tools
-9. **LLM Processing** - Stack / agent generates response, may invoke MCP tools, returns token counts
-10. **Post-Processing** - Generate conversation summary if new
-11. **Store Results** - Save to Cache DB, User DB, consume quota, update metrics
-12. **Return Response** - Complete LLM response with referenced documents, token usage, and remaining quota
+8. **Conversation compaction** - If enabled and estimated tokens exceed the threshold, summarize older turns and rebuild the context (see [Section 2.11](#211-conversation-compaction-utilscompactionpy-utilsconversation_compactionpy))
+9. **OGX / agent call** - Send request with system prompt, RAG context, and MCP tools
+10. **LLM Processing** - Stack / agent generates response, may invoke MCP tools, returns token counts
+11. **Post-Processing** - Generate conversation summary if new
+12. **Store Results** - Save to Cache DB, User DB, consume quota, update metrics
+13. **Return Response** - Complete LLM response with referenced documents, token usage, and remaining quota
 
 **Key Takeaways:**
 - RAG enhances responses with relevant documentation

docs/devel_doc/openapi.md

@@ -2738,7 +2738,7 @@ user's query to a selected Llama Stack LLM and returning the generated response.
 - mcp_headers: Headers that should be passed to MCP servers.
 
 ### Returns:
-- QueryResponse: Contains the conversation ID and the LLM-generated response.
+- QueryResponse: Contains the conversation ID, the LLM-generated response, and a `context_status` field indicating whether the conversation context is `"full"` or `"summarized"`.
 
 ### Raises:
 - HTTPException:
@@ -3021,7 +3021,7 @@ content type text/event-stream.
 - mcp_headers: Headers that should be passed to MCP servers.
 
 ### Returns:
-- SSE-formatted events for the query lifecycle.
+- SSE-formatted events for the query lifecycle. Includes a `context_status` field (`"full"` or `"summarized"`) in the `end` event payload indicating whether conversation compaction was applied. When compaction is triggered, a `compaction` SSE event is emitted before inference begins.
 
 ### Raises:
 - HTTPException:
@@ -8010,6 +8010,7 @@ Attributes:
 | available_quotas | object | Quota available as measured by all configured quota limiters |
 | tool_calls | array | List of tool calls made during response generation |
 | tool_results | array | List of tool results |
+| context_status | string | Indicates whether the conversation context sent to the LLM is `"full"` (complete history) or `"summarized"` (older turns were summarized). Only present in QueryResponse and StreamingQueryResponse; omitted from `/v1/responses` (OpenAI-compatible) and `/a2a` responses. |
 
 
 ## QuotaExceededResponse

docs/devel_doc/query_endpoint.md

@@ -145,6 +145,7 @@ The optional `solr` field configures Solr inline RAG behavior:
 | `tool_results` | array[object] | `[]` | Tool call results |
 | `rag_chunks` | array[object] | `[]` | *(Deprecated)* RAG chunks used |
 | `truncated` | boolean | `false` | *(Deprecated)* Always `false` |
+| `context_status` | string | `"full"` | Whether the conversation context is `"full"` (complete history) or `"summarized"` (older turns were summarized via conversation compaction) |
 
 **`referenced_documents` items:**
 
@@ -242,7 +243,7 @@ Emitted when the full response is assembled.
 
 #### 7. `end`
 
-Emitted last on success. Contains metadata.
+Emitted last on success. Contains metadata including `context_status` (`"full"` or `"summarized"`).
 
 ```json
 {
@@ -251,7 +252,8 @@ Emitted last on success. Contains metadata.
     "referenced_documents": [],
     "truncated": null,
     "input_tokens": 11,
-    "output_tokens": 19
+    "output_tokens": 19,
+    "context_status": "full"
   },
   "available_quotas": {"UserQuotaLimiter": 998911}
 }
@@ -327,9 +329,9 @@ Both endpoints share the same pre-processing pipeline:
 11. Prepare Responses API parameters (model, system prompt, tools, MCP headers)
 12. Extract image attachments separately for multimodal input construction
 
-**`/v1/query` then:** applies conversation compaction (blocking), calls the LLM, generates topic summary, consumes tokens, stores results, returns JSON.
+**`/v1/query` then:** applies conversation compaction (blocking), calls the LLM, generates topic summary, consumes tokens, stores results, returns JSON. When compaction is applied, the response includes `context_status: "summarized"`; otherwise `context_status: "full"`.
 
-**`/v1/streaming_query` then:** generates a `request_id`, starts the SSE stream, emits events as the LLM generates tokens, performs post-stream cleanup (topic summary, token consumption, persistence).
+**`/v1/streaming_query` then:** generates a `request_id`, starts the SSE stream, applies compaction if needed (emitting a `compaction` SSE event), emits events as the LLM generates tokens, performs post-stream cleanup (topic summary, token consumption, persistence). The `end` event includes `context_status` indicating whether compaction was applied.
 
 ---
 
@@ -409,7 +411,8 @@ curl -X POST http://localhost:8090/v1/query \
   "tool_calls": [],
   "tool_results": [],
   "rag_chunks": [],
-  "truncated": false
+  "truncated": false,
+  "context_status": "full"
 }
 ```
 
@@ -500,7 +503,7 @@ data: {"event": "token", "data": {"id": 2, "token": " an"}}
 
 data: {"event": "turn_complete", "data": {"id": 50, "token": "Kubernetes is an open-source..."}}
 
-data: {"event": "end", "data": {"referenced_documents": [], "truncated": null, "input_tokens": 11, "output_tokens": 50}, "available_quotas": {"UserQuotaLimiter": 998950}}
+data: {"event": "end", "data": {"referenced_documents": [], "truncated": null, "input_tokens": 11, "output_tokens": 50, "context_status": "full"}, "available_quotas": {"UserQuotaLimiter": 998950}}
 ```
 
 ### Streaming Query Interrupt

docs/index.md

@@ -34,6 +34,8 @@ product questions using backend LLM services, agents, and RAG databases.
 
 [OKP guide](https://lightspeed-core.github.io/lightspeed-stack/user_doc/okp_guide.html)
 
+[Conversation compaction](https://lightspeed-core.github.io/lightspeed-stack/user_doc/conversation_compaction.html)
+
 [Authentication and Authorization](https://lightspeed-core.github.io/lightspeed-stack/user_doc/auth.html)
 
 [User data collection](https://lightspeed-core.github.io/lightspeed-stack/user_doc/user_data_collection.html)

docs/user_doc/config.md

@@ -224,6 +224,51 @@ Attributes:
 | buffer_turns     | integer | Number of recent turns to keep verbatim.                                                                    |
 | buffer_max_ratio | number  | Maximum fraction of context window the buffer zone can occupy, regardless of buffer_turns.                  |
 
+### How to enable conversation compaction
+
+Compaction is disabled by default. To enable it, add a `compaction` section to your `lightspeed-stack.yaml` and set `enabled: true`. You must also register context window sizes for the models you use via the `inference.context_windows` map so the compaction trigger can calculate when older turns should be summarized.
+
+**Minimal configuration:**
+
+```yaml
+inference:
+  default_provider: openai
+  default_model: gpt-4o-mini
+  context_windows:
+    openai/gpt-4o-mini: 128000
+
+compaction:
+  enabled: true
+```
+
+**Full configuration with all options:**
+
+```yaml
+inference:
+  default_provider: openai
+  default_model: gpt-4o-mini
+  context_windows:
+    openai/gpt-4o-mini: 128000
+    openai/gpt-4o: 128000
+
+compaction:
+  enabled: true
+  threshold_ratio: 0.7       # trigger at 70% of context window (default)
+  token_floor: 4096           # minimum tokens before compaction can fire (default)
+  buffer_turns: 4             # recent turns kept verbatim (default)
+  buffer_max_ratio: 0.3       # buffer may use at most 30% of the window (default)
+```
+
+**Key considerations:**
+
+- `context_windows` is required. Models absent from this map have no registered window and compaction will not trigger for them.
+- `threshold_ratio` controls how aggressively compaction fires. Lower values compact sooner; higher values wait longer (closer to the window limit).
+- `buffer_turns` sets how many recent user/assistant turn pairs are kept in full. A degrading guard automatically reduces this if the buffer itself would exceed `buffer_max_ratio` of the window.
+- `token_floor` prevents compaction from triggering on very short conversations.
+- When compaction is disabled (the default), requests that exceed the context window surface as HTTP 413.
+
+For a comprehensive explanation of the feature, see the [Conversation Compaction Guide](conversation_compaction.md).
+
 
 ## Configuration