An intelligent AI assistant built with FastAPI, LangChain, Groq AI, and a modern glass-morphism web UI. JARVIS provides two chat modes (General and Realtime with web search), streaming responses, text-to-speech, voice input, and learns from your personal data files. Everything runs on one server with one command.
- Quick Start
- Features
- How It Works (Full Workflow)
- Architecture
- Project Structure
- API Endpoints
- Configuration
- Technologies Used
- Frontend Guide
- Troubleshooting
- Developer
- Python 3.10+ with pip
- OS: Windows, macOS, or Linux
- API Keys (set in
.envfile):GROQ_API_KEY(required) - Get from https://console.groq.com
You can use multiple Groq API keys (GROQ_API_KEY_2,GROQ_API_KEY_3, ...) for automatic fallback when one hits rate limits or fails.TAVILY_API_KEY(optional, for Realtime mode) - Get from https://tavily.com
-
Clone or download this repository.
-
Install dependencies:
pip install -r requirements.txt- Create a
.envfile in the project root:
GROQ_API_KEY=your_groq_api_key_here
# Optional: multiple keys for fallback when one hits rate limit
# GROQ_API_KEY_2=second_key
# GROQ_API_KEY_3=third_key
TAVILY_API_KEY=your_tavily_api_key_here
# Optional
GROQ_MODEL=openai/gpt-oss-120b
ASSISTANT_NAME=Jarvis
JARVIS_USER_TITLE=Sir
TTS_VOICE=en-GB-RyanNeural
TTS_RATE=+22%- Start the server:
python run.py- Open in browser: http://localhost:8000
That's it. The server hosts both the API and the frontend on port 8000.
- General Mode: Pure LLM responses using Groq AI. Uses your learning data and conversation history as context. No internet access.
- Realtime Mode: Searches the web via Tavily before answering. Smart query extraction converts messy conversational text into focused search queries. Uses advanced search depth with AI-synthesized answers.
- Server-side TTS using
edge-tts(Microsoft Edge's free cloud TTS, no API key needed). - Audio is generated on the server and streamed inline with text chunks via SSE.
- Sentences are detected in real time as text streams in, converted to speech in background threads (ThreadPoolExecutor), and sent to the client as base64 MP3.
- The client plays audio segments sequentially in a queue — speech starts as soon as the first sentence is ready, not after the full response.
- Works on all devices including iOS (uses a persistent
<audio>element with AudioContext unlock).
- Browser-native speech recognition (Web Speech API).
- Speak your question, and it auto-sends when you finish.
- Put
.txtfiles indatabase/learning_data/with any personal information, preferences, or context. - Past conversations are saved as JSON in
database/chats_data/. - At startup, all learning data and past chats are chunked, embedded with HuggingFace sentence-transformers, and stored in a FAISS vector index.
- For each question, only the most relevant chunks are retrieved (semantic search) and sent to the LLM. This keeps token usage bounded no matter how much data you add.
- Conversations are saved to disk after each message and survive server restarts.
- General and Realtime modes share the same session, so context carries over between modes.
- Configure multiple Groq API keys (
GROQ_API_KEY,GROQ_API_KEY_2,GROQ_API_KEY_3, ...). - Primary-first: every request tries the first key. If it fails (rate limit, timeout), the next key is tried automatically.
- Each key gets one retry for transient failures before falling back.
- Dark glass-morphism UI with animated WebGL orb in the background.
- The orb animates when the AI is speaking (TTS playing) and stays subtle when idle.
- Responsive: works on desktop, tablets, and mobile (including iOS safe area handling).
- No build tools, no frameworks — vanilla HTML/CSS/JS.
This section explains the complete journey of a user's message from the moment they press Send to the moment they hear the AI speak.
The user types a question (or speaks it via voice input) and presses Send. The frontend (script.js) does the following:
- Captures the text from the textarea.
- Adds the user's message bubble to the chat UI.
- Shows a typing indicator (three bouncing dots).
- If TTS is enabled, unlocks the audio context (required on iOS for programmatic playback).
- Sends a
POSTrequest to the backend with{ message, session_id, tts }.
The endpoint depends on the mode:
- General:
POST /chat/stream - Realtime:
POST /chat/realtime/stream
FastAPI validates the request body using the ChatRequest Pydantic model (checks message length 1-32,000 chars). The endpoint handler:
- Gets or creates a session via
ChatService.get_or_create_session(). - Calls
ChatService.process_message_stream()(general) orprocess_realtime_message_stream()(realtime), which returns a chunk iterator. - Wraps the iterator in
_stream_generator()and returns aStreamingResponsewithmedia_type="text/event-stream".
ChatService manages all conversation state:
- If no
session_idis provided, generates a new UUID. - If a
session_idis provided, checks in-memory first, then tries loading from disk (database/chats_data/chat_{id}.json). - Validates the session ID (no path traversal, max 255 chars).
- Adds the user's message to the session's message list.
- Formats conversation history into
(user, assistant)pairs, capped atMAX_CHAT_HISTORY_TURNS(default 20) to keep the prompt within token limits.
Before generating a response, the system retrieves relevant context:
- The user's question is embedded into a vector using the HuggingFace sentence-transformers model (runs locally, no API key needed).
- FAISS performs a nearest-neighbor search against the vector store (which contains chunks from learning data
.txtfiles and past conversations). - The top 10 most similar chunks are returned.
- These chunks are escaped (curly braces doubled for LangChain) and added to the system message.
For general chat:
_build_prompt_and_messages()assembles the system message:- Base personality prompt (from
config.py) - Current date and time
- Retrieved context chunks from the vector store
- General mode addendum ("answer from your knowledge, no web search")
- Base personality prompt (from
- The prompt is sent to Groq AI via LangChain's
ChatGroqwith streaming enabled. - Tokens arrive one by one and are yielded as an iterator.
- If the first API key fails (rate limit, timeout), the system automatically tries the next key.
For realtime chat, three additional steps happen before calling Groq:
-
Query Extraction: A fast LLM call (with
max_tokens=50,temperature=0) converts the user's raw conversational text into a clean search query. Example: "tell me about that website I mentioned" becomes "Jarvis for Everyone website". It uses the last 3 conversation turns to resolve references like "that", "him", "it". -
Tavily Web Search: The clean query is sent to Tavily's advanced search API:
search_depth="advanced"for thorough resultsinclude_answer=Trueso Tavily's AI synthesizes a direct answer- Up to 7 results with relevance scores
-
Result Formatting: Search results are structured with clear headers:
- AI-synthesized answer (marked as primary source)
- Individual sources with title, content, URL, and relevance score
-
These results are injected into the system message before the Realtime mode addendum (which explicitly instructs the LLM to USE the search data).
The _stream_generator function is the core of the streaming + TTS pipeline:
-
Text chunks are yielded immediately as SSE events (
data: {"chunk": "...", "done": false}). The frontend displays them in real time — TTS never blocks text display. -
If TTS is enabled, the generator also: a. Accumulates text in a buffer. b. Splits the buffer into sentences at punctuation boundaries (
. ! ? , ; :). c. Merges short fragments to avoid choppy speech. d. Submits each sentence to aThreadPoolExecutor(4 workers) for background TTS generation viaedge-tts. e. Checks the front of the audio queue for completed TTS jobs and yields them asdata: {"audio": "<base64 MP3>"}events — in order, without blocking. -
When the LLM stream ends, any remaining buffered text is flushed and all pending TTS futures are awaited (with a 15-second timeout per sentence).
-
Final event:
data: {"chunk": "", "done": true, "session_id": "..."}.
The frontend reads the SSE stream with fetch() + ReadableStream:
- Text chunks (
data.chunk): Appended to the message bubble in real time. A blinking cursor appears during streaming. - Audio events (
data.audio): Passed toTTSPlayer.enqueue(), which adds the base64 MP3 to a playback queue. - Done event (
data.done): Streaming is complete. The cursor is removed.
The TTSPlayer manages audio playback:
enqueue(base64Audio)adds audio to the queue and starts_playLoop()if not already running._playLoop()plays segments sequentially: converts base64 to a data URL, sets it as the<audio>element's source, plays it, and waits foronendedbefore playing the next segment.- When audio starts playing, the orb's
.speakingclass and WebGL animation are activated. - When all segments finish (or the user mutes TTS), the orb returns to its idle state.
After the stream completes:
- The full assistant response (accumulated from all chunks) is saved in the session.
- The session is written to
database/chats_data/chat_{id}.json. - During streaming, the session is also saved every 5 chunks for durability.
When the server restarts:
- All
.txtfiles indatabase/learning_data/are loaded. - All
.jsonfiles indatabase/chats_data/(past conversations) are loaded. - Everything is chunked, embedded, and indexed in the FAISS vector store.
- New conversations benefit from all previous context.
User (Browser)
|
| HTTP POST (JSON) + SSE response stream
v
+--------------------------------------------------+
| FastAPI Application (app/main.py) |
| - CORS middleware |
| - Timing middleware (logs all requests) |
| - _stream_generator (SSE + inline TTS) |
+--------------------------------------------------+
| |
v v
+------------------+ +------------------------+
| ChatService | | TTS Thread Pool |
| (chat_service) | | (4 workers, edge-tts) |
| - Sessions | +------------------------+
| - History |
| - Disk I/O |
+------------------+
|
v
+------------------+ +------------------------+
| GroqService | | RealtimeGroqService |
| (groq_service) | | (realtime_service) |
| - General chat | | - Query extraction |
| - Multi-key | | - Tavily web search |
| - LangChain | | - Extends GroqService |
+------------------+ +------------------------+
| |
v v
+--------------------------------------------------+
| VectorStoreService (vector_store.py) |
| - FAISS index (learning data + past chats) |
| - HuggingFace embeddings (local, no API key) |
| - Semantic search: returns top-k chunks |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| Groq Cloud API (LLM inference) |
| - openai/gpt-oss-120b (or configured model) |
| - Primary-first multi-key fallback |
+--------------------------------------------------+
JARVIS/
├── frontend/ # Web UI (vanilla HTML/CSS/JS, no build tools)
│ ├── index.html # Single-page app structure
│ ├── style.css # Dark glass-morphism theme, responsive
│ ├── script.js # Chat logic, SSE streaming, TTS player, voice input
│ └── orb.js # WebGL animated orb renderer (GLSL shaders)
│
├── app/ # Backend (FastAPI)
│ ├── __init__.py
│ ├── main.py # FastAPI app, all endpoints, inline TTS, SSE streaming
│ ├── models.py # Pydantic models (ChatRequest, ChatResponse, etc.)
│ ├── services/
│ │ ├── __init__.py
│ │ ├── chat_service.py # Session management, message storage, disk persistence
│ │ ├── groq_service.py # General chat: LangChain + Groq LLM + multi-key fallback
│ │ ├── realtime_service.py # Realtime chat: query extraction + Tavily search + Groq
│ │ └── vector_store.py # FAISS vector index, embeddings, semantic retrieval
│ └── utils/
│ ├── __init__.py
│ ├── retry.py # Retry with exponential backoff (for API calls)
│ └── time_info.py # Current date/time for the system prompt
│
├── database/ # Auto-created on first run
│ ├── learning_data/ # Your .txt files (personal info, preferences, etc.)
│ ├── chats_data/ # Saved conversations as JSON
│ └── vector_store/ # FAISS index files
│
├── config.py # All settings: API keys, paths, system prompt, TTS config
├── run.py # Entry point: python run.py
├── requirements.txt # Python dependencies
├── .env # Your API keys (not committed to git)
└── README.md # This file
General chat (non-streaming). Returns full response at once.
General chat with streaming. Returns Server-Sent Events.
Realtime chat (non-streaming). Searches the web first, then responds.
Realtime chat with streaming. Web search + SSE streaming.
Request body (all chat endpoints):
{
"message": "What is Python?",
"session_id": "optional-uuid",
"tts": true
}message(required): 1-32,000 characters.session_id(optional): omit to create a new session; include to continue an existing one.tts(optional, default false): set totrueto receive inline audio events in the stream.
SSE stream format:
data: {"session_id": "uuid-here", "chunk": "", "done": false}
data: {"chunk": "Hello", "done": false}
data: {"chunk": ", how", "done": false}
data: {"audio": "<base64 MP3>", "sentence": "Hello, how can I help?"}
data: {"chunk": "", "done": true, "session_id": "uuid-here"}
Non-streaming response:
{
"response": "Python is a high-level programming language...",
"session_id": "uuid-here"
}Returns all messages for a session.
Health check. Returns status of all services.
Standalone TTS endpoint. Send {"text": "Hello"}, receive streamed MP3 audio.
Redirects to /app/ (the frontend).
Returns list of available endpoints.
| Variable | Required | Default | Description |
|---|---|---|---|
GROQ_API_KEY |
Yes | - | Primary Groq API key |
GROQ_API_KEY_2, _3, ... |
No | - | Additional keys for fallback |
TAVILY_API_KEY |
No | - | Tavily search API key (for Realtime mode) |
GROQ_MODEL |
No | openai/gpt-oss-120b |
LLM model name |
ASSISTANT_NAME |
No | Jarvis |
Assistant's name |
JARVIS_USER_TITLE |
No | - | How to address the user (e.g. "Sir") |
TTS_VOICE |
No | en-GB-RyanNeural |
Edge TTS voice (run edge-tts --list-voices to see all) |
TTS_RATE |
No | +22% |
Speech speed adjustment |
The assistant's personality is defined in config.py. Key sections:
- Role: conversational face of the system; does not claim to have completed actions unless the result is visible
- Answering Quality: instructed to be specific, use context/search results, never give vague answers
- Tone: warm, intelligent, concise, witty
- Formatting: no asterisks, no emojis, no markdown, plain text only
Add .txt files to database/learning_data/:
- Files are loaded and indexed at startup.
- Only relevant chunks are sent to the LLM per question (not the full text).
- Restart the server after adding new files.
You can use multiple Groq API keys for automatic fallback. Set GROQ_API_KEY (required) and optionally GROQ_API_KEY_2, GROQ_API_KEY_3, etc. in your .env:
GROQ_API_KEY=first_key
GROQ_API_KEY_2=second_key
GROQ_API_KEY_3=third_keyEvery request tries the first key first. If it fails (rate limit, timeout, or error), the next key is tried automatically. Each key has its own daily limit on Groq's free tier, so multiple keys give you more capacity.
| Technology | Purpose |
|---|---|
| FastAPI | Web framework, async endpoints, SSE streaming |
| LangChain | LLM orchestration, prompt templates, message formatting |
| Groq AI | LLM inference (Llama 3.3 70B, extremely fast) |
| Tavily | AI-optimized web search with answer synthesis |
| FAISS | Vector similarity search for context retrieval |
| HuggingFace | Local embeddings (sentence-transformers/all-MiniLM-L6-v2) |
| edge-tts | Server-side text-to-speech (Microsoft Edge, free, no API key) |
| Pydantic | Request/response validation |
| Uvicorn | ASGI server |
| Technology | Purpose |
|---|---|
| Vanilla JS | Chat logic, SSE streaming, TTS playback queue |
| WebGL/GLSL | Animated orb (simplex noise, procedural lighting) |
| Web Speech API | Browser-native speech-to-text |
| CSS Glass-morphism | Dark translucent panels with backdrop blur |
| Poppins (Google Fonts) | Typography |
- General: Click "General" in the header. Uses the LLM's knowledge + your learning data. No internet.
- Realtime: Click "Realtime" in the header. Searches the web first, then answers with fresh information.
- Click the speaker icon to enable/disable TTS.
- When enabled, the AI speaks its response as it streams in.
- Click again to mute mid-speech (stops immediately, orb returns to idle).
- Click the microphone icon to start listening.
- Speak your question. It auto-sends when you finish.
- Click again to cancel.
- Idle: Subtle glow (35% opacity), slowly rotating.
- Speaking (TTS active): Full brightness, pulsing scale animation.
- The orb only animates when TTS audio is playing, not during text streaming.
On the welcome screen, click any chip ("What can you do?", "Open YouTube", etc.) to send a preset message.
- Ensure
GROQ_API_KEYis set in.env. - Run
pip install -r requirements.txtto install all dependencies. - Check that port 8000 is not in use.
- The server is not running. Start it with
python run.py. - Check the terminal for error messages.
- Ensure
TAVILY_API_KEYis set in.envand is valid. - Check the server logs for
[TAVILY]entries to see if search is working. - The query extraction LLM call should appear as
[REALTIME] Query extraction:in logs.
- Make sure TTS is enabled (speaker icon should be highlighted purple).
- On iOS: TTS requires a user interaction first (tap the speaker button before sending a message).
- Check server logs for
[TTS-INLINE]errors.
- Delete
database/vector_store/and restart — the index rebuilds automatically. - Check that
database/directories exist and are writable.
- Likely caused by
{or}in learning data files. The system escapes these automatically, but if you see errors, check your.txtfiles.
The server logs [TIMING] entries for every operation:
| Log Entry | What It Measures |
|---|---|
session_get_or_create |
Session lookup (memory/disk/new) |
vector_db |
Vector store retrieval |
tavily_search |
Web search (Realtime only) |
groq_api |
Full Groq API call |
first_chunk |
Time to first streaming token |
groq_stream_total |
Total stream duration + chunk count |
save_session_json |
Session save to disk |
Typical latencies:
- General mode first token: 0.3-1s
- Realtime mode first token: 2-5s (includes query extraction + web search)
- TTS first audio: ~1s after first sentence completes
- Session IDs are validated against path traversal (
..,/,\). - API keys are stored in
.env(never in code). - CORS allows all origins (
*) since this is a single-user server. - No authentication — add it if deploying for multiple users.
Start chatting: python run.py then open http://localhost:8000