Ask Lancaster University's academic regulations a question, and get an answer with the page it came from.
A retrieval-augmented (RAG) assistant for the Manual of Academic Regulations and Procedures (MARP),
built as event-driven Python microservices, with three LLMs you can compare side by side.
See it · How it works · Evaluation · Run it · For developers
The screenshots show the real interface replaying answers recorded during our December 2025 test runs. The analytics figures are sample data.
- Answers from the source. Questions are answered only from the MARP PDFs, found by semantic search over vector embeddings.
- Every answer cites its page. Each answer lists the document, page number and link, so anyone can check it.
- Says when it doesn't know. Questions outside MARP, like the weather in Lancaster, get a clear "not in MARP" instead of a made-up answer.
- Compare three LLMs. On your second question, GPT-4o Mini, DeepSeek Chat and Gemma 3n answer side by side, and you pick the one you prefer.
- Analytics. Admins see queries, response times, citations per answer, popular questions and per-model usage. Students see their own.
- Accounts. Registration and sign-in, with server-side sessions.
Compare models. The same question answered by three LLMs, each with its own sources. Your choice becomes your model for the rest of the chat.
PDFs ─► Ingestion ─► Extraction ─► Indexing ─────────► Qdrant
(discover) (text+pages) (chunks, embeddings) │
▼
Question ─► Chat service ─► Retrieval (top-k chunks) ─► LLM via OpenRouter ─► answer + citations
- Every step is its own service. They talk through RabbitMQ events, so a slow or failed step doesn't block the others.
- Embeddings:
all-MiniLM-L6-v2(384 dimensions, cosine similarity), stored in Qdrant. - Grounded answers: the model only sees the retrieved chunks and must cite every fact, and each citation carries its title, page and link.
- Around it: a React UI, an auth service (PostgreSQL + Redis sessions), an analytics service fed by events, Docker Compose for everything, and CI with lint, unit, integration and Docker build checks.
Full architecture: diagram, components and event flow
This system uses a microservices architecture with event-driven communication via RabbitMQ.
graph TB
subgraph Users["User Layer"]
Student[Students]
Staff[Staff]
end
subgraph Presentation["Presentation Layer"]
WebUI[Web UI<br/>React + Vite]
Auth[Authentication Service<br/>FastAPI]
end
subgraph Application["Application Layer"]
Chat[Chat Service<br/>RAG + Multi-Model]
Retrieval[Retrieval Service<br/>Semantic Search]
Analytics[Analytics Service<br/>Usage Tracking]
end
subgraph DataProcessing["Data Processing Layer"]
Ingestion[Ingestion Service<br/>PDF Discovery]
Extraction[Extraction Service<br/>Text Extraction]
Indexing[Indexing Service<br/>Embeddings]
end
subgraph Infrastructure["Infrastructure Layer"]
RabbitMQ[RabbitMQ<br/>Message Broker]
Qdrant[(Qdrant<br/>Vector Database)]
Storage[(File Storage<br/>PDFs + Events)]
PostgreSQL[(PostgreSQL<br/>User Data)]
Redis[(Redis<br/>Sessions)]
end
subgraph External["External Systems"]
MARP[Lancaster MARP<br/>Website]
OpenRouter[OpenRouter API<br/>LLM Provider]
end
%% User interactions
Student --> WebUI
Staff --> WebUI
WebUI --> Auth
WebUI --> Chat
WebUI --> Analytics
%% Auth connections
Auth --> PostgreSQL
Auth --> Redis
%% Application layer flows
Chat --> Retrieval
Chat --> OpenRouter
Chat --> Auth
Chat --> RabbitMQ
Analytics --> RabbitMQ
Retrieval --> Qdrant
%% Data processing pipeline
MARP -->|Scrape PDFs| Ingestion
Ingestion -->|DocumentDiscovered| RabbitMQ
RabbitMQ -->|Event| Extraction
Extraction -->|DocumentExtracted| RabbitMQ
RabbitMQ -->|Event| Indexing
Indexing -->|Store Vectors| Qdrant
%% Storage connections
Ingestion --> Storage
Extraction --> Storage
Indexing --> Storage
%% Analytics tracking
RabbitMQ -.->|Analytics Events| Analytics
%% Styling
style DataProcessing fill:#00A310,stroke:#333,stroke-width:2px
style Infrastructure fill:#A3A300,stroke:#333,stroke-width:2px
style Presentation fill:#00A310,stroke:#333,stroke-width:2px
style Application fill:#00A310,stroke:#333,stroke-width:2px
style External fill:#003BD1,stroke:#333,stroke-width:2px
style Users fill:#FF6B6B,stroke:#333,stroke-width:2px
Legend:
- ✅ Green - Operational (All services fully implemented)
- Yellow - Infrastructure (Always-on services)
- Blue - External Systems
- Red - End Users
Data Processing Pipeline:
- Ingestion Service - Discovers and downloads MARP PDFs from Lancaster's website
- Extraction Service - Extracts text and metadata from PDFs using pdfplumber
- Indexing Service - Chunks documents semantically and generates vector embeddings
Application Services:
4. Retrieval Service - REST API for semantic search over indexed documents
5. Chat Service - RAG-powered question answering with LLM integration and multi-model comparison
6. Authentication Service - User registration, login, session management, and admin account (admin@example.com)
7. Analytics Service - Tracks user interactions and provides usage insights (global for admins, personal for users)
8. Web UI - React-based frontend for chat interaction and analytics dashboard
Infrastructure: 9. Qdrant - Vector database for semantic search (384-dimensional embeddings) 10. RabbitMQ - Message broker for event-driven communication 11. PostgreSQL - User data storage (credentials, profiles) 12. Redis - Session token storage and caching
Data Processing:
Ingestion → DocumentDiscovered → Extraction → DocumentExtracted → Indexing → ChunksIndexed
User Query:
User → Auth (validate) → Chat → Retrieval → LLM (OpenRouter) → User
↓
Analytics (tracking)
For detailed architecture diagrams and service documentation, see:
- Microservices & Broker - Complete system architecture
- Ingestion Pipeline - PDF discovery flow
- Authentication Pipeline - User auth flows
- Multi-Model Pipeline - Comparison flow
- Web UI Pipeline - Frontend interaction flows
- Event Catalogue - All system events
We wrote a 17-question test catalogue covering basic knowledge, specific regulations, citation checks and edge cases: vague and ambiguous questions, typos, and out-of-scope questions that must be declined rather than answered. Each model was scored on accuracy, citation quality and hallucinations.
| Model | Passed | Quality | Cost |
|---|---|---|---|
| DeepSeek Chat | 17 / 17 | 9.5 / 10 | Free |
| GPT-4o Mini | 17 / 17 | 9.8 / 10 | Paid |
| Gemma 3n 2B | 14 / 17 | 7.5 / 10 | Free |
All three declined every out-of-scope question, with no hallucinated answers. The small Gemma model struggled most with ambiguous questions, which showed us where a cheaper model stops being good enough. Details: model comparison.
You need Docker (with Compose), about 4 GB of RAM for the containers, and a free OpenRouter API key.
git clone https://github.com/Orio-03/MARP-Guide-AI.git
cd MARP-Guide-AI
cp .env.example .env # then put your OpenRouter key in .env
docker compose up --buildOpen http://localhost:8080, register an account and ask a question. Your second question shows the three-model comparison. An admin account (admin@example.com / admin) is created on startup and sees analytics for all users.
Setup details: API key, faster builds, logs and health checks
-
Get an OpenRouter API Key:
- Visit openrouter.ai
- Sign up for a free account
- Get your API key from the dashboard
-
Configure environment variables:
Copy the example file and put your key in it:
cp .env.example .env
.envshould then contain:OPENROUTER_API_KEY=sk-or-v1-xxxxxxxxxxxxx
Note: If you experience authentication issues, ensure
OPENROUTER_API_KEYis not exported in your terminal environment (rununset OPENROUTER_API_KEYor use a fresh terminal). -
Clone the repository:
git clone https://github.com/Orio-03/MARP-Guide-AI.git cd MARP-Guide-AI -
Enable Docker BuildKit (Windows - for faster builds):
.\scripts\enable-buildkit.ps1 -
Start all services:
docker compose up --build
This will start:
- Infrastructure: RabbitMQ, Qdrant, PostgreSQL, Redis
- Data Pipeline: Ingestion, Extraction, Indexing services
- Application: Retrieval, Chat, Auth, Analytics services
- Frontend: React Web UI (port 8080)
-
Monitor logs:
# View all services docker compose logs -f # View specific service docker compose logs -f chat docker compose logs -f retrieval
-
Access the application:
- Web UI: http://localhost:8080 (Main interface - start here!)
- Ingestion Service: http://localhost:8001/health
- Retrieval Service: http://localhost:8002/health
- Chat Service: http://localhost:8003/health
- Auth Service: http://localhost:8004/health
- Analytics Service: http://localhost:8005/health
- RabbitMQ Management UI: http://localhost:15672 (guest/guest)
- Qdrant Dashboard: http://localhost:6333/dashboard
Using the app and the API directly (curl, Swagger docs)
The easiest way to use the system:
- Open the Web UI: http://localhost:8080
- Register an account: Click "Register" and create credentials
- Login: Use your credentials to get a session token
- Ask questions:
- First query: Get a single LLM answer with citations
- Second query: Automatic multi-model comparison (3 models side-by-side)
- Select your preferred answer to help improve the system
- View analytics:
- Navigate to Analytics page from sidebar
- Regular users see their personal query history and statistics
- Admin users see global analytics across all users
- Admin Login:
admin@example.com/admin(created automatically on startup)
Authentication Required: The Chat API now requires a Bearer token.
Step 1: Register a user
curl -X POST http://localhost:8004/auth/register \
-H "Content-Type: application/json" \
-d '{"email": "test@example.com", "password": "testpass123"}'Step 2: Login and get token
curl -X POST http://localhost:8004/auth/login \
-H "Content-Type: application/json" \
-d '{"email": "test@example.com", "password": "testpass123"}'Response: {"session_token": "your-session-token-here", "user_id": "...", "email": "test@example.com", "is_admin": false}
Step 3: Chat with token
curl -X POST http://localhost:8003/chat \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-session-token-here" \
-d '{"query": "What is MARP?", "top_k": 5}'Multi-Model Comparison:
curl -X POST http://localhost:8003/chat/compare \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-session-token-here" \
-d '{"query": "What happens if I am ill during exams?", "top_k": 5}'- Chat API: http://localhost:8003/docs
- Auth API: http://localhost:8004/docs
- Analytics API: http://localhost:8005/docs
- Retrieval API: http://localhost:8002/docs
Note: Use the "Authorize" button in Swagger UI to add your Bearer token.
Service URLs, ports and admin UIs
- URL: http://localhost:8080
- Features:
- User registration and login
- RAG-powered chat with citations
- Multi-model comparison (automatic on 2nd query)
- Analytics dashboard
- Lancaster University branding
- URL: http://localhost:8004
- Interactive Docs: http://localhost:8004/docs
- Default Admin Account:
- Email:
admin@example.com - Password:
admin - Created automatically on service startup
- Admin users can view global analytics and access all features
- Email:
- Endpoints:
POST /auth/register- Create new user accountPOST /auth/login- Get session token (24h expiry)POST /auth/logout- Invalidate session tokenGET /auth/validate- Verify Bearer token (internal use)GET /auth/preferences/model- Get user's preferred LLM modelPOST /auth/preferences/model- Set user's preferred LLM model
- URL: http://localhost:8003
- Interactive Docs: http://localhost:8003/docs
- Endpoints (All require
Authorization: Bearer <token>):GET /health- Health checkPOST /chat- Single model RAG with citationsPOST /chat/compare- Multi-model comparison (3 models)POST /chat/comparison/select- Record user's model preference
- URL: http://localhost:8005
- Interactive Docs: http://localhost:8005/docs
- Authentication: All endpoints require
Authorization: Bearer <token>header - Access Control:
- Admin Users (
admin@example.com): View global analytics across all users - Regular Users: View only their personal query history and statistics
- Admin Users (
- Endpoints:
GET /analytics/summary- Overall usage statistics- Admins: Total queries, average latency, and metrics across all users
- Users: Personal query count, average latency for their queries
GET /analytics/popular-queries- Most frequently asked questions- Admins: Popular queries across all users
- Users: User's own query history
GET /analytics/model-stats- Per-model performance metrics (admins only)GET /analytics/recent-queries- Recent queries with timestampsPOST /analytics/reset- Reset all analytics data (admin only, testing purposes)
- URL: http://localhost:8002
- Interactive Docs: http://localhost:8002/docs
- Endpoints:
GET /health- Health checkPOST /search- Semantic search endpoint (no auth required)
- URL: http://localhost:8001
- Type: Worker service (runs once at startup, not a REST API)
- Endpoints:
GET /health- Health check (RabbitMQ connection status)
- URL: http://localhost:15672
- Username:
guest - Password:
guest - Key queues:
documents.discovered,documents.extracted,documents.indexed,analytics.*
- HTTP API: http://localhost:6333
- Dashboard: http://localhost:6333/dashboard
- Collection:
marp-documents(384 dimensions, cosine similarity) - Embedding Model:
all-MiniLM-L6-v2
- Host:
localhost:5432 - Database:
marp_db - User:
marp_user - Tables:
users- User accounts with bcrypt password hashinguser_preferences- User's preferred LLM model settings
- Host:
localhost:6379 - Purpose: Session token storage (24h TTL)
Where the data is stored
Downloaded PDFs are stored in:
pdfs/
Intro-to-MARP.pdf
General-Regs.pdf
Study-Regs.pdf
...
All events and extracted data are stored in:
storage/extracted/{documentId}/
discovered.json # DocumentDiscovered event
pages.jsonl # Extracted page text (one JSON per line)
extracted.json # DocumentExtracted event
chunks.json # Document chunks for debugging
indexed.json # ChunksIndexed event
Stopping and resetting
docker compose downTo also remove volumes (PDFs and vector data):
docker compose down -vProject structure
MARP-Guide-AI/
├── .github/
│ └── workflows/
│ └── ci.yml # CI/CD pipeline configuration
├── services/
│ ├── ingestion/ # PDF discovery and download
│ │ ├── ingestion_service.py
│ │ └── worker.py
│ ├── extraction/ # PDF text extraction
│ │ ├── extraction_service.py
│ │ └── worker.py
│ ├── indexing/ # Chunking and embeddings
│ │ ├── indexing_service.py
│ │ └── worker.py
│ ├── retrieval/ # Semantic search API
│ │ ├── retrieval_service.py
│ │ ├── retrieval_utils.py
│ │ └── worker.py
│ ├── chat/ # RAG-powered Q&A
│ │ ├── chat_service.py
│ │ ├── retrieval_client.py
│ │ ├── openrouter_client.py
│ │ └── prompt_templates.py
│ ├── auth/ # User authentication
│ │ ├── auth_service.py
│ │ ├── database.py
│ │ └── models.py
│ ├── analytics/ # Usage tracking
│ │ ├── analytics_service.py
│ │ └── worker.py
│ └── ui/ # React web interface
│ ├── src/
│ │ ├── components/ # React components
│ │ ├── api/ # API clients
│ │ ├── hooks/ # Custom React hooks
│ │ ├── App.jsx # Main app component
│ │ └── index.css # Global styles
│ ├── Dockerfile # Multi-stage build
│ ├── nginx.conf # Nginx configuration
│ ├── package.json
│ └── vite.config.js
├── common/ # Shared modules
│ ├── events.py # Event schemas
│ ├── mq.py # RabbitMQ utilities
│ ├── config.py # Configuration
│ ├── health.py # Health checks
│ └── logging_config.py # Logging setup
├── tests/ # All test files
│ ├── test_ingestion.py
│ ├── test_extraction.py
│ ├── test_indexing.py
│ ├── test_retrieval.py
│ ├── test_chat.py
│ ├── test_common.py
│ ├── test_integration.py
│ └── README.md # Testing documentation
├── scripts/ # Utility scripts
│ ├── chat.sh # Chat CLI (Linux/Mac)
│ ├── chat.bat # Chat CLI (Windows CMD)
│ ├── chat.ps1 # Chat CLI (Windows PowerShell)
│ └── enable-buildkit.ps1 # Enable Docker BuildKit
├── docs/ # Documentation
│ ├── services/ # Service documentation
│ ├── events/ # Event catalogue
│ └── scrum/ # Sprint planning
├── pdfs/ # Downloaded MARP PDFs
├── storage/ # Extracted content storage
├── pyproject.toml # Python tool configuration
├── requirements.txt # Python dependencies
├── requirements-test.txt # Test dependencies
└── docker-compose.yml # Service orchestration
Code quality tools (black, isort, flake8, pre-commit)
Configuration is centralized in pyproject.toml:
Format code:
black services/ common/ tests/
isort services/ common/ tests/Lint code:
flake8 services/ common/ tests/Configuration:
- black: Code formatter (line length: 127)
- isort: Import sorter (compatible with black)
- flake8: Linter for code quality
- pytest: Test framework with coverage
- coverage: Code coverage tracking
Running the tests
Install test dependencies:
pip install -r requirements-test.txtRun all tests:
# Run all tests
pytest
# Run with coverage
pytest --cov=common --cov=services --cov-report=html
# Run specific test file
pytest tests/test_retrieval.py -v
# Run specific test function
pytest tests/test_retrieval.py::test_search_endpoint_success -vRun tests by marker:
# Run only unit tests (fast)
pytest -m unit
# Run only integration tests (requires services)
pytest -m integrationView coverage report:
# Generate HTML report
pytest --cov=common --cov=services --cov-report=html
# Open in browser
open htmlcov/index.html # macOS
start htmlcov/index.html # WindowsSee tests/README.md for comprehensive testing documentation including:
- Test structure and organization
- Writing new tests
- Mocking strategies
- CI/CD integration
- Troubleshooting guide
CI/CD pipeline
The project uses GitHub Actions for automated testing. The CI pipeline runs on every push and pull request.
Pipeline Stages:
-
Lint (fast, parallel)
- Code formatting checks (black, isort)
- Linting with flake8
- Syntax error detection
-
Unit Tests (parallel, depends on lint)
- All service tests run in parallel
- Coverage reports generated (HTML, XML, terminal)
- Artifacts uploaded for download
-
Integration Tests (sequential, depends on unit tests)
- Docker Compose starts RabbitMQ and Qdrant
- Proper health checks with 60-second timeouts
- Service logs displayed on failure
- Cleanup with
docker compose down -v
-
Docker Build (parallel with integration)
- All service images built and tested
- Verifies images are functional
-
Coverage Report (final summary)
- Aggregates coverage from all test jobs
- Displays summary in GitHub Actions
CI Configuration: .github/workflows/ci.yml
Trigger Conditions:
- Push to
main,develop,feature/**,hotfix/**branches - Pull requests to
mainanddevelop
Viewing Results:
- Go to GitHub repository → Actions tab
- Select latest workflow run
- View job results and logs
- Download coverage artifacts
Job Dependencies:
Lint (fast)
↓
Unit Tests (parallel) → Integration Tests + Docker Build (parallel)
↓
Coverage Report (summary)
This follows the fail-fast principle: linting errors stop the pipeline immediately, saving time and resources.
Faster Docker builds (BuildKit)
All Dockerfiles have been optimized with BuildKit cache mounts for significantly faster builds.
Windows (PowerShell):
.\scripts\enable-buildkit.ps1
docker compose up --buildLinux/macOS:
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1
docker compose up --buildMake it permanent (Windows):
- Press
Win + X→ System → Advanced system settings - Environment Variables → New (User variables)
- Add:
DOCKER_BUILDKIT=1 - Add:
COMPOSE_DOCKER_CLI_BUILD=1
| Build Type | Before | After | Improvement |
|---|---|---|---|
| First build | 10-15 min | 10-15 min | Same (needs to download) |
| Rebuild (code change) | 10-15 min | 3-5 min | 3-5x faster |
| Rebuild (no changes) | 8-10 min | 30 sec | 16-20x faster |
All Dockerfiles now use:
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements.txtThis caches pip downloads between builds without increasing the final image size.
Troubleshooting
docker compose logs rabbitmq
docker compose logs qdrant
docker compose logs ingestion- Check status:
docker compose ps rabbitmq - View logs:
docker compose logs rabbitmq | grep "started" - Restart:
docker compose restart rabbitmq
- Check status:
docker compose ps qdrant - Test API:
curl http://localhost:6333/collections - Restart:
docker compose restart qdrant
- Check logs:
docker compose logs ingestion - Check stats: Visit http://localhost:8001/stats
- Manually trigger: Visit http://localhost:8001/docs →
POST /ingest
- Check RabbitMQ queues: http://localhost:15672 (guest/guest)
- Verify service health: http://localhost:8001/health
- Check worker logs:
docker compose logs extraction indexing
- Ensure session token is valid (24h expiry)
- Check Redis connection:
docker compose ps redis - Re-login to get fresh token:
POST /login - View auth logs:
docker compose logs auth
- Check UI service:
docker compose ps ui - View logs:
docker compose logs ui - Verify port 8080 is not in use:
lsof -i :8080(Mac/Linux) - Clear browser cache and retry
- Ensure dependencies installed:
pip install -r requirements-test.txt - Run from project root:
cd MARP-Guide-AI && pytest - For integration tests:
docker compose up -d rabbitmq qdrant
Technology stack
| Technology | Version | Purpose |
|---|---|---|
| Docker | Latest | Containerization platform |
| Docker Compose | Latest | Multi-container orchestration |
| RabbitMQ | 3.12-management | Message broker |
| Qdrant | Latest | Vector database |
| PostgreSQL | 15-alpine | User data storage |
| Redis | 7-alpine | Session token storage |
| Technology | Version | Purpose |
|---|---|---|
| Python | 3.11 | Programming language |
| FastAPI | 0.104.1 | Web framework for REST APIs |
| Uvicorn | 0.24.0 | ASGI server |
| Starlette | 0.27.0 | ASGI framework |
| Pydantic | 2.0+ | Data validation |
| Pika | 1.3.2 | RabbitMQ client |
| psycopg2-binary | Latest | PostgreSQL adapter |
| redis | Latest | Redis client |
| bcrypt | Latest | Password hashing |
| Technology | Version | Purpose |
|---|---|---|
| sentence-transformers | 3.0.0+ | Generate embeddings |
| qdrant-client | 1.7.0 | Qdrant Python client |
| pdfplumber | 0.10.3 | PDF text extraction |
| BeautifulSoup4 | 4.12.2 | HTML parsing |
| lxml | 5.0.0+ | XML/HTML parser |
| httpx | 0.25.2 | Async HTTP client |
| requests | 2.31.0 | HTTP client |
| numpy | 1.24.3 | Numerical computing |
| openai | Latest | OpenAI SDK client |
- Embedding Model:
all-MiniLM-L6-v2(384 dimensions, cosine similarity) - LLM Provider: OpenRouter (free tier available)
- Default LLM:
openai/gpt-4o-mini(Temperature: 0.4, Max Tokens: 1200) - Multi-Model Comparison: 3 models compared in parallel (GPT-4o Mini, Gemma 3n, DeepSeek Chat)
- Max Context: 3500 tokens for comprehensive answers
| Technology | Version | Purpose |
|---|---|---|
| React | 18.2.0 | UI framework |
| React DOM | 18.2.0 | React rendering |
| Vite | 5.0.0 | Build tool & dev server |
| Tailwind CSS | 3.3.6 | Utility-first CSS |
| PostCSS | 8.4.32 | CSS processing |
| Autoprefixer | 10.4.16 | CSS vendor prefixing |
| Axios | 1.6.0 | HTTP client |
| react-markdown | 9.0.0 | Markdown rendering |
| prop-types | 15.8.1 | React prop validation |
| @vitejs/plugin-react | 4.2.0 | Vite React plugin |
| Nginx | Alpine | Production static server |
| Tool | Version | Purpose |
|---|---|---|
| pytest | 7.4.3 | Testing framework |
| pytest-cov | 4.1.0 | Coverage plugin |
| pytest-mock | 3.12.0 | Mocking support |
| pytest-asyncio | 0.21.1 | Async test support |
| responses | 0.24.1 | HTTP request mocking |
| black | 23.11.0 | Code formatter |
| isort | Latest | Import sorter |
| flake8 | 6.1.0 | Linter for code quality |
| pre-commit | Latest | Git hooks for formatting |
| GitHub Actions | N/A | CI/CD pipeline |
Built by a team of four as Lancaster University Software Engineering coursework (2025). See the contributors. Questions or issues? Open an issue.



