Network traffic monitoring, anomaly detection and threat analysis
Computer Networks project · BTech CSE · VIT Vellore · Drishty Raghav (@drishtyraghav0001)
FlowWatch reads raw PCAP captures or flow records (CSV/JSON, plus the public UNSW-NB15 and CIC-IDS2017 formats) and builds flows from them. It then runs eight detectors on those flows:
- a statistical mean + 2σ rule
- explainable behaviour rules
- four unsupervised models: Isolation Forest, LOF, One-Class SVM and an autoencoder
- trained Random Forest, Gradient Boosting and XGBoost attack classifiers
Their verdicts are combined into one ensemble verdict, and each alert is explained with SHAP and robust z-scores. Everything appears in a SOC-style dark dashboard.
You need Python 3.10+ (tested on 3.13) and git.
git clone https://github.com/drishtyraghav0001/FlowWatch.git
cd FlowWatch
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
streamlit run app.py # opens http://localhost:8501The trained models are already saved in models/, so the app works straight after cloning.
| Command | What it does | Time |
|---|---|---|
python -m pytest -q |
35 automated tests (parsing, detectors, ML, explanations, every app page) | ~45 s |
python scripts/train_models.py |
retrain and save all models to models/ |
~1 min |
python scripts/run_pipeline.py > docs/evaluation_results.txt |
full evaluation; also writes docs/metrics.json |
~30 s |
python -m src.generate |
regenerate the sample CSV / JSON / PCAP | ~10 s |
pip install -r requirements-dev.txt && python scripts/screenshots.py |
headless screenshots → docs/screenshots/ |
~45 s |
| Page | What you see |
|---|---|
| 🛰️ Overview | KPI cards with change vs the first half of the capture, traffic/alert timeline, severity donut, protocol mix, attack types, top talkers, highest-severity alerts |
| 📡 Live Analysis / Upload | ingestion summary (format detected, flows, time span), flow table, size distribution, arrival rate, how to capture with tcpdump |
| 🚨 Detection | sortable, filterable alert queue with colour-coded severity, which detectors fired, agreement heat-map, alerts per detector, scorecard against the labels, recall per attack type |
| 🧠 ML Lab | unsupervised models side by side (metrics, ROC curves, rank correlation, score distributions); supervised model card, classification report, confusion matrix, ROC and PR curves, feature importance; contamination and k sweeps |
| 🔍 Explainability | per-alert card with a plain-English reason, robust z-score contributions, classifier probabilities, local SHAP values, global mean SHAP values |
| 🌐 Threat Intel | host communication graph, port × time heat-map, attack × time heat-map, incidents grouped by attacker (or by target for floods), top ports, attack timeline |
| 📄 Reports | download full results (CSV), alerts (CSV), summary (JSON) and an incident report (HTML) |
Theme: .streamlit/config.toml (dark) plus the CSS, Plotly template and colour palette in src/ui.py.
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
flowchart LR
subgraph Ingest
A[CSV / JSON] --> P[parsing.py<br/>schema + aliases]
B[PCAP] --> Q[parsing.py<br/>scapy → 5-tuple flows] --> P
U[UNSW-NB15 / CIC-IDS2017] --> D[datasets.py<br/>column mapping] --> P
end
P --> F[features.py + behaviour.window_aggregates<br/>ratios · host context · 60 s windows]
F --> S[detection.py<br/>mean + kσ]
F --> R[behaviour.py<br/>explainable rules]
F --> M[ml_unsupervised.py<br/>IF · LOF · OC-SVM · autoencoder]
F --> C[supervised.py<br/>RF / GB / XGBoost<br/>models/supervised.joblib]
S & R & C --> E{Ensemble verdict}
M --> L[ML Lab comparison]
E --> X[explain.py + SHAP]
E --> I[insights.py<br/>attack type · severity · talkers]
X & I & L --> UI[app.py · SOC console]
G[generate.py<br/>labelled captures] --> T[scripts/train_models.py] --> C
FlowWatch/
├── app.py # Streamlit SOC console (7 pages)
├── .streamlit/config.toml # dark theme
├── assets/logo.svg
├── src/
│ ├── parsing.py # CSV / JSON / PCAP → flows
│ ├── datasets.py # UNSW-NB15 and CIC-IDS2017 loaders
│ ├── features.py # per-flow and host-context features
│ ├── behaviour.py # time-window rule detector
│ ├── detection.py # mean + kσ, Isolation Forest, method comparison
│ ├── ml_unsupervised.py # IF, LOF, One-Class SVM, autoencoder
│ ├── supervised.py # classifiers, grouped CV, search, SHAP
│ ├── explain.py # robust z-score explanations
│ ├── evaluate.py # metrics and sweeps
│ ├── insights.py # attack typing, severity, top talkers, timeline
│ ├── pipeline.py # runs everything; hybrid and ensemble verdicts
│ ├── generate.py # synthetic captures and sample PCAP
│ └── ui.py # CSS, palette, Plotly template, widgets
├── models/ # supervised.joblib, unsupervised.joblib, model_card.json
├── data/ # sample CSV / JSON / PCAP (+ labels), external/unsw_nb15_subset.csv
├── scripts/ # train_models.py, run_pipeline.py, screenshots.py
├── tests/ # 35 pytest tests
└── docs/ # evaluation_results.txt, metrics.json, screenshots/
Flows. Packets that share a unidirectional 5-tuple (src IP, dst IP, src port, dst port, protocol) form one flow. For each flow we record packets, bytes, duration, and SYN/ACK/RST/FIN counts.
Features.
- Per flow: packets, bytes, duration, bytes per packet, packet rate, SYN ratio and RST ratio.
- Whether the destination port is an authentication or well-known port.
- Host context: distinct ports contacted by the source, repeat connections to the same service, and distinct sources hitting one service.
- 60 s window aggregates: ports and hosts per source per window, sources and SYN ratio per service per window, and connections per (source, service) pair per window.
Counters are log-transformed (log1p) before going into a model.
Detectors.
| # | Detector | Type | Idea |
|---|---|---|---|
| 1 | mean + kσ (k = 2) | statistical | flag a flow if packets, bytes or duration has z > k |
| 2 | behaviour rules | rule-based | port scan: ≥ 20 ports per source in 60 s with ≤ 3 pkts/flow · SYN flood: ≥ 20 sources per service with ≥ 80 % SYN · brute force: ≥ 10 connections to an auth port · host sweep: ≥ 20 hosts |
| 3 | Isolation Forest | unsupervised | anomalies are isolated by fewer random splits |
| 4 | Local Outlier Factor | unsupervised | the flow's local density is low compared with its neighbours (novelty mode, k = 35) |
| 5 | One-Class SVM | unsupervised | RBF boundary around the bulk of the data (ν = contamination) |
| 6 | Autoencoder | unsupervised | MLPRegressor (k → k → 3 → k → k) trained to rebuild its input; score = reconstruction error |
| 7 | Random Forest / Gradient Boosting / XGBoost | supervised | 5-class attack classifier; the model with the best cross-validation score is deployed |
| 8 | Ensemble | combined | rules OR classifier OR (statistical volume alarm AND classifier p(attack) ≥ 0.2) |
All unsupervised models share the same alert threshold: the (1 − contamination) quantile of their scores on the fitting data.
Avoiding data leakage
- Time-based split. The classifiers are trained on hours 0–4 of capture A (seed 101) and tested on hours 4–6 of a different capture B (seed 202). Capture B uses new attacker IPs, victims and attack intensities. A test checks that the time ranges don't overlap and that fewer than 5 % of test attacker IPs appear in training.
- Grouped cross-validation. Inside the training data,
StratifiedGroupKFold(5)uses 10-minute time blocks as groups. Flows from one attack episode therefore never end up on both sides of a fold.RandomizedSearchCV(6 candidates per model, scored on macro-F1) picks the hyper-parameters, and the cross-validation score alone picks the deployed model. - No identifiers as features. IPs, source ports and timestamps are excluded.
- Look-alike normal traffic. The training and test captures include benign traffic that resembles attacks: nightly backups (large and long, like exfiltration) and a monitoring server polling many ports (like a scan).
- Honest caveat. The sample PCAP was meant as an out-of-distribution check, but the first two training runs did badly on it. Recall on brute force and SYN flood was 0–0.44, because training brute-force campaigns were too large, exfiltration sizes too narrow, and DNS went to random servers instead of a resolver. I widened those generator ranges and retrained, so the PCAP is no longer a fully blind test. The time-holdout capture B is the blind test.
Explanations.
- Model-agnostic: the robust z-score of each feature against the median/MAD baseline, normalised to a percentage share, plus an automatic sentence.
- Rules: each rule quotes the value it measured and the threshold it crossed.
- Classifier: SHAP
TreeExplainergives local values for the predicted class, and the mean |SHAP| over 300 flows gives a global view. If SHAP is unavailable, a perturbation explanation is used instead. - Feature importance: permutation importance (drop in macro-F1) plus impurity importance.
Attack type and severity. The ensemble decides the type in this order: rule type, then the classifier's class, then a heuristic. Severity is a base score for the type (exfiltration 90, SYN flood 80, brute force 70, port scan 50, unknown 30) plus up to 20 points from the ensemble score percentile. The levels are Critical ≥ 85, High ≥ 70, Medium ≥ 50, and Low below that.
These are real outputs of scripts/run_pipeline.py, with contamination 0.10, k = 2 and 60 s
windows. The full log is in docs/evaluation_results.txt and
the numbers are also in docs/metrics.json.
| Detector | Precision | Recall | F1 | ROC-AUC |
|---|---|---|---|---|
| Statistical mean + 2σ | 0.849 | 0.013 | 0.025 | 0.089 |
| Isolation Forest (base 4 features) | 0.490 | 0.189 | 0.272 | 0.661 |
| Isolation Forest (extended) | 0.883 | 0.340 | 0.491 | 0.905 |
| Local Outlier Factor¹ | 0.350 | 0.252 | 0.293 | 0.454 |
| One-Class SVM¹ | 0.299 | 0.105 | 0.156 | 0.831 |
| Autoencoder¹ | 0.113 | 0.059 | 0.078 | 0.204 |
| Unsupervised consensus (≥ 2 of 4)¹ | 0.338 | 0.163 | 0.220 | 0.597 |
| Behaviour rules | 0.9995 | 0.963 | 0.981 | 0.992 |
| Hybrid (rules OR stat OR IF) | 0.956 | 0.998 | 0.977 | 0.999 |
| Random Forest classifier | 0.998 | 1.000 | 0.999 | 1.000 |
| Ensemble (rules + classifier) | 0.997 | 1.000 | 0.999 | 1.000 |
¹ These models were fitted on the training capture and applied to the test capture, as they would be in deployment.
Recall per attack type (test capture)
| Attack | Statistical | Isolation Forest | Rules | Classifier | Ensemble |
|---|---|---|---|---|---|
| Port scan (551) | 0.00 | 0.79 | 0.967 | 1.00 | 1.00 |
| SYN flood (1,346) | 0.00 | 0.02 | 0.998 | 1.00 | 1.00 |
| Brute force (250) | 0.00 | 1.00 | 0.876 | 1.00 | 1.00 |
| Data exfiltration (28) | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| Model | CV macro-F1 (grouped, 5-fold) | Test macro-F1 | Test ROC-AUC (attack vs normal) | PCAP macro-F1² |
|---|---|---|---|---|
| Random Forest (selected by CV) | 0.998 ± 0.004 | 0.987 | 1.000 | 0.721 |
| Gradient Boosting (HistGB) | 0.993 ± 0.009 | 1.000 | 1.000 | 0.998 |
| XGBoost | 0.992 ± 0.010 | 0.993 | 1.000 | 0.905 |
- Random Forest per-class F1 on the test capture: normal 1.000, brute force 1.000, port scan 1.000, SYN flood 1.000, exfiltration 0.933. Four normal flows (backups) were predicted as exfiltration.
- Feature ablation: a Random Forest that uses only per-flow features (no host or window context) reaches macro-F1 0.969 on the test capture.
- Permutation importance:
src_distinct_dst_portsanddst_distinct_srcsmatter most.
² On the sample PCAP, the Random Forest misclassifies the 3 exfiltration flows as brute force, 14 normal flows as brute force and 10 normal flows as SYN flood. Gradient Boosting would have done better, but picking the model by its PCAP score would be test-set leakage, so the CV choice stands. This is listed as a limitation.
| Dataset | Statistical | IF (extended) | Rules | Hybrid | Classifier | Ensemble |
|---|---|---|---|---|---|---|
| Sample CSV (3,712 flows), F1 | 0.033 | 0.616 | 0.982 | 0.967 | 1.000 | 1.000 |
| Sample PCAP (693 flows), F1 | 0.267 | 0.524 | 0.949 | 0.949 | 0.923 | 0.923 |
| Sample PCAP, recall | 0.154 | 0.385 | 0.902 | 0.979 | 1.000 | 1.000 |
The sample CSV comes from the same generator family as the training data, so its perfect classifier score is in-distribution and should not be read as a claim of real-world accuracy.
This is a stratified subset of the official testing set: 6,922 records, 9 attack categories plus normal. The partition file has no IPs, ports or timestamps, so the only possible validation is per-record stratified 5-fold cross-validation, and the rule detector does not apply.
| Model (UNSW native features) | Precision | Recall | F1 | ROC-AUC | 10-class macro-F1 |
|---|---|---|---|---|---|
| Random Forest | 0.931 | 0.979 | 0.955 | 0.993 | 0.613 |
| HistGradientBoosting | 0.956 | 0.887 | 0.920 | 0.900 | 0.566 |
| Isolation Forest (flow features) | 0.359 | 0.064 | 0.108 | 0.172 | – |
| LOF / OC-SVM / Autoencoder | ≤ 0.58 | ≤ 0.10 | ≤ 0.18 | ≤ 0.46 | – |
On this real data, telling attack from normal works well (F1 0.95). Telling the attack families apart is hard: Analysis and Backdoor F1 ≈ 0.23, DoS 0.40, Generic 0.97. The unsupervised models score below random because UNSW attacks are mostly small flows, not volume outliers. That matches what the statistical rule shows on our own data.
- Per-flow detectors miss multi-flow attacks. Scans and floods are made of tiny flows, which is why mean + 2σ and the unsupervised models do poorly on them. Context features and window rules are what close the gap.
- The unsupervised models disagree with each other. Their rank correlations range from −0.05 to 0.77. LOF and the autoencoder learn the dense attack clusters as "normal" (AUC < 0.5).
- Supervised beats unsupervised only when labels exist. It also inherits the training distribution. The PCAP check exposed this, and the UNSW family-level F1 of 0.61 shows how much harder real data is.
- The ensemble keeps an explainable rule layer. Every alert that a rule raises has a human-readable reason, and SHAP explains the alerts that only the classifier raises.
- UNSW-NB15: a 1.3 MB subset ships in
data/external/unsw_nb15_subset.csv. Pick "UNSW-NB15 public subset" in the sidebar. To use the full file, downloadUNSW_NB15_testing-set.csvfrom the UNSW site and upload it; its format is detected automatically. - CIC-IDS2017: download any
MachineLearningCVE/*.csvfrom the CIC site and upload it. Its columns (Flow Duration, Total Fwd/Bwd Packets, flag counts, Label) are mapped bysrc/datasets.py::load_cicids. The mapping is unit-tested on a synthetic row only. The real files are several hundred MB, so they were not downloaded or evaluated here.
Citation: N. Moustafa and J. Slay, UNSW-NB15: a comprehensive data set for network intrusion detection systems, MilCIS 2015. The dataset is free for academic research.
- Run
streamlit run app.py. On Overview, point out the KPI deltas, the timeline (scan at ~10 min, flood at ~25, exfiltration at ~40, brute force at ~50) and the critical alerts. - Live Analysis: switch the sample to the PCAP to show raw packets turned into 693 flows.
- Detection: filter Critical alerts, sort by bytes, and show which detectors fired. Then show the agreement heat-map and explain why each detector sees different attacks.
- ML Lab → Unsupervised: four ROC curves, and why LOF and the autoencoder fail. Supervised: model card (grouped CV + search), confusion matrix, ROC/PR curves, importance. Sensitivity: move the contamination slider.
- Explainability: pick a SYN-flood alert and read the rule sentence, the z-score bars and the SHAP bars.
- Threat Intel: the communication graph (spoofed flood sources cluster around the victim), the heat-maps, and incidents grouped by actor.
- Reports: download the HTML incident report.
- In a terminal, run
python -m pytest -q(35 pass) andpython scripts/run_pipeline.py. The numbers reproduce exactly. - Optional: load the UNSW-NB15 subset to show the system on real lab traffic.
- Training data is synthetic. The classifiers are near-perfect in-distribution but drop on differently generated traffic (PCAP: Random Forest macro-F1 0.72). Training on real labelled captures would be the next step.
- Behaviour windows are fixed (tumbling) 60 s blocks, so attacks that cross a boundary can be split. Sliding windows would fix this.
- There is no live capture yet (
scapy.sniffcould feed the same pipeline). Flows are unidirectional. - The CIC-IDS2017 loader is tested only on a synthetic row.






