Skip to content
drishtyraghav0001Public

About

Network traffic monitoring and anomaly detection system

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

12 Commits

Folders and files

Repository files navigation

FlowWatch · SOC Console

Network traffic monitoring, anomaly detection and threat analysis
Computer Networks project · BTech CSE · VIT Vellore · Drishty Raghav (@drishtyraghav0001)

FlowWatch reads raw PCAP captures or flow records (CSV/JSON, plus the public UNSW-NB15 and CIC-IDS2017 formats) and builds flows from them. It then runs eight detectors on those flows:

  • a statistical mean + 2σ rule
  • explainable behaviour rules
  • four unsupervised models: Isolation Forest, LOF, One-Class SVM and an autoencoder
  • trained Random Forest, Gradient Boosting and XGBoost attack classifiers

Their verdicts are combined into one ensemble verdict, and each alert is explained with SHAP and robust z-scores. Everything appears in a SOC-style dark dashboard.

Overview


1. Quick start

You need Python 3.10+ (tested on 3.13) and git.

git clone https://github.com/drishtyraghav0001/FlowWatch.git
cd FlowWatch
python3 -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
streamlit run app.py                 # opens http://localhost:8501

The trained models are already saved in models/, so the app works straight after cloning.

Command What it does Time
python -m pytest -q 35 automated tests (parsing, detectors, ML, explanations, every app page) ~45 s
python scripts/train_models.py retrain and save all models to models/ ~1 min
python scripts/run_pipeline.py > docs/evaluation_results.txt full evaluation; also writes docs/metrics.json ~30 s
python -m src.generate regenerate the sample CSV / JSON / PCAP ~10 s
pip install -r requirements-dev.txt && python scripts/screenshots.py headless screenshots → docs/screenshots/ ~45 s

2. The dashboard

Page What you see
🛰️ Overview KPI cards with change vs the first half of the capture, traffic/alert timeline, severity donut, protocol mix, attack types, top talkers, highest-severity alerts
📡 Live Analysis / Upload ingestion summary (format detected, flows, time span), flow table, size distribution, arrival rate, how to capture with tcpdump
🚨 Detection sortable, filterable alert queue with colour-coded severity, which detectors fired, agreement heat-map, alerts per detector, scorecard against the labels, recall per attack type
🧠 ML Lab unsupervised models side by side (metrics, ROC curves, rank correlation, score distributions); supervised model card, classification report, confusion matrix, ROC and PR curves, feature importance; contamination and k sweeps
🔍 Explainability per-alert card with a plain-English reason, robust z-score contributions, classifier probabilities, local SHAP values, global mean SHAP values
🌐 Threat Intel host communication graph, port × time heat-map, attack × time heat-map, incidents grouped by attacker (or by target for floods), top ports, attack timeline
📄 Reports download full results (CSV), alerts (CSV), summary (JSON) and an incident report (HTML)

Theme: .streamlit/config.toml (dark) plus the CSS, Plotly template and colour palette in src/ui.py.

Detection ML Lab
Explainability Threat intel
Live analysis Reports

3. Architecture

flowchart LR
    subgraph Ingest
      A[CSV / JSON] --> P[parsing.py<br/>schema + aliases]
      B[PCAP] --> Q[parsing.py<br/>scapy → 5-tuple flows] --> P
      U[UNSW-NB15 / CIC-IDS2017] --> D[datasets.py<br/>column mapping] --> P
    end
    P --> F[features.py + behaviour.window_aggregates<br/>ratios · host context · 60 s windows]
    F --> S[detection.py<br/>mean + kσ]
    F --> R[behaviour.py<br/>explainable rules]
    F --> M[ml_unsupervised.py<br/>IF · LOF · OC-SVM · autoencoder]
    F --> C[supervised.py<br/>RF / GB / XGBoost<br/>models/supervised.joblib]
    S & R & C --> E{Ensemble verdict}
    M --> L[ML Lab comparison]
    E --> X[explain.py + SHAP]
    E --> I[insights.py<br/>attack type · severity · talkers]
    X & I & L --> UI[app.py · SOC console]
    G[generate.py<br/>labelled captures] --> T[scripts/train_models.py] --> C
Loading
FlowWatch/
├── app.py                     # Streamlit SOC console (7 pages)
├── .streamlit/config.toml     # dark theme
├── assets/logo.svg
├── src/
│   ├── parsing.py             # CSV / JSON / PCAP → flows
│   ├── datasets.py            # UNSW-NB15 and CIC-IDS2017 loaders
│   ├── features.py            # per-flow and host-context features
│   ├── behaviour.py           # time-window rule detector
│   ├── detection.py           # mean + kσ, Isolation Forest, method comparison
│   ├── ml_unsupervised.py     # IF, LOF, One-Class SVM, autoencoder
│   ├── supervised.py          # classifiers, grouped CV, search, SHAP
│   ├── explain.py             # robust z-score explanations
│   ├── evaluate.py            # metrics and sweeps
│   ├── insights.py            # attack typing, severity, top talkers, timeline
│   ├── pipeline.py            # runs everything; hybrid and ensemble verdicts
│   ├── generate.py            # synthetic captures and sample PCAP
│   └── ui.py                  # CSS, palette, Plotly template, widgets
├── models/                    # supervised.joblib, unsupervised.joblib, model_card.json
├── data/                      # sample CSV / JSON / PCAP (+ labels), external/unsw_nb15_subset.csv
├── scripts/                   # train_models.py, run_pipeline.py, screenshots.py
├── tests/                     # 35 pytest tests
└── docs/                      # evaluation_results.txt, metrics.json, screenshots/

4. Methodology

Flows. Packets that share a unidirectional 5-tuple (src IP, dst IP, src port, dst port, protocol) form one flow. For each flow we record packets, bytes, duration, and SYN/ACK/RST/FIN counts.

Features.

  • Per flow: packets, bytes, duration, bytes per packet, packet rate, SYN ratio and RST ratio.
  • Whether the destination port is an authentication or well-known port.
  • Host context: distinct ports contacted by the source, repeat connections to the same service, and distinct sources hitting one service.
  • 60 s window aggregates: ports and hosts per source per window, sources and SYN ratio per service per window, and connections per (source, service) pair per window.

Counters are log-transformed (log1p) before going into a model.

Detectors.

# Detector Type Idea
1 mean + kσ (k = 2) statistical flag a flow if packets, bytes or duration has z > k
2 behaviour rules rule-based port scan: ≥ 20 ports per source in 60 s with ≤ 3 pkts/flow · SYN flood: ≥ 20 sources per service with ≥ 80 % SYN · brute force: ≥ 10 connections to an auth port · host sweep: ≥ 20 hosts
3 Isolation Forest unsupervised anomalies are isolated by fewer random splits
4 Local Outlier Factor unsupervised the flow's local density is low compared with its neighbours (novelty mode, k = 35)
5 One-Class SVM unsupervised RBF boundary around the bulk of the data (ν = contamination)
6 Autoencoder unsupervised MLPRegressor (k → k → 3 → k → k) trained to rebuild its input; score = reconstruction error
7 Random Forest / Gradient Boosting / XGBoost supervised 5-class attack classifier; the model with the best cross-validation score is deployed
8 Ensemble combined rules OR classifier OR (statistical volume alarm AND classifier p(attack) ≥ 0.2)

All unsupervised models share the same alert threshold: the (1 − contamination) quantile of their scores on the fitting data.

Avoiding data leakage

  1. Time-based split. The classifiers are trained on hours 0–4 of capture A (seed 101) and tested on hours 4–6 of a different capture B (seed 202). Capture B uses new attacker IPs, victims and attack intensities. A test checks that the time ranges don't overlap and that fewer than 5 % of test attacker IPs appear in training.
  2. Grouped cross-validation. Inside the training data, StratifiedGroupKFold(5) uses 10-minute time blocks as groups. Flows from one attack episode therefore never end up on both sides of a fold. RandomizedSearchCV (6 candidates per model, scored on macro-F1) picks the hyper-parameters, and the cross-validation score alone picks the deployed model.
  3. No identifiers as features. IPs, source ports and timestamps are excluded.
  4. Look-alike normal traffic. The training and test captures include benign traffic that resembles attacks: nightly backups (large and long, like exfiltration) and a monitoring server polling many ports (like a scan).
  5. Honest caveat. The sample PCAP was meant as an out-of-distribution check, but the first two training runs did badly on it. Recall on brute force and SYN flood was 0–0.44, because training brute-force campaigns were too large, exfiltration sizes too narrow, and DNS went to random servers instead of a resolver. I widened those generator ranges and retrained, so the PCAP is no longer a fully blind test. The time-holdout capture B is the blind test.

Explanations.

  • Model-agnostic: the robust z-score of each feature against the median/MAD baseline, normalised to a percentage share, plus an automatic sentence.
  • Rules: each rule quotes the value it measured and the threshold it crossed.
  • Classifier: SHAP TreeExplainer gives local values for the predicted class, and the mean |SHAP| over 300 flows gives a global view. If SHAP is unavailable, a perturbation explanation is used instead.
  • Feature importance: permutation importance (drop in macro-F1) plus impurity importance.

Attack type and severity. The ensemble decides the type in this order: rule type, then the classifier's class, then a heuristic. Severity is a base score for the type (exfiltration 90, SYN flood 80, brute force 70, port scan 50, unknown 30) plus up to 20 points from the ensemble score percentile. The levels are Critical ≥ 85, High ≥ 70, Medium ≥ 50, and Low below that.

5. Results

These are real outputs of scripts/run_pipeline.py, with contamination 0.10, k = 2 and 60 s windows. The full log is in docs/evaluation_results.txt and the numbers are also in docs/metrics.json.

5.1 Time-holdout test capture (8,371 flows, 2,175 attacks; never seen in training)

Detector Precision Recall F1 ROC-AUC
Statistical mean + 2σ 0.849 0.013 0.025 0.089
Isolation Forest (base 4 features) 0.490 0.189 0.272 0.661
Isolation Forest (extended) 0.883 0.340 0.491 0.905
Local Outlier Factor¹ 0.350 0.252 0.293 0.454
One-Class SVM¹ 0.299 0.105 0.156 0.831
Autoencoder¹ 0.113 0.059 0.078 0.204
Unsupervised consensus (≥ 2 of 4)¹ 0.338 0.163 0.220 0.597
Behaviour rules 0.9995 0.963 0.981 0.992
Hybrid (rules OR stat OR IF) 0.956 0.998 0.977 0.999
Random Forest classifier 0.998 1.000 0.999 1.000
Ensemble (rules + classifier) 0.997 1.000 0.999 1.000

¹ These models were fitted on the training capture and applied to the test capture, as they would be in deployment.

Recall per attack type (test capture)

Attack Statistical Isolation Forest Rules Classifier Ensemble
Port scan (551) 0.00 0.79 0.967 1.00 1.00
SYN flood (1,346) 0.00 0.02 0.998 1.00 1.00
Brute force (250) 0.00 1.00 0.876 1.00 1.00
Data exfiltration (28) 1.00 1.00 0.00 1.00 1.00

5.2 Supervised classifiers (5 classes)

Model CV macro-F1 (grouped, 5-fold) Test macro-F1 Test ROC-AUC (attack vs normal) PCAP macro-F1²
Random Forest (selected by CV) 0.998 ± 0.004 0.987 1.000 0.721
Gradient Boosting (HistGB) 0.993 ± 0.009 1.000 1.000 0.998
XGBoost 0.992 ± 0.010 0.993 1.000 0.905
  • Random Forest per-class F1 on the test capture: normal 1.000, brute force 1.000, port scan 1.000, SYN flood 1.000, exfiltration 0.933. Four normal flows (backups) were predicted as exfiltration.
  • Feature ablation: a Random Forest that uses only per-flow features (no host or window context) reaches macro-F1 0.969 on the test capture.
  • Permutation importance: src_distinct_dst_ports and dst_distinct_srcs matter most.

² On the sample PCAP, the Random Forest misclassifies the 3 exfiltration flows as brute force, 14 normal flows as brute force and 10 normal flows as SYN flood. Gradient Boosting would have done better, but picking the model by its PCAP score would be test-set leakage, so the CV choice stands. This is listed as a limitation.

5.3 Sample datasets

Dataset Statistical IF (extended) Rules Hybrid Classifier Ensemble
Sample CSV (3,712 flows), F1 0.033 0.616 0.982 0.967 1.000 1.000
Sample PCAP (693 flows), F1 0.267 0.524 0.949 0.949 0.923 0.923
Sample PCAP, recall 0.154 0.385 0.902 0.979 1.000 1.000

The sample CSV comes from the same generator family as the training data, so its perfect classifier score is in-distribution and should not be read as a claim of real-world accuracy.

5.4 Public dataset: UNSW-NB15 (real lab traffic)

This is a stratified subset of the official testing set: 6,922 records, 9 attack categories plus normal. The partition file has no IPs, ports or timestamps, so the only possible validation is per-record stratified 5-fold cross-validation, and the rule detector does not apply.

Model (UNSW native features) Precision Recall F1 ROC-AUC 10-class macro-F1
Random Forest 0.931 0.979 0.955 0.993 0.613
HistGradientBoosting 0.956 0.887 0.920 0.900 0.566
Isolation Forest (flow features) 0.359 0.064 0.108 0.172 –
LOF / OC-SVM / Autoencoder ≤ 0.58 ≤ 0.10 ≤ 0.18 ≤ 0.46 –

On this real data, telling attack from normal works well (F1 0.95). Telling the attack families apart is hard: Analysis and Backdoor F1 ≈ 0.23, DoS 0.40, Generic 0.97. The unsupervised models score below random because UNSW attacks are mostly small flows, not volume outliers. That matches what the statistical rule shows on our own data.

5.5 What this means (for the viva)

  • Per-flow detectors miss multi-flow attacks. Scans and floods are made of tiny flows, which is why mean + 2σ and the unsupervised models do poorly on them. Context features and window rules are what close the gap.
  • The unsupervised models disagree with each other. Their rank correlations range from −0.05 to 0.77. LOF and the autoencoder learn the dense attack clusters as "normal" (AUC < 0.5).
  • Supervised beats unsupervised only when labels exist. It also inherits the training distribution. The PCAP check exposed this, and the UNSW family-level F1 of 0.61 shows how much harder real data is.
  • The ensemble keeps an explainable rule layer. Every alert that a rule raises has a human-readable reason, and SHAP explains the alerts that only the classifier raises.

6. Using a public dataset

  • UNSW-NB15: a 1.3 MB subset ships in data/external/unsw_nb15_subset.csv. Pick "UNSW-NB15 public subset" in the sidebar. To use the full file, download UNSW_NB15_testing-set.csv from the UNSW site and upload it; its format is detected automatically.
  • CIC-IDS2017: download any MachineLearningCVE/*.csv from the CIC site and upload it. Its columns (Flow Duration, Total Fwd/Bwd Packets, flag counts, Label) are mapped by src/datasets.py::load_cicids. The mapping is unit-tested on a synthetic row only. The real files are several hundred MB, so they were not downloaded or evaluated here.

Citation: N. Moustafa and J. Slay, UNSW-NB15: a comprehensive data set for network intrusion detection systems, MilCIS 2015. The dataset is free for academic research.

7. Demo walkthrough for faculty review (≈ 12 min)

  1. Run streamlit run app.py. On Overview, point out the KPI deltas, the timeline (scan at ~10 min, flood at ~25, exfiltration at ~40, brute force at ~50) and the critical alerts.
  2. Live Analysis: switch the sample to the PCAP to show raw packets turned into 693 flows.
  3. Detection: filter Critical alerts, sort by bytes, and show which detectors fired. Then show the agreement heat-map and explain why each detector sees different attacks.
  4. ML Lab → Unsupervised: four ROC curves, and why LOF and the autoencoder fail. Supervised: model card (grouped CV + search), confusion matrix, ROC/PR curves, importance. Sensitivity: move the contamination slider.
  5. Explainability: pick a SYN-flood alert and read the rule sentence, the z-score bars and the SHAP bars.
  6. Threat Intel: the communication graph (spoofed flood sources cluster around the victim), the heat-maps, and incidents grouped by actor.
  7. Reports: download the HTML incident report.
  8. In a terminal, run python -m pytest -q (35 pass) and python scripts/run_pipeline.py. The numbers reproduce exactly.
  9. Optional: load the UNSW-NB15 subset to show the system on real lab traffic.

8. Limitations and future work

  • Training data is synthetic. The classifiers are near-perfect in-distribution but drop on differently generated traffic (PCAP: Random Forest macro-F1 0.72). Training on real labelled captures would be the next step.
  • Behaviour windows are fixed (tumbling) 60 s blocks, so attacks that cross a boundary can be split. Sliding windows would fix this.
  • There is no live capture yet (scapy.sniff could feed the same pipeline). Flows are unidirectional.
  • The CIC-IDS2017 loader is tested only on a synthetic row.

About

Network traffic monitoring and anomaly detection system

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages