A full-featured, client-side analytics dashboard for Spotify Extended Streaming History exports, with advanced behavioral analysis, F1-style ranking systems, session/chaining intelligence, interactive filtering, and deep per-entity drill-down.
- Project Overview
- Data Sources and Input Contracts
- Architecture and System Design
- ETL and Analytics Pipeline
- State Management, Filters, and Interaction Model
- Detailed Feature Breakdown by Tab
- Visualization Layer
- Data Engineering Details
- Machine Learning and Advanced Analytics Notes
- Technical Stack
- How to Run the Project
- Configuration and Tuning Parameters
- Performance Characteristics
- Limitations and Known Issues
- Security and Privacy Notes
- Future Improvements
- Project Structure Reference
Spotify Stats Analyzer ingests a Spotify "Extended streaming history" ZIP export and builds an interactive multi-tab analytics experience over your complete listening timeline. It transforms raw event-level playback logs into:
- KPI dashboards
- temporal and behavioral distributions
- streak and gap analysis
- ranking systems (including Formula 1-inspired championship scoring)
- discovery and loyalty insights
- exploratory raw-history tools
- comparative artist-versus-artist intelligence
- annual "Wrapped" narratives
- quiz/game mechanics derived from your own data
- Personal analytics users who want deeper insight than Spotify's standard yearly summary
- Data enthusiasts who want to inspect listening behavior at event granularity
- Users interested in longitudinal trends across multiple years
- Power users who want configurable skip detection, streak thresholds, and ranking logic
Spotify exports are raw JSON event streams that are difficult to inspect at scale. This project solves:
- high-friction manual analysis of large JSON history archives
- lack of custom definitions (skip logic, minimum play threshold)
- inability to perform cross-sectional and longitudinal filtering interactively
- lack of rich drill-down from aggregate charts into entity-level detail
The main product is a browser-based static app. Data processing happens in-memory in the client, so uploads are not transmitted to a backend service.
A secondary Streamlit script (app.py) exists as an alternative/prototype workflow.
Spotify Extended Streaming History ZIP export containing JSON files such as:
endsong_*.jsonStreaming_History_Audio_*.jsonStreaming_History_Video_*.json
The pipeline maps each record into a canonical event object with fields including:
ts(parsed timestamp)date(localYYYY-MM-DD)trackName,artistName,albumName- podcast-specific
episodeName,episodeShowName isPodcastmsPlayed,durationMinyear,month,hour,weekdayreasonStart,reasonEndplatform(normalized label)countryseason,timeOfDayskipped(derived by configurable policy)
- JSON files contain Spotify event objects with
tsand play duration information - corrupted JSON or invalid timestamps are dropped
- entries below configured minimum play time are discarded
- No GPX, CSV, or external APIs are required for the browser dashboard
- Optional external script in
index.htmlreferences Crazy Egg (analytics script)
The production app is a modular, client-side architecture:
index.html: shell, tab layout, controls, canvas targetsjs/main.js: bootstrap, ingestion trigger, filters, tab routingjs/store.js: core data processing + metric/insight computation enginejs/charts.js: chart rendering adapters (Chart.js wrappers)js/ui.js: top-level rendering orchestrationjs/detail.js: entity modal drill-down and chart renderingjs/tabs/*.js: per-tab presentation + interactions
flowchart LR
A[User uploads Spotify ZIP] --> B[processSpotifyZip in store.js]
B --> C[Normalize events + apply config filters]
C --> D[window.spotifyData.full]
D --> E[UI filter panel in main.js]
E --> F[window.spotifyData.filtered]
F --> G[Tab renderers in js/tabs]
G --> H[Chart layer in charts.js]
G --> I[Detail modal in detail.js]
store.js- deterministic analytics and aggregations
- reusable computation primitives shared by all tabs
- no direct DOM mutation
tabs/*- translate store outputs into HTML and chart invocations
- handle tab-local controls and user actions
main.js- global controls, filter lifecycle, tab activation/lazy render, reset flow
charts.js- Chart.js instance lifecycle (
destroy/recreate pattern) - consistent palette, axis styles, tooltip formatting
- Chart.js instance lifecycle (
- Global filters alter
window.spotifyData.filtered - Most tabs recompute against filtered dataset
- Detail modal opens from clickable rows/cards and computes against full dataset for context richness
- External API calls: none for analytics core
- Authentication: none required
- Rate limits: not applicable
- Caching:
- browser app: no persistent cache, in-memory only
- Streamlit prototype: uses
@st.cache_datafor weekly ranking computation
- Pagination:
- Explorer table paginates client-side (
PAGE_SIZE = 200)
- Explorer table paginates client-side (
This project follows a clear in-browser ETL pattern.
processSpotifyZip (store.js):
- opens ZIP with JSZip
- scans entries for supported Spotify history filename patterns
- parses each JSON payload and concatenates arrays
processEntry (store.js) applies strict filtering:
- reject if
ms_played < minPlayMs - reject invalid timestamps
- reject based on config flags:
- podcasts off
- offline plays off
- incognito plays off
For each retained entry:
- timestamp decomposition to year/month/hour/weekday/date
- season and time-of-day derivation
- platform normalization (device class mapping)
- skip classification using configurable policy:
- by reason
- by duration threshold
- or combined
Examples implemented in store.js:
- KPI features: totals, uniques, active days, average/day, skip rate
- temporal buckets: day/week/month/year aggregates
- distribution vectors: hour, weekday, month, season, platform, country, reasons
- sequence/session features: transitions and session segmentation by inactivity gap
- competition features: F1-style points, podiums, streaks, fastest laps
- comparison features: overlap metrics, duel points, quantile distributions
Tab modules request derived datasets and route them into Chart.js render functions, with interactive controls triggering recomputation.
window.spotifyData:
full: immutable canonical processed timeline after uploadfiltered: current working subset after active filters
Read from upload settings and injected into store config:
- minimum counted play duration
- skip detection mode and threshold
- top-N sizes
- streak gap threshold
- include/exclude podcasts/offline/incognito
- F1 minutes-vs-plays weight
- date range (
from,to) - year
- season
- platform
- countries (searchable multi-select; no selection means all countries)
- time of day
- skip state (all/completed/skipped)
- multi-select entities: artists, albums, tracks (with search and independent Include selected / Exclude selected modes; no selection means all)
applyFilters in main.js performs deterministic predicate filtering and then triggers full rerender of dependent tabs.
- searchable multi-selects with retained selections during search
- Ctrl/Cmd-click selects multiple countries or entities; Exclude selected keeps all entries except the chosen entities
- active filter pills with one-click removal
- removal and Reset clear selections, searches and exclusion modes; a new upload starts with fresh multi-select filters
- panel collapse/expand mechanics
- KPI card grid
- quick-facts cards
- timeline chart (daily/weekly/monthly/yearly)
- top tracks/artists/albums tables with sortable metrics and top-N controls
Filtered music events (!isPodcast && trackName).
calculateGlobalKPIscalculateTopItemswith optional metric:- plays
- minutes
- F1-derived points
- timeline via
calculateAggregatedTimeline - quick-facts custom computations (peak hour/day, one-hit ratio, weekend share)
- broad scale and intensity of listening behavior
- concentration and replay behavior
- comparative ranking shifts by selected metric
- timeline aggregation toggle
- top-N buttons (10/25/50)
- sort selectors per top list
- row click opens detail modal
- listening clock (polar)
- weekday/month/season distributions
- start/end reason distributions
- platform/country distributions (country plays use a logarithmic X axis, including countries with just one play)
- yearly evolution
- skip-rate trend chart with granularity controls
- weekday-hour heatmap bubble plot
Filtered full dataset (music + podcast where relevant to generic dimensions).
calculateTemporalDistributioncalculateSeasonDistributioncalculateDistributionPercentcalculateSkipRateTrendcalculateWeekdayHourMatrix
- when/where/on-what-device listening occurs
- behavior around starts/ends/skips
- macro temporal rhythm and seasonality
- skip trend granularity: day/week/month/year
- tooltips expose counts and percentages
- hero cards for longest/current streak and best periods
- milestone badges
- calendar heatmap with year selector
- top streaks and longest absence lists for artists/tracks/albums
Filtered dataset and date-level calendar aggregates.
calculateListeningStreakscalculateArtistDailyStreaks,calculateTrackDailyStreaks,calculateAlbumDailyStreakscalculateArtistGapStreaks,calculateTrackGapStreaks,calculateAlbumGapStreakscalculateBestPeriodsbuildCalendarData
- consistency patterns
- comeback gaps for top entities
- best-day/week/month/year performance windows
- calendar year navigation
- expandable streak and gap views
- behavior cards (personality, loyalty, hidden gems, skipped tracks, replay kings)
- diversity over time bars
- listening habits summary
- chain analysis sections
- session pattern analysis
Filtered music events and sessionized sequence data.
calculateDeepInsightscalculateListeningChainscalculateListeningSessions
- loyalty: distinct-year coverage by artist
- hidden gems: plays >= 20 and avg duration < 2.5 min
- abandoned tracks: high skip rate with minimum play count
- replay kings: same track repeated >= 3 times/day
- chain transitions: previous-to-next transitions within <= 30 min gaps
- sessionization: contiguous plays under inactivity threshold
- many list rows are detail-clickable to open entity modal
- championship mode selector (artists/tracks/albums)
- season selector
- score weight slider (minutes vs plays)
- standings table with sortable columns
- week-by-week top-10 inspection
- evolution charts (monthly/weekly cumulative)
- all-time records and yearly podium table
Filtered music events grouped weekly.
calculateF1Championship- weekly rank score:
- normalize minutes and plays per week
- weighted composite score
- F1 points map:
[25,18,15,12,10,8,6,4,2,1] - fastest-lap bonus:
- largest single-session play in week
- bonus applies only if entity is in weekly top 10
- season and all-time dominance
- contender consistency and streaks
- weekly competition context and turning points
- mode/year/week navigation
- sortable standings/all-time/week tables
- dynamic weight slider with immediate recomputation
- yearly story hero metrics
- trend pills vs previous year
- persona, peak, consistency/discovery cards
- quarter momentum, daypart DNA, monthly new-artist rhythm
- top songs/artists/albums lists
- monthly listening volume chart for the selected year only (no previous-year series)
- top 5 songs or artists by plays for each month of the selected year, with artist names, play counts and clickable details; months without music are explicitly marked
Selected year from full dataset.
calculateWrappedStats- year-over-year deltas
- quarter splits and half-year comparisons
- streak and concentration metrics
- discovery percentages vs previously seen artists/tracks
- compact narrative summary of a selected year
- growth/decline vs previous year
- taste concentration and behavior profile
- wrapped year selector
- clickable top lists open detail modal
Four quiz modes:
- Higher or Lower
- Guess the Year
- Complete the Chain
- First Listen Order
window.spotifyData.full (primarily music entries).
- pulls top entities and chain transitions from store
- builds randomized challenge sets with correctness scoring
- tracks score, streak, accuracy, best streak
- gamified self-validation of listening knowledge
- mode/category/round setup
- per-round choice buttons
- end-screen with stats and replay options
- explorer summary stats
- discovery timeline bars
- recent listening sessions
- full streaming history table with search/sort/filter/pagination
- CSV export
Filtered music dataset.
- sessionization via
calculateListeningSessions - first-play discovery derivation by entity key
- client-side table operations with page windowing
- row-level auditability and exportability
- session-level context on top of event logs
- search input and sort selectors
- completed/skipped filter
- page controls
- CSV export action
Animated progression/race chart system for top entities over time.
Filtered music dataset with user-selected parameters.
calculateViewerAccumulatedSeries- supports:
- entity: artist/album/track
- metric: minutes/plays/points
- value mode: accum/rolling/period/simple
- granularity: day/week/month/year
- date range and top-X
- trajectory dominance changes over time
- ranking dynamics and accumulation behavior
- build/play/pause/reset
- speed slider
- scrubber + step navigation
- chart mode line/bar
- lock camera scaling option
- podcast KPI grid
- top shows and top episodes charts
- podcast timeline with granularity controls
- hour/weekday charts
- yearly growth bars
- replayed episodes, binge days, discovered shows lists
Podcast entries (isPodcast or episode/show fields).
- show and episode aggregations
- completion rates
- listening streaks and active-day summaries
- timeline grouping day/week/month/year
- podcast-specific consumption profile
- show loyalty, binge behavior, replay tendencies
- chart click-to-show-detail
- timeline granularity toggles
- clickable show/episode lists
- artist A vs artist B selector
- head-to-head and weighted winner hero cards
- metric-by-metric verdict table with weights
- cumulative race chart and comparative profile charts
- recent weekly duel table
- overlap/platform/timeline extras
Filtered music entries for selected artists.
calculateArtistComparison- summary feature vectors for each artist
- weekly duel scoring (3/1/0 style)
- overlap (shared/exclusive tracks/albums, Jaccard-like percentages)
- weighted rule-based verdict engine
- distribution quantiles for session-like daily minutes
- objective side-by-side profile and winner framing
- ability to tune verdict via metric weighting
- random similar pair selection
- strict winner mode
- metric weight editing and reset
- race metric switching (minutes/plays/points)
- year/month/week calendar modes
- heat/intensity coloring by plays/day
- period stats ribbon
- per-day detail drawer with full event list
Filtered dataset mapped to date-indexed aggregates.
buildDayMap-style day aggregation in tab module- period summaries: plays, duration, active days, unique entities
- day detail includes track/podcast names, reason start/end, durations
- calendar-native view of consistency and burst days
- high-resolution daily audit trail
- prev/next/today navigation
- year/month/week toggles
- day-click detail overlay
Chart.js v4 with:
chartjs-plugin-datalabelschartjs-adapter-date-fns
- line charts (timelines, evolution curves)
- bar charts (top lists, grouped comparisons)
- doughnut charts (reason/platform distributions)
- polar area (hour clock)
- bubble chart (weekday-hour density)
- hover tooltips with computed context
- dynamic chart recreation to prevent stale instances
- clickable list/chart items feeding detail modal
- control-driven re-render (granularity, top-N, metric mode)
- bars for ranking clarity and magnitude comparison
- lines for temporal progression and trend continuity
- doughnut/polar for proportional and cyclical interpretation
- bubble matrix for quick hotspot detection in periodic behavior
- invalid timestamps removed
- entries below minimum duration removed
- optional exclusion of podcast/offline/incognito events
- null/empty entity names filtered in entity-level ranking paths
- unknown platform/country/reason values mapped to fallback labels
- track/album/artist nulls excluded from relevant computations
- defensive defaults avoid divide-by-zero issues
- session boundary threshold (default 30 min) reduces artificial cross-session transitions
- skip detection configurable by reason, time threshold, or union of both
- daily, weekly (Monday-start), monthly, yearly rollups
- per-entity grouped stats (plays, minutes, skips, points)
- per-bucket distribution percentages
Viewer tab supports rolling-mean mode with configurable window to smooth noisy period signals.
F1 scoring normalizes weekly plays and minutes before weighted blending:
$score = w_{min} \cdot \frac{minutes}{max(minutes)} + w_{play} \cdot \frac{plays}{max(plays)}$
- skip rate
- active-day intensity
- streak lengths and longest absences
- year-over-year deltas
- overlap percentages and weighted comparison verdicts
- fastest-lap weekly bonus in F1 mode
No predictive machine learning models are currently implemented.
A robust rules-based and statistics-driven analytics layer provides:
- ranking engines
- weighted scoring systems
- distributional profiling
- sequence/session mining
- comparative scorecards
- regression forecasting
- clustering/PCA embeddings
- SHAP/feature attribution
- survival/churn modeling
- next-track recommendation baseline with Markov models
- temporal demand forecasting for listening volume
- unsupervised taste segmentation (artist/genre vectors)
- anomaly detection for unusual listening periods
The Forecast tab uses the last available listening date (not an invented current-year snapshot) and projects that year-end plus the following three years. Select artists/songs, minutes/plays/F1 points and annual/all-time totals. Finish probability is always top 5, with no selector. Forecast uses full imported history rather than dashboard filters. Upload a new ZIP to invalidate the cache.
js/forecast.js: pure weekly aggregation, EWMA, seasonal volume projection, Monte Carlo and backtesting.js/forecast-worker.js: background computation and per-dataset/metric/entity/weight result cache.js/tabs/forecast.js: controls, rankings, simulation ranges, actual/predicted plays charts and validation tables.tests/forecast.test.mjs: synthetic tests; runnode --test tests/forecast.test.mjs(Node 22+).
Artist and song detail modals have an optional, initially collapsed Forecast - artist/song plays section. Expanding it computes that entity's cumulative and monthly actual/predicted plays, with year and annual/all-time controls. The entity is included even when outside the main simulation candidate pool; same-named songs are distinguished by artist. Predictions use the full history's cutoff and validated model/baseline, run in a background worker, and are cached by entity and import settings. Closing or changing the detail invalidates pending UI requests without discarding reusable cached results.
Weekly buckets use local Mondays through Sundays. Short plays are retained for Forecast plays/skips, but plays under 30 seconds contribute zero minutes. Other import settings still apply. The eight latest complete weeks drive total volume, multiplied by prior-year monthly indices. Entity EWMA has a four-week half-life, brief peaks decay over eight to ten weeks, and 5,000 seeded simulations include variance, persistent shocks, obsession spikes and anonymous Q4 newcomers estimated from earlier annual top tens. Unknown future names cannot be predicted.
The simulation pool contains up to 80 known entities, including annual/all-time leaders and recent movers, plus newcomer slots. Other known entities remain a volume tail; their individual position probabilities are not simulated. Ranges are conditional P10-P90 simulations, not empirically calibrated confidence intervals. Cumulative charts show observed plays as a solid line and median predicted plays as a dashed line. Monthly charts use separate actual/predicted bars with one category per month, for the selected year or all time. Monthly predicted values are differences of cumulative medians, including observed plays in the partial cutoff month. Allocations are approximate at week boundaries; annual volume boundaries are exact. Future F1 fastest laps use historical session-size proxies.
Backtesting predicts 2023, 2024 and 2025 from data through October 1 inclusive and compares with observed December 31 rankings. Incomplete years are explicitly unavailable. Top-3/top-10 set overlap and tie-aware Spearman on the union of top tens are displayed against the frozen-current-ranking baseline. The hybrid is selected only if its average weighted score improves without reducing top-3 or top-10 overlap; otherwise the ranking and entity shares are kept at baseline, with position probabilities unavailable rather than fictitious certainty. Seasonal volume simulation still runs. This short-horizon validation does not establish three-year forecast accuracy.
- HTML5/CSS3/JavaScript (ES modules)
- Chart.js + plugins
- JSZip for client-side ZIP parsing
- WordCloud2.js (word cloud capability)
- None required for browser app
- Streamlit (
app.py) - pandas, matplotlib, seaborn, plotly
- static-site compatible (any static file host)
- local run via lightweight static server
- modern browser (Chrome/Edge/Firefox)
- local static server (recommended for module/script compatibility)
- Clone or download this repository.
- Start a static server from project root:
python -m http.server 8000- Open:
http://localhost:8000
- Upload your Spotify Extended Streaming History ZIP from the landing screen.
- Python 3.9+
pip install -r requirements.txtstreamlit run app.pyThe Streamlit app and browser app are separate implementations. The browser app is the primary, feature-complete experience.
Upload-time settings control ingestion semantics:
Minimum play time to count(seconds)Skip detection method:- reason-only
- time-only
- both
Skip time threshold(seconds)Top lists sizeListening streak gap(days)F1 score weight(plays vs minutes)- include toggles:
- podcasts
- offline plays
- incognito plays
These settings materially change downstream KPIs and should be considered part of experiment/config metadata when sharing results.
- one-time parse and normalization on upload
- in-memory filtered view for fast iterative interactions
- lazy render for heavy tabs (streaks, deep dive, F1, explorer, viewer, compare, calendar, podcasts)
- chart instance reuse policy via explicit destroy/recreate to prevent memory leaks
- paginated table rendering in Explorer
Very large history exports can increase:
- initial parse latency
- memory footprint
- rerender cost when many filters/tabs are triggered together
- web worker offloading for ETL and heavy aggregations
- memoized selectors for repeated derived datasets
- incremental chart updates where feasible
- virtualized tables for very large row counts
- Calendar streak year extraction in
streakstab currently referencesendTimefor year derivation; this may not exist on normalized events and can affect year toggles. - Deep Dive "shuffle" and "offline" habit percentages rely on properties not explicitly persisted in normalized output, so reported values may default to zero unless source model is extended.
- Explorer tab includes a word-cloud function that may not be wired in all render paths.
- No automated test suite is included yet.
- No server-side persistence; all analytics reset on page refresh.
- Core analytics processing is client-side and local in browser memory.
- Uploaded ZIP contents are not sent to an app backend by this project.
index.htmlincludes an external Crazy Egg script tag; if strict local privacy is required, remove that script reference.
- Add automated unit/integration tests for
store.jsanalytics functions. - Introduce typed schema validation for Spotify event payloads.
- Move heavy transforms to Web Workers for large datasets.
- Add snapshot/export for dashboard state and reproducible analysis configs.
- Add true ML modules (forecasting, clustering, recommendation prototypes).
- Add CI checks and linting/format pipeline.
- Provide explicit data dictionary page in-app.
SpotifyStats/
├── index.html # Main static dashboard shell
├── style.css # Global styles
├── js/
│ ├── main.js # Bootstrap, upload flow, global filters, tab routing
│ ├── store.js # ETL + analytics engine
│ ├── charts.js # Chart rendering helpers
│ ├── ui.js # High-level render orchestration
│ ├── detail.js # Detail modal and entity drill-down
│ └── tabs/
│ ├── overview.js # KPIs + top lists + timeline
│ ├── trends.js # Temporal and distribution analytics
│ ├── streaks.js # Streaks, gaps, achievements, heatmap
│ ├── deepdive.js # Behavioral insights + chains + sessions
│ ├── f1.js # Championship ranking experience
│ ├── wrapped.js # Annual wrapped narrative
│ ├── game.js # Gamified quiz modes
│ ├── explorer.js # History table + discovery + CSV export
│ ├── viewer.js # Animated progression/race visualizer
│ ├── podcasts.js # Podcast-specific analytics
│ ├── compare.js # Artist-vs-artist analytics duel
│ └── calendar.js # Calendar navigation and day detail
├── app.py # Streamlit alternative/prototype implementation
└── requirements.txt # Python dependencies for Streamlit path
If you want, the next step can be a second document focused on contributor workflows (coding standards, adding a new tab, extending store.js safely, and writing regression tests for metric functions).