A data science project that collects, cleans, and analyzes millions of Amazon customer reviews for digital devices (laptops, tablets, desktops). The project covers exploratory data analysis, statistical hypothesis testing, and a two-stage Aspect-Based Sentiment Analysis (ABSA) pipeline - all presented through an interactive web dashboard.
Requirements: Python 3.11, a Google Cloud project, and GCP credentials configured locally.
# Create and activate a virtual environment
python -m venv .venv
.venv\Scripts\Activate.ps1 # Windows
source .venv/bin/activate # macOS/Linux
# Install dependencies
pip install -r requirements.txtEnvironment variables (set in shell or .env):
| Variable | Description |
|---|---|
GCP_PROJECT |
GCP project ID (default: cs163-amazon-analysis) |
GCS_BUCKET_NAME |
Cloud Storage bucket name |
ABSA_API_URL |
URL of the deployed ABSA inference service |
Run the web app locally:
python -m app.mainRun the data pipeline:
python -m data_cleaning_pipeline.run_pipeline # all steps
python -m data_cleaning_pipeline.run_pipeline --steps 2 3 # specific steps onlyAmazon Electronics Data + Metadata
│
▼
[Step 1] ML Filter (data_cleaning_pipeline/step1_ml_filter.py)
TF-IDF + Logistic Regression classifies product titles
as digital devices or not.
│
▼
[Step 2] data_cleaning_pipeline/step2_bq_queries.py
Joins reviews ↔ product metadata, deduplicates rows,
and produces a clean combined table and save under dataset/digital_devices_reviews_no_duplicates.csv.
(initially we did joins the reviews in BigQuery but I also make this step 2 to run locally)
│
▼
[Step 3] EDA Data Prep (data_cleaning_pipeline/step3_eda_data.py)
Cleans timestamps, assigns device categories (laptop/tablet/desktop),
extracts brands, translates non-English reviews, applies VADER
sentiment scoring, assigns price tiers. Output: dataset/eda_ready.csv
(or BigQuery table in cloud mode).
│
▼
[Step 4] Exploratory Data Analysis (eda/)
Analyzes eda_ready.csv across multiple dimensions: category
distribution, rating trends, price tiers, review volume over time,
COVID-period impact, text/word patterns, and feature correlations.
Each analysis module (eda/overview.py, category.py, ratings.py,
price.py, time.py, text.py, covid.py, correlation.py) produces
interactive Plotly figures rendered live in the Dash app
at app/pages/analytics/eda.py.
│
▼
[ABSA Training] — Google Colab (external)
Domain-Adaptive Pre-Training (DAPT) on RoBERTa, then fine-tuned
as a multitask model (M1) for aspect detection and a DeBERTa model
(M2) for aspect-level sentiment classification.
Colab notebook: https://colab.research.google.com/drive/1bem5kz0FBWC22zAFnU1D_xa6oHxdUZPW
│
▼
[ABSA Inference API] (absa-api/)
FastAPI service containerized with Docker and deployed on Cloud Run.
Exposes /predict and /predict/batch endpoints.
│
▼
[Dash Web App] (app/)
Interactive dashboard deployed on Google App Engine.
Visualizes EDA findings, hypothesis tests, and live ABSA demos.
Amazon_Electronic_Analysis/
├── data_cleaning_pipeline/ # Three-step data processing pipeline
│ ├── run_pipeline.py # Orchestrator — runs all steps in order
│ ├── step1_ml_filter.py # ML classifier to identify digital device products
│ ├── step2_bq_queries.py # BigQuery SQL: join reviews ↔ products, deduplicate
│ └── step3_eda_data.py # Feature engineering and EDA data preparation
│
├── absa-api/ # Containerized ABSA inference service (FastAPI + Docker)
│ ├── app/
│ │ ├── main.py # FastAPI app with /predict and /predict/batch endpoints
│ │ ├── models.py # Model architectures (M1: RoBERTa+LoRA, M2: DeBERTa)
│ │ └── schemas.py # Pydantic request/response schemas
│ ├── dapt_roberta/ # Domain-adapted RoBERTa backbone (weights via Drive)
│ ├── model1_multitask/ # Multitask aspect-detection model (weights via Drive)
│ ├── model2_sentiment/ # Aspect-sentiment classifier (weights via Drive)
│ ├── Dockerfile
│ └── requirements.txt
│
├── app/ # Dash web application (Google App Engine)
│ ├── main.py # Entry point; initializes Dash server
│ ├── layout.py # Shared layout components and helpers
│ ├── callbacks.py # Dash callback logic
│ └── pages/ # One module per dashboard page
│ ├── overview.py # Project overview and research questions
│ ├── dataset.py # Dataset description page
│ ├── methods.py # Methods and pipeline explanation
│ ├── analytics/ # EDA and hypothesis analysis pages
│ │ ├── analytics.py # Analytics section landing page
│ │ ├── eda.py # Exploratory data analysis visualizations
│ │ ├── hypothesis1.py # Hypothesis 1 test and results
│ │ ├── hypothesis2.py # Hypothesis 2 test and results
│ │ └── hypothesis3.py # Hypothesis 3 test and results
│ └── models/ # ABSA model pages and live demo
│ ├── data_overview.py # Labeled training data overview
│ ├── demo.py # Live ABSA inference demo
│ ├── models.py # Model architecture description page
│ └── models_detail.py # Detailed model breakdown
│
├── eda/ # EDA visualization modules (used by the Dash app)
│ ├── overview.py, category.py, ratings.py, price.py, time.py
│ ├── text.py, correlation.py, covid.py
│ └── hypothesis1.py, hypothesis2.py, hypothesis3.py
│
├── dataset/ # Local CSV data files (intermediate pipeline outputs)
│ ├── digital_devices_reviews_no_duplicates.csv # Post-Step-2 deduplicated data
│ ├── eda_ready.csv # Post-Step-3 feature-engineered data
│ ├── final_nonlabel.csv
│ └── final.csv
│
├── sql/ # BigQuery SQL scripts
│ ├── link_reviews_products.sql # Joins reviews with product metadata
│ └── duplicate_and_null_handling.sql
│
├── models/ # Supplementary model utilities and training helpers (used by Dash app)
│ ├── augmentation.py
│ └── data_processing.py
│
├── notebooks/ # Jupyter notebooks for model development, training, and analysis
│ ├── absa_pipeline_vfinal.ipynb # Full end-to-end ABSA training pipeline (canonical version)
│ ├── absa_model1.ipynb # Standalone notebook for Model 1 (aspect detection, RoBERTa+LoRA)
│ ├── absa_model2.ipynb # Standalone notebook for Model 2 (sentiment classification, DeBERTa)
│ ├── data_separation.ipynb # Splits and prepares labeled datasets for M1 and M2 training
│ └── hypothesis.ipynb # Statistical hypothesis testing experiments
│
├── app.yaml # Google App Engine deployment configuration
└── requirements.txt # Python dependencies for the web app
┌──────────────────────────────────────────────────────────────────────┐
│ Amazon Electronics Dataset │
│ (HuggingFace — reviews + product metadata) │
└──────────────────────────────┬───────────────────────────────────────┘
│
▼
╔═════════════════════════════════════════╗
║ DATA CLEANING PIPELINE ║
║ ║
║ ┌──────────────────────────────────┐ ║
║ │ Step 1 · ML Filter │ ║
║ │ TF-IDF + Logistic Regression │ ║
║ │ identifies digital device titles │ ║
║ └─────────────────┬────────────────┘ ║
║ │ ║
║ ┌─────────────────▼────────────────┐ ║
║ │ Step 2 · BigQuery SQL │ ║
║ │ Join reviews ↔ product metadata │ ║
║ │ Deduplicate rows │ ║
║ └─────────────────┬────────────────┘ ║
║ │ ║
║ ┌─────────────────▼────────────────┐ ║
║ │ Step 3 · EDA Data Prep │ ║
║ │ VADER sentiment · brand extract │ ║
║ │ translations · price tiers │ ║
║ └─────────────────┬────────────────┘ ║
╚════════════════════╪════════════════════╝
│
▼
┌──────────────┐
│ GCS │
│ (bucket) │
└──────┬───────┘
│
┌─────────────────────────┘
│
│ ┌────────────────────────────────────────┐
│ │ ABSA Training (Google Colab) │
│ │ │
│ │ DAPT pre-training → RoBERTa backbone │
│ │ ↓ │
│ │ M1 · RoBERTa + LoRA (multitask) │
│ │ aspect detection per category │
│ │ ↓ │
│ │ M2 · DeBERTa (sentiment classifier) │
│ │ positive / negative per aspect │
│ └────────────────────┬───────────────────┘
│ │
│ ▼
│ ┌───────────────────────────────┐
│ │ ABSA Inference API │
│ │ FastAPI · Docker │
│ │ Google Cloud Run │
│ │ │
│ │ GET /health │
│ │ POST /predict │
│ │ POST /predict/batch │
│ └───────────────┬───────────────┘
│ │ HTTP REST
│ ▼
│ ┌───────────────────────────────┐
└───────▶│ Dash Web App │
│ Google App Engine │
│ │
│ Overview · Dataset · EDA │
│ Hypothesis · ABSA Live Demo │
└───────────────┬───────────────┘
│
▼
┌──────────┐
│ User │
│ Browser │
└──────────┘
Scalability:
- App Engine scales automatically from 0 to 3 instances based on traffic, serving the Dash dashboard.
- Cloud Run handles the ABSA API as a stateless container; it scales independently and only runs when requests arrive (scale-to-zero).
- BigQuery is serverless and handles large-scale SQL queries without infrastructure management.
- The two services (dashboard and inference API) are decoupled — the API can be updated or scaled without redeploying the frontend.
The inference service lives in absa-api/ and is packaged as a Docker container.
- Docker Desktop installed and running
- Model weights downloaded from Google Drive and placed under
absa-api/:
absa-api/
├── dapt_roberta/ ← download from [Drive]
├── model1_multitask/ ← download from [Drive]
├── model2_sentiment/ ← download from [Drive]
├── app/
├── ├──main.py
├── ├──models.py
├── └──schemas.py
├── Dockerfile
└── requirements.txt
cd absa-api
docker build -t absa-api .The Dockerfile:
- Uses
python:3.11-slimas the base image - Installs CPU-only PyTorch (
torch==2.3.0) to keep the image lean - Pre-downloads the DeBERTa base model (
yangheng/deberta-v3-base-absa-v1.1) at build time so the container has no outbound internet dependency at runtime - Copies the three model weight directories into
/app/models_artifacts/ - Exposes port
8080and starts Uvicorn with a single worker (models are ~1.5 GB each; multiple workers would OOM)
docker run --rm -p 8080:8080 absa-apiTest the running container:
curl http://localhost:8080/health
curl -X POST http://localhost:8080/predict \
-H "Content-Type: application/json" \
-d '{"text": "The battery life is excellent but the display is dim."}'# Authenticate and set project
gcloud auth login
gcloud config set project cs163-amazon-analysis
# Build and push to Google Container Registry
gcloud builds submit --tag gcr.io/cs163-amazon-analysis/absa-api .
# Deploy
gcloud run deploy absa-api \
--image gcr.io/cs163-amazon-analysis/absa-api \
--platform managed \
--region us-central1 \
--memory 8Gi \
--cpu 2 \
--timeout 300 \
--allow-unauthenticatedThe ABSA API runs as a FastAPI application on Google Cloud Run:
URL: https://absa-api-467984634090.us-central1.run.app
Endpoints:
| Method | Path | Description |
|---|---|---|
GET |
/health |
Liveness check |
POST |
/predict |
Run ABSA on a single review text |
POST |
/predict/batch |
Run ABSA on a list of review texts |
Inference pipeline (two-stage):
- M1 — RoBERTa + LoRA (multitask): Determines if a review is product-related and technical, then predicts which product aspects (e.g., battery, display, performance) are mentioned using per-aspect confidence thresholds.
- M2 — DeBERTa (sentiment): For each detected aspect, classifies its sentiment as positive or negative using a (review, aspect) text-pair encoding.
The models are loaded once at container startup and kept in memory. A single Uvicorn worker is used to avoid out-of-memory errors (~1.5 GB per model).
Google BigQuery (cs163-amazon-analysis):
| Dataset | Table | Description |
|---|---|---|
amazon_electronics |
meta_Electronics |
Raw product metadata |
amazon_electronics |
ml_sample2 |
Labeled training data for the ML filter |
amazon_digital_devices_cleaned |
metadata_digital_device_result |
ML-filtered product metadata |
amazon_digital_devices_cleaned |
digital_devices_reviews_no_duplicates |
Deduplicated reviews joined with product data |
amazon_digital_devices_cleaned |
eda_ready |
Feature-engineered dataset ready for analysis |
Google Cloud Storage — bucket cs163-amazon-review-analysis-data: stores intermediate and processed data files.
Google Drive — trained model weights (not committed to the repo due to file size): Model Files on Google Drive
Download and place the model directories under absa-api/:
absa-api/
├── dapt_roberta/
├── model1_multitask/
└── model2_sentiment/
To prepare the data for Aspect-Based Sentiment Analysis (ABSA), we created both labeled and unlabeled datasets from the product review corpus.
The labeled dataset was constructed by sampling 500 product reviews from the review data. To ensure that the sample was representative of the original dataset, the reviews were sampled based on the rating distribution. This helped preserve the proportion of low-rated, neutral, and high-rated reviews in the labeled subset.
After sampling, each review was split into individual sentences. These sentence-level samples were then manually annotated with the following labels:
is_related: whether the sentence is related to the product or user experienceis_technical: whether the sentence discusses a technical feature or technical issuetarget_aspect: the specific product aspect mentioned in the sentencetarget_sentiment: the sentiment expressed toward the target aspect
This labeled dataset was used for supervised training and evaluation of the ABSA models.
The unlabeled dataset was created from the remaining reviews after removing the reviews used in the labeled dataset. These reviews were also split into sentence-level samples to match the format of the labeled data.
Unlike the labeled dataset, the unlabeled data does not contain manual annotations. It was used to improve the model’s understanding of domain-specific language in electronic product reviews. The final_nonlabel dataset (unlabeled data) can be obtained by using final.csv as the labeled data and eda_ready.csv, then running the data_separation.ipynb notebook.
This folder contains the main Jupyter notebooks used during model development, experimentation, and analysis.
The notebooks absa_model1.ipynb and absa_model2.ipynb were separated from the original absa_pipeline_vfinal.ipynb notebook.
They can be executed independently if you only need to reproduce or work with one specific ABSA model:
-
absa_pipeline_vfinal.ipynb
Full final ABSA pipeline notebook. -
absa_model1.ipynb
Standalone notebook for ABSA Model 1. -
absa_model2.ipynb
Standalone notebook for ABSA Model 2.
Because Model 1 and Model 2 were split from the full ABSA pipeline, each notebook can be run separately without having to execute the entire absa_pipeline_vfinal.ipynb.
These notebooks were not originally executed in the local project directory structure.
Therefore, if you want to run them locally using the current repository layout, you need to update the dataset paths.
Since all notebooks are located inside the notebooks/ folder, dataset files should be accessed by moving one level up to the project root, then entering the dataset/ folder.
Use this path format:
../dataset/<file_name>