CS2 Analytics Platform
Live, end-to-end CS2 analytics platform - Python ingestion pipeline into PostgreSQL, a public FastAPI on Render, and a React dashboard on GitHub Pages.
Overview
CS2 Analytics is an open-source platform that scrapes, parses, and normalizes professional Counter-Strike 2 match data into a queryable PostgreSQL schema. I built it end-to-end - data model, scrapers, ingestion-state orchestration, API, and frontend - with reliability as the primary goal. It is deployed end to end: the FastAPI service runs on Render and a React dashboard on GitHub Pages serves player statistics from the production database.
Tech Stack
View more →Challenges & Solutions
Bypassing bot detection to scrape reliably
Standard requests/Selenium were blocked by anti-bot measures, breaking ingestion and causing intermittent failures.
Solution: Switched to SeleniumBase with human-like interaction settings and added retry/backoff logic. Centralized selectors, added health checks, and built a small harness to verify page states before extraction.
Impact: Achieved stable scrape sessions across long runs with far fewer blocks and timeouts.
Migrating from a single long-running script to ingestion-state tables
Initial ingestion ran as one long process, making failures costly and restarts painful; no isolation between work units and no durable record of what had been processed.
Solution: Refactored around PostgreSQL-backed ingestion-state tables for matches, maps, and demos, with source IDs as primary keys and explicit lifecycle states (rediscovery, retry, processed, failed, skipped) so rediscovery refreshes existing rows instead of duplicating work. Batch controllers own retry policy, scraper reset/rotation, and run summaries, while per-item stage services own fetch/parse/persist outcomes; scrapers only fetch and parsers only parse.
Impact: The pipeline is resumable and idempotent - failed items can be retried without re-running the entire pipeline.
Designing inserts to avoid duplicates while normalizing
Data was mostly complete but required consistent keys and relationships; reprocessing risked duplicate rows.
Solution: Defined a normalized Postgres schema with unique constraints and implemented idempotent upserts (ON CONFLICT) for matches, teams, and players. Added lightweight validation before writes.
Impact: Enabled safe reprocessing of records without duplication while maintaining referential integrity.
Maintaining reproducible local environments before containerization
Different developer machines and OS setups led to inconsistent Python deps, env vars, and DB connections.
Solution: Introduced `.env`-driven configuration, uv-managed dependencies with a committed lockfile, Alembic migrations behind a one-command DB init (`manage_db.py --init`), and Makefile-style entrypoints for setup, runs, and tests.
Impact: Reduced setup friction and made runs predictable; the setup evolved into the Docker Compose and CI baseline the project runs on today.