# Milestone v1.0: Data Pipeline

**Status:** SHIPPED 2026-01-10
**Phases:** 1-7 (+ 2.1 inserted)
**Total Plans:** 15

## Overview

Transform the monolithic data scraping scripts into a maintainable, sport-organized pipeline that ensures every game correctly links to its teams and stadium. Starting with script restructuring, we completed the stadium database, added alias systems for name variations, established correct canonical linking, implemented full CloudKit CRUD operations, and finished with comprehensive validation reports.

## Phases

### Phase 1: Script Architecture
**Goal**: Reorganize monolithic scraping scripts into sport-specific modules (MLB, NBA, NHL, NFL) for easier debugging and maintenance
**Depends on**: Nothing (first phase)
**Plans**: 3 plans

Plans:
- [x] 01-01: Create core.py shared module + mlb.py sport module
- [x] 01-02: Create nba.py + nhl.py sport modules
- [x] 01-03: Create nfl.py + refactor scrape_schedules.py orchestrator

**Details:**
- Created `core.py` with shared Game/Stadium dataclasses and scraper utilities
- Each sport module exports: {SPORT}_TEAMS, scrape_{sport}_games, {SPORT}_GAME_SOURCES
- Reduced orchestrator from 3359 to 733 lines (78% reduction)

### Phase 2: Stadium Foundation
**Goal**: Complete stadium database with correct coordinates, names, and venue data for all 4 sports
**Depends on**: Phase 1
**Plans**: 2 plans

Plans:
- [x] 02-01: Audit & complete hardcoded stadium data in sport modules
- [x] 02-02: Regenerate canonical data and verify pipeline

**Details:**
- Audited and completed hardcoded stadium data for MLB, NBA, NHL, NFL
- Used original opening years (not renovation years) for year_opened field
- Regenerated canonical JSON files with complete stadium coverage

### Phase 2.1: Additional Sports Stadiums (INSERTED)
**Goal**: Add hardcoded stadium data for secondary sports: MLS, WNBA, NWSL
**Depends on**: Phase 2
**Plans**: 3 plans

Plans:
- [x] 02.1-01: Create MLS module with 30 hardcoded stadiums
- [x] 02.1-02: Create WNBA module with 13 hardcoded arenas
- [x] 02.1-03: Create NWSL module with 13 hardcoded stadiums

**Details:**
- Created mls.py with 30 MLS stadiums including shared NFL venues
- Created wnba.py with 13 arenas cross-referenced from NBA/NHL
- Created nwsl.py with 13 stadiums cross-referenced from MLS
- CBB deferred (350+ D1 teams requires separate scoped phase)

### Phase 3: Alias Systems
**Goal**: Implement alias systems for both stadiums and teams to handle name variations across data sources
**Depends on**: Phase 2.1
**Plans**: 2 plans

Plans:
- [x] 03-01: Add NFL to canonicalization pipeline with aliases
- [x] 03-02: Add MLS, WNBA, NWSL to canonicalization pipeline with aliases

**Details:**
- Added TEAM_ABBREV_ALIASES for cross-source team name variations
- Stadium aliases handle historical names and source-specific variations
- All 7 sports now have alias support

### Phase 4: Canonical Linking
**Goal**: Ensure every game correctly links to its home/away teams and stadium via canonical IDs
**Depends on**: Phase 3
**Plans**: 1 plan

Plans:
- [x] 04-01: Generate canonical games with resolved team/stadium links

**Details:**
- All games correctly link to teams and stadiums via canonical IDs
- Team abbreviation aliases discovered during canonicalization added iteratively

### Phase 5: CloudKit CRUD
**Goal**: Implement full create, read, update, delete operations for CloudKit management
**Depends on**: Phase 4
**Plans**: 2 plans

Plans:
- [x] 05-01: Smart sync with change detection (diff reporting, differential upload)
- [x] 05-02: Verification and record management (sync verification, individual CRUD)

**Details:**
- New records use forceReplace; updates use recordChangeTag for conflict detection
- Orphan deletion requires explicit --delete-orphans flag (safe by default)
- Triple lookup fallback: direct recordName -> deterministic UUID -> canonicalId query

### Phase 6: Validation Reports
**Goal**: Generate validation reports showing record counts, data gaps, orphan records, and relationship integrity
**Depends on**: Phase 5
**Plans**: 1 plan

Plans:
- [x] 06-01: Comprehensive validation with orphan listing and completeness metrics

**Details:**
- Health score formula: avg_completeness - orphan_penalty (max -30) - unknown_penalty (max -10)
- --list-orphans requires CloudKit connection; --validate works offline
- Completeness metrics per sport with expected game counts

### Phase 7: Testing & Documentation
**Goal**: Complete pipeline documentation and finalize project status
**Depends on**: Phase 6
**Plans**: 1 plan

Plans:
- [x] 07-01: Create Scripts/README.md and update PROJECT.md with completion status

**Details:**
- Created comprehensive Scripts/README.md with usage examples
- Updated PROJECT.md with completion status and validated requirements

---

## Milestone Summary

**Decimal Phases:**
- Phase 2.1: Additional Sports Stadiums (inserted after Phase 2 for MLS/WNBA/NWSL coverage)

**Key Decisions:**
- Split by sport, not function (user preference for organization)
- Validation reports over automated tests (faster feedback, easier debugging)
- Full CRUD over upload-only (enable data corrections without full rebuild)
- Each sport module independent with own team abbrev functions
- Non-core sports remain inline with TODO markers for future extraction

**Issues Resolved:**
- Game linking failures (games now correctly link to teams/stadiums)
- Missing stadium data (148 stadiums complete with coordinates)
- Name variation mismatches (alias systems handle cross-source differences)

**Issues Deferred:**
- CBB support (350+ D1 teams requires separate scoped phase)

**Technical Debt Incurred:**
- None significant

---

_For current project status, see .planning/ROADMAP.md_