e5c6d0fec7
Phase 5: CloudKit CRUD - 2 plans created - 4 total tasks defined - Ready for execution Plan 05-01: Smart sync with change detection - Change detection with diff reporting - Differential sync (upload only changed records) Plan 05-02: Verification and record management - Sync verification (CloudKit vs local comparison) - Individual record CRUD operations Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
111 lines
4.5 KiB
Markdown
111 lines
4.5 KiB
Markdown
# Roadmap: SportsTime Data Pipeline
|
|
|
|
## Overview
|
|
|
|
Transform the monolithic data scraping scripts into a maintainable, sport-organized pipeline that ensures every game correctly links to its teams and stadium. Starting with script restructuring, we'll complete the stadium database, add alias systems for name variations, establish correct canonical linking, implement full CloudKit CRUD operations, and finish with comprehensive validation reports.
|
|
|
|
## Domain Expertise
|
|
|
|
None
|
|
|
|
## Phases
|
|
|
|
**Phase Numbering:**
|
|
- Integer phases (1, 2, 3): Planned milestone work
|
|
- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)
|
|
|
|
- [x] **Phase 1: Script Architecture** - Split monolithic scripts into sport-specific modules (3/3 plans)
|
|
- [x] **Phase 2: Stadium Foundation** - Complete stadium database with coordinates and names (2/2 plans)
|
|
- [x] **Phase 2.1: Additional Sports Stadiums** - Add stadium data for MLS, WNBA, NWSL, CBB (INSERTED) (3/3 plans)
|
|
- [x] **Phase 3: Alias Systems** - Stadium and team alias systems for name variations (2/2 plans)
|
|
- [x] **Phase 4: Canonical Linking** - Correct game→team→stadium relationships (1/1 plans)
|
|
- [ ] **Phase 5: CloudKit CRUD** - Full create, read, update, delete operations
|
|
- [ ] **Phase 6: Validation Reports** - Reports showing counts, gaps, orphan records
|
|
|
|
## Phase Details
|
|
|
|
### Phase 1: Script Architecture
|
|
**Goal**: Reorganize monolithic scraping scripts into sport-specific modules (MLB, NBA, NHL, NFL) for easier debugging and maintenance
|
|
**Depends on**: Nothing (first phase)
|
|
**Research**: Unlikely (internal refactoring, Python module patterns)
|
|
**Plans**: 3 plans
|
|
|
|
Plans:
|
|
- [x] 01-01: Create core.py shared module + mlb.py sport module
|
|
- [x] 01-02: Create nba.py + nhl.py sport modules
|
|
- [x] 01-03: Create nfl.py + refactor scrape_schedules.py orchestrator
|
|
|
|
### Phase 2: Stadium Foundation
|
|
**Goal**: Complete stadium database with correct coordinates, names, and venue data for all 4 sports
|
|
**Depends on**: Phase 1
|
|
**Research**: No (hardcoded data exists in sport modules, internal pipeline work)
|
|
**Plans**: 2 plans
|
|
|
|
Plans:
|
|
- [x] 02-01: Audit & complete hardcoded stadium data in sport modules
|
|
- [x] 02-02: Regenerate canonical data and verify pipeline
|
|
|
|
### Phase 2.1: Additional Sports Stadiums (INSERTED)
|
|
**Goal**: Add hardcoded stadium data for secondary sports: MLS, WNBA, NWSL (CBB deferred - 350+ D1 teams requires separate scoped phase)
|
|
**Depends on**: Phase 2
|
|
**Research**: No (stadium data compilation following established patterns)
|
|
**Plans**: 3 plans
|
|
|
|
Plans:
|
|
- [x] 02.1-01: Create MLS module with 30 hardcoded stadiums
|
|
- [x] 02.1-02: Create WNBA module with 13 hardcoded arenas
|
|
- [x] 02.1-03: Create NWSL module with 13 hardcoded stadiums
|
|
|
|
### Phase 3: Alias Systems
|
|
**Goal**: Implement alias systems for both stadiums and teams to handle name variations across data sources
|
|
**Depends on**: Phase 2.1
|
|
**Research**: No (internal mapping logic)
|
|
**Plans**: 2 plans
|
|
|
|
Plans:
|
|
- [x] 03-01: Add NFL to canonicalization pipeline with aliases
|
|
- [x] 03-02: Add MLS, WNBA, NWSL to canonicalization pipeline with aliases
|
|
|
|
### Phase 4: Canonical Linking
|
|
**Goal**: Ensure every game correctly links to its home/away teams and stadium via canonical IDs
|
|
**Depends on**: Phase 3
|
|
**Research**: Unlikely (existing model relationships)
|
|
**Plans**: 1 plan
|
|
|
|
Plans:
|
|
- [x] 04-01: Generate canonical games with resolved team/stadium links
|
|
|
|
### Phase 5: CloudKit CRUD
|
|
**Goal**: Implement full create, read, update, delete operations for CloudKit management
|
|
**Depends on**: Phase 4
|
|
**Research**: No (existing patterns in cloudkit_import.py sufficient)
|
|
**Plans**: 2 plans
|
|
|
|
Plans:
|
|
- [ ] 05-01: Smart sync with change detection (diff reporting, differential upload)
|
|
- [ ] 05-02: Verification and record management (sync verification, individual CRUD)
|
|
|
|
### Phase 6: Validation Reports
|
|
**Goal**: Generate validation reports showing record counts, data gaps, orphan records, and relationship integrity
|
|
**Depends on**: Phase 5
|
|
**Research**: Unlikely (internal reporting logic)
|
|
**Plans**: TBD
|
|
|
|
Plans:
|
|
- [ ] 06-01: TBD
|
|
|
|
## Progress
|
|
|
|
**Execution Order:**
|
|
Phases execute in numeric order: 1 → 2 → 2.1 → 3 → 4 → 5 → 6
|
|
|
|
| Phase | Plans Complete | Status | Completed |
|
|
|-------|----------------|--------|-----------|
|
|
| 1. Script Architecture | 3/3 | Complete | 2026-01-10 |
|
|
| 2. Stadium Foundation | 2/2 | Complete | 2026-01-10 |
|
|
| 2.1. Additional Sports Stadiums | 3/3 | Complete | 2026-01-10 |
|
|
| 3. Alias Systems | 2/2 | Complete | 2026-01-10 |
|
|
| 4. Canonical Linking | 1/1 | Complete | 2026-01-10 |
|
|
| 5. CloudKit CRUD | 0/2 | In progress | - |
|
|
| 6. Validation Reports | 0/TBD | Not started | - |
|