docs(scripts): add comprehensive README for data scraping pipeline
Documents the complete sportstime_parser package including architecture, multi-source scraping, name normalization with aliases, CloudKit uploads, and workflows for manual review and adding new sports. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Trey t
833
Scripts/README.md
Normal file
833
Scripts/README.md
Normal file
@@ -0,0 +1,833 @@
|
|||||||
|
# SportsTime Parser
|
||||||
|
|
||||||
|
A Python package for scraping, normalizing, and uploading sports schedule data to CloudKit for the SportsTime iOS app.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Overview](#overview)
|
||||||
|
- [Installation](#installation)
|
||||||
|
- [Quick Start](#quick-start)
|
||||||
|
- [Architecture](#architecture)
|
||||||
|
- [Directory Structure](#directory-structure)
|
||||||
|
- [Configuration](#configuration)
|
||||||
|
- [Data Models](#data-models)
|
||||||
|
- [Normalizers](#normalizers)
|
||||||
|
- [Scrapers](#scrapers)
|
||||||
|
- [Uploaders](#uploaders)
|
||||||
|
- [Utilities](#utilities)
|
||||||
|
- [Manual Review Workflow](#manual-review-workflow)
|
||||||
|
- [Adding a New Sport](#adding-a-new-sport)
|
||||||
|
- [Troubleshooting](#troubleshooting)
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The `sportstime_parser` package provides a complete pipeline for:
|
||||||
|
|
||||||
|
1. **Scraping** game schedules from multiple sources (Basketball-Reference, ESPN, MLB API, etc.)
|
||||||
|
2. **Normalizing** raw data to canonical identifiers (teams, stadiums, games)
|
||||||
|
3. **Resolving** team/stadium names using exact matching, historical aliases, and fuzzy matching
|
||||||
|
4. **Uploading** data to CloudKit with diff-based sync and resumable uploads
|
||||||
|
|
||||||
|
### Supported Sports
|
||||||
|
|
||||||
|
| Sport | Code | Sources | Season Format |
|
||||||
|
|-------|------|---------|---------------|
|
||||||
|
| NBA | `nba` | Basketball-Reference, ESPN, CBS | Oct-Jun (split year) |
|
||||||
|
| MLB | `mlb` | Baseball-Reference, MLB API, ESPN | Mar-Nov (single year) |
|
||||||
|
| NFL | `nfl` | ESPN, Pro-Football-Reference, CBS | Sep-Feb (split year) |
|
||||||
|
| NHL | `nhl` | Hockey-Reference, NHL API, ESPN | Oct-Jun (split year) |
|
||||||
|
| MLS | `mls` | ESPN, FBref | Feb-Nov (single year) |
|
||||||
|
| WNBA | `wnba` | ESPN | May-Oct (single year) |
|
||||||
|
| NWSL | `nwsl` | ESPN | Mar-Nov (single year) |
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd Scripts
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
|
||||||
|
- `requests` - HTTP requests with session management
|
||||||
|
- `beautifulsoup4` + `lxml` - HTML parsing
|
||||||
|
- `rapidfuzz` - Fuzzy string matching
|
||||||
|
- `pyjwt` + `cryptography` - CloudKit JWT authentication
|
||||||
|
- `rich` - Terminal UI (progress bars, logging)
|
||||||
|
- `pytz` / `timezonefinder` - Timezone detection
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Scrape a Single Sport
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sportstime_parser.scrapers import create_nba_scraper
|
||||||
|
|
||||||
|
scraper = create_nba_scraper(season=2025)
|
||||||
|
result = scraper.scrape_all()
|
||||||
|
|
||||||
|
print(f"Games: {result.game_count}")
|
||||||
|
print(f"Teams: {result.team_count}")
|
||||||
|
print(f"Stadiums: {result.stadium_count}")
|
||||||
|
print(f"Needs review: {result.review_count}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Upload to CloudKit
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sportstime_parser.uploaders import CloudKitClient, RecordDiffer
|
||||||
|
|
||||||
|
client = CloudKitClient(environment="development")
|
||||||
|
differ = RecordDiffer()
|
||||||
|
|
||||||
|
# Compare local vs remote
|
||||||
|
diff = differ.diff_games(local_games, remote_records)
|
||||||
|
|
||||||
|
# Upload changes
|
||||||
|
records = diff.get_records_to_upload()
|
||||||
|
result = await client.save_records(records)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ DATA SOURCES │
|
||||||
|
│ Basketball-Reference │ ESPN API │ MLB API │ Hockey-Reference │ etc. │
|
||||||
|
└────────────────────────────────┬────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ SCRAPERS │
|
||||||
|
│ NBAScraper │ MLBScraper │ NFLScraper │ NHLScraper │ MLSScraper │ etc. │
|
||||||
|
│ │
|
||||||
|
│ Features: │
|
||||||
|
│ • Multi-source fallback (try sources in priority order) │
|
||||||
|
│ • Automatic rate limiting with exponential backoff │
|
||||||
|
│ • Doubleheader detection │
|
||||||
|
│ • International game filtering (NFL London, NHL Global Series) │
|
||||||
|
└────────────────────────────────┬────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ NORMALIZERS │
|
||||||
|
│ TeamResolver │ StadiumResolver │ CanonicalIdGenerator │ AliasLoader │
|
||||||
|
│ │
|
||||||
|
│ Resolution Strategy (in order): │
|
||||||
|
│ 1. Exact match against canonical mappings │
|
||||||
|
│ 2. Date-aware alias lookup (handles renames/relocations) │
|
||||||
|
│ 3. Fuzzy matching with confidence threshold (85%) │
|
||||||
|
│ 4. Flag for manual review if unresolved or low confidence │
|
||||||
|
└────────────────────────────────┬────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ DATA MODELS │
|
||||||
|
│ Game │ Team │ Stadium │ ManualReviewItem │
|
||||||
|
│ │
|
||||||
|
│ All models use canonical IDs: │
|
||||||
|
│ • team_nba_lal (Los Angeles Lakers) │
|
||||||
|
│ • stadium_nba_los_angeles_lakers (Crypto.com Arena) │
|
||||||
|
│ • game_nba_2025_20251022_bos_lal (specific game) │
|
||||||
|
└────────────────────────────────┬────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ UPLOADERS │
|
||||||
|
│ CloudKitClient │ RecordDiffer │ StateManager │
|
||||||
|
│ │
|
||||||
|
│ Features: │
|
||||||
|
│ • JWT authentication with Apple's CloudKit Web Services │
|
||||||
|
│ • Batch operations (up to 200 records per request) │
|
||||||
|
│ • Diff-based sync (only upload changes) │
|
||||||
|
│ • Resumable uploads with persistent state │
|
||||||
|
└────────────────────────────────┬────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ CLOUDKIT │
|
||||||
|
│ Public Database: Games, Teams, Stadiums, Aliases │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
Scripts/
|
||||||
|
├── README.md # This file
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── pyproject.toml # Package configuration
|
||||||
|
├── league_structure.json # League hierarchy (conferences, divisions)
|
||||||
|
├── team_aliases.json # Historical team name mappings
|
||||||
|
├── stadium_aliases.json # Historical stadium name mappings
|
||||||
|
├── logs/ # Runtime logs (auto-created)
|
||||||
|
├── output/ # Scrape output files (auto-created)
|
||||||
|
└── sportstime_parser/ # Main package
|
||||||
|
├── __init__.py
|
||||||
|
├── config.py # Configuration constants
|
||||||
|
├── SOURCES.md # Data source documentation
|
||||||
|
├── models/ # Data classes
|
||||||
|
│ ├── game.py # Game model
|
||||||
|
│ ├── team.py # Team model
|
||||||
|
│ ├── stadium.py # Stadium model
|
||||||
|
│ └── aliases.py # Alias and ManualReviewItem models
|
||||||
|
├── normalizers/ # Name resolution
|
||||||
|
│ ├── canonical_id.py # ID generation
|
||||||
|
│ ├── alias_loader.py # Alias loading and resolution
|
||||||
|
│ ├── fuzzy.py # Fuzzy string matching
|
||||||
|
│ ├── timezone.py # Timezone detection
|
||||||
|
│ ├── team_resolver.py # Team name resolution
|
||||||
|
│ └── stadium_resolver.py # Stadium name resolution
|
||||||
|
├── scrapers/ # Sport-specific scrapers
|
||||||
|
│ ├── base.py # Abstract base scraper
|
||||||
|
│ ├── nba.py # NBA scraper
|
||||||
|
│ ├── mlb.py # MLB scraper
|
||||||
|
│ ├── nfl.py # NFL scraper
|
||||||
|
│ ├── nhl.py # NHL scraper
|
||||||
|
│ ├── mls.py # MLS scraper
|
||||||
|
│ ├── wnba.py # WNBA scraper
|
||||||
|
│ └── nwsl.py # NWSL scraper
|
||||||
|
├── uploaders/ # CloudKit integration
|
||||||
|
│ ├── cloudkit.py # CloudKit Web Services client
|
||||||
|
│ ├── diff.py # Record diffing
|
||||||
|
│ └── state.py # Resumable upload state
|
||||||
|
└── utils/ # Shared utilities
|
||||||
|
├── logging.py # Rich-based logging
|
||||||
|
├── http.py # Rate-limited HTTP client
|
||||||
|
└── progress.py # Progress tracking
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### config.py
|
||||||
|
|
||||||
|
Key configuration constants:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Directories
|
||||||
|
SCRIPTS_DIR = Path(__file__).parent.parent # Scripts/
|
||||||
|
OUTPUT_DIR = SCRIPTS_DIR / "output" # JSON output
|
||||||
|
STATE_DIR = SCRIPTS_DIR / ".parser_state" # Upload state
|
||||||
|
|
||||||
|
# CloudKit
|
||||||
|
CLOUDKIT_CONTAINER = "iCloud.com.sportstime.app"
|
||||||
|
CLOUDKIT_ENVIRONMENT = "development" # or "production"
|
||||||
|
|
||||||
|
# Rate Limiting
|
||||||
|
DEFAULT_REQUEST_DELAY = 3.0 # seconds between requests
|
||||||
|
MAX_RETRIES = 3 # retry attempts
|
||||||
|
BACKOFF_FACTOR = 2.0 # exponential backoff multiplier
|
||||||
|
INITIAL_BACKOFF = 5.0 # initial backoff duration
|
||||||
|
|
||||||
|
# Fuzzy Matching
|
||||||
|
FUZZY_THRESHOLD = 85 # minimum match confidence (0-100)
|
||||||
|
|
||||||
|
# Expected game counts (for validation)
|
||||||
|
EXPECTED_GAME_COUNTS = {
|
||||||
|
"nba": 1230, # 30 teams × 82 games ÷ 2
|
||||||
|
"mlb": 2430, # 30 teams × 162 games ÷ 2
|
||||||
|
"nfl": 272, # Regular season only
|
||||||
|
"nhl": 1312, # 32 teams × 82 games ÷ 2
|
||||||
|
"mls": 544, # 29 teams × ~34 games ÷ 2
|
||||||
|
"wnba": 228, # 12 teams × 40 games ÷ 2
|
||||||
|
"nwsl": 182, # 14 teams × 26 games ÷ 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# Geography (for filtering international games)
|
||||||
|
ALLOWED_COUNTRIES = {"USA", "Canada"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### league_structure.json
|
||||||
|
|
||||||
|
Defines the hierarchical structure of each league:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"nba": {
|
||||||
|
"name": "National Basketball Association",
|
||||||
|
"conferences": {
|
||||||
|
"Eastern": {
|
||||||
|
"divisions": {
|
||||||
|
"Atlantic": ["BOS", "BKN", "NYK", "PHI", "TOR"],
|
||||||
|
"Central": ["CHI", "CLE", "DET", "IND", "MIL"],
|
||||||
|
"Southeast": ["ATL", "CHA", "MIA", "ORL", "WAS"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Western": { ... }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"mlb": { ... },
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### team_aliases.json / stadium_aliases.json
|
||||||
|
|
||||||
|
Historical name mappings with validity dates:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"team_mlb_athletics": [
|
||||||
|
{
|
||||||
|
"alias": "Oakland Athletics",
|
||||||
|
"alias_type": "full_name",
|
||||||
|
"valid_from": "1968-01-01",
|
||||||
|
"valid_until": "2024-12-31"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"alias": "Las Vegas Athletics",
|
||||||
|
"alias_type": "full_name",
|
||||||
|
"valid_from": "2028-01-01",
|
||||||
|
"valid_until": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Models
|
||||||
|
|
||||||
|
### Game
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class Game:
|
||||||
|
id: str # Canonical ID: game_{sport}_{season}_{date}_{away}_{home}
|
||||||
|
sport: str # Sport code (nba, mlb, etc.)
|
||||||
|
season: int # Season start year
|
||||||
|
home_team_id: str # Canonical team ID
|
||||||
|
away_team_id: str # Canonical team ID
|
||||||
|
stadium_id: str # Canonical stadium ID
|
||||||
|
game_date: datetime # UTC datetime
|
||||||
|
game_number: Optional[int] # 1 or 2 for doubleheaders
|
||||||
|
home_score: Optional[int] # None if not played
|
||||||
|
away_score: Optional[int]
|
||||||
|
status: str # scheduled, final, postponed, cancelled
|
||||||
|
source_url: Optional[str] # For manual review
|
||||||
|
raw_home_team: Optional[str] # Original scraped value
|
||||||
|
raw_away_team: Optional[str]
|
||||||
|
raw_stadium: Optional[str]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Team
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class Team:
|
||||||
|
id: str # Canonical ID: team_{sport}_{abbrev}
|
||||||
|
sport: str
|
||||||
|
city: str # e.g., "Los Angeles"
|
||||||
|
name: str # e.g., "Lakers"
|
||||||
|
full_name: str # e.g., "Los Angeles Lakers"
|
||||||
|
abbreviation: str # e.g., "LAL"
|
||||||
|
conference: Optional[str] # e.g., "Western"
|
||||||
|
division: Optional[str] # e.g., "Pacific"
|
||||||
|
stadium_id: Optional[str] # Home stadium
|
||||||
|
primary_color: Optional[str]
|
||||||
|
secondary_color: Optional[str]
|
||||||
|
logo_url: Optional[str]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stadium
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class Stadium:
|
||||||
|
id: str # Canonical ID: stadium_{sport}_{city_team}
|
||||||
|
sport: str
|
||||||
|
name: str # Current name (e.g., "Crypto.com Arena")
|
||||||
|
city: str
|
||||||
|
state: Optional[str]
|
||||||
|
country: str
|
||||||
|
latitude: Optional[float]
|
||||||
|
longitude: Optional[float]
|
||||||
|
capacity: Optional[int]
|
||||||
|
surface: Optional[str] # grass, turf, ice, hardwood
|
||||||
|
roof_type: Optional[str] # dome, retractable, open
|
||||||
|
opened_year: Optional[int]
|
||||||
|
image_url: Optional[str]
|
||||||
|
timezone: Optional[str]
|
||||||
|
```
|
||||||
|
|
||||||
|
### ManualReviewItem
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class ManualReviewItem:
|
||||||
|
item_type: str # "team" or "stadium"
|
||||||
|
raw_value: str # Original scraped value
|
||||||
|
suggested_id: Optional[str] # Best fuzzy match (if any)
|
||||||
|
confidence: float # 0.0 - 1.0
|
||||||
|
reason: str # Why review is needed
|
||||||
|
source_url: Optional[str] # Where it came from
|
||||||
|
sport: str
|
||||||
|
check_date: Optional[date] # For date-aware alias lookup
|
||||||
|
```
|
||||||
|
|
||||||
|
## Normalizers
|
||||||
|
|
||||||
|
### Canonical ID Generation
|
||||||
|
|
||||||
|
IDs are deterministic and immutable:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Team ID
|
||||||
|
generate_team_id("nba", "LAL")
|
||||||
|
# → "team_nba_lal"
|
||||||
|
|
||||||
|
# Stadium ID
|
||||||
|
generate_stadium_id("nba", "Los Angeles", "Lakers")
|
||||||
|
# → "stadium_nba_los_angeles_lakers"
|
||||||
|
|
||||||
|
# Game ID
|
||||||
|
generate_game_id(
|
||||||
|
sport="nba",
|
||||||
|
season=2025,
|
||||||
|
away_abbrev="BOS",
|
||||||
|
home_abbrev="LAL",
|
||||||
|
game_date=datetime(2025, 10, 22),
|
||||||
|
game_number=None
|
||||||
|
)
|
||||||
|
# → "game_nba_2025_20251022_bos_lal"
|
||||||
|
|
||||||
|
# Doubleheader Game ID
|
||||||
|
generate_game_id(..., game_number=2)
|
||||||
|
# → "game_nba_2025_20251022_bos_lal_2"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Team Resolution
|
||||||
|
|
||||||
|
The `TeamResolver` uses a three-stage strategy:
|
||||||
|
|
||||||
|
```python
|
||||||
|
resolver = get_team_resolver("nba")
|
||||||
|
result = resolver.resolve(
|
||||||
|
"Los Angeles Lakers",
|
||||||
|
check_date=date(2025, 10, 22),
|
||||||
|
source_url="https://..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Result:
|
||||||
|
# - canonical_id: "team_nba_lal"
|
||||||
|
# - confidence: 1.0 (exact match)
|
||||||
|
# - review_item: None
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution stages:**
|
||||||
|
|
||||||
|
1. **Exact Match**: Check against canonical team mappings
|
||||||
|
- Full name: "Los Angeles Lakers"
|
||||||
|
- City + Name: "Los Angeles" + "Lakers"
|
||||||
|
- Abbreviation: "LAL"
|
||||||
|
|
||||||
|
2. **Alias Lookup**: Check historical aliases with date awareness
|
||||||
|
- "Oakland Athletics" → "team_mlb_athletics" (valid until 2024-12-31)
|
||||||
|
- Handles relocations: "Oakland" → "Las Vegas" transition
|
||||||
|
|
||||||
|
3. **Fuzzy Match**: Use rapidfuzz with 85% threshold
|
||||||
|
- "LA Lakers" → "Los Angeles Lakers" (92% match)
|
||||||
|
- Low-confidence matches flagged for review
|
||||||
|
|
||||||
|
### Stadium Resolution
|
||||||
|
|
||||||
|
Similar three-stage strategy with additional location awareness:
|
||||||
|
|
||||||
|
```python
|
||||||
|
resolver = get_stadium_resolver("nba")
|
||||||
|
result = resolver.resolve(
|
||||||
|
"Crypto.com Arena",
|
||||||
|
check_date=date(2025, 10, 22)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key features:**
|
||||||
|
- Handles naming rights changes (Staples Center → Crypto.com Arena)
|
||||||
|
- Date-aware: "Staples Center" resolves correctly for historical games
|
||||||
|
- Location-based fallback using latitude/longitude
|
||||||
|
|
||||||
|
## Scrapers
|
||||||
|
|
||||||
|
### Base Scraper
|
||||||
|
|
||||||
|
All scrapers extend `BaseScraper` with these features:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class BaseScraper(ABC):
|
||||||
|
def __init__(self, sport: str, season: int): ...
|
||||||
|
|
||||||
|
# Required implementations
|
||||||
|
def _get_sources(self) -> list[str]: ...
|
||||||
|
def _scrape_games_from_source(self, source: str) -> list[RawGameData]: ...
|
||||||
|
def _normalize_games(self, raw_games) -> tuple[list[Game], list[ManualReviewItem]]: ...
|
||||||
|
def scrape_teams(self) -> list[Team]: ...
|
||||||
|
def scrape_stadiums(self) -> list[Stadium]: ...
|
||||||
|
|
||||||
|
# Built-in features
|
||||||
|
def scrape_games(self) -> ScrapeResult:
|
||||||
|
"""Multi-source fallback - tries each source in order."""
|
||||||
|
...
|
||||||
|
|
||||||
|
def scrape_all(self) -> ScrapeResult:
|
||||||
|
"""Scrapes games, teams, and stadiums with progress tracking."""
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
### NBA Scraper
|
||||||
|
|
||||||
|
```python
|
||||||
|
class NBAScraper(BaseScraper):
|
||||||
|
"""
|
||||||
|
Sources (in priority order):
|
||||||
|
1. Basketball-Reference - HTML tables, monthly pages
|
||||||
|
2. ESPN API - JSON, per-date queries
|
||||||
|
3. CBS Sports - Backup (not implemented)
|
||||||
|
|
||||||
|
Season: October to June (split year, e.g., 2025-26)
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Basketball-Reference parsing:**
|
||||||
|
- URL: `https://www.basketball-reference.com/leagues/NBA_{year}_games-{month}.html`
|
||||||
|
- Table columns: date_game, visitor_team_name, home_team_name, visitor_pts, home_pts, arena_name
|
||||||
|
|
||||||
|
### MLB Scraper
|
||||||
|
|
||||||
|
```python
|
||||||
|
class MLBScraper(BaseScraper):
|
||||||
|
"""
|
||||||
|
Sources:
|
||||||
|
1. Baseball-Reference - Single page per season
|
||||||
|
2. MLB Stats API - Official API with date range queries
|
||||||
|
3. ESPN API - Backup
|
||||||
|
|
||||||
|
Season: March to November (single year)
|
||||||
|
Handles: Doubleheaders with game_number
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
|
||||||
|
### NFL Scraper
|
||||||
|
|
||||||
|
```python
|
||||||
|
class NFLScraper(BaseScraper):
|
||||||
|
"""
|
||||||
|
Sources:
|
||||||
|
1. ESPN API - Week-based queries
|
||||||
|
2. Pro-Football-Reference - Single page per season
|
||||||
|
|
||||||
|
Season: September to February (split year)
|
||||||
|
Filters: International games (London, Mexico City, Frankfurt)
|
||||||
|
Scrapes: Preseason (4 weeks), Regular (18 weeks), Postseason (4 rounds)
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
|
||||||
|
### NHL Scraper
|
||||||
|
|
||||||
|
```python
|
||||||
|
class NHLScraper(BaseScraper):
|
||||||
|
"""
|
||||||
|
Sources:
|
||||||
|
1. Hockey-Reference - Single page per season
|
||||||
|
2. NHL API - New API (api-web.nhle.com)
|
||||||
|
3. ESPN API - Backup
|
||||||
|
|
||||||
|
Season: October to June (split year)
|
||||||
|
Filters: International games (Prague, Stockholm, Helsinki)
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
|
||||||
|
### MLS / WNBA / NWSL Scrapers
|
||||||
|
|
||||||
|
All use ESPN API as primary source with similar structure:
|
||||||
|
- Single calendar year seasons
|
||||||
|
- Conference-based organization (MLS) or single table (WNBA, NWSL)
|
||||||
|
|
||||||
|
## Uploaders
|
||||||
|
|
||||||
|
### CloudKit Client
|
||||||
|
|
||||||
|
```python
|
||||||
|
class CloudKitClient:
|
||||||
|
"""CloudKit Web Services API client with JWT authentication."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
container_id: str = CLOUDKIT_CONTAINER,
|
||||||
|
environment: str = "development", # or "production"
|
||||||
|
key_id: str = None, # From CloudKit Dashboard
|
||||||
|
private_key: str = None, # EC P-256 private key
|
||||||
|
): ...
|
||||||
|
|
||||||
|
async def fetch_records(
|
||||||
|
self,
|
||||||
|
record_type: RecordType,
|
||||||
|
filter_by: Optional[dict] = None,
|
||||||
|
sort_by: Optional[str] = None,
|
||||||
|
) -> list[dict]: ...
|
||||||
|
|
||||||
|
async def save_records(
|
||||||
|
self,
|
||||||
|
records: list[CloudKitRecord],
|
||||||
|
batch_size: int = 200,
|
||||||
|
) -> BatchResult: ...
|
||||||
|
|
||||||
|
async def delete_records(
|
||||||
|
self,
|
||||||
|
record_names: list[str],
|
||||||
|
record_type: RecordType,
|
||||||
|
) -> BatchResult: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Authentication:**
|
||||||
|
- Uses EC P-256 key pair
|
||||||
|
- JWT tokens signed with private key
|
||||||
|
- Tokens valid for 30 minutes
|
||||||
|
|
||||||
|
### Record Differ
|
||||||
|
|
||||||
|
```python
|
||||||
|
class RecordDiffer:
|
||||||
|
"""Compares local records with CloudKit records."""
|
||||||
|
|
||||||
|
def diff_games(self, local: list[Game], remote: list[dict]) -> DiffResult: ...
|
||||||
|
def diff_teams(self, local: list[Team], remote: list[dict]) -> DiffResult: ...
|
||||||
|
def diff_stadiums(self, local: list[Stadium], remote: list[dict]) -> DiffResult: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**DiffResult:**
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class DiffResult:
|
||||||
|
creates: list[RecordDiff] # New records to create
|
||||||
|
updates: list[RecordDiff] # Changed records to update
|
||||||
|
deletes: list[RecordDiff] # Remote records to delete
|
||||||
|
unchanged: list[RecordDiff] # Records with no changes
|
||||||
|
|
||||||
|
def get_records_to_upload(self) -> list[CloudKitRecord]:
|
||||||
|
"""Returns creates + updates ready for upload."""
|
||||||
|
```
|
||||||
|
|
||||||
|
### State Manager
|
||||||
|
|
||||||
|
```python
|
||||||
|
class StateManager:
|
||||||
|
"""Manages resumable upload state."""
|
||||||
|
|
||||||
|
def load_session(self, sport, season, environment) -> Optional[UploadSession]: ...
|
||||||
|
def save_session(self, session: UploadSession) -> None: ...
|
||||||
|
def get_session_or_create(
|
||||||
|
self,
|
||||||
|
sport, season, environment,
|
||||||
|
record_names: list[tuple[str, str]],
|
||||||
|
resume: bool = False,
|
||||||
|
) -> UploadSession: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**State persistence:**
|
||||||
|
- Stored in `.parser_state/upload_state_{sport}_{season}_{env}.json`
|
||||||
|
- Tracks: pending, uploaded, failed records
|
||||||
|
- Supports retry with backoff
|
||||||
|
|
||||||
|
## Utilities
|
||||||
|
|
||||||
|
### HTTP Client
|
||||||
|
|
||||||
|
```python
|
||||||
|
class RateLimitedSession:
|
||||||
|
"""HTTP session with rate limiting and exponential backoff."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
delay: float = 3.0, # Seconds between requests
|
||||||
|
max_retries: int = 3,
|
||||||
|
backoff_factor: float = 2.0,
|
||||||
|
): ...
|
||||||
|
|
||||||
|
def get(self, url, **kwargs) -> Response: ...
|
||||||
|
def get_json(self, url, **kwargs) -> dict: ...
|
||||||
|
def get_html(self, url, **kwargs) -> str: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Features:**
|
||||||
|
- User-agent rotation (5 different Chrome/Firefox/Safari agents)
|
||||||
|
- Per-domain rate limiting
|
||||||
|
- Automatic 429 handling with exponential backoff + jitter
|
||||||
|
- Connection pooling
|
||||||
|
|
||||||
|
### Logging
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sportstime_parser.utils import get_logger, log_success, log_error
|
||||||
|
|
||||||
|
logger = get_logger() # Rich-formatted logger
|
||||||
|
logger.info("Starting scrape")
|
||||||
|
|
||||||
|
log_success("Scraped 1230 games") # Green checkmark
|
||||||
|
log_error("Failed to parse") # Red X
|
||||||
|
```
|
||||||
|
|
||||||
|
**Log output:**
|
||||||
|
- Console: Rich-formatted with colors
|
||||||
|
- File: `logs/parser_{timestamp}.log`
|
||||||
|
|
||||||
|
### Progress Tracking
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sportstime_parser.utils import ScrapeProgress, track_progress
|
||||||
|
|
||||||
|
# Specialized scrape tracking
|
||||||
|
progress = ScrapeProgress("nba", 2025)
|
||||||
|
progress.start()
|
||||||
|
|
||||||
|
with progress.scraping_schedule(total_months=9) as advance:
|
||||||
|
for month in months:
|
||||||
|
fetch(month)
|
||||||
|
advance()
|
||||||
|
|
||||||
|
progress.finish() # Prints summary
|
||||||
|
|
||||||
|
# Generic progress bar
|
||||||
|
for game in track_progress(games, "Processing games"):
|
||||||
|
process(game)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual Review Workflow
|
||||||
|
|
||||||
|
When the system can't confidently resolve a team or stadium:
|
||||||
|
|
||||||
|
1. **Low confidence fuzzy match** (< 85%):
|
||||||
|
```
|
||||||
|
ManualReviewItem(
|
||||||
|
item_type="team",
|
||||||
|
raw_value="LA Lakers",
|
||||||
|
suggested_id="team_nba_lal",
|
||||||
|
confidence=0.82,
|
||||||
|
reason="Fuzzy match below threshold"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **No match found**:
|
||||||
|
```
|
||||||
|
ManualReviewItem(
|
||||||
|
raw_value="Unknown Team FC",
|
||||||
|
suggested_id=None,
|
||||||
|
confidence=0.0,
|
||||||
|
reason="No match found in canonical mappings"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Ambiguous match** (multiple candidates):
|
||||||
|
```
|
||||||
|
ManualReviewItem(
|
||||||
|
raw_value="LA",
|
||||||
|
suggested_id="team_nba_lac",
|
||||||
|
confidence=0.5,
|
||||||
|
reason="Ambiguous: could be Lakers or Clippers"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution:**
|
||||||
|
- Review items are exported to JSON
|
||||||
|
- Manually verify and add to `team_aliases.json` or `stadium_aliases.json`
|
||||||
|
- Re-run scrape - aliases will be used for resolution
|
||||||
|
|
||||||
|
## Adding a New Sport
|
||||||
|
|
||||||
|
1. **Create scraper** in `scrapers/{sport}.py`:
|
||||||
|
```python
|
||||||
|
class NewSportScraper(BaseScraper):
|
||||||
|
def __init__(self, season: int, **kwargs):
|
||||||
|
super().__init__("newsport", season, **kwargs)
|
||||||
|
self._team_resolver = get_team_resolver("newsport")
|
||||||
|
self._stadium_resolver = get_stadium_resolver("newsport")
|
||||||
|
|
||||||
|
def _get_sources(self) -> list[str]:
|
||||||
|
return ["primary_source", "backup_source"]
|
||||||
|
|
||||||
|
def _scrape_games_from_source(self, source: str) -> list[RawGameData]:
|
||||||
|
# Implement source-specific scraping
|
||||||
|
...
|
||||||
|
|
||||||
|
def _normalize_games(self, raw_games) -> tuple[list[Game], list[ManualReviewItem]]:
|
||||||
|
# Use resolvers to normalize
|
||||||
|
...
|
||||||
|
|
||||||
|
def scrape_teams(self) -> list[Team]:
|
||||||
|
# Return canonical team list
|
||||||
|
...
|
||||||
|
|
||||||
|
def scrape_stadiums(self) -> list[Stadium]:
|
||||||
|
# Return canonical stadium list
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Add team mappings** in `normalizers/team_resolver.py`:
|
||||||
|
```python
|
||||||
|
TEAM_MAPPINGS["newsport"] = {
|
||||||
|
"ABC": ("team_newsport_abc", "Full Team Name", "City"),
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Add stadium mappings** in `normalizers/stadium_resolver.py`:
|
||||||
|
```python
|
||||||
|
STADIUM_MAPPINGS["newsport"] = {
|
||||||
|
"stadium_newsport_venue": StadiumInfo(
|
||||||
|
name="Venue Name",
|
||||||
|
city="City",
|
||||||
|
state="State",
|
||||||
|
country="USA",
|
||||||
|
latitude=40.0,
|
||||||
|
longitude=-74.0,
|
||||||
|
),
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Add to league_structure.json** (if hierarchical)
|
||||||
|
|
||||||
|
5. **Update config.py**:
|
||||||
|
```python
|
||||||
|
EXPECTED_GAME_COUNTS["newsport"] = 500
|
||||||
|
```
|
||||||
|
|
||||||
|
6. **Export from `__init__.py`**
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Rate Limiting (429 errors)
|
||||||
|
|
||||||
|
The system handles these automatically with exponential backoff. If persistent:
|
||||||
|
- Increase `DEFAULT_REQUEST_DELAY` in config.py
|
||||||
|
- Check if source has changed their rate limits
|
||||||
|
|
||||||
|
### Missing Teams/Stadiums
|
||||||
|
|
||||||
|
1. Check scraper logs for raw values
|
||||||
|
2. Add to `team_aliases.json` or `stadium_aliases.json`
|
||||||
|
3. Or add to canonical mappings if it's a new team/stadium
|
||||||
|
|
||||||
|
### CloudKit Authentication Errors
|
||||||
|
|
||||||
|
1. Verify key_id matches CloudKit Dashboard
|
||||||
|
2. Check private key format (EC P-256, PEM)
|
||||||
|
3. Ensure container identifier is correct
|
||||||
|
|
||||||
|
### Incomplete Scrapes
|
||||||
|
|
||||||
|
The system discards partial data on errors. Check:
|
||||||
|
- `logs/` for error details
|
||||||
|
- Network connectivity
|
||||||
|
- Source website availability
|
||||||
|
|
||||||
|
### International Games Appearing
|
||||||
|
|
||||||
|
NFL and NHL scrapers filter these automatically. If new locations emerge:
|
||||||
|
- Add to `INTERNATIONAL_LOCATIONS` in the scraper
|
||||||
|
- Or add filtering logic for neutral site games
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
1. Follow existing patterns for new scrapers
|
||||||
|
2. Always use canonical IDs
|
||||||
|
3. Add aliases for historical names
|
||||||
|
4. Include source URLs for traceability
|
||||||
|
5. Test with multiple seasons
|
||||||
Reference in New Issue
Block a user