63acf7accb
Adds the full Django application layer on top of sportstime_parser: - core: Sport, Team, Stadium, Game models with aliases and league structure - scraper: orchestration engine, adapter, job management, Celery tasks - cloudkit: CloudKit sync client, sync state tracking, sync jobs - dashboard: staff dashboard for monitoring scrapers, sync, review queue - notifications: email reports for scrape/sync results - Docker setup for deployment (Dockerfile, docker-compose, entrypoint) Game exports now use game_datetime_utc (ISO 8601 UTC) instead of venue-local date+time strings, matching the canonical format used by the iOS app. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
132 lines
4.0 KiB
Markdown
132 lines
4.0 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides context for Claude Code when working on this project.
|
|
|
|
## Project Overview
|
|
|
|
SportsTime is a Django-based sports data pipeline that scrapes game schedules from official sources, normalizes the data, stores it in PostgreSQL, and syncs to CloudKit for iOS app consumption.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────┐
|
|
│ Data Sources │ ──▶ │ Scrapers │ ──▶ │ PostgreSQL │ ──▶ │ CloudKit │
|
|
│ (ESPN, leagues) │ │ (sportstime_ │ │ (Django) │ │ (iOS) │
|
|
└─────────────────┘ │ parser) │ └─────────────┘ └──────────┘
|
|
└──────────────┘
|
|
```
|
|
|
|
## Key Directories
|
|
|
|
- `core/` - Django models: Sport, Team, Stadium, Game, Conference, Division, Aliases
|
|
- `scraper/` - Scraper orchestration, adapter, job management
|
|
- `sportstime_parser/` - Standalone scraper library (ESPN, league APIs)
|
|
- `cloudkit/` - CloudKit sync client and job management
|
|
- `dashboard/` - Staff dashboard for monitoring and controls
|
|
- `templates/` - Django templates for dashboard UI
|
|
|
|
## Data Flow
|
|
|
|
1. **Scraper runs** (manual or scheduled via Celery Beat)
|
|
2. **sportstime_parser** fetches from ESPN/league APIs
|
|
3. **Adapter** normalizes data and resolves team/stadium names
|
|
4. **Django models** store normalized data with CloudKit sync flags
|
|
5. **CloudKit sync** pushes pending records to iCloud
|
|
|
|
## Models Hierarchy
|
|
|
|
```
|
|
Sport
|
|
├── Conference
|
|
│ └── Division
|
|
│ └── Team (has TeamAliases)
|
|
├── Stadium (has StadiumAliases)
|
|
└── Game (references Team, Stadium)
|
|
```
|
|
|
|
## Name Resolution
|
|
|
|
Team and stadium names from scraped data are resolved via:
|
|
1. Direct ID match (canonical IDs from scraper)
|
|
2. Database aliases (TeamAlias/StadiumAlias with date validity)
|
|
3. Direct name/abbreviation match
|
|
|
|
Aliases support validity dates for historical names (e.g., team relocations, stadium naming rights).
|
|
|
|
## Common Tasks
|
|
|
|
### Run a scraper
|
|
```bash
|
|
docker-compose exec web python manage.py shell
|
|
>>> from scraper.tasks import run_scraper_task
|
|
>>> run_scraper_task.delay(config_id)
|
|
```
|
|
|
|
### Check scraper status
|
|
Visit `/dashboard/scraper-status/` or check `ScrapeJob` model.
|
|
|
|
### Add team/stadium alias
|
|
Use Django admin at `/admin/core/teamalias/` or `/admin/core/stadiumalias/`.
|
|
|
|
### Export/Import data
|
|
All admin models support import/export (JSON, CSV, XLSX) via django-import-export.
|
|
|
|
### Sync to CloudKit
|
|
```bash
|
|
docker-compose exec web python manage.py shell
|
|
>>> from cloudkit.tasks import run_cloudkit_sync
|
|
>>> run_cloudkit_sync.delay(config_id)
|
|
```
|
|
|
|
## Environment
|
|
|
|
- **Docker Compose** for local development
|
|
- **PostgreSQL** database
|
|
- **Redis** for Celery broker
|
|
- **Celery** for async tasks and scheduled jobs
|
|
|
|
## Key Files
|
|
|
|
- `sportstime/settings.py` - Django settings
|
|
- `scraper/engine/adapter.py` - Bridges sportstime_parser to Django
|
|
- `scraper/engine/db_alias_loader.py` - Database alias resolution
|
|
- `core/resources.py` - Import/export resource definitions
|
|
- `docker-compose.yml` - Container orchestration
|
|
|
|
## Supported Sports
|
|
|
|
| Code | Sport | Season Type |
|
|
|------|-------|-------------|
|
|
| nba | NBA Basketball | split (Oct-Jun) |
|
|
| mlb | MLB Baseball | calendar (Mar-Oct) |
|
|
| nfl | NFL Football | split (Sep-Feb) |
|
|
| nhl | NHL Hockey | split (Oct-Jun) |
|
|
| mls | MLS Soccer | calendar (Feb-Nov) |
|
|
| wnba | WNBA Basketball | calendar (May-Sep) |
|
|
| nwsl | NWSL Soccer | calendar (Mar-Nov) |
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
docker-compose exec web pytest
|
|
```
|
|
|
|
## Useful Commands
|
|
|
|
```bash
|
|
# Restart containers
|
|
docker-compose restart
|
|
|
|
# Rebuild after requirements change
|
|
docker-compose down && docker-compose up -d --build
|
|
|
|
# View logs
|
|
docker-compose logs -f web
|
|
|
|
# Django shell
|
|
docker-compose exec web python manage.py shell
|
|
|
|
# Database shell
|
|
docker-compose exec db psql -U sportstime -d sportstime
|
|
```
|