feat: add Django web app, CloudKit sync, dashboard, and game_datetime_utc export
Adds the full Django application layer on top of sportstime_parser: - core: Sport, Team, Stadium, Game models with aliases and league structure - scraper: orchestration engine, adapter, job management, Celery tasks - cloudkit: CloudKit sync client, sync state tracking, sync jobs - dashboard: staff dashboard for monitoring scrapers, sync, review queue - notifications: email reports for scrape/sync results - Docker setup for deployment (Dockerfile, docker-compose, entrypoint) Game exports now use game_datetime_utc (ISO 8601 UTC) instead of venue-local date+time strings, matching the canonical format used by the iOS app. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Trey t
176
templates/dashboard/export.html
Normal file
176
templates/dashboard/export.html
Normal file
@@ -0,0 +1,176 @@
|
||||
{% extends 'base.html' %}
|
||||
|
||||
{% block content %}
|
||||
<div class="page-header">
|
||||
<h2>Export Data</h2>
|
||||
</div>
|
||||
|
||||
<!-- Overview Stats -->
|
||||
<div class="stat-grid mb-2">
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ counts.teams }}</div>
|
||||
<div class="stat-label">Teams</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ counts.stadiums }}</div>
|
||||
<div class="stat-label">Stadiums</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ counts.games|floatformat:0 }}</div>
|
||||
<div class="stat-label">Games</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ counts.team_aliases }}</div>
|
||||
<div class="stat-label">Team Aliases</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ counts.stadium_aliases }}</div>
|
||||
<div class="stat-label">Stadium Aliases</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">
|
||||
<!-- Export Options -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Export Options</h3>
|
||||
</div>
|
||||
<form id="exportForm" method="get" action="{% url 'dashboard:export_download' %}">
|
||||
<div style="margin-bottom: 1.5rem;">
|
||||
<label style="display: block; font-weight: 600; margin-bottom: 0.5rem;">Data Types</label>
|
||||
<div style="display: grid; gap: 0.5rem;">
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="sports" checked>
|
||||
<span>Sports</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.sports }} sports)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="league_structure" checked>
|
||||
<span>League Structure</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.conferences }} conferences, {{ counts.divisions }} divisions)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="teams" checked>
|
||||
<span>Teams</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.teams }} teams)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="stadiums" checked>
|
||||
<span>Stadiums</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.stadiums }} stadiums)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="games" checked>
|
||||
<span>Games</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.games }} games)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="team_aliases" checked>
|
||||
<span>Team Aliases</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.team_aliases }} aliases)</span>
|
||||
</label>
|
||||
<label style="display: flex; align-items: center; gap: 0.5rem; cursor: pointer;">
|
||||
<input type="checkbox" name="type" value="stadium_aliases" checked>
|
||||
<span>Stadium Aliases</span>
|
||||
<span class="text-muted" style="font-size: 0.85rem;">({{ counts.stadium_aliases }} aliases)</span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style="margin-bottom: 1.5rem;">
|
||||
<label style="display: block; font-weight: 600; margin-bottom: 0.5rem;">Filter by Sport</label>
|
||||
<select name="sport" style="width: 100%; padding: 0.5rem; border: 1px solid #ddd; border-radius: 6px;">
|
||||
<option value="">All Sports</option>
|
||||
{% for sport in sports %}
|
||||
<option value="{{ sport.code }}">{{ sport.short_name }} - {{ sport.name }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div style="margin-bottom: 1.5rem;">
|
||||
<label style="display: block; font-weight: 600; margin-bottom: 0.5rem;">Filter Games by Year</label>
|
||||
<select name="year" style="width: 100%; padding: 0.5rem; border: 1px solid #ddd; border-radius: 6px;">
|
||||
<option value="">All Years</option>
|
||||
{% for year in years %}
|
||||
<option value="{{ year }}">{{ year }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
<span class="text-muted" style="font-size: 0.85rem; display: block; margin-top: 0.25rem;">All games played in this calendar year, regardless of season</span>
|
||||
</div>
|
||||
|
||||
<button type="submit" class="btn btn-primary" style="width: 100%; padding: 0.75rem;">
|
||||
Download Export
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
<!-- Export Files Info -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Output Files</h3>
|
||||
</div>
|
||||
<div style="display: grid; gap: 1rem;">
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">sports_canonical.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Sport metadata: display names, SF Symbol icons, brand colors, and season month ranges.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">league_structure.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
League hierarchy: sports as leagues, conferences, and divisions with parent_id references.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">teams_canonical.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Teams with canonical IDs, abbreviations, stadium references, and conference/division IDs.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">stadiums_canonical.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Stadiums with coordinates, capacity, timezone, and primary team abbreviations.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">games_canonical.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Games with local date/time (converted from UTC), team names, abbreviations, and canonical IDs.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">team_aliases.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Historical team names and abbreviations with validity date ranges.
|
||||
</p>
|
||||
</div>
|
||||
<div style="padding: 1rem; background: #f8f9fa; border-radius: 8px;">
|
||||
<strong style="color: #1a1a2e;">stadium_aliases.json</strong>
|
||||
<p class="text-muted mb-0" style="font-size: 0.85rem; margin-top: 0.25rem;">
|
||||
Stadium name aliases (e.g., former naming rights) with validity dates.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Quick Export Buttons -->
|
||||
<div class="card mt-2">
|
||||
<div class="card-header">
|
||||
<h3>Quick Export</h3>
|
||||
</div>
|
||||
<div style="display: flex; gap: 1rem; flex-wrap: wrap;">
|
||||
<a href="{% url 'dashboard:export_download' %}?type=sports&type=league_structure&type=teams&type=stadiums&type=games&type=team_aliases&type=stadium_aliases" class="btn btn-success">
|
||||
Export All Data
|
||||
</a>
|
||||
<a href="{% url 'dashboard:export_download' %}?type=sports&type=league_structure&type=teams&type=stadiums" class="btn btn-primary">
|
||||
Export Structure Only (No Games)
|
||||
</a>
|
||||
<a href="{% url 'dashboard:export_download' %}?type=games" class="btn btn-secondary">
|
||||
Export Games Only
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
188
templates/dashboard/index.html
Normal file
188
templates/dashboard/index.html
Normal file
@@ -0,0 +1,188 @@
|
||||
{% extends 'base.html' %}
|
||||
|
||||
{% block content %}
|
||||
<div class="page-header">
|
||||
<h2>Dashboard</h2>
|
||||
<div>
|
||||
<a href="{% url 'admin:index' %}" class="btn btn-secondary">Admin Panel</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Overview Stats -->
|
||||
<div class="stat-grid">
|
||||
<div class="stat-card primary">
|
||||
<div class="stat-value">{{ sports_count }}</div>
|
||||
<div class="stat-label">Active Sports</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ teams_count }}</div>
|
||||
<div class="stat-label">Teams</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ stadiums_count }}</div>
|
||||
<div class="stat-label">Stadiums</div>
|
||||
</div>
|
||||
<div class="stat-card success">
|
||||
<div class="stat-value">{{ games_count|floatformat:0 }}</div>
|
||||
<div class="stat-label">Games</div>
|
||||
</div>
|
||||
<div class="stat-card {% if pending_reviews > 0 %}warning{% endif %}">
|
||||
<div class="stat-value">{{ pending_reviews }}</div>
|
||||
<div class="stat-label">Pending Reviews</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid-2 mt-2">
|
||||
<!-- Sport Stats -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Stats by Sport</h3>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Sport</th>
|
||||
<th>Teams</th>
|
||||
<th>Stadiums</th>
|
||||
<th>Games</th>
|
||||
<th>Reviews</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for stat in sport_stats %}
|
||||
<tr>
|
||||
<td><strong>{{ stat.sport.short_name }}</strong></td>
|
||||
<td>{{ stat.teams }}</td>
|
||||
<td>{{ stat.stadiums }}</td>
|
||||
<td>{{ stat.games }}</td>
|
||||
<td>
|
||||
{% if stat.pending_reviews > 0 %}
|
||||
<span class="badge badge-warning">{{ stat.pending_reviews }}</span>
|
||||
{% else %}
|
||||
<span class="text-muted">0</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="5" class="text-muted">No sports configured</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Recent Scrape Jobs -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Recent Scrape Jobs</h3>
|
||||
<a href="{% url 'dashboard:scraper_status' %}" class="btn btn-primary">View All</a>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Sport</th>
|
||||
<th>Status</th>
|
||||
<th>Games</th>
|
||||
<th>Time</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for job in recent_jobs %}
|
||||
<tr>
|
||||
<td>{{ job.config.sport.short_name }}</td>
|
||||
<td>
|
||||
{% if job.status == 'completed' %}
|
||||
<span class="badge badge-success">Completed</span>
|
||||
{% elif job.status == 'running' %}
|
||||
<span class="badge badge-info">Running</span>
|
||||
{% elif job.status == 'failed' %}
|
||||
<span class="badge badge-danger">Failed</span>
|
||||
{% else %}
|
||||
<span class="badge badge-secondary">{{ job.status }}</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>{{ job.games_found }}</td>
|
||||
<td class="text-muted">{{ job.created_at|timesince }} ago</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="4" class="text-muted">No recent jobs</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">
|
||||
<!-- Recent Syncs -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Recent CloudKit Syncs</h3>
|
||||
<a href="{% url 'dashboard:sync_status' %}" class="btn btn-primary">View All</a>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Type</th>
|
||||
<th>Status</th>
|
||||
<th>Records</th>
|
||||
<th>Time</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for sync in recent_syncs %}
|
||||
<tr>
|
||||
<td>{{ sync.sync_type }}</td>
|
||||
<td>
|
||||
{% if sync.status == 'completed' %}
|
||||
<span class="badge badge-success">Completed</span>
|
||||
{% elif sync.status == 'running' %}
|
||||
<span class="badge badge-info">Running</span>
|
||||
{% elif sync.status == 'failed' %}
|
||||
<span class="badge badge-danger">Failed</span>
|
||||
{% else %}
|
||||
<span class="badge badge-secondary">{{ sync.status }}</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>{{ sync.records_synced }}</td>
|
||||
<td class="text-muted">{{ sync.created_at|timesince }} ago</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="4" class="text-muted">No recent syncs</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Quick Actions -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Quick Actions</h3>
|
||||
</div>
|
||||
<div style="display: grid; gap: 1rem;">
|
||||
<a href="{% url 'admin:core_game_changelist' %}" class="btn btn-primary" style="text-align: center;">
|
||||
Manage Games
|
||||
</a>
|
||||
<a href="{% url 'admin:core_team_changelist' %}" class="btn btn-primary" style="text-align: center;">
|
||||
Manage Teams
|
||||
</a>
|
||||
<a href="{% url 'admin:core_stadium_changelist' %}" class="btn btn-primary" style="text-align: center;">
|
||||
Manage Stadiums
|
||||
</a>
|
||||
<a href="{% url 'dashboard:review_queue' %}" class="btn {% if pending_reviews > 0 %}btn-danger{% else %}btn-secondary{% endif %}" style="text-align: center;">
|
||||
Review Queue {% if pending_reviews > 0 %}({{ pending_reviews }}){% endif %}
|
||||
</a>
|
||||
<form method="post" action="{% url 'dashboard:run_sync' %}">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="btn btn-success" style="width: 100%;">
|
||||
Trigger CloudKit Sync
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
96
templates/dashboard/review_queue.html
Normal file
96
templates/dashboard/review_queue.html
Normal file
@@ -0,0 +1,96 @@
|
||||
{% extends 'base.html' %}
|
||||
|
||||
{% block content %}
|
||||
<div class="page-header">
|
||||
<h2>Review Queue</h2>
|
||||
<span class="badge {% if total_pending > 0 %}badge-warning{% else %}badge-success{% endif %}" style="font-size: 1rem; padding: 0.5rem 1rem;">
|
||||
{{ total_pending }} Pending
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{% if total_pending == 0 %}
|
||||
<div class="alert alert-success">
|
||||
No items pending review. All data has been validated.
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<!-- Summary by Sport and Type -->
|
||||
{% if review_summary %}
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Review Summary</h3>
|
||||
</div>
|
||||
<div class="stat-grid">
|
||||
{% for item in review_summary %}
|
||||
<div class="stat-card warning">
|
||||
<div class="stat-value">{{ item.count }}</div>
|
||||
<div class="stat-label">{{ item.sport__short_name }} - {{ item.item_type|title }}</div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<!-- Pending Items -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Pending Reviews</h3>
|
||||
<a href="{% url 'admin:scraper_manualreviewitem_changelist' %}?status__exact=pending" class="btn btn-primary">View in Admin</a>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Type</th>
|
||||
<th>Sport</th>
|
||||
<th>Raw Value</th>
|
||||
<th>Matched To</th>
|
||||
<th>Confidence</th>
|
||||
<th>Created</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for item in pending_items %}
|
||||
<tr>
|
||||
<td>
|
||||
{% if item.item_type == 'team' %}
|
||||
<span class="badge badge-info">Team</span>
|
||||
{% elif item.item_type == 'stadium' %}
|
||||
<span class="badge badge-secondary">Stadium</span>
|
||||
{% elif item.item_type == 'game' %}
|
||||
<span class="badge badge-warning">Game</span>
|
||||
{% else %}
|
||||
<span class="badge badge-secondary">{{ item.item_type }}</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>{{ item.sport.short_name }}</td>
|
||||
<td><strong>{{ item.raw_value }}</strong></td>
|
||||
<td>
|
||||
{% if item.matched_value %}
|
||||
{{ item.matched_value }}
|
||||
{% else %}
|
||||
<span class="text-muted">No match</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
<div class="progress-bar" style="width: 80px;">
|
||||
<div class="fill" style="width: {{ item.confidence }}%; background: {% if item.confidence >= 85 %}#28a745{% elif item.confidence >= 70 %}#ffc107{% else %}#dc3545{% endif %};"></div>
|
||||
</div>
|
||||
<small class="text-muted">{{ item.confidence }}%</small>
|
||||
</td>
|
||||
<td class="text-muted">{{ item.created_at|timesince }} ago</td>
|
||||
<td>
|
||||
<a href="{% url 'admin:scraper_manualreviewitem_change' item.id %}" class="btn btn-primary" style="padding: 0.25rem 0.5rem; font-size: 0.85rem;">
|
||||
Review
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="7" class="text-muted">No pending reviews</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% endblock %}
|
||||
256
templates/dashboard/scraper_status.html
Normal file
256
templates/dashboard/scraper_status.html
Normal file
@@ -0,0 +1,256 @@
|
||||
{% extends 'base.html' %}
|
||||
|
||||
{% block content %}
|
||||
<div class="page-header">
|
||||
<h2>Scraper Status</h2>
|
||||
</div>
|
||||
|
||||
<!-- Status Overview -->
|
||||
<div class="stat-grid mb-2">
|
||||
<div class="stat-card {% if running_jobs > 0 %}primary{% endif %}">
|
||||
<div class="stat-value">{{ running_jobs }}</div>
|
||||
<div class="stat-label">Running Jobs</div>
|
||||
</div>
|
||||
<div class="stat-card {% if pending_jobs > 0 %}warning{% endif %}">
|
||||
<div class="stat-value">{{ pending_jobs }}</div>
|
||||
<div class="stat-label">Pending Jobs</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Scraper Configs -->
|
||||
<div class="card">
|
||||
<div class="card-header" style="display: flex; justify-content: space-between; align-items: center;">
|
||||
<h3 style="margin: 0;">Scraper Configurations</h3>
|
||||
<form method="post" action="{% url 'dashboard:run_all_scrapers' %}" style="display: inline;">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="btn btn-primary">Run All Enabled</button>
|
||||
</form>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Sport</th>
|
||||
<th>Season</th>
|
||||
<th>Status</th>
|
||||
<th>Last Run</th>
|
||||
<th>Games Found</th>
|
||||
<th>Source</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for config in configs %}
|
||||
<tr>
|
||||
<td><strong>{{ config.sport.short_name }}</strong></td>
|
||||
<td>{{ config.season }}</td>
|
||||
<td>
|
||||
{% if config.is_enabled %}
|
||||
<span class="badge badge-success">Active</span>
|
||||
{% else %}
|
||||
<span class="badge badge-secondary">Inactive</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
{% if config.last_run %}
|
||||
{{ config.last_run|timesince }} ago
|
||||
{% if config.last_run_status == 'completed' %}
|
||||
<span class="text-success">✓</span>
|
||||
{% elif config.last_run_status == 'failed' %}
|
||||
<span class="text-danger">✗</span>
|
||||
{% endif %}
|
||||
{% else %}
|
||||
<span class="text-muted">Never</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
{% if config.last_run_games %}
|
||||
{{ config.last_run_games }} games
|
||||
{% else %}
|
||||
<span class="text-muted">-</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>{{ config.primary_source|default:"auto" }}</td>
|
||||
<td>
|
||||
<form method="post" action="{% url 'dashboard:run_scraper' config.sport.code config.season %}" style="display: inline;">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="btn btn-primary" style="padding: 0.25rem 0.5rem; font-size: 0.85rem;">
|
||||
Run Now
|
||||
</button>
|
||||
</form>
|
||||
</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="7" class="text-muted">No scraper configurations</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Recent Jobs -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Recent Jobs</h3>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Sport</th>
|
||||
<th>Season</th>
|
||||
<th>Status</th>
|
||||
<th>Games</th>
|
||||
<th>Started</th>
|
||||
<th>Duration</th>
|
||||
<th>Error</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for job in recent_jobs %}
|
||||
<tr>
|
||||
<td><strong>{{ job.config.sport.short_name }}</strong></td>
|
||||
<td>{{ job.config.season }}</td>
|
||||
<td>
|
||||
{% if job.status == 'completed' %}
|
||||
<span class="badge badge-success">Completed</span>
|
||||
{% elif job.status == 'running' %}
|
||||
<span class="badge badge-info">Running</span>
|
||||
{% elif job.status == 'failed' %}
|
||||
<span class="badge badge-danger">Failed</span>
|
||||
{% elif job.status == 'pending' %}
|
||||
<span class="badge badge-warning">Pending</span>
|
||||
{% else %}
|
||||
<span class="badge badge-secondary">{{ job.status }}</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
{% if job.status == 'completed' %}
|
||||
{{ job.games_found }} found, +{{ job.games_new }} new
|
||||
{% else %}
|
||||
-
|
||||
{% endif %}
|
||||
</td>
|
||||
<td class="text-muted">{{ job.created_at|timesince }} ago</td>
|
||||
<td>{{ job.duration_display }}</td>
|
||||
<td>
|
||||
{% if job.error_message %}
|
||||
<span class="text-danger" title="{{ job.error_message }}">{{ job.error_message|truncatechars:50 }}</span>
|
||||
{% else %}
|
||||
-
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="7" class="text-muted">No recent jobs</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Documentation -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>How Scrapers Work</h3>
|
||||
</div>
|
||||
<div style="padding: 1rem;">
|
||||
<h4 style="margin-top: 0;">What Gets Updated Automatically</h4>
|
||||
<p>When a scraper runs, it fetches schedule data from official sources and updates the following:</p>
|
||||
<table class="table" style="margin-bottom: 1.5rem;">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Data Type</th>
|
||||
<th>Behavior</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>Games</strong></td>
|
||||
<td>Creates new games, updates scores/status for existing games. Uses canonical IDs to match.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Teams</strong></td>
|
||||
<td>Auto-created from scraper's built-in team mappings. New teams are added automatically.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Stadiums</strong></td>
|
||||
<td>Auto-created from scraper's built-in stadium mappings. New venues are added automatically.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Conferences</strong></td>
|
||||
<td>Auto-created based on team data (e.g., Eastern, Western).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Divisions</strong></td>
|
||||
<td>Auto-created based on team data (e.g., Atlantic, Pacific).</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h4>New Team Scenario</h4>
|
||||
<p>If a league adds a new team (e.g., expansion team):</p>
|
||||
<ol>
|
||||
<li>Add the team via <a href="/admin/core/team/add/">Admin → Teams</a></li>
|
||||
<li>Add <a href="/admin/core/teamalias/add/">Team Aliases</a> for any names/abbreviations used by data sources</li>
|
||||
<li>Add the stadium via <a href="/admin/core/stadium/add/">Admin → Stadiums</a> (if it's a new venue)</li>
|
||||
<li>Add <a href="/admin/core/stadiumalias/add/">Stadium Aliases</a> for any alternate names used by data sources</li>
|
||||
<li>Run the scraper - it will automatically import all the new team's games</li>
|
||||
</ol>
|
||||
<p class="text-muted" style="font-size: 0.9rem;">If the scraper encounters an unknown team or stadium name, it creates a <strong>Review Item</strong> for manual resolution.</p>
|
||||
|
||||
<h4>What Requires Manual Action</h4>
|
||||
<table class="table" style="margin-bottom: 1.5rem;">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Situation</th>
|
||||
<th>Action Required</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Unknown team name in schedule</td>
|
||||
<td>Add a <a href="/admin/core/teamalias/add/">Team Alias</a> in the admin, or resolve in <a href="{% url 'dashboard:review_queue' %}">Review Queue</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Unknown stadium name</td>
|
||||
<td>Add a <a href="/admin/core/stadiumalias/add/">Stadium Alias</a> in the admin, or resolve in Review Queue</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>New expansion team</td>
|
||||
<td>Add a <a href="/admin/core/team/add/">new Team</a> in the admin, then add aliases for any alternate names</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Team relocation/rename</td>
|
||||
<td>Add a <a href="/admin/core/teamalias/add/">Team Alias</a> with validity dates for the old name</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Stadium rename (naming rights)</td>
|
||||
<td>Add a <a href="/admin/core/stadiumalias/add/">Stadium Alias</a> with validity dates (e.g., "Staples Center" valid until 2021)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h4>Managing Aliases via Admin</h4>
|
||||
<p>Team and stadium name mappings can be managed directly in the admin interface:</p>
|
||||
<ul>
|
||||
<li><a href="/admin/core/teamalias/">Team Aliases</a> - Map alternate team names, abbreviations, historical names</li>
|
||||
<li><a href="/admin/core/stadiumalias/">Stadium Aliases</a> - Map alternate stadium names, former names (naming rights changes)</li>
|
||||
</ul>
|
||||
<p>Aliases support <strong>validity dates</strong> - useful for historical names like "Washington Redskins" (valid until 2020) or stadium naming rights changes.</p>
|
||||
|
||||
<h4>Data Flow</h4>
|
||||
<p style="font-family: monospace; background: #f5f5f5; padding: 0.75rem; border-radius: 4px;">
|
||||
Scraper runs → Fetches from source (ESPN, league API, etc.) → Normalizes team/stadium names → Creates/updates records → Marks changed records for CloudKit sync → Creates review items for unresolved names
|
||||
</p>
|
||||
|
||||
<h4>Tips</h4>
|
||||
<ul style="margin-bottom: 0;">
|
||||
<li>Run scrapers regularly to keep scores and game statuses current</li>
|
||||
<li>Check the <a href="{% url 'dashboard:review_queue' %}">Review Queue</a> after scrapes for items needing attention</li>
|
||||
<li>Scrapers are idempotent - running multiple times is safe and won't duplicate data</li>
|
||||
<li>Each sport uses multiple data sources with automatic fallback if one fails</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
111
templates/dashboard/stats.html
Normal file
111
templates/dashboard/stats.html
Normal file
@@ -0,0 +1,111 @@
|
||||
{% extends 'base.html' %}
|
||||
|
||||
{% block content %}
|
||||
<div class="page-header">
|
||||
<h2>Statistics</h2>
|
||||
</div>
|
||||
|
||||
<!-- Game Stats -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Game Statistics</h3>
|
||||
</div>
|
||||
<div class="stat-grid">
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ game_stats.total|floatformat:0 }}</div>
|
||||
<div class="stat-label">Total Games</div>
|
||||
</div>
|
||||
<div class="stat-card success">
|
||||
<div class="stat-value">{{ game_stats.final|floatformat:0 }}</div>
|
||||
<div class="stat-label">Completed</div>
|
||||
</div>
|
||||
<div class="stat-card primary">
|
||||
<div class="stat-value">{{ game_stats.scheduled|floatformat:0 }}</div>
|
||||
<div class="stat-label">Scheduled</div>
|
||||
</div>
|
||||
<div class="stat-card warning">
|
||||
<div class="stat-value">{{ game_stats.today }}</div>
|
||||
<div class="stat-label">Today</div>
|
||||
</div>
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ game_stats.this_week }}</div>
|
||||
<div class="stat-label">This Week</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Sync Stats -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>CloudKit Sync Statistics</h3>
|
||||
</div>
|
||||
<div class="stat-grid">
|
||||
<div class="stat-card">
|
||||
<div class="stat-value">{{ sync_stats.total|floatformat:0 }}</div>
|
||||
<div class="stat-label">Total Records</div>
|
||||
</div>
|
||||
<div class="stat-card success">
|
||||
<div class="stat-value">{{ sync_stats.synced|floatformat:0 }}</div>
|
||||
<div class="stat-label">Synced</div>
|
||||
</div>
|
||||
<div class="stat-card warning">
|
||||
<div class="stat-value">{{ sync_stats.pending|floatformat:0 }}</div>
|
||||
<div class="stat-label">Pending</div>
|
||||
</div>
|
||||
<div class="stat-card danger">
|
||||
<div class="stat-value">{{ sync_stats.failed|floatformat:0 }}</div>
|
||||
<div class="stat-label">Failed</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% if sync_stats.total > 0 %}
|
||||
<div class="mt-2">
|
||||
<div style="display: flex; justify-content: space-between; margin-bottom: 0.5rem;">
|
||||
<span>Sync Progress</span>
|
||||
<span>{{ sync_stats.synced }} / {{ sync_stats.total }} ({% widthratio sync_stats.synced sync_stats.total 100 %}%)</span>
|
||||
</div>
|
||||
<div class="progress-bar">
|
||||
<div class="fill" style="width: {% widthratio sync_stats.synced sync_stats.total 100 %}%; background: #28a745;"></div>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<!-- Sport Breakdown -->
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<h3>Statistics by Sport</h3>
|
||||
</div>
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Sport</th>
|
||||
<th>Teams</th>
|
||||
<th>Stadiums</th>
|
||||
<th>Games</th>
|
||||
<th>Pending Reviews</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for stat in sport_stats %}
|
||||
<tr>
|
||||
<td>
|
||||
<strong>{{ stat.sport.short_name }}</strong>
|
||||
<div class="text-muted" style="font-size: 0.8rem;">{{ stat.sport.name }}</div>
|
||||
</td>
|
||||
<td>{{ stat.teams }}</td>
|
||||
<td>{{ stat.stadiums }}</td>
|
||||
<td>{{ stat.games }}</td>
|
||||
<td>
|
||||
{% if stat.pending_reviews > 0 %}
|
||||
<span class="badge badge-warning">{{ stat.pending_reviews }}</span>
|
||||
{% else %}
|
||||
<span class="badge badge-success">0</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% endblock %}
|
||||
Reference in New Issue
Block a user