Deployment
The site is published at https://list.sturgis.me/ as a static site on GitHub Pages (see ADR 0002). The old address, https://jsturgis.github.io/the_list/, redirects there. Diagrams of the Deploy workflow’s triggers and of a new edition’s run are in Architecture.
The data branch
An orphan branch holding the SQLite database (the_list.db), the FAISS index (faiss/) and Band photos
(images/). main never contains data files. It keeps only its newest 4 commits (the current data plus 3 to roll
back to): after each ingest, older history is squashed into the oldest kept commit and the branch is force-pushed
(.github/scripts/prune-history.sh).
The Deploy workflow
.github/workflows/deploy.yml has five jobs, in four stages:
- Check (the
checkjob): runspython -m app.cli checkagainst thedatabranch’s database. It fetches Steve’s page (stevelist.com/list) and compares its SHA-256 with the page the last successful ingest read (changed), and the week (Friday to Thursday) of the date in its header with the newest edition ingested (new_edition). On the hourly cron, an unchanged page ends the run here. - Ingest (the
ingestjob): runspython -m app.cli ingestagainst thedatabranch’s database (with an Ollama service container for embeddings) and commits the changed database, index and Band photos back todata. Each run’s row iningestion_runsrecords the edition’s date, the page’s hash, its new Venues, its new Bands and, as a percentage, how many of them have no photo (app/ingestion/ingest_stats.py; also in theingestionRunsGraphQL query and the ingest log), to follow the lookups’ coverage. - Build and deploy (the
buildanddeployjobs): checks outmainand thedatabranch, runspython -m app.cli export, copiesimages/into the site’s static files, builds the static site and deploys it to Pages. - Alerts (the
alertsjob): after the successful ingest and deploy of a new edition only, runspython -m app.cli alerts. That emails each person whose Saved Filters match Upcoming Shows (see Weekly Alerts). It never runs for a correction to the current edition, or on deploy-only runs. If it fails, the site stays deployed and the run shows red.
Only one run at a time: a newer run waits for the one in progress.
- An unchanged List costs one short
checkjob an hour: no ingest, no deploy, no row iningestion_runs. - A page that can’t be fetched, or isn’t the List (no edition date, or fewer than 20 listings) fails the run; the next hour’s check tries again.
- A failed ingest fails the run: the
databranch is left untouched and nothing is deployed. The page still counts as changed, so the next hour’s run tries it again. GitHub emails you about failed scheduled runs (Settings → Notifications → Actions).
The maintainer’s emails
Besides GitHub’s own emails about failed runs, the workflow emails the maintainer (MAINTAINER_EMAIL) about the
ingest, through Resend from ALERTS_FROM (app/notifications.py). These aren’t Alerts: they go to the maintainer
only, and say how the ingest went.
-
After each successful ingest: “The List: ingest succeeded: 12 new bands, 2 new venues”, with the edition, its Shows parsed, upserted and new, the share of new Bands without a photo, and a link to the site. The
ingestjob sends it (python -m app.cli notify-success) once the data is committed to thedatabranch, from the committed database, so it never reports data that wasn’t saved. An ingest that brought no new Shows sends nothing. -
While the ingest has been failing for two weeks: “The List: ingest failing for 15 days: ListError: …”, with the root cause (the final exception, one line), which job failed (
checkoringest), when the last successful ingest was and its edition, the last 60 lines of the failing job’s log, and a link to the Actions run. A step withif: failure()at the end of thecheckandingestjobs sends it (python -m app.cli notify-failure --stage check|ingest --log PATH); the jobs’ commands also write their output to that log.A failed ingest isn’t committed to the
databranch (the job stops before the commit), and a failed check records nothing, so the database can’t count failures. “Failing for two weeks” is: this attempt just failed, and the newest successful ingest finished 14 days ago or more (FAILING_AFTER_DAYS). A List that hasn’t changed for two weeks isn’t a failure: its checks succeed and nothing is sent. A database with no successful ingest sends nothing. The root cause of a failed ingest comes from its run’s traceback, in the job’s own copy of the database; otherwise from the log.The cron is hourly, so a lasting failure would email every hour: the email goes at most once a day (the Bay Area’s day). Its Resend
Idempotency-Keyis the recipient and the date, and Resend refuses the day’s later, different emails with that key. The root cause and the log are sent with credentials taken out (query-stringapi_key=,key=,token=,secret=, Authorization and Bearer values, and every key in the settings).
Without MAINTAINER_EMAIL or RESEND_API_KEY both commands log that they’re skipping and exit 0. A failed send
fails only its own step, which has continue-on-error, so it can’t change the ingest’s result. Both commands take
--dry-run, which prints the email instead (notify-failure --dry-run prints it whether or not it’s due).
When it runs
- Every hour, at 17 minutes past (
17 * * * *): check, and only when Steve’s page has changed, ingest and deploy. Steve posts each Friday’s edition ahead of it (the November 13, 2026 edition went up the Thursday night), so a new edition’s Alerts go out within about an hour of his posting it, at whatever hour that is. GitHub disables a public repository’s scheduled workflows after 60 days without activity, so each hourly run first re-enables the workflow through GitHub’s API (the “Keep the hourly schedule enabled” step). GitHub doesn’t document that this resets the 60 days. If the schedule ever stops anyway, the Actions tab shows the workflow as disabled: click Enable workflow, or rungh workflow enable deploy.yml. No failure email can warn you of this, as no run happens. - By hand from the Actions tab (Deploy → Run workflow) or
gh workflow run deploy.yml. A manual run ingests the List whether or not it has changed, and sends Alerts only if it’s a new edition. Tick Skip ingestion (gh workflow run deploy.yml -f skip_ingest=true) to rebuild from the existing data without sending Alerts. - On pushes to
mainthat touchfrontend/,backend/app/,docs/guide/(these docs) or the workflow’s own file: deploy only, no ingest. - After the Backfill workflow, which starts a deploy-only run when it has committed to
data.
Settings
- Pages: the source must be GitHub Actions (Settings → Pages).
- Custom domain: set in Settings → Pages, with a
CNAMErecordlist→jsturgis.github.ioin thesturgis.meDNS. The build takes its base path and site URL fromactions/configure-pages(PAGES_BASE_PATH,PAGES_SITE_URL):""andhttps://list.sturgis.mewith the domain,/the_listandhttps://jsturgis.github.io/the_listwithout it. Local builds default to/the_list. - The ingest uses the
GOOGLE_MAPS_API_KEY,ANTHROPIC_API_KEY,LASTFM_API_KEY,DISCOGS_CONSUMER_KEYandDISCOGS_CONSUMER_SECRETsecrets (Settings → Secrets and variables → Actions). Steve’s page needs none. The Anthropic key is only for reading Venues’ own calendars (flyers and more-info links); without it that step is skipped and nothing else changes. - Alerts have their own secrets and variables: see Weekly Alerts.
- The maintainer’s emails need the
MAINTAINER_EMAILsecret (where they go) and the Alerts’RESEND_API_KEYsecret andALERTS_FROMvariable. WithoutMAINTAINER_EMAILnone are sent.
The Backfill workflow
.github/workflows/backfill.yml brings Bands ingested before a lookup existed up to what new Bands get. Run it by
hand: Backfill → Run workflow, or gh workflow run backfill.yml.
- What it does:
python -m app.cli backfilllooks each Band up on MusicBrainz, Last.fm, Discogs, Wikimedia Commons and Bandcamp by the ingest’s rules (app/ingestion/band_sources.py; what the Band already has is the last fallback), re-embeds it when its genres change, and stores its new photo. - It’s resumable: each Band looked up gets
enriched_at. A run takes the Bands not yet looked up (those on Upcoming Shows first), stops after max minutes (default 300, under the 6-hour job limit) or limit Bands, commits todataand starts a deploy-only Deploy. Run it again until its commit message says none are still to do. Discogs’ rate limit (one request a second) sets the pace. - What it needs:
LASTFM_API_KEY, both Discogs keys and the genre model. It checks first that Last.fm and Discogs accept the keys and the model answers. - A service that’s down mid-run returns nothing, and those Bands are still marked looked up. The exception is Bandcamp: a Band whose page Bandcamp won’t serve stays pending for the next run.
- Options that first mark Bands to look up again:
- recheck genres (
--recheck-genres): Bands whose stored genres hold a tag that isn’t a genre. - recheck photos (
--recheck-photos): Bands with a Bandcamp page and no photo from a service. - redo (
--redo "<name>", names separated by;in the workflow): Bands matched to the wrong artist, looked up again from scratch. Use it after adding a Band’s correction (see How ingestion works).
- recheck genres (
- Locally (with the keys in
backend/.env):docker compose exec api python -m app.cli backfill --limit 20.
Genres and photo focal points
- Genres are filtered (
app/ingestion/genre_filter.py): Last.fm’s tags and MusicBrainz’s free-form tags keep only genres. A tag in MusicBrainz’s genre list (app/ingestion/genres.txt) or a known shorthand is kept; any other is judged once by thetev1:0.8bdecision model on Ollama, and the answer is stored in thegenre_tagstable. To overrule it, edit that row on thedatabranch (is_genre, anddecided_by = 'hand'). - Photo focal points (
app/ingestion/photo_focus.py): each Band photo the ingest or backfill stores gets a focal point (bands.image_focus) from the faces in it, and the site crops the photo around it. Photos stored before focal points need one run ofpython -m app.cli photo-focus, which works them out from the files inimages/(no network; it only looks at Bands without one, commits every 100, and can be run again;--allredoes every photo). The next section shows how to run it on thedatabranch.
Changing the data by hand
To run a command against the published data, check the data branch out beside the repository and point the
backend at it. This example runs photo-focus, from a checkout of main:
git fetch origin datagit worktree add -B data ../the_list-data origin/datadocker compose build api # the image needs the current codedocker compose run --rm --no-deps -v "$PWD/../the_list-data:/data" \ -e DATABASE_URL=sqlite:////data/the_list.db -e IMAGES_PATH=/data/images -e FAISS_INDEX_PATH=/data/faiss \ api sh -c "alembic upgrade head && python -m app.cli photo-focus"cd ../the_list-data && git add -A && git commit -m "Focal points for stored Band photos" && git push origin datagh workflow run deploy.yml -f skip_ingest=trueThe branch is rewritten when it’s pruned, so if the worktree already exists, bring it up to date first with
git fetch origin data && git reset --hard origin/data (never pull). To replace the data outright, commit a new
the_list.db and faiss/ to the branch, then run the workflow with Skip ingestion.
Rolling back
Revert the bad commit on the data branch (git revert <sha> on an up-to-date checkout of data, then push) and
run the workflow with Skip ingestion. Only the last 3 changes can be rolled back this way.