How ingestion works
python -m app.cli ingest runs the steps below (app/scheduler.py). The List says which Shows there are, where
and when, with Steve’s flags and notes; everything else about a new Band or Venue comes from the services. A Band
or Venue already in the database isn’t looked up again at ingest. The Backfill workflow looks stored Bands up again (see
Deployment); a Venue is refreshed by hand (see
Fixing data by hand). For how this fits into the hourly
check, see Architecture.
flowchart TD
accTitle: The ingest, step by step
accDescr: Daily maintenance, then Steve's page is fetched and its listings parsed into Shows. New Venues are looked up on Google Places, and the result kept only when it's the listed Venue. Joint billings are split, Band corrections applied, and new Bands looked up on MusicBrainz, then for a photo on Wikimedia Commons, Discogs and Bandcamp, and for genres on Last.fm and Discogs, then MusicBrainz's free-form tags, then a correction's. Image URLs and links are checked; Shows, Venues, Bands and Acts are upserted; Band photos are stored; Shows and headliners are embedded and indexed in FAISS.
maintenance["Daily maintenance<br/>mark Past Shows,<br/>delete old ones"]
list[("stevelist.com/list<br/>plain text")]
map["list_shows()<br/>one dict per listing"]
places["Google Places<br/>new Venues only,<br/>checked by venue_check.py"]
joint["joint_bands.py<br/>split joint billings"]
corrections["band_corrections.py"]
mb["MusicBrainz<br/>new Bands only"]
photo["Photo: Wikimedia Commons,<br/>else Discogs, else Bandcamp"]
genres["Genres: MusicBrainz, else<br/>Last.fm, else Discogs, else<br/>MusicBrainz's tags, else its correction's"]
filter["genre_filter.py<br/>keeps only genres"]
checks["images.py, links.py<br/>check image URLs, links"]
upsert["upsert_shows()<br/>Show, Venue, Band, Act rows"]
photos["band_photos.py<br/>store each photo"]
embed["batch_embed_and_index()<br/>Ollama → FAISS"]
maintenance --> list --> map
map --> places & joint
joint --> corrections --> mb
mb --> photo & genres
genres --> filter
places & photo & filter --> checks
checks --> upsert --> photos --> embed- The List is the plain text of stevelist.com/list (
app/ingestion/steve_list.py). An edition is the page as of the date in its header; Steve also corrects the page between editions. The page’s SHA-256 is kept on the ingestion run, which is how the hourly check (python -m app.cli check) tells a changed page. A page with no edition date, or with fewer than 20 listings, isn’t taken for the List: the run fails. - Each listing is parsed by
app/ingestion/parser.py: the date, the lineup, the Venue, city and street, age, price, door and set times, flags, andCANCELLED:,POSTPONED:orINCORRECT LISTING:(also Cancelled). A listing’s year is the edition’s, or the next when that would put it more than 30 days before the edition (“jan 15” in a November edition); for a date in the 30 days before the edition, it’s the year the listing’s weekday falls on (“oct 30 sat” in the Nov 16, 2026 edition is 2027’s), and the edition’s when the weekday fits neither. A parenthetical after a band (“Charger (record release)”) is a note on its Act; one after the details (“(sold out)”) is a note on the Show. The page is typed by hand, so typos are expected: a misspelt weekday or city is read as the one meant, and what can’t be read goes to the Show’s notes. Lines that aren’t a Show (“record store day”) are left out. - Venue names are matched to stored Venues ignoring case, a leading “the”, punctuation and theater/theatre,
then by Steve’s other names for a Venue (
_VENUE_ALIASESinapp/ingestion/upsert.py: “the Castro” is The Castro Theatre; “Hopmonk” is a different tavern in Novato and in Sebastopol), then by a one-typo match (“the Indpendent”). Only a name that matches nothing is looked up on Google Places, and joins a Venue at the same Google place. - Google’s answer is checked (
app/ingestion/venue_check.py), because Places always offers a best guess: a neighbourhood for “the Castro”, a bare address for a studio it doesn’t know, another church for “First Church of the Buzzard”. A result is used only when it’s a place (not an area or an address), in the listing’s city (Google’s city, or a part of its address), and named like the listing: every word of one name is in the other, allowing Steve’s typos (“the Indpendent”) and short names (“the Arena” for Oakland Arena). When Places offers several guesses, only its exact name counts. A result that is just the listed street address keeps the address and coordinates, but no place id, website or phone. Anything else is dropped: the Venue is stored as listed, and the ingest logsnot confirmed on Google Placeswith what Google offered. A nickname Google can’t tie to its place (“the Giants Ballpark” for Oracle Park) belongs in_VENUE_ALIASES. The manual Venue refresh applies the same check before overwriting anything. - Joint billings (“Dying Fetus And Sanguisugabogg”) become two Bands; a Band with “and” in its name (“Belle and
Sebastian”) stays one. MusicBrainz decides (
app/ingestion/joint_bands.py). - A Band’s correction (
app/ingestion/band_corrections.py) gives the genres and links to use when the services have none. - MusicBrainz finds the artist (the Bay Area one, when several share the name) and its links.
- A Band’s photo is its Wikimedia Commons photo, found through MusicBrainz’s Wikidata link and kept with its
credit (
app/ingestion/wikimedia.py); else its Discogs photo (discogs.py); else the band photo on its Bandcamp page (bandcamp.py). Members come from Discogs. - Genres are MusicBrainz’s curated ones, else Last.fm’s tags (
app/ingestion/lastfm.py), else Discogs’ genres and styles (app/ingestion/discogs.py), else MusicBrainz’s free-form tags, and only then its correction’s.genre_filter.pykeeps only genres: thetev1:0.8bmodel on Ollama judges a tag it hasn’t seen. - Image URLs are kept, repaired (Wikimedia paths) or dropped; links that don’t work are dropped.
- Band photos still at a remote URL are downloaded, resized to ≤800px WebP and saved to the images folder
(
app/ingestion/band_photos.py), with a focal point from the faces in them (photo_focus.py). - Embeddings (
nomic-embed-texton Ollama) of each Show and its headliner go into the FAISS indices.
Show identity (upsert key): (date, venue_id, door_time), and the headliner where Shows share that: a Venue
with two rooms lists a Show in each at the same time (7pm/8pm (upstairs), 7pm/8pm (downstairs)), and each is its
own Show. Listings with the same identity and headliner are one Show (Steve sometimes lists a Show twice). A listing
takes the stored Show with its identity and headliner, else one with its identity that no other listing has taken,
so a lone listing whose lineup Steve corrected still updates its Show. Re-running ingestion on the same page is
idempotent. When Steve adds or corrects a door time, or lists a Show again without one (POSTPONED:), the listing
updates the stored Show at that Venue that day with the same headliner rather than starting another.
What the List doesn’t say is kept: a Show’s ticket link and provider, benefit cause and special event, and a Venue’s neighbourhood, type and transit came from the formatted editions ingested until November 2026, and stay on the Shows and Venues that have them. Sold out, matinee (doors before 5pm) and benefit are read from the listing.
Bands: DJs (dj …) and back-to-back sets (… b2b …) are excluded. Band details from the services
only fill empty fields, so corrections made in the database aren’t overwritten.
Data retention: shows older than DATA_RETENTION_DAYS (90) are hard-deleted by the daily maintenance
that runs before each ingest.
Fixing data by hand
These change the database, so to fix the published site run them against a checkout of the data branch, then
commit it and redeploy, as Deployment describes. The scripts are dry runs
unless given --apply.
- A Band matched to the wrong artist, or with the wrong genres or links: add a correction for it to
app/ingestion/band_corrections.py(its MusicBrainz artist, or that it’s never matched by name, and the genres and links to use). Then look it up again:python -m app.cli backfill --redo "<name>", or the Backfill workflow’s redo input. - A tag that isn’t a genre (or one wrongly dropped): edit its row in the
genre_tagstable (is_genre, anddecided_by = 'hand'), then run the backfill with--recheck-genres. - A Venue’s details (address, coordinates, description): the
refreshVenueEnrichmentGraphQL mutation on the local API looks one Venue up again (venueId), or every Venue without one, overwriting what’s stored. It usesANTHROPIC_API_KEYto tidy the name and write the description (the ingest uses it too, for Venues’ calendars). - The same Band stored twice under different spellings:
python -m app.ingestion.dedupe_bands. Thenpython -m app.ingestion.index_stored_vectors, which adds any stored vector the FAISS indexes are missing, so the kept Band still comes up as a Similar Band. - The same Venue stored twice:
python -m app.ingestion.dedupe_venues. It merges names that differ by case or a leading “the”, and records at the same Google place. For a typo or another name (“Hotel Utan”), first add it to_VENUE_ALIASESinapp/ingestion/upsert.py, so later listings under that name join the Venue too. - A Band name that includes a set time or show details (“The Coverups (2:30pm)”):
python -m app.ingestion.clean_band_names.