# Benchmark — the two neural generations, scored against one held-out race How the hand-built architecture eras (v1 2024, v2 2025) were re-scored on one common modern holdout (Giro d'Italia 2026), and the story of getting the 2024 checkpoint to run again. Status: **COMPLETE** — both neural generations scored on Giro 2026 (v1 0.214 / v2 0.310 NDCG@5). **Deliverables** - Scripts (`scripts/legacy_benchmark/`): `scorer.py` (shared eval) · `score_v2.py` (v2) · `score_v1_b4.py` (v1, the actual 2024 winning checkpoint) · `build_neural_seq.py` (sequence cache). - Extracted 2024 pipeline: `v1_src/{preprocess_b4, sequences_np_b4, model_combo_3}.py` (commit b4df757). - Viz: `docs/viz/analysis.html` (version-by-version, with architecture flowcharts) · `docs/viz/build_log.html` (stage-profile timeline of the eras). Results (gitignored): `results/*.json`, `*_pred_gc.csv`. - Full method + the story below (Updates 1–7; the v1 recovery hinged on one `'one_day'` typo). ## Results **Common eval unit (pinned):** per rider-stage; groups per (year, tour, stage); relevance `_pos_to_label` (top-25 graded: pos 1 → 25 ... pos 25 → 1, else 0); NDCG mean over stage-groups; AUC on binary top-10. Scorer: `scripts/legacy_benchmark/scorer.py` (self-tested, `python scorer.py`). **FINAL — both scored on Giro 2026, out-of-sample:** | Model | NDCG@5 | AUC(top10) | dominance | riders scored | |---|---|---|---|---| | v2 image-encoder (2025) | **0.310** | 0.744 | 1.00× | 123 / 184 | | v1 flagship (2024, ACTUAL winning checkpoint) | **0.214** | 0.595 | 1.14× | 183 / 184 | - Clean progression 0.21 → 0.31. v1's GC top picks are sensible climbers (Gall, Vingegaard, Ciccone, Hindley, Bernal, Kuss) — not garbage; the features/alignment are right. - v1 = the REAL checkpoint (`score_v1_b4.py`), out-of-sample, **images masked** (2026 not in the 2024 OCR store) so 0.21 is a floor; 183/184 riders, 3277/3448 rows. - **Coverage caveat:** v2 was scored in encoded-image mode, which drops riders without encodable stage images → 123/184 riders (2,212/3,448 rows, 20/21 stage-groups). So v1 and v2 are **not on the identical rider set**; treat the gap as indicative, not exact. (Fixable by encoding the missing riders' images.) - **Flat favorites:** v2's per-stage probabilities are similar across riders → GC-sum dominance 1.00× (no favorite separation), unlike v1's 1.14× here and its reported 1.9× (Pogačar) on Giro 2024. - **Scorer caveat:** `dominance_ratio` is only meaningful for probability scores (sigmoid/softmax outputs). Both neural nets output probabilities, so they compare cleanly. --- ## P1 progress + the hardware wall (2026-07-11) Established (good): - **Alignment solved:** neural `get_seq` produces the **identical 184 riders** for Giro 2026 as the flat rider-stage table → neural per-rider-stage outputs map straight onto the scorer meta. - `preproc` is fast (~0.7 min); the split is clean (train 2020-24 / val 2025 / test Giro 2026). - **v2 needs no bug fix** — active code already uses `MASK_VALUE=-9999` consistently in masking layers AND padding; preprocessing defects already handled. So v2's "corrected variant" = active code as-is. Blocker (real): - Building sequences in **raw-image mode** (128×128) makes the training set **~17.7 GB** vs this machine's **16 GB RAM** → full-image training will thrash/OOM. (Test set alone = 1.2 GB, loads fine.) - The shipped v2 was almost certainly trained in **encoded-image mode** (autoencoder latents, `encoded_profiles_100`) which is tiny and fits RAM — but `encoded_*_100` has **no 2026 images**, so a Giro-2026 encode pass (autoencoder over raw 2026 images) is required for the test set. Paths forward (user decision): - **A — numeric-only neural rows:** train encoder-decoder with `img=False`. Fits RAM, fast, clean; drops the image tower (a design decision) — labeled honestly. - **B — encoded-image mode + 2026 encode pass:** the "right" way v2 was trained; preserves the image tower; needs ENCODER paths + an autoencoder pass over Giro-2026 raw images. More setup, fits RAM. - **C — pause neural rows:** fold findings into the viz now, do the neural rows in a dedicated session. Kept: `scripts/legacy_benchmark/seq_cache/test.pkl` (Giro 2026, 1.2 GB, alignment-verified). Deleted the 17.7 GB train + 3.6 GB val raw pickles. ### Update 2 — the encoded-mode fix hit an architecture coupling - Encoded the 2026 Giro images (via `scripts/encode_images_to_mongo.py`) → `encoded_*_100` now have the 21 Giro-2026 stages. Rebuilt sequences in encoded mode → **100× smaller** (test 11.5 MB, train 250 MB). - **But the v2 model (`image_encoder_base.CyclingModel`) EMBEDS the autoencoder as a layer** (`TimeDistributed(self.encoder_profiles_model)` in `encoder()`), so it expects **raw (128,128,1)** input and encodes internally — it will NOT accept pre-encoded 100-dim latents (shape mismatch at build). Feeding encoded latents fails; feeding raw images is back to the 17 GB > 16 GB RAM wall. - Net: on this 16 GB machine the v2 image-tower model can't be retrained on the full holdout by materializing sequences. Real options: - **(i) Streaming batch generator** — load raw images from Mongo per batch (never all resident). Proper fix, ~an hour of engineering. Enables exact-holdout retrain. - **(ii) As-shipped checkpoint inference** — run an existing v2 checkpoint on the small raw Giro-2026 test set (fits), score out-of-sample. Fast; not exact-holdout-trained (label it). - **(iii) maxlen_img=15 + float16 raw** — ~6–12 GB, borderline; may still thrash. - v0/v1 note: v1's keras-OCR images are 1499-dim (small) and v0 may be numeric-only — so the RAM wall is **specific to v2's raw-128×128 autoencoder tower**, not the other eras. --- ### Update 3 — v2 SCORED (identity-passthrough fix) The RAM wall was beaten by swapping the model's embedded autoencoder for an identity pass-through — the encoded `_100` latents ARE that autoencoder's output, so it's the same network on tiny inputs. `scripts/legacy_benchmark/score_v2.py` trains on ≤2024 and scores Giro 2026 via the shared scorer. First pass: v2 NDCG@5 **0.322**, NDCG@10 0.304, AUC(top10) 0.736, dominance 1.00×, 123 riders. (Re-run with a fixed seed for the final table above: 0.310 / 0.744.) - **Caveat 1 — coverage:** encoded mode drops riders without encodable stage images → v2 scored on 123/184 riders (2,212/3,448 rows, 20/21 stage-groups). AUC 0.74 is real signal, but NDCG is not on the identical rider set as v1. (Fixable by encoding the missing riders' images.) - **Caveat 2 — flat favorites:** v2's per-stage probs are similar across riders → GC-sum dominance 1.00× (no favorite separation), unlike v1's reported 1.9×. ### Update 4 — v1 flagship investigated; live re-score is a dig (2026-07-11) Traced the full v1 chain. It's a separate 2024 stack: old `preprocessing/preprocess.py` (still in tree, produces the checkpoint's 28-enc/21-dec features) + `sequence/sequences_np.py` + `models/model_combo_3` + the old JSON image store (`get_all_json` / `treated_good_json` / `db.finish`). Findings: - Old `preprocess(get_data(), flow='only_tours')` runs but returns **0 rows** — its "json cut" (`preprocess.py:464-494`) keeps only records whose `img_url` ∈ (old JSON store ∩ `db.finish`), which the current DB / 2026 data don't match. This is the coupling to the custom-OCR 1499-dim image store. - **Bypass exists:** the cut only runs when `sequences=True` (line 461 returns early otherwise), so `preprocess(..., sequences=False)` returns all years incl. Giro 2026 with the numeric features intact. - **Tractable path = numeric v1 + masked images:** bypass cut → old `sequences_np` numeric path → manual split → feed the verified-loading flagship checkpoint `20240627-155334` with zeroed 1499-dim image inputs → 4-class softmax → P(band-1)=P(top-25) per stage → shared scorer + favorite-separation. - **Risk:** the old numeric features must match the checkpoint's 28-enc/21-dec exactly; old `sequences_np` + manual split + masked-image feed each carry drift/mismatch risk vs a 2-year-old checkpoint. Several steps, each can need iteration. This is real archaeology, not a quick continuation. - **Note:** v1's favorite-separation story (1.9× Pogačar) is ALREADY git-committed as the 2024 Giro/Tour receipts — the live Giro-2026 re-score would *add* a scored cell, not create the story from scratch. ### Update 5 — v1 DEFINITIVE blocker found (2026-07-11): feature-recipe drift Pushed the v1 dig all the way through. Extracted 2024 code (`v1_src/`), patched the plumbing (missing `moving_average_dict` arg; malformed `img=False` unpack line in the 2024 source; neutralized 27 `ipdb` breakpoints; swapped the missing-2026 image fetch for masked -9999 blanks; bypassed `get_all_json`). The old pipeline then **ran cleanly** (5.8 min, 357 riders) and returned real arrays. **But the shapes don't match the checkpoint — the real, final blocker:** ``` checkpoint 20240627-155334 wants extracted code produces encoder num (20, 28) (20, 15) decoder num (21, 21) (21, 6) ``` `sequences_np.py:162` builds only 6 decoder feats [weight, height, startlist_quality, distance, vert_meters, profile_score]; the richer variant (line 163, ~13 feats) is **commented out**, and even it doesn't reach 21. So the June-2024 checkpoint was trained with a feature recipe that is NOT fully present in the `590a483` code — the exact 28-enc/21-dec column set + order is undocumented. **Conclusion:** v1's live re-score is blocked not by effort or missing logic but by **feature-recipe drift between the saved model and its own code**. Reconstructing the exact 28/21 columns is uncertain trial-and-error against a black-box checkpoint; a wrong guess yields a garbage score (worse than none). The honest stopping point. v1's story stands on its git-committed 2024 receipts (already in the viz). Recommended if ever resumed: dig MLflow run params / `registry_2` artifacts for `20240627-155334` to recover the exact feature list, OR retrain a fresh v1 on the 15/6 features (a NEW model, not the winner). ### Update 6 — v1 ENCODER recovered from git; decoder was UNCOMMITTED (2026-07-11) Followed the git trail (user's steer). Found the exact commit: **`b4df757`** "all added params encoder and decoder", Jun 27 2024 14:06 — files at ROOT (`cycling_manager/{preprocess,sequences_np}.py`), reads Mongo `tabular_giro24_fix` (2024 schema, still present). Extracted → `v1_src/{preprocess_b4,sequences_np_b4}.py`, patched (point get_data at `tabular_production_fixed` for 2026 data; keep `decoder` col on the `sequences=False` return; mask images; tolerant imports for removed `get_array_ls_mongo_db`/`get_all_json`). Ran clean (1.4 min, 184 riders = exact match with the flat rider-stage table). **Result — encoder EXACT, decoder short:** ``` checkpoint wants b4df757 code produces encoder num (20, 28) (20, 28) ✓ EXACT MATCH decoder num (21, 21) (21, 8) ✗ 8 not 21 ``` - Encoder recipe fully recovered from git. Decoder line (`sequences_np_b4.py:60`) builds 8 feats; NO richer/commented decoder variant exists anywhere in history. - **Timing proves it:** `b4df757` committed 14:06; checkpoint `20240627-155334` trained **15:53** (1h47m later). The decoder's extra 13 columns were LOCAL UNCOMMITTED edits at train time — never in git. - **Definitive:** the winning v1 was trained on uncommitted code; its exact decoder recipe is unrecoverable. An exact live re-score is impossible. (Best-effort: guess the 13 decoder cols + eyeball sanity — but wrong order feeds the checkpoint garbage; not presentable as "the flagship's score".) v1 stands on git receipts. ### Update 7 — v1 SOLVED: the decoder was a one-word typo, not uncommitted (2026-07-11) Update 6's "unrecoverable" conclusion was WRONG. The decoder's missing 13 features ARE in the b4df757 code: `sequences_np.py` builds `pretour_performance` (13 cols: fl/hi_fl/hi_hi/mo_fl/mo_mo, bunch/reduced sprint, solo, tt, other, gt_score, one_day, stage_race) and adds them to X_decoder_num — BUT the source line selects `'one_day'` while the column is `'one_day_race'`, so it throws, gets swallowed by a `try/except`, and the decoder silently stays at 8. **Fixing that one word → decoder = (21,21) → the actual winning checkpoint `20240627-155334` LOADS and SCORES.** Result: the final table at the top. - Viz updated with both generations + the typo story. `results/v1_score.json` + `v1_pred_gc.csv`. - (The fresh-retrain path for v1 is redundant — we scored the actual winner instead.) P2 (v0, the 2023 LSTM-binary era) deprioritized. --- Methodology + coverage settled 2026-07-11. --- ## Goal Two things, side by side, per architecture era: 1. **As-shipped receipts** — the model *as it actually ran*, with its git-committed predictions. The provenance/story anchor (the "then"). **Unchanged, never retrained.** 2. **Corrected variant** — the same architecture with only its **unintended bugs** fixed, retrained, and **scored on Giro 2026** (the "now"). Keeping both is deliberate: it makes "I found and fixed my own bugs" a strength, not a contradiction of the shipped artifact. Deliverable per era: `{era, NDCG@5, NDCG@10, top10-AUC, favorite-separation}` on Giro 2026, plus the preserved as-shipped receipts. --- ## Methodology (settled) **Narrow corrected-retrain.** NOT a rebuild, NOT a rearchitecture. - **"Bugs" means unintended defects only** — scoped to: - the **mask inconsistency** (`Masking(-1000)` layer vs `-9999` tensor padding — gotcha #1) - **data-preprocessing defects** (from REVIEW.md): invalid `parcours_type` (`mg_rp4`, `p0`), `won_how` placeholder text, `birth_year` format, duplicate records, silent image-coverage filtering, bare `except: pass` masking bad records. - **Every design decision is preserved** — the 4-class band target, the LSTM 40·30·15 skeleton, the keras-OCR image tower, class-prior bias init. We fix what he *didn't intend*, not what he *chose*. - **Retrain is required** (not optional): the mask fix only takes effect if the weights are retrained with consistent masking. This is why it's a retrain and not fix-and-run. ### Why retrain doesn't endanger the flagship story The advisor's earlier warning — "a retrained v1 is a different model, could contradict 'the model that won'" — is resolved by keeping **both**: the as-shipped checkpoint + receipts stay as the winning-model provenance; the corrected variant is clearly labeled as a *corrected* reconstruction. In the end the actual checkpoint itself was scored (Update 7), so no corrected v1 retrain was needed. --- ## Common evaluation (all models, identical) - **Holdout:** train ≤2024, val 2025, **test = Giro 2026** (real GC labels, 21 stages, 3,448 rows). Every architecture retrained or scored under this exact split. - **Evaluation unit — PIN BEFORE ANY SCORING CODE:** one row per rider-stage, grouped per stage. Per-stage → per-rider aggregation (sum over 21 stages of P(top-band)) is **identical** across models. - **Label:** finishing position per stage. Binary top-10 (AUC) + graded top-25 rank (NDCG). v1's native band is top-25; fine for order-based metrics — stated in viz. - **Metrics:** `NDCG@5` (headline) + `NDCG@10` + `top-10 AUC` + **favorite-separation** (predicted #1/#2 score ratio; actual GC winner's predicted rank) so the headline scalar doesn't wash out the portfolio's own narrative (v1 spread Pogačar 1.9× on Giro 2024). --- ## Feasibility (verified 2026-07-11) - **Checkpoints on disk (as-shipped receipts source):** v0 `2023-08*`, v1 `20240627-155334` (load-verified: 4 inputs → 21×4 softmax, 8.3M params), v2 `2025-05*`. - **Training data through 2025 exists** in `tabular_production_fixed` for the retrain. - **Giro 2026 images:** `raw_profiles_images` +187 2026, `raw_finish_images` +180. **`encoded_*_100` has 0 2026** → v2 uses raw mode or an autoencoder encode pass. v1's keras-OCR tower uses raw. - **Custom-OCR dependency:** v1's image tower may rely on the retrained OCR models (`cycling_ocr_detector.h5` 250MB, `cycling_ocr_recognizer.h5` 71MB, on disk) — confirm in Phase 3. --- ## Custom image experiments — timeline notes (NOT fixed, just told) Portfolio "roads explored" beats, surfaced in the viz as tried-branches: - **Custom OCR retrain** (`ocr_full_retrain.ipynb`, Sep 2023) — retrained keras-OCR detector + recognizer on *synthetic* stage-profile text (`"1850m"`, `"14.2km"`, `"9.5%"`) to read elevation / distance / gradient numbers off the profile images. Ambitious; didn't pay off over the flatten path. - **Masked convolution** (`maskedconv.ipynb`, Aug 2024) — custom masked-conv image branch on the full-season model. - ~~Stacked capsule autoencoders~~ — dropped; not part of the remembered story. --- ## Phases (each leaves a presentable number — stop after any one) - **Phase 0 — Benchmark harness.** Pin eval unit + label; build `score_rider_stage(...)` → {NDCG@5/10, AUC, favorite-sep}. *Out: scorer.* - **Phase 1 — v2 (active image-encoder).** Lowest risk (active code). Apply mask + preprocessing fixes, retrain on ≤2024, score Giro 2026. *Out: v2 row.* - **Phase 2 — v0 (LSTM binary).** No images, simplest. Fixes, retrain, score. *Out: v0 row.* (deprioritized) - **Phase 3 — v1 (flagship dense_multi).** Highest value/risk. Recover the feature recipe, load the actual checkpoint, adapt 4-class → GC score, score + favorite-separation. *Out: v1 row + narrative number.* - **Phase 4 — Viz.** Add scored rows + custom-image tried-branch notes to the build-log artifact. --- ## Risks - **Eval-unit mismatch = silent invalidation.** Pin the per-stage → per-rider aggregation in Phase 0. - **v1 keras-OCR reproduction on 2026 images** may not match 2024. Fallback = masked branch, labeled. - **Scope creep on "bugs."** Hold the line: mask + the catalogued preprocessing defects only. Anything that changes a *design decision* is out (that's a different, later experiment). - **v2 encoded-image gap.** 2026 not in `encoded_profiles_100`; raw mode or encode pass. - **Retrain data recency.** v2 trains ≤2024 (not its original era) so the holdout is identical across models — architecture-fair but means the scored v2 ≠ the 2025 model's exact training era. Acceptable and labeled; the as-shipped receipts cover the era-faithful story.