Measuring a model that loses
Football 1X2 prediction service · public scorecard · walk-forward archive
# summary
> A football 1X2 prediction service whose own public scorecard says it is less accurate than the prices it is measured against, built so that the answer stays checkable rather than flattering.
# the problem
> Anyone can publish football predictions. Almost nobody publishes the number that says whether those predictions are any good, because the number is usually bad: the closing price on a liquid 1X2 market is a very strong forecast, and beating it is the whole difficulty. A system that reports accuracy only when accuracy is favourable is not a measurement system, and a reader has no way to tell one from the other from the outside. The design question here was not how to make the model better. It was where to put the number that says the model is worse, and what has to be true of the code for that number to stay honest as the model retrains itself every week.
# what we built
> A FastAPI service on Python with an XGBoost pooled meta-model over 22 leagues, a Dixon-Coles strength layer and an ELO component underneath it, and a walk-forward archive in which every scored row was produced by a model that had not seen that match. The public front of it is a scorecard: /karne compares the model against the de-margined market on hundreds of thousands of historical matches and reports log loss, RPS and Brier for both, plus a paired test on the matches where both had a number.
> A single-match page reconstructs one fixture — what the archive said before kick-off, what the market said, what happened, and a TreeSHAP decomposition of the served model's own output. A paper bet ledger records what the bot would have staked, hash-chained, with no real money in it. Around that sit a model registry that decides which artefact is served, a weekly job that settles and places paper bets, and a daily bulletin that publishes yesterday's "what the model said, what actually happened" cards. 2,619 tests collect in the suite, counted 2026-08-24 at commit 4986dd78.
# technical decisions
The headline number is the one against us, and it is the page that needs no account.
Every figure in this section is quoted with the run or the commit that produced it, because the scorecard is re-run every few days and an undated number is a claim about today rather than a measurement. From backtest_results/karne_20260823.json, the file the page reads, generated 2026-08-23 22:53 UTC and re-read on 2026-08-24: the model's log loss is 1.0079 over 201,268 scored matches, against 1.0054 for the de-margined pre-match price over the 194,782 of those that carry one; paired on those same 194,782 matches, the model is 0.00229 worse, with t p = 6.038e-25 and Wilcoxon p = 1.169e-19.
Against the closing price, on the 105,152 matches that have one, it is worse by more: 1.00506 against 1.00037, a gap of 0.00469, t p = 1.938e-47. The model loses to the bookmakers on both lines, and by the wider margin against the sharper one. The page says so in plain language before it shows a table, and draws the conclusion out loud: the gap is small but real, the margin sits on top of it, and long-run profit from betting does not look available on this evidence.
Two structural choices hold that in place. Every figure on the page is read from backtest_results/karne_20260823.json — the page computes nothing, and it states that if the file and the page disagree the file is right, which makes the claim falsifiable by one git diff rather than by re-deriving a pipeline. And the access design points the same way: /karne is reachable with no account at all, while the interactive tools sit behind a sign-in, so the one page a stranger can check is the one that reports the loss.
The alternative — put the accuracy work behind the login and the coupons in front of it — is the arrangement the reader has been trained to expect, which is exactly why it would not have been believed.
A model that retrains itself needs a champion pointer, a ledger, and rules with numbers in them.
services/model_registry.py replaced latest_artifact(), which picked the alphabetically last file in the model store. That is fine while retraining is a person typing a command and wrong the moment it is a scheduled task: the retrained model would be served the instant it was written, and a shadow that is served is not on trial. Promotion now requires the shadow to have been on trial at least 28 days, to have at least 200 scored matches, and to have lower log loss on the matches the two share; rollback triggers when the champion's last two weeks against the market degrade by 2σ of its own history.
The module names the gate for what it is — a staleness guard, not a significance test — and says why: 200 matches cannot resolve a 0.002 log-loss difference, the paired p-value is reported next to the decision and would almost never clear 0.05, and requiring it would freeze the champion permanently. What the gate does buy is that a model must survive a month of genuinely unseen football before it serves anyone, which is the failure that actually bites — a refit that looks good because it has already seen the data.
Read from models_store/ on 2026-08-24: xgb_20260810.json is the champion (pointer set 2026-08-11 14:07 UTC, 105 features, fitted on 233,660 matches across 22 leagues and seasons 9394–2425), xgb_20260814 is the shadow on trial, xgb_20260811 is retired.
A tamper-evident ledger that writes down what it cannot detect.
Each paper bet in services/paper_ledger.py carries the hash of the row before it. verify() walks the chain and separates two failures deliberately, because they mean different things: a bad hash means that row's contents were altered, a bad previous_hash means a row was inserted, removed or reordered. Its docstring then states the case it is blind to. Deleting rows from the end leaves the survivors internally consistent, so verify() returns valid and the missing bets appear nowhere.
▶▼ read the full reasoning
That is not a thought experiment, and it is not a past incident either — it is a loop that is still running, and the mechanism is the interesting part. data/paper_ledger.jsonl and data/predictions.jsonl are runtime data that git tracks, in a repository deployed by hard reset, so every sync overwrites what the machine wrote with whatever was committed. On 2026-08-24 that happened twice.
At 01:50:04 +03 a git reset --hard replaced a 129-row ledger with the 32-row blob committed on 14 August; the nightly job appended 23 more at 04:43; at 13:36:27 a second reset replaced those too, and both files carry that second as their inode birth time. 120 rows destroyed in twelve hours, and verify() returned valid after each one, because a chain pruned from the end is a valid chain.
The two files that survived are the two nobody tracked: paper_ledger.watermark.json, which .gitignore excludes by name, and paper_settlements.jsonl, which was never added — their inodes date from 14 and 15 August and no reset has touched them. watermark() is the answer to the part the chain cannot see — it keeps the row count and head hash in a separate file, and updates only when the ledger grows, so a shrink stays visible until a person looks rather than being erased by the next successful run.
The weekly job checks the watermark before it places anything, logs the alarm, and then still settles and places the night's bets, because a shortfall is a statement about the past and stopping would cost a second night as well. Read again 2026-08-24 14:16 +03, and the two still disagree, which is the whole point: verify() returns valid over 32 entries, while the watermark holds a baseline of 112 written at 00:36 and reports shrunk: true, 80 rows lost.
Even that under-reports — 17 of the destroyed rows were placed after the baseline was taken and were never watermarked, and the second reset is outside its arithmetic entirely. The chain is internally consistent and has nothing to report; a stale, under-counting watermark is still the only thing on the machine that knows any rows are missing.
The return figure is pre-registered, and the page refuses to compute it.
/kasa lays a paper bankroll of 30,000 TL over that ledger and passes no profit and no ROI into the template at all. services/bankroll.roi_gate() measures the ledger against a threshold written in docs/ON_KAYIT_R11.md §4 before the measurements existed: 200 settled bets, on the stated reasoning that a 100-bet record swings by roughly ±10% on chance alone, so a number read earlier is noise wearing a percentage sign.
Below the threshold the realised figures are null and the curves are empty — there is no aggregate return anywhere in the payload, so there is nothing for a template change or a careless caller to leak. Read from services/bankroll.summary() on 2026-08-24: 32 bets on the ledger, 28 of them settled against the 200 required, so every realised figure in the payload is null.
The tempting alternative was to show the running return with a caveat next to it. A caveat next to a number does not stop the number being read; not computing it does.
One page, two models, and it used to contradict itself.
The single-match page shows pre-match probabilities from the walk-forward archive, where each row came from a fold model that never saw that match, and below them a TreeSHAP decomposition recomputed from the artefact currently being served, which was fitted on the whole archive. Those are different models by construction and they disagree often enough to be seen: measured on 2026-08-15 against the exact files the page serves, 303 of 9,880 Premier League rows diverged (3.07%) and 151 of 3,000 rows sampled across all 22 leagues (5.03%).
On a diverging match the page named one outcome in its headline and highlighted a different one three lines below, with nothing on screen saying which number came from which model. The instructive part is what the fix was not. Swapping the underlying archive file did not help — the contradiction survived that change, measured before and after — because staleness was never the cause.
The fix was to label both halves and print an explicit line when they disagree. The champion's own metadata carries the same warning in one sentence: walk-forward numbers come from per-season refits, not from that artefact.
# stack
├─ Python with FastAPI, Jinja2 templates, server-rendered HTML, no client │ framework ├─ XGBoost multi-class softprob champion with per-league encodings, over a │ Dixon-Coles strength layer with time decay and ridge regularisation, and │ an ELO component ├─ TreeSHAP for the per-match decomposition, computed from the served artefact ├─ Shin de-margining for the market baseline, with pre-match and closing │ prices measured on separate subsets and never read against each other ├─ pandas and Parquet over a layered local data lake (raw, bronze, silver, │ gold), 1.4 GB on disk on 2026-08-24 ├─ A file-backed model registry — champion pointer, shadow directory, retired │ directory and a Parquet ledger — with promotion and rollback thresholds as │ named constants ├─ Append-only JSONL for the paper bet ledger and for served predictions, │ hash-chained, with a separate watermark file for the truncation case the │ chain cannot see ├─ SQLite for accounts, with scrypt password hashing and email verification ├─ pytest, 2,619 tests collected at commit 4986dd78 on 2026-08-24 └─ pm2 on a Mac mini we own, bound to loopback only and published through a Cloudflare Tunnel
# outcome
> Live at tahmin.onbers.com. Verified from outside the network on 2026-08-24: the scorecard returns 200, and the origin sets its own Content-Security-Policy (default-src 'self', frame-ancestors 'none'), X-Frame-Options: DENY, X-Content-Type-Options: nosniff and Referrer-Policy: strict-origin-when-cross-origin and Strict-Transport-Security. The service listens on 127.0.0.1:8000 and nowhere else, so the tunnel is the only route in; .env, the accounts database and the data-lake and administration endpoints all return 404 to an anonymous request from the internet. On the same date the accounts database holds one row, the owner's.
> Measured 2026-08-24 at commit 4986dd78: the repository is 170 commits, first commit 10 August 2026, 286 tracked Python files and 81,443 lines, and 119 files whose names contain "test" -- the count shifts by a few either way on a different definition. Those counters move most days, which is why the commit is named next to them. Two things about that state are worth saying rather than leaving to be discovered.
> Its origin is a local bundle file, not a hosted remote, so there is exactly one copy of the history and it is on the same machine as the running service. And the working tree is dirty in the direction that costs most. Re-read 2026-08-24 at 14:16 +03, seven commits later at 9118c957, the three modified tracked files are all source — web/routes.py, services/geo_strip.py and tests/test_geo_split.py, the bookmaker-name fix described below — while the paper ledger and the served-prediction log are tracked, clean, and identical to a blob committed ten days earlier, because a reset put them back. The code that is running is not the code that is committed, and the evidence it reports is whatever the last reset restored. Why it is uncommitted is worth stating rather than leaving to look like negligence: something drives a git reset --hard followed by a fast-forward in this repository every few minutes — four times on the afternoon of 2026-08-24 alone — and committing the fix would break that sync's fast-forward. The patched files are kept under ~/backups/ so the running state can be restored after the next one.
> Two findings were open and unfixed when this was checked, on 2026-08-24. /docs, /redoc and /openapi.json answer 200 to anyone and enumerate every route the service has — 49 of them that day — which is at odds with the same service returning 404 for its administration pages: the pages are hidden and the map to them is published.
▶▼ read the full reasoning
A login gate is switched on for the interactive pages, which redirect anonymous visitors with a 303, and the page they land on still renders the sentence "membership is not currently required, the whole site is open" directly above the notice telling the visitor to sign in, because that sentence is keyed on a separate paid-tier flag that is still off.
Each is a small change; neither had been made on that date, and neither should be described as done. A third finding — the compliance layer that strips foreign bookmaker names for Turkish visitors reached the coupon calculator and the league pages but not the home page, the first page a visitor sees — was closed later the same day, by moving the strip out of the individual routes and into the single template layer every page renders through, so that it is on by default instead of being something each new route has to remember.
Measured from outside the network at 14:15 +03 on 2026-08-24: an external Turkish request to / returns 0 bookmaker names where it returned 10, /sozluk and /bulten return 0, /bugun and /guncel 307 to the home page, and the same request over loopback still returns 10, because the operator is supposed to see the real prices. What remains at that point was not a legal question but an unfinished filter, and this page described it wrongly until it was counted.
An external Turkish request to /karne returned 13 bookmaker mentions at 14:47 +03 on 2026-08-24, counted against a fixed 35-name list — pinnacle, bet365, b365, williamhill, william hill, betfair, bwin, unibet, ladbrokes, 1xbet, marathonbet, betsson, betway, coral, paddypower, skybet, betvictor, 10bet, 888sport, interwetten, leonbets, nordicbet, sbobet, smarkets, matchbook, betfred, boylesports, betclic, betano, misli, nesine, iddaa, bilyoner, tuttur, panel — matched case-insensitively on non-alphanumeric boundaries after script and style blocks are dropped and the tags stripped.
Loopback returned the same 13, split pinnacle 5, b365 4, williamhill 4. Exactly one of the thirteen was prose: the literal caption "Pinnacle kapanışı" written into karne.html:529, which is the keep the code documents. The other twelve were data, and all of them came from one line — karne.html:454 printing row.portfolio for the nine price-shopping rows, four each for pinnacle, b365 and williamhill, the avg and max rows being aggregation methods rather than brands.
The strip did run on that page — it is hooked into the shared template environment rather than called route by route — so it walked the whole context and changed nothing, because it neutralises an allow-list of key names, odds_source, bookmaker and bookmakers, while the leaking list is stored under portfolio. Twelve of the thirteen were therefore precisely what the layer says it strips: a value repackaged under a key the list does not contain defeats the filter silently, and no test read rendered /karne output.
Calling them prose made an unfinished filter read as finished. It was fixed the same afternoon, in the same place as the finding before it: portfolio became a mixed list key in the strip, book and price_source joined the scalar keys, and the two aggregation tokens are passed through so that stripping does not leave three differently-scored rows sharing one label.
The mechanism is still the default-on template layer rather than a fourth hand-written call site, and book is on the list only because kombine_pazarlar.py:288 copies the already-covered odds_source value into an uncovered key — the quiet way past a filter that matches names. The guard is now a test that renders the page in the Turkish view and counts the 35 names in the output instead of calling the function: five of its assertions failed before the change and all pass after, alongside 353 tests in the geo, bankroll, coupon, match and scorecard suites.
Measured from outside the network at 14:54 +03 on 2026-08-24: /karne returns 1 bookmaker name where it returned 13, the twelve data cells now reading panel with the avg and max rows unchanged, while loopback still returns all 13 real names. What is left is hand-written provenance in three places — that one caption, two sentences in ligler.html (lines 151 and 304), and one in lig.html (line 574) which renders on every one of the 22 league pages, so it is the widest of the three.
That is the keep the code documents, and a question for a lawyer rather than for a filter. Those three files are uncommitted for the same reason the earlier ones were, and copied to ~/backups/karne-leak-fix-20260824-1447/ against the next reset.