The model is not the point. The point is that it survives an audit.

The source changed its vocabulary and nobody noticed

The SBA rebuilt the business-age category scheme between FY2018 and FY2021. It did not rename them: it changed the classification. Today most values fall into categories the model has no evidence about, and serving maps them to “unknown”.

The model does not degrade: it loses the variable entirely and keeps returning HTTP 200. No performance metric would have shown it, because performance cannot be measured on young cohorts — a charge-off takes a median of 51 months to appear.

And what it cost to fix

Harmonizing the vocabulary loses resolution: four age bands collapse into one. Arguing that is easy; measuring it means training the model twice on the same split with the same seed.

cost in AUC

0.0015

coverage recovered

+74.7 pp

irreducible

9.7%

The qualitative argument was right in direction and negligible in magnitude. What does not move is the residual: “Change of Ownership” is not an age but a form of acquisition, and mapping it would be inventing the data.

FY2024–2026 · 15.4% → 90.1% · AUC 0.7005 → 0.6990 · latest cohort FY2026 (35,641 loans, year in progress): 82.9%

The decision, in dollars and with its counterweight

Declining the riskiest 10% of the test portfolio would have avoided real losses. A sales deck would stop there.

FY2017–2018 · 89,313 loans · absorbed by the SBA $942M

The control blocks my own model

Of the ten promotion gates, the access model fails the fairness one: the disparate impact ratio sits below the four-fifths threshold. The threshold did not move.

worst group: Native Hawaiian or Other Pacific Islander

The gap did not widen — it came back

Between 2021 and 2023 the racial denial gap grew, and I published that figure myself. With three more years of data — FY2018 and FY2019 are ordinary-rate years — the baseline turns out to have been the wrong one: 2020–21 was the refi boom, an anomalous regime.

FY2018–2019 · ordinary rates

15.70pp

FY2020–2021 · refi boom

13.14pp

FY2023–2025 · post-shock

15.73pp

The post-shock gap sits three hundredths of a point from the pre-pandemic one. The number in circulation measures the boom ending, not the rate shock. (+0.03 pp vs FY2018–2019; +2.59 pp vs 2021)

And still, no causal effect is published

The estimator compares the gap within county, purpose, lien and occupancy, on a balanced panel with errors clustered by county. The pre-shock coefficients should be zero. They are not.

Three measured reasons, not three excuses

1. 1

   Pre-trends are not flat

   The 2018 coefficient exceeds the threshold, and the violation is monotone: a trend, not noise.

2. 2

   The textbook fix manufactures the effect

   Extrapolating the pre-trend yields a large, significant effect — because what it extrapolates is the boom the shock ends.

3. 3

   The two pools did not empty alike

   One group had far fewer applications drop away than the other. The 2023 pool is not the 2021 pool with fewer people: it is a different pool.

The conclusion resembles the SBA causal diagnostic — no effect published — but the content is the opposite. There, there was nothing to falsify with. Here there was, the test ran, and the falsification is what closes the case.

92,860,926 applications · FY2018–FY2025 · differential retention 17.2 pp

What sits behind this page

The repository ships the full ledger of what broke, including the defects I found in my own tooling while building this. It is the part you cannot fake.

• See the code
• The defect ledger
• The decision records
• What is missing, prioritized

No number on this page is written by hand. Every one comes from the bundle the repository publishes and verifies against its own exports. 0f1f958efacecb47