Decision system · credit-risk-mlops

The model is not the point. The point is that it survives an audit.

A credit model nobody can audit does not get deployed, however good it is. This page shows the four things a validator asks — and almost no portfolio shows — with the measurement beside each one: what happens when the source changes its language, what the decision costs in dollars, what the control does when the model does not comply, and what can honestly be claimed about a gap that moves.

data: SBA 7(a) FOIA + HMDA (FFIEC/CFPB) · metrics recomputed from saved predictions · open source

The model, running in your browser

Not a video and not a screenshot: the production ONNX artifact scoring locally, with no server and no data leaving the page. Try setting business age to “Change of Ownership” — the model answers with the same confidence, and the next section explains why that is the problem.

Open the demo1.9 MB · numerical parity verified against the original model

The source changed its vocabulary and nobody noticed

The SBA rebuilt the business-age category scheme between FY2018 and FY2021. It did not rename them: it changed the classification. Today most values fall into categories the model has no evidence about, and serving maps them to “unknown”.

Share of each cohort whose business_age value falls into categories holding less than 0.5% of the training mass. Hollow bars are healthy cohorts.

The model does not degrade: it loses the variable entirely and keeps returning HTTP 200. No performance metric would have shown it, because performance cannot be measured on young cohorts — a charge-off takes a median of 51 months to appear.

And what it cost to fix

Harmonising the vocabulary loses resolution: four age bands collapse into one. Arguing that is easy; measuring it means training the model twice on the same split with the same seed.

cost in AUC: 0.0015
coverage recovered: +74.7 pp
irreducible: 9.7%

The qualitative argument was right in direction and negligible in magnitude. What does not move is the residual: “Change of Ownership” is not an age but a form of acquisition, and mapping it would be inventing the data.

FY2024–2026 · 15.4% → 90.1% · AUC 0.7005 → 0.6990 · latest cohort FY2026 (35,641 loans, year in progress): 82.9%

The decision, in dollars and with its counterweight

Declining the riskiest 10% of the test portfolio would have avoided real losses. A sales deck would stop there.

Both figures travel in the same payload in the repository. A headline that shows only the numerator is not a headline.

FY2017–2018 · 89,313 loans · absorbed by the SBA $942M

The control blocks my own model

Of the ten promotion gates, the access model fails the fairness one: the disparate impact ratio sits below the four-fifths threshold. The threshold did not move.

The verdict separates two things that are usually conflated: the build not breaking, and the model complying. Here they differ, and the report says so in those words.

worst group: Native Hawaiian or Other Pacific Islander

The gap did not widen — it came back

Between 2021 and 2023 the racial denial gap grew, and I published that figure myself. With three more years of data — FY2018 and FY2019 are ordinary-rate years — the baseline turns out to have been the wrong one: 2020–21 was the refi boom, an anomalous regime.

FY2018–2019 · ordinary rates: 15.70pp
FY2020–2021 · refi boom: 13.14pp
FY2023–2025 · post-shock: 15.73pp

The post-shock gap sits three hundredths of a point from the pre-pandemic one. The number in circulation measures the boom ending, not the rate shock. (+0.03 pp vs FY2018–2019; +2.59 pp vs 2021)

And still, no causal effect is published

The estimator compares the gap within county, purpose, lien and occupancy, on a balanced panel with errors clustered by county. The pre-shock coefficients should be zero. They are not.

Hollow point: pre-period. Bar: 95% confidence interval, clustered by county. The band is the magnitude declared meaningful before any result was seen.

Three measured reasons, not three excuses

1
Pre-trends are not flat
The 2018 coefficient exceeds the threshold, and the violation is monotone: a trend, not noise.
2
The textbook fix manufactures the effect
Extrapolating the pre-trend yields a large, significant effect — because what it extrapolates is the boom the shock ends.
3
The two pools did not empty alike
One group had far fewer applications drop away than the other. The 2023 pool is not the 2021 pool with fewer people: it is a different pool.

The conclusion resembles the SBA causal diagnostic — no effect published — but the content is the opposite. There, there was nothing to falsify with. Here there was, the test ran, and the falsification is what closes the case.

92,860,926 applications · FY2018–FY2025 · differential retention 17.2 pp

What sits behind this page

The repository ships the full ledger of what broke, including the defects I found in my own tooling while building this. It is the part you cannot fake.

No number on this page is written by hand. Every one comes from the bundle the repository publishes and verifies against its own exports. 0f1f958efacecb47