Model Calibration Drift: When a PD Model Still Ranks Risk Correctly but Predicts the Wrong Risk Level

Discrimination and calibration are different model objects

Ranking asks whether PD_a>PD_b corresponds probabilistically to Risk_a>Risk_b. Calibration asks whether borrowers assigned p default at approximately p over the governed horizon and population.

Every band remains correctly ordered, so AUC/Gini can remain strong. Yet absolute risk is approximately doubled. A ranking metric cannot detect that error because monotonic probability transformations preserve ordering.

O/E finds level error; intercept and slope diagnose its form

O/E≈1 indicates broad alignment; 1.5 means observed risk is 50% above expected; 0.7 means the model overpredicts. Portfolio O/E can conceal offsetting segment errors and says little about shape.

α captures broad level bias conditional on the fitted calibration model. β captures whether the spread of predicted risk is appropriate. Under the stated convention, β<1 often signals overly extreme predictions and β>1 insufficient dispersion, but range, sample, estimation uncertainty and misspecification matter; never interpret slope as a slogan.

Brier and log loss evaluate probability quality but mix discrimination and calibration. Calibration curves compare predicted and observed rates by decile, score or PD band; grouping changes the picture. Hosmer–Lemeshow is grouping- and sample-size-sensitive: huge samples reject trivial deviations while small samples miss material ones.

A 250,000-account portfolio can rank well and understate risk

AUC remains 0.76 versus 0.77 at validation, the estimated calibration slope is 0.98 and intercept is positive. The uniform band pattern supports an intercept-type challenger. It does not prove the cause or authorise production change: vintage, segment, policy, macro and data evidence still matter.

PSI and calibration answer orthogonal questions

A stable score distribution and low PSI can coexist with rising defaults after unemployment, interest-rate or income stress. Conversely, a riskier applicant mix can raise PSI and average PD while each score band remains correctly calibrated. Population movement is not calibration failure.

Likewise, higher realised loss can originate from PD, LGD or EAD: EL=PD×LGD×EAD. Funding costs and margins can change lending economics while PD remains calibrated. Keep probability validity separate from decision economics.

Aggregate alignment can hide segment and maturity failure

Changing mix can make aggregate calibration appear correct while both segments are wrong—an intuitive Simpson's-paradox pattern. Diagnose product, channel, customer type, band, justified geography, risk grade and vintage, with uncertainty controls against noisy slicing.

A 12-month PD requires mature 12-month outcomes. Recent cohorts are censored; comparing annual PD with partial performance creates maturity bias. Use mature cohorts, disclose partial maturity, or use governed survival methods where appropriate. Low-default portfolios may require longer aggregation; high-risk short-tenor books can provide faster but more volatile feedback.

Recalibration method must match the diagnosed failure

Intercept-only recalibration is plausible when ranking, slope and ordering remain stable, a broad level shift persists and mature outcomes are sufficient. Champion/challenger comparisons should include existing, intercept-only, full logistic and justified segmented mappings on out-of-time mature samples.

Frequent rolling recalibration is responsive but can chase noise, destabilise PDs and expand governance burden. Structural policy, product, channel or regime breaks can invalidate simple adjustment.

Recalibration is not a substitute for redevelopment

Falling Gini/AUC, unstable slope, variable-relationship drift, segment inversion, weak support, new product mechanics or structural policy change point beyond simple calibration. Data defects demand remediation, not statistical polishing.

Large samples make tiny deviations statistically significant; small segments create wide uncertainty. Assess confidence intervals, absolute PD error, exposure, expected-loss impact and decision impact.

A stable score cut-off can conceal a changed risk appetite

A monotonic score may remain unchanged while recalibration updates Score→PD. If score 620 once implied 5% PD and now implies 7.5%, preserving the numeric cut-off accepts a different absolute risk level. SameScore ≠ SameRisk across calibration versions.

Score Scaling & PDO explains the mapping; Cut-Off Strategy connects it to economics and appetite. Direct-PD strategies can change approval immediately after recalibration; score strategies require explicit remapping and governance review.

Understated PD understates expected loss all else equal, distorting pricing, provisioning, portfolio forecasts and capital planning. Application-scoring PD, regulatory PD and IFRS 9 lifetime PD have different horizons and calibration architectures; never transfer a recalibration mechanically between uses.

A four-quarter case changes diagnosis as evidence matures

Q2 is population movement with stable performance. Q3 adds persistent level drift while ranking and slope largely survive: recalibration becomes a credible challenger. Q4 introduces slope and Gini deterioration, so intercept adjustment alone would hide structural failure.

Operational calibration control joins predictions to mature outcomes

Record model, calibration, score-scale and effective-date versions; test old and new score-to-PD maps; quantify cut-off, approval, expected-loss and segment impacts; preserve rollback and historical reconstruction. PD Model Monitoring provides the broader surveillance layer, and PD Ranking & Calibration develops the core validation distinction.

Fast-turning non-bank portfolios can learn sooner—and confound faster

Fintech, consumer-finance and instalment lenders may see faster turnover, higher default incidence and shorter outcome horizons, enabling responsive calibration monitoring. But frequent pricing, policy and channel changes alter selection and mix, making cause attribution harder. High-risk tails require granular evidence; low-default books require patience and pooled uncertainty.

Where calibrated parameters feed impairment, calibration drift can affect ECL. This does not make application PD equivalent to IFRS 9 lifetime PD; horizon, forward-looking scenarios and use-specific architecture remain distinct.

Seventeen failures turn recalibration into false assurance

A PD Calibration & Drift Agent can make challenger evidence recurring

Entimema's Credit Risk work connects PD calibration, validation, monitoring and redevelopment diagnostics. Decision Automation can operationalise controlled surveillance while accountable humans approve model changes.

Related foundations include Credit Scorecard Development, Logistic Regression for Credit Risk and Early Warning Indicators.