+7 495 989 9610, 9am6pm (GMT+3), Monday – Friday
ИД «Финансы и кредит»






Finance and Credit

Benchmark test for calculated default probability values derived from the application of rating-based models

Vol. 27, Iss. 12, DECEMBER 2021

Received: 11 October 2021

Received in revised form: 25 October 2021

Accepted: 8 November 2021

Available online: 27 December 2021

Subject Heading: Financial control

JEL Classification: C58, G17, G28

Pages: 2719–2745


Mikhail V. POMAZANOV National Research University – Higher School of Economics (NRU – HSE), Moscow, Russian Federation


Subject. This article deals with the issues of validation of the consistency of rating-based model forecasts.
Objectives. The article aims to provide developers and validators of rating-based models with a practical fundamental test for benchmarking study of the estimated default probability values obtained as a result of the application of models used in the rating system.
Methods. For the study, I used the classical interval approach to testing of statistical hypotheses focused on the subject area of calibration of rating systems.
Results. In addition to the generally accepted tests for the correspondence of the predicted probabilities of default of credit risk objects to the historically realized values, the article proposes a new statistical test that corrects the shortcomings of the generally accepted ones, focused on "diagnosing" the consistency of the implemented discrimination of objects by the rating model. Examples of recognizing the reasons for a negative test result and negative consequences for lending are given while maintaining the current settings of the rating model. In addition to the bias in the assessment of the total frequency of defaults in the loan portfolio, the proposed method makes it possible to objectively reveal the inadequacy of discrimination against borrowers with a calibrated rating model, diagnose the “disease” of the rating model.
Conclusions and Relevance. The new practical benchmark test allows to reject the hypothesis about the consistency of assessing the probability of default by the rating model at a given level of confidence and available historical data. The test has the advantage of practical interpretability based on its results, it is possible to draw a conclusion about the direction of the model correction. The offered test can be used in the process of internal validation by the bank of its own rating models, which is required by the Bank of Russia for approaches based on internal ratings.

Keywords: credit risk, probability of default, statistical test, Gini, ROC curve


  1. Tasche D. Validation of Internal Rating Systems and PD Estimates. In: The Analytics of Risk Model Validation. Elsevier, 2008, pp. 169–196. URL: Link
  2. González F., Coppens F., Winkler G. The Performance of Credit Rating Systems in the Assessment of Collateral Used in Eurosystem Monetary Policy Operations. European Central Bank Occasional Paper Series, 2007, no. 65, 42 p. URL: Link
  3. Miu P., Ozdemir B. Estimating and Validating Long-Run Probability of Default with Respect to Basel II Requirements. Journal of Risk Model Validation, 2008, vol. 2, no. 2, pp. 3–41. URL: Link
  4. Sauer S., Coppens F., Mayer M. et al. Advances in Multivariate Back-Testing for Credit Risk Underestimation. European Central Bank Working Paper Series, 2016, no. 1885, 35 p. URL: Link
  5. Westfall P.H., Wolfinger R.D. Multiple Tests with Discrete Distributions. The American Statistician, 1997, vol. 51, iss. 1, pp. 3–8. URL: Link
  6. Hosmer D.W., Lemeshow S. Applied Logistic Regression. New York, John Wiley & Sons, Inc., 2000. URL: Link
  7. Sokal R.R., Rohlf F.J. Biometry: The Principles and Practice of Statistics in Biological Research. New York, Freeman, 1994, 880 p.
  8. McDonald J.H. Small numbers in chi-square and G–tests. Handbook of Biological Statistics. Baltimore, Maryland, Sparky House Publishing, 2014, pp. 86–89.
  9. Spiegelhalter D.J. Probabilistic Prediction in Patient Management and Clinical Trials. Statistics in Medicine, 1986, vol. 5, iss. 5, pp. 421–433. URL: Link
  10. Geary R.C. The Frequency Distribution of the Quotient of Two Normal Variates. Journal of the Royal Statistical Society, 1930, vol. 93, no. 3, pp. 442–446. URL: Link
  11. Hinkley D.V. On the Ratio of Two Correlated Normal Random Variables. Biometrika, 1969, vol. 56, iss. 3, pp. 635–639. URL: Link
  12. Hayya J., Armstrong D., Gressis N. A Note on the Ratio of Two Normally Distributed Variables. Management Science, 1975, vol. 21, no. 11, pp. 1338–1341. URL: Link
  13. Pomazanov M.V. [ROC Analysis and Calibration of Scoring Models Based on Second Order Accuracy Metrics]. Upravlenie finansovymi riskami = Financial Risk Management, 2021, no. 2, pp. 100–121. (In Russ.) URL: Link
  14. Hong Ch.-S., Lee W.-Y. ROC Curve Fitting with Normal Mixtures. The Korean Journal of Applied Statistics, 2011, vol. 24, iss. 2, pp. 269–278. URL: Link
  15. Engelmann B., Hayden E., Tasche D. Measuring the Discriminative Power of Rating Systems. Discussion Paper Series 2: Banking and Financial Supervision, 2003, no. 01, 24 p. URL: Link
  16. Hanley J.A., McNeil B.J. The Meaning and Use of the Area under a Receiver Operating Characteristics (ROC) Curve. Radiology, 1982, vol. 143, no. 1, pp. 29–36. URL: Link

View all articles of issue


ISSN 2311-8709 (Online)
ISSN 2071-4688 (Print)

Journal current issue

Vol. 27, Iss. 12
December 2021