How to check the quality of new features and hypotheses in risk models and what metrics to use?

To assess the quality of new features and hypotheses in risk models, it is important to use a comprehensive approach that includes both statistical evaluation and business metrics.

Main steps:

1. Data splitting: use train/test or cross-validation to evaluate the model's generalization ability.

2. Model quality metrics:

   • AUC-ROC — to assess the model's ability to distinguish between classes.
   • KS statistic — often used in credit risk models to evaluate separation between defaulting and non-defaulting clients.
   • Gini coefficient — closely related to AUC, used to assess ranking quality.
   • Log Loss / Brier Score — to evaluate the quality of probabilistic forecasts.

3. Stability metrics:

   • Population Stability Index (PSI) — to assess changes in feature distribution or scoring over time.

4. Business metrics:

   • Loss Given Default (LGD), Probability of Default (PD), Exposure at Default (EAD) — if applicable.
   • Economic efficiency — for example, reduction in losses, profit growth.

5. A/B testing or pilot launch of new features to test hypotheses in real conditions.

Example: if you add a new feature to a credit scoring model, first evaluate the change in AUC and KS on the test sample, then check PSI to ensure the new feature does not lead to instability. After that, you can conduct a pilot test on a small group of clients and evaluate the impact on business metrics, such as default rate and profitability.

Thus, combining statistical and business metrics allows for an objective assessment of the quality of new features and hypotheses in risk models.