If the AUC of a new model is 0.82 compared to 0.80 for the production model — can we confidently say that the new one is better? How to check statistical significance?

A difference in AUC between 0.82 and 0.80 does not necessarily mean that the new model is statistically significantly better. To make a confident conclusion, you need to test the statistical significance of the difference.

How to test:

1. Statistical test — for example, the DeLong test, which compares ROC curves of two models on the same dataset.
2. Bootstrap — repeated random sampling with AUC calculation to obtain the distribution and confidence intervals for each model, then check if the intervals overlap.

If the p-value of the test is less than the chosen significance level (e.g., 0.05), it can be considered that the new model is statistically significantly better.

Without such a test, it is not possible to confidently assert that 0.82 is better than 0.80, as the difference could be due to randomness.