What is deep ensembles as a method of uncertainty estimation?

Deep ensembles are a method for estimating uncertainty in neural network predictions, based on training multiple models with different initializations and/or different data subsets. The idea is that each model in the ensemble is trained independently, and the final prediction is obtained by averaging the results of all models.

This approach allows for a more reliable estimate of uncertainty, as the spread of predictions among models reflects the confidence level in the answer. If models vary significantly, it indicates that the model is uncertain about the prediction.

Example:

• Train 5 neural networks with different random initializations.
• Obtain 5 predictions for a new input.
• The average value is the final prediction.
• The variance of the predictions is a measure of uncertainty.

This is a simple and effective method that does not require complex Bayesian calculations and often outperforms more complex methods such as variational Bayesian neural networks.