What is ELBO and why is it maximized?

ELBO (Evidence Lower Bound) is a lower bound on the logarithm of the marginal likelihood of data in variational inference. In Bayesian inference tasks, where exact computation of the posterior distribution is impossible, ELBO serves as the objective function for optimizing the approximate distribution.

Why maximize ELBO:

• Maximizing ELBO is equivalent to minimizing the KL divergence between the approximate and true posterior distributions.
• It helps find the approximation closest to the true posterior distribution.
• ELBO balances the accuracy of the approximation and the complexity of the model.

In the context of Gaussian Processes (GP) and Bayesian Optimization (BO), maximizing ELBO helps efficiently train models and make predictions considering uncertainty, thereby improving the quality of optimization and generalization.