What is variational inference?

Variational inference is an approximate Bayesian inference method used to estimate complex posterior distributions that are difficult to compute directly.

The idea is to replace the true posterior distribution of unknown form with a simpler parametric distribution from a certain family (e.g., Gaussian), and to tune the parameters of this distribution so that it is as close as possible to the true one (minimizing the Kullback-Leibler divergence).

This allows for efficient computation of approximate probabilities and inference in models with a large number of parameters, where classical methods like MCMC are too slow.

In the context of Gaussian Processes (GP) and Bayesian Optimization (BO), variational inference helps approximate complex distributions over functions and hyperparameters, ensuring scalability and efficiency.

The process roughly looks like this:

1. A family of variational distributions q(θ; φ) is chosen.
2. The variational lower bound (ELBO) is formulated.
3. The parameters φ are optimized to maximize ELBO.

Thus, variational inference transforms the Bayesian inference problem into an optimization problem.