What is gradient boosting? Where does the gradient appear in it?

Gradient boosting is an ensemble learning method that builds a model sequentially, adding new weak models (usually decision trees) to correct the errors of the previous ones.

The gradient appears because each new model is trained on the gradient of the loss function with respect to the predictions of the current ensemble model. In simpler terms, we minimize the error using gradient descent in the function space.

At each step, the gradient of the error (the direction and magnitude in which predictions need to be improved) is calculated, and a new model is trained to predict this gradient, thereby adjusting the final prediction.

This allows for effectively reducing the error and improving the model quality through sequential learning from the errors of previous models.