What is the DART mode in boosting?

The DART (Dropouts meet Multiple Additive Regression Trees) mode is a boosting method that improves upon classic gradient boosting by adding a mechanism of random dropout of individual trees at each training step. The idea is to prevent overfitting and enhance the model's generalization ability by randomly 'turning off' part of the ensemble during updates.

In traditional gradient boosting, each new tree is added to the ensemble to correct errors of the previous ones. In DART, at each step, some of the already built trees are randomly excluded, and the new tree is trained on the remaining ones, which reduces correlation between trees and improves model stability.

An example of using DART in XGBoost (Python):

import xgboost as xgb

params = {
    'booster': 'dart',
    'objective': 'binary:logistic',
    'rate_drop': 0.1,  # probability of dropping a tree
    'skip_drop': 0.5   # probability of skipping dropout
}
dtrain = xgb.DMatrix(X_train, label=y_train)
model = xgb.train(params, dtrain, num_boost_round=100)

Thus, DART helps make the model more resistant to overfitting by randomly excluding trees during training.