What is Dropout and how does it behave during training and inference?

Dropout is a regularization method in deep neural networks that helps prevent overfitting. During training, a random subset of neurons is "dropped out" (set to zero) with a specified probability (e.g., 0.5), which forces the network not to rely too heavily on any particular neurons and improves generalization.

During the training phase (train), dropout is active: neurons are randomly turned off, and weights are updated accordingly.

During inference (inference), dropout is turned off, all neurons are active, and outputs are scaled (usually multiplied by the probability of keeping a neuron) to compensate for the dropout during training.

Example in PyTorch:

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 20),
    nn.Dropout(p=0.5),
    nn.ReLU(),
    nn.Linear(20, 1)
)

model.train()  # dropout enabled
output_train = model(torch.randn(1, 10))

model.eval()   # dropout disabled
output_eval = model(torch.randn(1, 10))