What is Dropout and how does it behave during training and inference?

Dropout is a regularization method in deep neural networks that helps prevent overfitting. During training, a random subset of neurons is "dropped out" (set to zero) with a specified probability (e.g., 0.5), which forces the network not to rely too heavily on any particular neurons and improves generalization.

During the training phase (train), dropout is active: neurons are randomly turned off, and weights are updated accordingly.

During inference (inference), dropout is turned off, all neurons are active, and outputs are scaled (usually multiplied by the probability of keeping a neuron) to compensate for the dropout during training.

Example in PyTorch:

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 20),
    nn.Dropout(p=0.5),
    nn.ReLU(),
    nn.Linear(20, 1)
)

model.train()  # dropout enabled
output_train = model(torch.randn(1, 10))

model.eval()   # dropout disabled
output_eval = model(torch.randn(1, 10))

Dropout is a regularization technique for neural networks that randomly "turns off" (zeros out) some neurons during training to prevent overfitting.

During training:

• A certain percentage of neurons (e.g., 20-50%) are randomly turned off for each input example.
• This forces the network not to rely on specific neurons and improves generalization.

During inference (prediction):

• Dropout is not applied, all neurons are active.
• To compensate for the absence of neuron deactivation, weights are scaled (usually multiplied by the probability of a neuron being active), or scaling is used during training (inverted dropout).

Thus, dropout helps make the model more robust and less prone to overfitting, but during inference, the network operates fully without random deactivations.