What is dropout?

Dropout is a regularization technique in neural networks that helps prevent overfitting. During training, a random subset of neurons is "dropped out" (set to zero) with a specified probability, which forces the network not to rely too heavily on any particular neurons and improves the model's generalization ability.

For example, with a dropout probability of 0.5, half of the neurons are temporarily disabled at each training step.

In popular libraries like TensorFlow or PyTorch, dropout is implemented as a layer:

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.dropout = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(256, 10)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

Thus, dropout helps make the model more robust and improves its performance on new data.