What is a residual connection and what is it used for?

Residual connection — this is an architectural technique in deep neural networks where the input to a layer is added to its output, bypassing the main transformation block. This helps combat the vanishing gradient problem and facilitates the training of very deep networks.

The idea is that the layer learns not to directly map input to output, but only to correct the input (learn the residual). This allows the gradient to flow freely through the network, improving convergence and model quality.

Example in PyTorch:

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = self.conv(x)
        out += x  # residual connection
        return self.relu(out)

Residual connections are widely used in ResNet architectures and their derivatives.