What is global average pooling and why does it replace a fully-connected layer?

Global Average Pooling (GAP) is an operation that computes the average value across each channel of the input tensor, typically after convolutional layers in a neural network. Instead of using a fully-connected layer, which transforms a multi-dimensional tensor into a fixed-length vector, GAP reduces each channel to a single number — the mean value over the spatial dimensions (height and width).

Reasons for replacing a fully-connected layer with GAP:

• Reducing the number of parameters: Fully-connected layers have many parameters, which increases the risk of overfitting and requires more computational resources.
• Invariance to spatial shifts: GAP aggregates information over the entire spatial domain, making the model less sensitive to the position of features.
• Simplifying architecture: GAP allows for variable input image sizes, as it does not depend on a fixed input size, unlike fully-connected layers.

Example of using GAP in PyTorch:

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(16, 32, kernel_size=3, padding=1),
            nn.ReLU()
        )
        self.global_avg_pool = nn.AdaptiveAvgPool2d(1)  # GAP
        self.classifier = nn.Linear(32, num_classes)  # replaces fully-connected layer

    def forward(self, x):
        x = self.features(x)
        x = self.global_avg_pool(x)  # size [batch, 32, 1, 1]
        x = x.view(x.size(0), -1)    # size [batch, 32]
        x = self.classifier(x)
        return x

Thus, GAP simplifies the model and reduces the number of parameters while preserving important information for classification.