What is ConvNeXt and why did it catch up with ViT in quality?

ConvNeXt is a convolutional neural network architecture developed to modernize classic CNNs so that they can compete with transformers like ViT (Vision Transformer) in terms of quality and efficiency.

Main reasons why ConvNeXt caught up with ViT in quality:

• Modern architectural solutions: ConvNeXt incorporates ideas from transformers, such as large kernel sizes, LayerNorm instead of BatchNorm, simplified blocks, and an improved structure, which increases the model's expressiveness.
• Optimization for modern tasks: ConvNeXt applies best practices from transformers while maintaining the efficiency of convolutional operations, providing a balance between performance and quality.
• Training on large datasets: ConvNeXt is trained on large datasets using modern optimization techniques, which helps the model generalize better.

An example of a simplified ConvNeXt block (PyTorch-like pseudocode):

class ConvNeXtBlock(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dwconv = nn.Conv2d(dim, dim, kernel_size=7, padding=3, groups=dim)  # depthwise convolution
        self.norm = nn.LayerNorm(dim)
        self.pwconv1 = nn.Linear(dim, 4 * dim)
        self.act = nn.GELU()
        self.pwconv2 = nn.Linear(4 * dim, dim)

    def forward(self, x):
        input = x
        x = self.dwconv(x)
        x = x.permute(0, 2, 3, 1)  # NCHW -> NHWC
        x = self.norm(x)
        x = self.pwconv1(x)
        x = self.act(x)
        x = self.pwconv2(x)
        x = x.permute(0, 3, 1, 2)  # NHWC -> NCHW
        return x + input

Thus, ConvNeXt combines the advantages of convolutional networks and modern architectural solutions, enabling it to compete with ViT in quality.