What is the problem with fine-tuning CLIP?

The problem of fine-tuning CLIP is related to its architecture and training process features:

• Catastrophic forgetting — when fine-tuning on a new task, the model may forget previously learned representations, which degrades quality.
• Data balancing complexity — CLIP is trained on a vast diversity of data; fine-tuning on a narrow dataset can lead to overfitting and reduced generalization ability.
• Computational resource requirements — fine-tuning large models like CLIP requires significant resources.
• Architecture features — CLIP combines text and visual embeddings, which complicates correct weight updates without breaking consistency.

Because of this, methods such as freezing some layers or adaptive techniques are often used to minimize negative effects.