What is cross-modal retrieval?

Cross-modal retrieval is a task of information retrieval where the query and objects to be searched are presented in different modalities (types of data). For example, a user inputs a text query, and the system must find relevant images, videos, or audio.

In the context of computer vision and multimodal systems, this means creating a common representation (embedding) for different data types so that objects can be compared and searched across modalities.

Example:

• Searching for an image by a textual description ("red car against a mountain background")
• Searching for a video by an audio clip

Implementation often involves neural network models trained on paired data (e.g., text and corresponding image) to project them into a shared feature space. Then, the search reduces to finding nearest neighbors in this space.

Thus, cross-modal retrieval enables efficient linking and searching of information across heterogeneous data sources, which is important for multimodal applications and artificial intelligence systems.