What is Mask2Former and its query-based approach?

Mask2Former is a modern approach to image segmentation tasks that combines various types of segmentation (e.g., semantic, instance, and panoptic) into a single architecture. The main idea is to use a query-based mechanism, inspired by transformers, to predict object masks.

Instead of a traditional pixel-wise or class-wise classifier, Mask2Former formulates segmentation as a task of finding a set of masks using a set of learnable queries. Each query is responsible for predicting one mask and its class. This allows the model to effectively separate objects from the background and handle different types of segmentation within a single framework.

Here's roughly how it works:

• The input image passes through a backbone (e.g., CNN or transformer) to extract features.
• These features are fed into a transformer, where learnable queries interact with the features.
• The output for each query is a mask and a class.

This approach simplifies the architecture and improves segmentation quality by leveraging global context and query flexibility.