◆ Mixture of Prompt Learning (MoCoOp) for Vision-Language Models

What is MoCoOp?

Mixture of Prompt Learning (MoCoOp) is a technique designed to improve the performance of pre-trained vision-language models (VLMs) like CLIP in various downstream tasks. This method enhances prompt learning, a technique that fine-tunes text prompts (inputs) to guide the VLMs toward better results in tasks like image classification.

Traditionally, prompts can be hard prompts (manually designed, static templates) or soft prompts (learnable vectors). Soft prompts are more adaptable but face two key challenges:

1. Dataset Style Variations: A single soft prompt cannot capture the variety of styles and patterns within a dataset.
2. Overfitting: Soft prompts are prone to overfitting, meaning they may perform well on training data but struggle on new or unseen classes.

MoCoOp tackles these issues by incorporating multiple soft prompts, each representing different styles, and uses a routing module to dynamically select the most appropriate prompt for each image. This is combined with a gating mechanism that ensures the selected prompts align with the knowledge from hard prompts, helping reduce overfitting.

How it works:

1. Multiple Prompts: Different soft prompts are used to capture the diverse styles within a dataset, instead of relying on a single prompt.
2. Routing Module: For each image, the router selects the most suitable prompts by evaluating their compatibility with the image’s features.
3. Gating Mechanism: Ensures that the chosen soft prompts maintain alignment with hard prompts, retaining valuable knowledge.
4. Semantically Grouped Supervision: Each soft prompt is initialized from semantically grouped hard prompts, helping to preserve the initial structure and knowledge, and reducing overfitting.

How it Works: Step by Step

1. Image Processing: The input image is processed through the image encoder to get its features.
2. Routing and Prompt Selection: The routing module selects the top $k$ soft prompts that best match the image features. These soft prompts are concatenated with class names and passed through the text encoder.
3. Weighted Combination: The resulting class text features are weighted and averaged to produce a final feature set.
4. Classification: This final feature set is compared to the image features to predict the class.

How does MoCoOp differ from existing techniques?

MoCoOp stands out by dynamically selecting soft prompts to better match the diversity of the dataset, while also incorporating hard prompts to maintain generalizable knowledge. Here’s a comparison with existing approaches:

Technique	Approach	Key Features	Generalization
CoOp	Static soft prompts	Learnable prompts for better task adaptation	Prone to overfitting on new classes
CoCoOp	Dynamic soft prompts	Instance-specific prompt generation	Improved unseen class performance
MoCoOp	Mixture of soft prompts	Multiple prompts with routing for better match	Strongest, especially in complex datasets

Key Differences:

• MoCoOp incorporates multiple prompts, representing different styles, while CoOp and CoCoOp rely on a single or instance-specific prompt.
• The routing mechanism in MoCoOp ensures the most suitable prompts are selected dynamically for each image.
• Hard prompt guided supervision ensures that the soft prompts retain core knowledge and mitigate overfitting, a problem often seen in CoOp.

How to use MoCoOp

MoCoOp is particularly useful for tasks that involve datasets with a wide variety of styles or patterns and for applications where the model needs to generalize to new or unseen classes. Typical applications include:

• Image classification with varied visual styles (e.g., artistic or medical images).
• Few-Shot learning: Learning from a small number of labeled samples.
• Domain generalization: Adapting a model trained on one dataset to perform well on others.

Example Input/Output:

• Input: An image of a rare bird species with a varied visual style.
• Output: The correct bird species class, identified using dynamically selected prompts that match the image style.

Results of MoCoOp

MoCoOp has shown significant improvements in Few-Shot learning, base-to-new class generalization, and domain generalization across multiple benchmarks.

Base-to-New Class Generalization:

Method	Base Accuracy	New Accuracy	Harmonic Mean (H)
CLIP	70.25%	74.22%	71.57%
CoOp	82.64%	68.00%	74.02%
MoCoOp	83.32%	77.34%	80.17%

MoCoOp consistently outperforms other methods like CoOp, LASP, and CoCoOp in Few-Shot learning scenarios, with significant gains across datasets like Caltech101, Flowers102, and Stanford Cars.

Conclusion

MoCoOp effectively addresses key challenges in prompt learning for vision-language models by introducing a mixture of soft prompts and dynamic selection via a routing module. This approach significantly improves performance in few-shot learning, base-to-new class generalization, and domain generalization, outperforming existing methods across various datasets.

Valeriia Kuka

Valeriia Kuka, Head of Content at Learn Prompting, is passionate about making AI and ML accessible. Valeriia previously grew a 60K+ follower AI-focused social media account, earning reposts from Stanford NLP, Amazon Research, Hugging Face, and AI researchers. She has also worked with AI/ML newsletters and global communities with 100K+ members and authored clear and concise explainers and historical articles.

Footnotes

1. Du, Y., Niu, T., & Zhao, R. (2024). Mixture of Prompt Learning for Vision Language Models. https://arxiv.org/abs/2409.12011 ↩