Instance-adaptive Zero-Shot Chain-of-Thought Prompting (IAP)

What is Instance-adaptive Zero-Shot CoT Prompting?

Instance-adaptive Zero-Shot Chain-of-Thought (CoT) Prompting (IAP) is a strategy designed to improve reasoning performance in Large Language Models (LLMs) by selecting the most effective prompt on a per-instance basis. Unlike traditional approaches that use a single prompt for all questions in a task (task-level prompting), IAP tailors prompts to each individual question, enhancing the accuracy and effectiveness of Zero-Shot CoT reasoning.

How It Works:

1. Prompt Selection Based on Information Flow: IAP analyzes how well a prompt helps the LLM aggregate information from the question to the prompt and from the question and prompt to the reasoning steps (rationale). By using saliency scores, the model can identify which prompts are better at transferring the necessary information to get the correct answer.

2. Adaptive Mechanism: IAP employs two strategies for prompt selection:

   • Sequential Substitution (IAP-ss): This method evaluates each prompt sequentially, stopping when a good prompt is found for the instance.
   • Majority Vote (IAP-mv): This method evaluates all candidate prompts and selects the answer based on the highest-scoring prompts.

3. Saliency Score Calculation: Saliency scores measure the importance of each token's contribution to the reasoning. IAP computes saliency scores for three key interactions:

   • Question-to-Prompt: How well the prompt reflects the question.
   • Question-to-Rationale: How much information from the question contributes directly to the reasoning.
   • Prompt-to-Rationale: How well the prompt influences the reasoning process.

How IAP Differs from Existing Techniques

• Task-Level vs. Instance-Level: Previous methods like Plan-and-Solve or OPPR aim to find the best prompt for an entire task. IAP, however, selects the best prompt for each individual question, making it more flexible and accurate.
• Saliency-Based Approach: While task-level methods focus on crafting optimal prompts for general use, IAP uses saliency score analysis to dynamically select prompts that better suit the specific information flow of each instance.
• Improved Long-Range Reasoning: IAP's ability to adapt to individual questions makes it especially effective in tasks that require complex, multi-step reasoning.

How to Use IAP

IAP is particularly useful for reasoning tasks such as math problems, logic puzzles, and commonsense reasoning. To implement it, you would:

1. Prepare Prompts: Collect a set of diverse prompts with different reasoning styles, such as "Let’s think step by step" or "Don’t think. Just feel."
2. Analyze Information Flow: Use saliency score analysis to evaluate how well each prompt helps the LLM handle the question at hand.
3. Select or Combine Prompts: Depending on the strategy (IAP-ss or IAP-mv), either sequentially test prompts or aggregate results to find the best reasoning path.

Results of IAP

IAP was tested on various reasoning tasks, including math (GSM8K, SVAMP), logic (Causal Judgment), and commonsense reasoning (CommonsenseQA, MMLU), using multiple LLMs such as LLaMA-3, Qwen, and LLaMA-2. It consistently outperformed task-level prompting approaches like OPPR and Self-discover, achieving significant accuracy improvements across different models and datasets.

Key Results:

These results highlight that IAP significantly boosts accuracy, particularly in tasks involving complex reasoning and multi-step problem-solving.

Conclusion

Instance-adaptive Zero-shot Chain-of-Thought Prompting (IAP) offers a dynamic, saliency-based approach to prompt selection for reasoning tasks. By tailoring prompts to each question rather than applying a one-size-fits-all solution, IAP improves LLM performance across math, logic, and commonsense reasoning tasks. Its ability to adaptively enhance the information flow between the question, prompt, and rationale makes it a powerful tool for zero-shot reasoning with LLMs.