Announcing our new Paper: The Prompt Report, with Co-authors from OpenAI & Microsoft!
Check it out →
Learn Prompting🟦 Contrastive Chain-of-Thought (CoT) Prompting
Last updated on August 19, 2024 by Bhuwan Bhatt-
Learning from Negative Examples: Contrastive Chain-of-Thought (CoT) enhances LLM reasoning by incorporating both positive and negative examples
-
Prompt Structure: Contrastive CoT prompts include a question, correct/incorrect answers, and the query.
-
Generating Examples: Automatically create contrastive examples by shuffling key entities from correct answers.
What is Contrastive Chain-of-thought (CoT) Prompting?
Chain-of-thought (CoT) prompting1 improves the accuracy of language models when solving mathematical and reasoning tasks by compelling the models to think. Adding a simple statement like "Let's think step by step" to the prompt can significantly improve the model's performance. Adding a positive example, a question-answer pair, before the actual question further improves the reasoning of the language model. Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.
Inspired by human's ability to learn from positive and negative examples, Contrastive Chain-of-Thought (CoT) prompting2, employs negative examples along with positive ones to enhance the reasoning capabilities of language models. Making the models aware of what mistakes to avoid via negative examples also makes them more accurate.
How to Use Contrastive Chain-of-thought (CoT) Prompting?
The prompt in the contrastive CoT compromises of the following components:
- Sample question or example question
- Correct explanation question in 1
- Incorrect explanation for question in 1
- Actual question/query
The example below shows contrastive chain-of-thought prompting in action.
Prompt
Question : James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?
Explanation: He writes each friend 3[object Object]2=12 pages every week. That means he writes 12*52=624 pages a year.
Wrong Explanation: He writes each friend 12[object Object]2=6 pages every week. That means he writes 6*2=12 pages a year.
Question: James has 30 teeth. His dentist drills 4 of them and caps 7 more teeth than he drills. What percentage of James' teeth does the dentist fix?
AI Output
Explanation: The dentist drills 4 teeth, so there are 30 - 4 = 26 teeth left. The dentist caps 7 more teeth than he drills, so he caps 4 + 7 = 11 teeth. Therefore, the dentist fixes a total of 4 + 11 = 15 teeth. To find the percentage of teeth the dentist fixes, we divide the number of teeth fixed by the total number of teeth and multiply by 100: 15/30 × 100 = 50%
How to Use an Automated Method to Generate Contrastive Examples?
The authors devise an automated method to generate contrastive examples in the paper. It involves the following steps:
- Curate examples with correct reasoning steps from the existing dataset.
- Within the example in step 1, use the entity recognition model to identify numbers, equations, or persons.
- Shuffle the position of the entities extracted from step 2.
Contrastive example generation2
The image below shows an instance where a contrastive example is generated using a positive example.
From positive example to contrastive example2
The example below shows contrastive CoT in action. Feel free to modify it and test your inputs.
Conclusion
Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.