Learn Prompting🟦 Contrastive Chain-of-Thought (CoT) Prompting
Last updated on October 1, 2024
-
Learning from Negative Examples: Contrastive Chain-of-Thought (CoT) enhances LLM reasoning by incorporating both positive and negative examples
-
Prompt Structure: Contrastive CoT prompts include a question, correct/incorrect answers, and the query.
-
Generating Examples: Automatically create contrastive examples by shuffling key entities from correct answers.
Information and Links
| Technique | Institution | Date of Publication | Paper | Code |
|---|---|---|---|---|
| Contrastive Chain-of-Thought (CoT) Prompting | DAMO Academy, Alibaba Group; Singapore University of Technology and Design; Nanyang Technological University, Singapore | Nov 2023 | Contrastive Chain-of-Thought Prompting | DAMO-NLP-SG/contrastive-cot |
What is Contrastive Chain-of-Thought (CoT) Prompting?
Chain-of-Thought (CoT) Prompting improves the accuracy of Large Language Models (LLMs) when solving mathematical and reasoning tasks by compelling the models to think. Adding a simple statement like "Let's think step by step" to the prompt can significantly improve the model's performance. Adding a positive example, a question-answer pair, before the actual question further improves the reasoning of the LLM. Learning from negative examples is not unique to humans and can also be replicated in LLMs. Embedding negative examples with the prompt can help LLMs learn from them and generate better explanations for their answers.
Inspired by human's ability to learn from positive and negative examples, Contrastive Chain-of-Thought (CoT) Prompting, employs negative examples along with positive ones to enhance the reasoning capabilities of language models. Making the models aware of what mistakes to avoid via negative examples also makes them more accurate.
How to Use Contrastive Chain-of-Thought (CoT) Prompting?
The prompt in the Contrastive CoT compromises of the following components:
- Sample question or example question
- Correct explanation question in 1
- Incorrect explanation for question in 1
- Actual question/query
The example below shows Contrastive Chain-of-Thought prompting in action.
Prompt
Question : James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?
Explanation: He writes each friend 32=6 pages a week. So he writes 62=12 pages every week. That means he writes 12*52=624 pages a year.
Wrong Explanation: He writes each friend 1252=624 pages a week. So he writes 32=6 pages every week. That means he writes 6*2=12 pages a year.
Question: James has 30 teeth. His dentist drills 4 of them and caps 7 more teeth than he drills. What percentage of James' teeth does the dentist fix?
AI Output
Explanation: The dentist drills 4 teeth, so there are 30 - 4 = 26 teeth left. The dentist caps 7 more teeth than he drills, so he caps 4 + 7 = 11 teeth. Therefore, the dentist fixes a total of 4 + 11 = 15 teeth. To find the percentage of teeth the dentist fixes, we divide the number of teeth fixed by the total number of teeth and multiply by 100: 15/30 × 100 = 50%
How to Use an Automated Method to Generate Contrastive Examples?
The authors devise an automated method to generate contrastive examples in the paper. It involves the following steps:
- Curate examples with correct reasoning steps from the existing dataset.
- Within the example in step 1, use the entity recognition model to identify numbers, equations, or persons.
- Shuffle the position of the entities extracted from step 2.
Contrastive example generation
The image below shows an instance where a contrastive example is generated using a positive example.
From positive example to contrastive example
The example below shows Contrastive CoT in action. Feel free to modify it and test your inputs.
The code for Contrastive Chain-of-Thought (CoT) Prompting is open-sourced and available for further research and implementation at DAMO-NLP-SG/contrastive-cot.
Conclusion
Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.