Announcing our new Course: AI Red-Teaming and AI Safety Masterclass

Check it out →
🧠 Advanced
🧠 AdvancedThought Generation🟦 Contrastive Chain-of-Thought

🟦 Contrastive Chain-of-Thought (CoT) Prompting

Last updated on September 27, 2024 by Bhuwan Bhatt
Overview of Contrastive Chain-of-Thought (CoT) Prompting
Takeaways
  • Learning from Negative Examples: Contrastive Chain-of-Thought (CoT) enhances LLM reasoning by incorporating both positive and negative examples

  • Prompt Structure: Contrastive CoT prompts include a question, correct/incorrect answers, and the query.

  • Generating Examples: Automatically create contrastive examples by shuffling key entities from correct answers.

Information and Links

TechniqueInstitutionDate of PublicationPaperCode
Contrastive Chain-of-thought (CoT) PromptingDAMO Academy, Alibaba Group; Singapore University of Technology and Design; Nanyang Technological University, SingaporeNov 2023Contrastive Chain-of-Thought PromptingCode

What is Contrastive Chain-of-thought (CoT) Prompting?

Chain-of-thought (CoT) prompting1 improves the accuracy of language models when solving mathematical and reasoning tasks by compelling the models to think. Adding a simple statement like "Let's think step by step" to the prompt can significantly improve the model's performance. Adding a positive example, a question-answer pair, before the actual question further improves the reasoning of the language model. Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.

Inspired by human's ability to learn from positive and negative examples, Contrastive Chain-of-Thought (CoT) Prompting2, employs negative examples along with positive ones to enhance the reasoning capabilities of language models. Making the models aware of what mistakes to avoid via negative examples also makes them more accurate.

How to Use Contrastive Chain-of-Thought (CoT) Prompting?

The prompt in the contrastive CoT compromises of the following components:

  1. Sample question or example question
  2. Correct explanation question in 1
  3. Incorrect explanation for question in 1
  4. Actual question/query

The example below shows contrastive chain-of-thought prompting in action.

Astronaut

Prompt


Question : James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?

Explanation: He writes each friend 3[object Object]2=12 pages every week. That means he writes 12*52=624 pages a year.

Wrong Explanation: He writes each friend 12[object Object]2=6 pages every week. That means he writes 6*2=12 pages a year.

Question: James has 30 teeth. His dentist drills 4 of them and caps 7 more teeth than he drills. What percentage of James' teeth does the dentist fix?
Robot

AI Output


Explanation: The dentist drills 4 teeth, so there are 30 - 4 = 26 teeth left. The dentist caps 7 more teeth than he drills, so he caps 4 + 7 = 11 teeth. Therefore, the dentist fixes a total of 4 + 11 = 15 teeth. To find the percentage of teeth the dentist fixes, we divide the number of teeth fixed by the total number of teeth and multiply by 100: 15/30 × 100 = 50%

How to Use an Automated Method to Generate Contrastive Examples?

The authors devise an automated method to generate contrastive examples in the paper. It involves the following steps:

  1. Curate examples with correct reasoning steps from the existing dataset.
  2. Within the example in step 1, use the entity recognition model to identify numbers, equations, or persons.
  3. Shuffle the position of the entities extracted from step 2.

Contrastive example generation2

The image below shows an instance where a contrastive example is generated using a positive example.

From positive example to contrastive example2

The example below shows contrastive CoT in action. Feel free to modify it and test your inputs.

Tip

The code for Contrastive Chain-of-Thought (CoT) Prompting is open-sourced and available for further research and implementation on GitHub.

Conclusion

Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.

Footnotes

  1. Jason Wei. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

  2. Yew Ken Chia. (2023). Contrastive Chain-of-Thought Prompting. In arXiv preprint arXiv:1907.11692. 2 3

Edit this page
Word count: 0
Copyright © 2024 Learn Prompting.