🟦 Contrastive Chain-of-Thought (CoT) Prompting

Information and Links

What is Contrastive Chain-of-Thought (CoT) Prompting?

Chain-of-Thought (CoT) Prompting^{1Jason Wei. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.} improves the accuracy of Large Language Models (LLMs) when solving mathematical and reasoning tasks by compelling the models to think. Adding a simple statement like "Let's think step by step" to the prompt can significantly improve the model's performance. Adding a positive example, a question-answer pair, before the actual question further improves the reasoning of the LLM. Learning from negative examples is not unique to humans and can also be replicated in LLMs. Embedding negative examples with the prompt can help LLMs learn from them and generate better explanations for their answers.

Inspired by human's ability to learn from positive and negative examples, Contrastive Chain-of-Thought (CoT) Prompting^{2Yew Ken Chia. (2023). Contrastive Chain-of-Thought Prompting. In arXiv preprint arXiv:1907.11692.} , employs negative examples along with positive ones to enhance the reasoning capabilities of language models. Making the models aware of what mistakes to avoid via negative examples also makes them more accurate.

How to Use Contrastive Chain-of-Thought (CoT) Prompting?

The prompt in the Contrastive CoT compromises of the following components:

1. Sample question or example question
2. Correct explanation question in 1
3. Incorrect explanation for question in 1
4. Actual question/query

The example below shows Contrastive Chain-of-Thought prompting in action.

 Copy

Prompt

---

Question : James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?

Explanation: He writes each friend 32=6 pages a week. So he writes 62=12 pages every week. That means he writes 12*52=624 pages a year.

Wrong Explanation: He writes each friend 1252=624 pages a week. So he writes 32=6 pages every week. That means he writes 6*2=12 pages a year.

Question: James has 30 teeth. His dentist drills 4 of them and caps 7 more teeth than he drills. What percentage of James' teeth does the dentist fix?

 Copy

AI Output

---

Explanation: The dentist drills 4 teeth, so there are 30 - 4 = 26 teeth left. The dentist caps 7 more teeth than he drills, so he caps 4 + 7 = 11 teeth. Therefore, the dentist fixes a total of 4 + 11 = 15 teeth. To find the percentage of teeth the dentist fixes, we divide the number of teeth fixed by the total number of teeth and multiply by 100: 15/30 × 100 = 50%

How to Use an Automated Method to Generate Contrastive Examples?

The authors devise an automated method to generate contrastive examples in the paper. It involves the following steps:

1. Curate examples with correct reasoning steps from the existing dataset.
2. Within the example in step 1, use the entity recognition model to identify numbers, equations, or persons.
3. Shuffle the position of the entities extracted from step 2.

Contrastive example generation^{2Yew Ken Chia. (2023). Contrastive Chain-of-Thought Prompting. In arXiv preprint arXiv:1907.11692.}

The image below shows an instance where a contrastive example is generated using a positive example.

From positive example to contrastive example^{2Yew Ken Chia. (2023). Contrastive Chain-of-Thought Prompting. In arXiv preprint arXiv:1907.11692.}

The example below shows Contrastive CoT in action. Feel free to modify it and test your inputs.

Conclusion

Learning from negative examples is not unique to humans and can also be replicated in language models. Embedding negative examples with the prompt can help language models learn from them and generate better explanations for their answers.