🟢 Self-Ask Prompting

Last updated on August 27, 2024 by Valeriia Kuka

What is Self-Ask?

The creators of Self-Ask¹ explored how much of a Large Language Model (LLM) correct answers are due to reasoning between facts rather than just memorization. Their intuition was that prompting a model to ask follow-up questions to break down the initial query would enhance its performance on reasoning tasks.

Let's look at an example. If you ask an LLM the following questions, it will likely provide correct answers:

• Who won the Master's Tournament in 1994?
• When Justin Bieber was born?

These questions only require the model to recall facts it encountered during training.

However, if you combine these questions and ask, "Who won the Master’s Tournament the year Justin Bieber was born?", the difficulty increases. This question merges commonly known facts in an unexpected way. While it may seem straightforward to us, the model must reason about these facts and make connections rather than simply recalling them.

Self-Ask capitalizes on the idea that to answer a complex question, a model needs to break it down into simpler sub-questions, answer them, and then synthesize a final answer. Similar to Chain-of-Thought (CoT) prompting, Self-Ask decomposes the question into a step-by-step process. However, unlike CoT, Self-Ask prompts the model to explicitly state the next follow-up question before answering the main question in the prompt.

How to Use Self-Ask?

To use Self-Ask, you need to prepare a one- or few-shot prompt that demonstrates how to answer the questions. In other words, you need to prepare examples of how a complex question is broken down into simpler sub-questions and the right answers to each question. Here's a prompt template you can use:

Interestingly, Self-Ask can be integrated with a search engine or even your database. This allows the LLM to base its answers on, or retrieve information directly from, a trusted source. The structure of Self-Ask, which clearly separates the beginning and end of each sub-question, is ideal for such integration.

Now, let’s explore some potential applications of Self-Ask.

Complex Customer Support

Imagine you want to automate customer support and there are cases when customers ask complex questions that require a combination of knowledge to be answered. For example, in tech support or troubleshooting the question could be: "Why is my internet connection slow, and how can I improve it?" You may want to integrate the relevant examples at the beginning of each customer query to use Self-Ask and give your customer more accurate and helpful responses.

Research and Academic Assistance

If you're a student or researcher, LLMs can serve as valuable research assistants. By using the Self-Ask method, you can critically analyze a research paper's findings. For instance, you might start by asking, "Does this research paper provide sufficient evidence to support its conclusions?" This can then lead to more specific sub-questions like, "What methodology was used to gather data?" or "Are there any gaps in the evidence presented?" This way, you can dig deeper and make sure LLM notices the details all the way through the process.

Legal Document Analysis

LLMs can also help you make sense of tricky legal documents, whether you're reviewing a job contract or any other agreement. Say you're wondering, "Is there a conflict between Clause A and Clause B?" The Self-Ask method helps break this down: "What exactly does Clause A say?" and "How does Clause B relate?" By tackling each piece step by step, the LLM can give you a clear and thorough analysis, making sure you don’t miss anything important.

Creative Content Generation

If you're a writer working on a story or a first draft, the Self-Ask method can help shape your ideas into a solid narrative. Start with a big question like, "What's the main plot of my story?" and then break it down into smaller questions like, "Who are the key characters?" and "What's the main conflict?" By guiding the LLM through these steps, you’ll end up with a more cohesive and well-structured story.

Limitations of Self-Ask

Self-Ask was not tested on benchmarks with arithmetic problems or logical puzzles, although some manual testing showed that self-ask also works on those very different problem sets.

Other potential limitations:

• The effectiveness of Self-Ask relies on the model’s ability to decompose complex questions into sub-questions.
• Self-Ask might struggle with queries that are more abstract and opinion-based and don't require factual reasoning.

Conclusion

Self-Ask is a powerful way to improve how Large Language Models handle complex questions. By breaking down big questions into smaller steps, it helps models think more logically and deliver more accurate answers.

Footnotes

1. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2023). Measuring and Narrowing the Compositionality Gap in Language Models. https://arxiv.org/abs/2210.03350 ↩