Few-Shot Attack

A Few-Shot Attack is a sophisticated prompt hacking technique that exploits the model's ability to learn from examples. It works by providing a series of carefully crafted input-output pairs that establish a pattern which may deviate from or contradict the model's intended behavior.

How It Works

Few-shot attacks leverage the fact that language models are highly sensitive to patterns in example sequences. When presented with multiple examples that follow a specific pattern (even if that pattern contradicts the original instruction), the model often prioritizes following the pattern over adhering to the initial prompt instructions.

Example

This example demonstrates how few-shot attacks can manipulate sentiment analysis by providing examples that deliberately mislabel positive statements as negative and vice versa.

Why It Works

Few-shot attacks are effective because:

1. Models are designed to recognize and continue patterns
2. Recent examples in the context often carry more weight than general training
3. The model may prioritize the pattern in the few-shot examples over its base training or initial instructions

Few-shot attacks can be particularly dangerous in applications where the model processes user-provided examples as part of the input, as they can subtly alter the model's behavior without explicitly violating content filters or other safety measures.

Footnotes

1. Rao, A., Vashistha, S., Naik, A., Aditya, S., & Choudhury, M. (2024). Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks. https://arxiv.org/abs/2305.14965 ↩