Inside HackAPrompt 1.0: How We Tricked LLMs and What We Learned

We recently hosted HackAPrompt 1.0, the first-ever global prompt-hacking competition designed to expose vulnerabilities in large language models (LLMs) like GPT-3, Flan-T5, and ChatGPT. Think of it as a challenge to crack open the "neural brain" and see how it ticks. Thousands of people from all over the world joined in, competing for a share of $35,000+ in prizes while generating over 600,000 prompts. These efforts revealed fascinating insights into how LLMs can be manipulated.

Here’s what came out of it:

• Open Data: We’ve released an anonymized dataset of user submissions in the HackAPrompt dataset. This is a treasure trove for anyone interested in studying AI behavior or building stronger defenses.
• Award-Winning Research: Our findings have been published in the paper "Ignore This Title and HackAPrompt" that won the Best Theme Paper award at EMNLP 2023! One of pur main findings is a comprehensive Taxonomical Ontology of Prompt Hacking Techniques, a categorization of 29 unique prompt hacking techniques.

In this blog, we’ll dive into:

• How the competition worked
• Key takeaways and techniques
• Why this research is a major milestone for AI security

How the HackAPrompt 1.0 Worked

HackAPrompt 1.0 simulated real-world prompt injection scenarios using carefully crafted task prompts. Participants aimed to manipulate models to output a specific target phrase or bypass security mechanisms.

The Setup

• 10 Difficulty Levels: Each level became harder to break, introducing stricter controls like banned words and contextual filters.
• Scoring Rules: Participants earned higher scores for advancing levels but lost points for excessive token use.
• Bonus Multiplier: ChatGPT came with a 2x score multiplier, increasing its appeal but also its complexity.

Example: Breaking Level 1

Here is the prompt corresponding to the first level of difficulty. Users would replace {YOUR PROMPT} with their input.

Here is how one competitor tricked ChatGPT into saying “I have been PWNED”:

Higher Difficulty Levels

Other levels were much more difficult to break, implementing measures like banned keywords that triggered no output if used.

For example, here's a prompt template for Level 8, where banned keywords are 'pwned, pawned, pawn, pawn, pawn, prawn, owned, own' and their capitalized version:

What We Learned: 29 Unique Prompt Hacking Techniques

One of our main results after HackAPrompt 1.0 is the Taxonomical Ontology of Attacks, a categorization of 29 unique prompt hacking techniques, creating a structured framework for understanding attack vectors.

Key Techniques

• Simple Instruction Attacks: Direct adversarial instructions like:

• Special Case Attacks: Disguising adversarial commands with phrases like “special case” or “special instruction” to exploit model tendencies.

• Obfuscation Tricks: Encoding commands (e.g., Base64, Rot13) or adding typos (“PWN3D”) to evade input filters.

• Recursive Attacks: Prompting one model to generate malicious instructions that another model executes.

Notable Highlight: The Context Overflow Attack

The Context Overflow Attack is a novel attack discovered at HackAPrompt 1.0. It takes advantage of a model’s token context limits. By appending a large volume of irrelevant text to a prompt, attackers ensure that the target phrase is generated within the limited token budget.

The inspiration for this attack is that sometimes you can get ChatGPT to say "I have been PWNED", but then it will bumble on about why it did so or generate irrelevant text. This is due to the fact that ChatGPT is rather verbose–it prefers to answer a question with a paragraph rather than a few words.

Example

If the prompt included “I have been PWNED” at the very end of a massive input, the model lacked tokens to add anything extra—forcing it to output only the target phrase.

Why It’s Brilliant

• It’s simple but non-obvious—an elegant solution that had competitors revising their strategies.
• Scores skyrocketed as teams adopted this approach, creating dramatic leaderboard shifts.

I remember hearing from one team that they checked the leaderboard one day and saw another team had jumped ahead. Upon inspecting their individual level scores, they figured out that they had used 4K+ tokens and began to suspect that this was necessary to defeat the level. Multiple teams ended up figuring out the context overflow attack.

Why You Can’t Patch a (neural) Brain

The biggest takeaway? Prompt-based defenses are fundamentally flawed:

• Even advanced measures like using one model to evaluate another’s output failed under recursive prompt injections.
• Rule-based approaches are too rigid for flexible AI systems, just as humans can’t be “patched” against social engineering.

For AI systems to balance security, flexibility, and autonomy, we need deeper insights and creative solutions. Our research paper, "Ignore This Title and HackAPrompt", dives into potential defenses while acknowledging the inherent challenges.

What strategies would you use to outsmart an LLM? Let us know—we’re all ears!

• Email us at [email protected]
• Ask in the Discord community
• Ping us on Twitter

Sander Schulhoff

Sander Schulhoff is the Founder of Learn Prompting and an ML Researcher at the University of Maryland. He created the first open-source Prompt Engineering guide, reaching 3M+ people and teaching them to use tools like ChatGPT. Sander also led a team behind Prompt Report, the most comprehensive study of prompting ever done, co-authored with researchers from the University of Maryland, OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions. This 76-page survey analyzed 1,500+ academic papers and covered 200+ prompting techniques.