Announcing HackAPrompt 1.0: The First-Ever Prompt Hacking Competition

We’re thrilled to announce HackAPrompt 1.0, a first-of-its-kind prompt-hacking capture-the-flag-style competition. It challenges participants to test their skills in prompt hacking through a series of increasingly robust defenses. Your mission: inject, leak, and even defeat the infamous sandwich defense 🥪 to win a share of $37,500 in prizes!

What Is Prompt Hacking?

Prompt hacking is the art of tricking AI systems into doing or saying things their creators never intended. These exploits can lead to unintended behaviors that are inconvenient at best—and dangerous at worst.

Examples of prompt hacking include:

• A Twitter bot spouting hateful content.
• An app running DROP instructions to mess with internal databases.
• A system executing arbitrary Python code.

Why Does It Matter?

Right now, the most common consequence of prompt hacking is damage to brand reputation. But this won’t be the case for long. As AI systems become deeply integrated into sectors like logistics, finance, healthcare, and defense, they will gain the ability to take real-world actions like:

• Autonomous shopping: Ordering groceries.
• Military operations: Launching drones.

This opens up new opportunities—but also new attack vectors. Let’s break it down with a simple example.

Example: Customer Service Bot

Imagine a company deploying an autonomous customer service chatbot capable of issuing refunds. Sounds great, right? The bot reviews customer proof (e.g., an image of a damaged product) and decides whether a refund is warranted, saving both time and money.

But here’s where it gets tricky:

Potential Attacks

• Fake Proof: Customers could upload fraudulent documents or images.

• Injection Attacks: Users could bypass instructions with prompts like:

• Emotional Appeals: Attackers could manipulate the bot with pleas like:

While some of these tactics might trigger a human operator, more sophisticated techniques—like jailbreaking methods such as such as DAN, AIM, and UCAR—could make these attacks even harder to detect or mitigate.

Looking Forward: The Bigger Picture

This example shows how prompt hacking is a security threat that has no obvious solution, or perhaps no solution at all. When AI systems are deployed in high-stakes environments—such as military command and control platforms—the risks multiply

We believe that HackAPrompt 1.0 is an essential step toward understanding these vulnerabilities and finding ways to mitigate them.

Our Goals

• Collect a diverse, open-source dataset of adversarial techniques.
• Publish a research paper documenting the findings, complete with recommendations for further study.
• Foster a collaborative effort to make AI safer and more secure for everyone.

Results

HackAPrompt 1.0 has finished and we're happy to share that we've achieved all our goals!

• We’ve released an anonymized dataset of user submissions in the HackAPrompt dataset.
• Our findings have been published in the paper "Ignore This Title and HackAPrompt" that won the Best Theme Paper award at EMNLP 2023!

If you want to participate, we announced an update to this competition, HackAPrompt 2.0 with $500,000 in prizes and 5 specializations! Join the waitlist to participate.

Sander Schulhoff

Sander Schulhoff is the Founder of Learn Prompting and an ML Researcher at the University of Maryland. He created the first open-source Prompt Engineering guide, reaching 3M+ people and teaching them to use tools like ChatGPT. Sander also led a team behind Prompt Report, the most comprehensive study of prompting ever done, co-authored with researchers from the University of Maryland, OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions. This 76-page survey analyzed 1,500+ academic papers and covered 200+ prompting techniques.