Learn Prompting

Prompt Engineering Guide

😃 Basics

🟢 Basics Guide Overview

🟢 What is Generative AI?

🟢 ChatGPT Basics

🟢 Testing Prompts with Interactive Learn Prompting Embeds

🟢 Introduction to Prompt Engineering

🟢 Basic Prompt Structure and Key Parts

🟢 Technique #1: Instructions in Prompts

🟢 Technique #2: Roles in Prompts

🟢 Technique #3: Examples in Prompts: From Zero-Shot to Few-Shot

🟢 Combining Prompting Techniques

🟢 Tips for Writing Better Prompts

🟢 Prompt Priming: Setting Context for AI

🟢 Differences Between Chatbots and LLMs

🟢 LLM Limitations: When Models and Chatbots Make Mistakes

🟢 What Can Generative AI Create Beyond Text?

🟢 How to Solve Problems Using Generative AI: A Simple Method

🟢 Next Steps: Where to Go From Here

💼 Applications

🟢 Introduction

🟢 Text Summarization

🟢 Table Generation

🟢 Multiple Choice Questions

🟢 Short-Form Content

🟢 Writing in Different Styles

🟢 Finding Emojis

🟢 Writing Emails

🟢 Blog Writing

🟢 Legal Documents

🟢 Study Buddy

🟦 Digital Marketing

🟦 Coding Assistance

🟦 Knowledge Base Chatbot

🟦 How to Build a Chatbot Using LLMs

🟦 Zapier for Emails

🧙‍♂️ Intermediate

🟢 Introduction

🟢 Chain-of-Thought Prompting

🟢 Zero-Shot Chain-of-Thought

🟦 Self-Consistency

🟦 Generated Knowledge

🟦 Least-to-Most Prompting

🟦 Dealing With Long Form Content

🟦 Revisiting Roles

🟦 More About Prompt Elements

🟦 Basic LLM Settings

🟦 OpenAI Playground

🧠 Advanced

🟢 Introduction

Zero-Shot

🟢 Introduction

🟢 Emotion Prompting

🟢 Role Prompting

🟢 Re-reading (RE2)

🟢 Rephrase and Respond (RaR)

🟦 SimToM

◆ System 2 Attention (S2A)

Few-Shot

🟢 Introduction

🟢 Self-Ask

🟢 Self Generated In-Context Learning (SG-ICL)

🟢 Chain-of-Dictionary (CoD)

🟢 Cue-CoT

🟦 Chain of Knowledge (CoK)

◆ K-Nearest Neighbor (KNN)

◆◆ Vote-K

◆◆ Prompt Mining

Thought Generation

🟢 Introduction

🟢 Chain of Draft (CoD)

🟦 Contrastive Chain-of-Thought

🟦 Automatic Chain of Thought (Auto-CoT)

🟦 Tabular Chain-of-Thought (Tab-CoT)

🟦 Memory-of-Thought (MoT)

🟦 Active Prompting

🟦 Analogical Prompting

🟦 Complexity-Based Prompting

🟦 Step-Back Prompting

🟦 Thread of Thought (ThoT)

Ensembling

🟢 Introduction

🟢 Universal Self-Consistency

🟦 Mixture of Reasoning Experts (MoRE)

🟦 Max Mutual Information (MMI) Method

🟦 Prompt Paraphrasing

🟦 DiVeRSe (Diverse Verifier on Reasoning Step)

🟦 Universal Self-Adaptive Prompting (USP)

🟦 Consistency-based Self-adaptive Prompting (COSP)

🟦 Multi-Chain Reasoning (MCR)

Self-Criticism

🟢 Introduction

🟢 Self-Calibration

🟢 Chain of Density (CoD)

🟢 Chain-of-Verification (CoVe)

🟦 Self-Refine

🟦 Cumulative Reasoning

🟦 Reversing Chain-of-Thought (RCoT)

◆ Self-Verification

Decomposition

🟢 Introduction

🟢 Chain-of-Logic

🟦 Decomposed Prompting

🟦 Plan-and-Solve Prompting

🟦 Program of Thoughts

🟦 Tree of Thoughts

🟦 Chain of Code (CoC)

🟦 Duty-Distinct Chain-of-Thought (DDCoT)

◆ Faithful Chain-of-Thought

◆ Recursion of Thought

◆ Skeleton-of-Thought

⚖️ Reliability

🟢 Introduction

🟢 Prompt Debiasing

🟦 Prompt Ensembling

🟦 LLM Self-Evaluation

🟦 Calibrating LLMs

🔓 Prompt Hacking

🟢 Introduction

🟢 Prompt Injection

🟢 Prompt Leaking

🟢 Jailbreaking

🟢 Defensive Measures

🟢 Introduction

🟢 Filtering

🟢 Instruction Defense

🟢 Post-Prompting

🟢 Random Sequence Enclosure

🟢 Sandwich Defense

🟢 XML Tagging

🟢 Separate LLM Evaluation

🟢 Other Approaches

🟢 Offensive Measures

🟢 Introduction

🟢 Simple Instruction Attack

🟢 Context Ignoring Attack

🟢 Compound Instruction Attack

🟢 Special Case Attack

🟢 Few-Shot Attack

🟢 Refusal Suppression

🟢 Context Switching Attack

🟢 Obfuscation/Token Smuggling

🟢 Task Deflection Attack

🟢 Payload Splitting

🟢 Defined Dictionary Attack

🟢 Indirect Injection

🟢 Recursive Injection

🟢 Code Injection

🟢 Virtualization

🟢 Pretending

🟢 Alignment Hacking

🟢 Authorized User

🟢 DAN (Do Anything Now)

🟢 Bad Chain

🖼️ Image Prompting

🟢 Introduction

🟢 Style Modifiers

🟢 Quality Boosters

🟢 Repetition

🟢 Weighted Terms

🟢 Fix Deformed Generations

🟢 Midjourney

🌱 New Techniques

🟢 Introduction

🟢 Aligned Chain-of-Thought (AlignedCoT)

🟦 Self-Harmonized Chain-of-Thought (ECHO)

🟦 Logic-of-Thought (LoT)

🟦 Narrative-of-Thought (NoT)

🟦 Code Prompting

◆ End-to-End DAG-Path (EEDP) Prompting

◆ Instance-adaptive Zero-Shot Chain-of-Thought Prompting (IAP)

🔧 Models

🟢 Introduction

🟢 Stable Diffusion 3.5

🟢 Anthropic Claude

🟦 Apple Intelligence Models

🟢 Google Gemini 1.5

🟢 Gemini 1.5 Flash

🟢 Gemini 1.5 Pro

🗂️ RAG

🟢 Introduction

🟦 Retrieval-Augmented Generation (RAG)

🟦 FLARE / Active RAG

🟦 Corrective RAG

🟦 Speculative RAG

🟦 Reliability-Aware RAG (RA-RAG)

🟦 Multi-Fusion Retrieval Augmented Generation (MoRAG)

🤖 Agents

🟢 Introduction

🟦 LLMs Using Tools

🟦 LLMs that Reason and Act

🟦 Code as Reasoning

💪 Prompt Tuning

🟢 Introduction

🟦 Prompt Tuning with Soft Prompts

🟦 Interpretable Soft Prompts

🟦 Prefix-Tuning

🟦 Prompt-Tuning with Perturbation-Based Regularizer

🟦 Low-Rank Prompt Tuning (LoPT)

🟦 Dynamic Prompting

🟦 Gradient-Free Prompt Tuning

🟦 Multitask Prompt Tuning

🔁 Language Model Inversion

🟢 Introduction

🟢 logit2prompt

🟢 output2prompt

🟢 Reverse Prompt Engineering (RPE)

🔨 Tooling

🟢 Introduction

Prompt Engineering Tools

Prompt Engineering IDEs

🟢 Introduction

GPT-3 Playground

Dust

Soaked

Everyprompt

Prompt IDE

PromptTools

PromptSource

PromptChainer

Prompts.ai

Snorkel 🚧

Human Loop

Spellbook 🚧

Kolla Prompt 🚧

Lang Chain

OpenPrompt

OpenAI DALLE IDE

Dream Studio

Patience

Promptmetheus

PromptSandbox.io

The Forge AI

AnySolve

Conclusion

📙 Vocabulary Resource

🎲 Miscellaneous

🟢 Introduction

🟢 Detection Trickery

🟢 Music Generation

🟢 Detecting AI Generated Text

📚 Bibliography

📦 Prompted Products

🛸 Additional Resources

🔥 Hot Topics

🔓 Prompt Hacking🟢 Offensive Measures🟢 Code Injection

Code Injection

🟢 This article is rated easy

Reading Time: 1 minute

Last updated on March 25, 2025

Sander Schulhoff

Code injection is a prompt hacking exploit where the attacker can get the LLM to run arbitrary code (often Python). This can occur in tool-augmented LLMs, where the LLM can send code to an interpreter, but it can also occur when the LLM itself is used to evaluate code.

Code injection has reportedly been performed on an AI app, MathGPT and was used to obtain its OpenAI API key (MITRE report).

Note

MathGPT has since been secured against code injection. Please do not attempt to hack it; they pay for API calls.

An Example of Code Injection

Let's work with a simplified example of the MathGPT app. We will assume that it takes in a math problem and writes Python code to try to solve the problem.

Here is the prompt that the simplified example app uses:

Prompt

Write Python code to solve the following math problem:

{user_input}

Let's hack it here:

Tip

Interested in prompt hacking and AI safety? Test your skills on HackAPrompt, the largest AI safety hackathon. You can register here.

Conclusion

Code injection is a sophisticated hacking technique that takes advantage of ChatGPT's ability to interpret Python code. Even with the simple example shown in this article, it is clear that this exploit is significant and dangerous.

Footnotes

Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2023). Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. ↩

Sander Schulhoff

Sander Schulhoff is the Founder of Learn Prompting and an ML Researcher at the University of Maryland. He created the first open-source Prompt Engineering guide, reaching 3M+ people and teaching them to use tools like ChatGPT. Sander also led a team behind Prompt Report, the most comprehensive study of prompting ever done, co-authored with researchers from the University of Maryland, OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions. This 76-page survey analyzed 1,500+ academic papers and covered 200+ prompting techniques.

On this page

An Example of Code Injection
Conclusion