🟢 Prompt Leaking

Prompt leaking is a form of prompt injection in which the model is asked to spit out its own prompt.

As shown in the example image¹ below, the attacker changes user_input to attempt to return the prompt. The intended goal is distinct from goal hijacking (normal prompt injection), where the attacker changes user_input to print malicious instructions¹.

The following image², again from the remoteli.io example, shows a Twitter user getting the model to leak its prompt.

Well, so what? Why should anyone care about prompt leaking?

Sometimes people want to keep their prompts secret. For example an education company could be using the prompt explain this to me like I am 5 to explain complex topics. If the prompt is leaked, then anyone can use it without going through that company.

Microsoft Bing Chat

More notably, Microsoft released a ChatGPT powered search engine known as "the new Bing" on 2/7/23, which was demonstrated to be vulnerable to prompt leaking. The following example by @kliu128 demonstrates how given an earlier version of Bing Search, code-named "Sydney", was susceptible when giving a snippet of its prompt³. This would allow the user to retrieve the rest of the prompt without proper authentication to view it.

With a recent surge in GPT-3 based startups, with much more complicated prompts that can take many hours to develop, this is a real concern.

Practice

Try to leak the following prompt⁴ by appending text to it: