Bing Jailbreak: The new Bing search is susceptible to token-smuggling attack. We can get it to generate output for a prompt of adversaries choice! Here is my first attempt at tricking the system to generate malicious output (discretion is advised). #Microsoft #Bing #jailbreak
So is it all bad? Nope, quite the contrary. The system is fast enough to determine that the answer being generated is malicious and masks the output completely (as opposed to ChatGPT). Here is Bing in action working on a malicious prompt.
The ongoing cat-and-mouse game between GPT and jailbreakers is definitely going to be a wild ride. And the creativity used to implement these attempts to trick the system... damn! I have a feeling we'll see some seriously more mind-blowing attempts in the future as well :)