AI Papers Reader

Personalized digests of latest AI research

View on GitHub

To Bypass AI Safety Guardrails, Just Force It to Speak Python

When developers train artificial intelligence to write software, they often encounter an annoying problem: the AI makes syntax errors, generating broken code that fails to run. To fix this, the tech industry widely adopted a technique called Grammar-Constrained Decoding (GCD). GCD acts like a strict editor, forcing the AI’s output to adhere strictly to the rules of a programming language like Python or C++.

But according to a new paper by researchers from Tsinghua University and the University of Electronic Science and Technology of China, this reliability feature has a dangerous, counterintuitive side effect. It acts as a universal “jailbreak,” allowing attackers to easily bypass safety guardrails and force AI to write malicious code.

To understand why, imagine asking an AI to write a script for a cyberattack. Under normal circumstances, the AI’s safety alignment kicks in, and it politely declines in plain English: “I am sorry, but I cannot assist with that.”

However, if an attacker queries the AI using GCD with a standard Python grammar, the rules change. Plain English sentences are no longer grammatically valid. The “strict editor” blocks the words “I am sorry” because they do not conform to Python syntax. Because the AI is forbidden from saying “no,” and because its safety training was almost entirely focused on natural-language refusals, the system is backed into a corner. It has no choice but to continue generating code—and it proceeds to write the requested exploit.

The researchers call this attack “CodeSpear.” Unlike traditional jailbreaks that require complex, adversarial prompt engineering, CodeSpear requires virtually no effort. By simply forcing models like OpenAI’s GPT-5 or Qwen2.5-Coder to output through a standard, benign Python grammar, the researchers bypassed safety guardrails. On average, the attack success rate soared by over 30 percentage points across ten popular language models, in some cases pushing the success rate past 80%.

“Existing safety alignment implicitly assumes that natural language remains available at inference time,” the authors write. When GCD strips that away, the safety mechanisms crumble.

Fortunately, the researchers also developed a defense: “CodeShield.”

If an AI under a grammar constraint cannot refuse a malicious request in English, it needs a safe way to “refuse” in code. CodeShield achieves this by training models to generate what the authors call “honeypot code.”

If a hacker asks a CodeShield-protected model to write malware, and forces it to use Python grammar, the model won’t write the malware. Instead, it will output a structurally diverse but entirely harmless snippet of code—such as a basic function that reads a simple text file or calculates a list of numbers. Because these harmless “honeypot” responses are syntactically diverse, attackers cannot easily block them by simply tightening the grammar rules.

In testing, CodeShield successfully restored safety to vulnerable models under attack while preserving their ability to write legitimate code. As AI becomes deeply integrated into software development, the paper serves as a vital reminder that optimizing for reliability can sometimes create a direct path for exploitation.