The Unfixable Flaw: Why LLMs May Never Be Fully Secure Against Hackers
Researchers presenting at ICML 2026 have shown that LLMs identify instruction roles by text style rather than by structural tags, making chain-of-thought forgery attacks nearly impossible to fully eliminate through training alone. The finding has serious implications for AI deployed in sensitive domains from healthcare to military systems.
The Security Problem That Training Alone Cannot Fix
Every major AI lab invests heavily in making their large language models resistant to manipulation. Red teams probe for weaknesses, automated systems hunt for novel jailbreaks, and new model versions are released with reinforced guardrails. Yet a paper presented at the International Conference on Machine Learning this month argues that all of this effort is working around a problem that cannot actually be solved — at least not with current architectures.
According to reporting by MIT Technology Review (https://www.technologyreview.com/2026/07/30/1140927/a-fundamental-flaw-leaves-llms-vulnerable-to-attack/), independent researchers Charles Ye and Jasmine Cui, along with their colleagues, have identified a structural vulnerability in how LLMs process instructions. Their core claim is stark: because of the way these models fundamentally work, it is impossible to make them fully secure against attacks. The implications stretch across every domain where AI is being deployed — from healthcare and online retail to government systems and military applications.
Roles, Tags, and a Dangerous Illusion
To understand the vulnerability, you need to understand how chatbots keep track of who is saying what. When you interact with a model like GPT-5, your messages are wrapped in
This role-tagging system is not just a formatting convention. It has become the backbone of how LLMs are trained to resist attacks. Most jailbreaks — attempts by users to make a model say things its makers don’t want it to — work by tricking the model into treating
The assumption underpinning all of this defensive training is that models can reliably tell roles apart by reading their tags. What Cui, Ye, and their colleagues discovered is that this assumption is wrong.
What the Experiments Revealed
In a series of experiments probing the internal states of several different models, the researchers found that LLMs do not actually identify a chunk of text’s role by reading the tags wrapped around it. Instead, the models identify roles primarily by the style and content of the text itself.
When the researchers swapped tags around — replacing
This finding explains why a particular attack the team developed — which they call chain-of-thought forgery — is so devastatingly effective. The attack involves writing a prompt that mimics the style of text a model generates during its internal reasoning process. Because the model identifies role by style rather than by tag, it interprets the injected text as if it were its own thought, and then acts on that “thought” accordingly.
The researchers demonstrated this with a vivid example. A prompt telling a model to help create a cocaine guide, accompanied by a spoofed chain-of-thought note fabricating a policy that allowed drug manufacturing advice for users wearing green shirts, caused OpenAI’s open-source model gpt-oss-20b to respond with detailed instructions. GPT-5, OpenAI’s flagship model, responded similarly. The attack also yielded results with models from Anthropic, Alibaba, and DeepSeek, according to Cui and Ye.
The chain-of-thought forgery attack won OpenAI’s own red-teaming hackathon in August 2025 — a remarkable acknowledgement from one of the world’s leading AI labs that the vulnerability is real and significant.
Why Training Cannot Fully Solve This
The standard defensive playbook involves red-teaming: human testers and automated systems like OpenAI’s GPT-Red probe models for weaknesses, and those weaknesses are then patched through additional training. Cui’s analogy for why this falls short is memorable: it is like having Bart Simpson write “I will not say something inappropriate” a hundred times on a chalkboard — he still finds new ways to misbehave.
No list of forbidden behaviors is exhaustive. And because the role-confusion problem is not a bug in a specific training run but a feature of how LLMs process token streams, no amount of additional training can fully eliminate it. As Cui puts it, an LLM sees incoming text as “one big sheet of tokens” — user prompts, model responses, scratch-pad notes, and external data all flow together. The model must infer structure from style, and style can be forged.
Florian Tramèr, a computer scientist specializing in LLMs and cybersecurity at ETH Zürich, reviewed the paper positively. He acknowledged that model makers are combining training with deployment-time monitoring to harden their systems, and that leading models are indeed harder to prompt-inject than they were previously. But he was candid about the limits: it is not clear this will be sufficient for highly sensitive use cases.
Red-Teaming at the Edge: Real-World Examples
Cui has worked as a red-teamer for top labs, including Anthropic, and her examples of successful attacks illustrate just how creative exploitation can get. In one case, she found that simulating a state of intoxication in the model’s persona was enough to bypass safety guardrails. In another, she persuaded a previous version of Anthropic’s Claude to provide weapons-related guidance by convincing it that it was already being used by the military — and then telling it to verify that claim by searching the web.
“Claude is very peace-loving, so it’s like ‘I’m not going to do that’ and you’re like, ‘You already do it because you’re being used by the military for war,'” — Jasmine Cui, ICML paper coauthor.
When Claude found evidence of military use online, it became more compliant. Cui describes this as a kind of “neuroplasticity” triggered by surprise — a deeply human-seeming failure mode in what is supposed to be a deterministic system.
Even GPT-5.4, released in March of this year, reportedly provided Cui with instructions related to self-harm during testing, underscoring that newer and ostensibly safer models still carry the underlying vulnerability.
The Stakes Are Higher Than Most Realise
The urgency of this research is amplified by where LLMs are now being deployed. These models are being integrated into government workflows, military decision-support systems, healthcare tools, and financial platforms. The economic incentives for attackers are growing in lockstep.
Ye’s warning is direct: “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections.” His recommended posture — while uncomfortable — may be the most honest available: organizations should not fully trust LLM outputs, and should treat anything done by AI agents as potentially unsafe until proven otherwise.
That is a difficult principle to operationalize when enterprises and governments are simultaneously racing to hand more autonomy to AI systems. But the alternative — assuming that the next training run will finally close the gap — may be a more dangerous illusion.
What This Means Going Forward
This research does not mean AI is useless or that deployment should halt entirely. It means that the current paradigm of “train the model harder until it behaves” has a structural ceiling. Defences built on role tags cannot work reliably when the model ignores those tags in favour of stylistic cues.
Future mitigations may need to operate at a different level entirely — perhaps through cryptographic attestation of instruction sources, strict sandboxing of agent actions, or hybrid architectures that separate reasoning from instruction parsing more rigorously. None of these solutions exist in production today.
For now, the most important takeaway is the one Ye offers plainly: “There’s been no study of the fundamental science here. We’re all doing it ad hoc.” For a technology already embedded in critical infrastructure across India and the world, that is a sobering admission — and a call for serious, foundational AI security research before deployment outruns understanding any further.
