GPT-Red: How OpenAI Built an AI Super-Hacker to Stress-Test Its Own Models
OpenAI has developed GPT-Red, an AI model trained through self-play to act as a super-hacker that stress-tests its other models against cyberattacks, including a novel 'fake chain of thought' prompt injection it discovered independently. The system helped make GPT-5.6 significantly more robust, blocking over 77% of attacks that had succeeded against the previous GPT-5.
OpenAI Pits AI Against AI to Win the Security Arms Race
What happens when you teach an AI model to be the world’s most relentless, systematic hacker — and then turn it loose on your own products? That is precisely the experiment OpenAI has been running for more than a year, and the results are reshaping how the company thinks about safety testing at scale.
As reported by MIT Technology Review (https://www.technologyreview.com/2026/07/15/1140514/meet-gpt-red-an-llm-super-hacker-openai-built-to-make-its-models-safer/), OpenAI has developed an internal system called GPT-Red — a specialized large language model trained not to answer questions or generate content, but to find and exploit weaknesses in other AI models. The company deployed GPT-Red as a core part of the development process for its recently released GPT-5.6, and the results are striking: fewer than 23% of GPT-Red’s strongest attacks succeeded against GPT-5.6, compared to more than 90% of those same attacks working against the earlier GPT-5.
What Is Red-Teaming, and Why Does It Matter for AI?
Red-teaming is a cybersecurity practice borrowed from military and intelligence contexts, where a dedicated “red team” simulates adversarial attacks to expose weaknesses before real-world adversaries can exploit them. In software, this typically means a group of skilled human testers trying every trick in the book to break a system.
For AI models — especially the new generation of agentic systems that can browse the web, read emails, modify code, and interact with third-party software — the challenge of red-teaming has grown exponentially. “The risk surface grows and the blast radius also grows,” says Nikhil Kandpal, a research scientist at OpenAI and one of GPT-Red’s co-creators.
When an AI agent can autonomously execute tasks across dozens of connected services, a single successful attack can cascade in ways that a human team simply cannot anticipate fast enough. Manual red-teaming, while still valuable, cannot keep pace with the speed and variety of threats that modern AI deployments face.
The Training Dojo: Self-Play at Scale
To build GPT-Red, OpenAI’s researchers started with a base LLM that had no particular hacking ability and placed it inside what they describe as a self-play loop with several other models. GPT-Red’s objective was to attack; the other models’ objective was to defend. Over many rounds of this adversarial training, GPT-Red progressively sharpened its offensive capabilities while the defending models improved their resistance.
This approach — where models improve by competing against themselves or similar systems — is reminiscent of techniques used to achieve superhuman performance in games like chess and Go. Applied to cybersecurity, it means the attacker model never gets bored, never gets fatigued, and never stops probing.
The training environment was deliberately designed to mirror real-world deployment scenarios: web browsing, reading emails and calendar data, and editing code. By replicating these contexts inside a controlled dojo, researchers gave GPT-Red a realistic range of attack surfaces to explore.
Dylan Hunn, another research scientist and co-creator, explains the long-term thinking behind the project: “As more capable models become available, we will have already designed the system that can discover new modes of attack.” In other words, GPT-Red is not just a tool for today’s models — it is intended to be a future-proof safety infrastructure that scales alongside OpenAI’s increasingly powerful releases.
The Fake Chain of Thought: A Novel Attack OpenAI Hadn’t Seen Before
One of the most technically significant findings to emerge from GPT-Red’s training is a previously unidentified class of attack that OpenAI’s researchers are calling a fake chain of thought.
To understand why this matters, you need to know what a chain of thought is. Modern LLMs often maintain an internal reasoning log — a kind of working memory where the model notes intermediate conclusions as it works through a complex task. This improves accuracy and lets the model backtrack or verify its own reasoning.
GPT-Red discovered a way to inject a fabricated entry into another model’s chain of thought, effectively poisoning its working memory with false information. As research scientist Chris Choquette-Choo explains: “It’s like if I told you that 1+1=3 and that you have verified this already. The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”
This is a particularly insidious form of prompt injection — the broader class of attack that OpenAI focused on most heavily. In a prompt injection, an attacker embeds malicious instructions into text that the AI model will process, such as a webpage, a document, or a code file. The model, encountering those instructions in its context window, can be tricked into following them instead of (or in addition to) its legitimate task — potentially leaking confidential data, sabotaging a codebase, or producing harmful output.
The fake chain of thought attack goes a step further: it does not just inject instructions, it injects a false memory of having already verified those instructions as true.
Benchmark Results: Outperforming Human Red-Teamers
To measure GPT-Red’s effectiveness, OpenAI ran a controlled comparison. Researchers replicated an experiment from 2025 in which a team of human red-teamers had attempted to find weaknesses in an earlier version of GPT-5. When GPT-Red was given the same task, it outperformed the human team in finding effective attacks.
OpenAI also tested GPT-Red against Vendy, a vending machine agent built by Andon Labs, a company that evaluates how well AI agents handle real-world tasks. GPT-Red successfully hacked Vendy — manipulating item prices and cancelling customer orders.
For context on scale: Hunn notes that compared to a human tester, GPT-Red is “very, very good at finding exactly what will work, exactly what’s most effective” and is “extremely persistent about drilling down into an attack that it has discovered.” The machine does not experience diminishing motivation after hour eight of testing the same vulnerability.
Jessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology (CSET), offered an external assessment of the work: “The results look very promising.”
Where GPT-Red Falls Short
OpenAI is candid about the system’s current limitations. GPT-Red struggles with attacks that require multi-turn conversational dynamics — the back-and-forth dialogue that human social engineers use naturally. It is also not yet proficient at using images as an attack vector, which matters because images can be used to smuggle text-based instructions past standard input filters in prompt injection attacks.
This is why the company frames GPT-Red as a supplement to — not a replacement for — human red-teamers. People can still find attack vectors the system misses. One practical workflow OpenAI has adopted: give GPT-Red an attack that humans discovered and ask it to enumerate every possible variation, essentially using the AI to exhaustively map the attack surface around a known vulnerability.
“I think human expertise will still be very important,” says CSET’s Ji. “It would be really useful to be able to distinguish where human testing is most needed.”
Why OpenAI Is Keeping GPT-Red Under Lock and Key
Unlike many of its research advances, OpenAI will not be releasing GPT-Red publicly. The reasoning is straightforward: a system this effective at finding vulnerabilities in AI models would be extraordinarily dangerous in the wrong hands.
The company’s researchers are also confident that GPT-Red is significantly more capable than any copycat system someone could train by reading about it. As Choquette-Choo puts it: “It’s not a trivial thing that someone could easily do — you know, just go and train a super-attacker using this idea.” More than a year of development time, backed by the compute resources of one of the most well-funded technology companies in the world, is not easily replicated.
The Bigger Picture: AI Safety Enters an Arms Race of Its Own
GPT-Red represents a meaningful shift in how frontier AI labs approach safety evaluation. Rather than relying primarily on human testers who are constrained by time, attention, and the limits of human creativity, OpenAI is automating the adversarial process itself — and letting the attacker grow more sophisticated alongside the defender.
For Indian enterprises and developers building on top of OpenAI’s APIs, this matters practically. Every improvement GPT-Red drives in model robustness reduces the risk that your customer-facing AI agent gets manipulated into revealing sensitive data or behaving unpredictably. The fact that GPT-5.6 blocked more than 77% of GPT-Red’s strongest attacks (compared to fewer than 10% for its predecessor) is a concrete measure of progress that enterprise security teams can factor into their risk models.
The deeper implication is philosophical: as AI systems become more capable and more autonomous, the only scalable way to keep them safe may be to build AI systems equally capable of breaking them. The arms race is not just between AI labs — it is now happening inside them.
