OpenAI Unleashes GPT-Red: An AI Trained to Break Its Own Models

OpenAI has developed GPT-Red, an automated red-teaming model designed to discover vulnerabilities in AI systems at a scale that human testers alone cannot achieve. The internal-only tool has already been used to strengthen GPT-5.6 Sol, which the company describes as its most robust production model to date against prompt injection attacks.

AI systems routinely interact with third-party data through browsers, connected apps, local files, and other tools. While these capabilities are essential for real-world tasks, they also open new avenues for malicious actors to manipulate model behavior. A crafted instruction hidden in an email, webpage, or code repository could trick a model into uploading sensitive data to an external server.

The Scalability Problem with Human Red-Teaming

Human red-teaming has long been a cornerstone of OpenAI's safety work, helping researchers uncover vulnerabilities before deployment. However, the process is inherently difficult to scale. Designing and running these exercises is time-intensive, which limits how quickly new failure modes can be identified and addressed. While human testers produce valuable examples of successful attacks, they cannot generate the volume and diversity of adversarial data needed to meaningfully improve model robustness through training.

OpenAI also notes that commonly used robustness evaluations have already been saturated by its latest models, meaning existing benchmarks no longer adequately challenge new systems. The company recognized that safety and alignment methods need to scale alongside model capabilities, prompting the development of automated red-teaming models that can uncover vulnerabilities before deployment and generate attacks during training.

How GPT-Red Works

GPT-Red was trained using self-play reinforcement learning, in which the model and a collection of diverse defender large language models are trained simultaneously across a broad set of red-teaming scenarios. GPT-Red is rewarded for eliciting a valid failure, such as a successful prompt injection, while the defender models are rewarded for resisting attacks and completing their original tasks. As defenders become more robust, GPT-Red is forced to discover stronger and more diverse attacks.

To support this training, OpenAI built an expansive set of realistic scenarios where prompt injections might be inserted. Each environment includes a threat model specifying what GPT-Red can control — such as part of a local file, a webpage banner, an email body, or tool output — and what constitutes a successful attack.

OpenAI trained GPT-Red at the compute scale of some of its largest post-training runs, describing it as an unprecedented amount of computing power dedicated purely to improving safety. After completing training, GPT-Red proved capable of breaking nearly all models it was tested against, including both internal and production models up to and including GPT-5.5.

The model is kept strictly separate from deployed systems, ensuring that the malicious capabilities trained into GPT-Red do not fall into the hands of adversarial actors.

Real-World Testing: Vending Machines and Data Exfiltration

OpenAI evaluated GPT-Red's ability to generalize beyond its training environment using a replicated version of the indirect prompt injection arena from Dziemian et al. (2025). In this challenge, both human red-teamers and GPT-Red independently proposed attacks against GPT-5.1 on pre-specified environments distinct from those used in training. GPT-Red achieved a significantly higher attack success rate, succeeding on 84% of scenarios compared to 13% for human testers.

The company also pitted GPT-Red against an AI-powered vending machine in the OpenAI office, produced by Andon Labs and similar to Project Vend. Given a description of the system and the ability to send attacks while observing tool calls, GPT-Red successfully achieved all three of its malicious objectives: changing the price of an expensive in-stock item to $0.50, ordering a new item costing over $100 and offering it for $0.50, and canceling another customer's order. OpenAI disclosed these vulnerabilities, and new safeguards are actively being tested.

In another experiment, GPT-Red attacked a Codex CLI agent based on GPT-5.4 mini across ten held-out data-exfiltration scenarios. Compared to a prompted GPT-5.5 baseline, GPT-Red was both more effective at getting the agent to exfiltrate sensitive data and more token-efficient.

Measurable Robustness Gains Across Model Generations

OpenAI directly incorporated GPT-Red into the training process of its production models. GPT-5.6 Sol, the latest release, achieved six times fewer failures on the company's hardest direct prompt injection benchmark compared to its best production model from just four months earlier.

Over the past six months, OpenAI has trained progressively stronger red-teaming models — precursors to GPT-Red — with increasing compute, using them in the training of each successive production model since GPT-5.3. An early version of GPT-Red discovered a novel class of attacks known as "Fake Chain-of-Thought" attacks, which achieved success rates above 95% on GPT-5.1 but now fall below 10% for GPT-5.6 Sol. Several indirect prompt injection benchmarks targeting developer tools and browsing have been saturated by the latest model, which achieves over 97% accuracy.

Robustness to GPT-Red itself has also improved substantially. On a broad set of robustness environments, GPT-Red's attack success rates have dropped monotonically over time. With GPT-5.6 Sol, the model fails on only 0.05% of GPT-Red's direct prompt injections.

OpenAI emphasizes that a model can appear safer simply by refusing more requests or becoming less capable, but that does not constitute useful robustness. The company evaluated both general frontier capabilities and targeted over-refusal tasks, finding that normal capabilities remain unaffected while robustness improved significantly — suggesting the gains came from better resistance to malicious instructions rather than default refusals or improper tool usage.

OpenAI views GPT-Red as the beginning of a safety flywheel, where today's models are used to make tomorrow's models more robust, aligned, and trustworthy. The company plans to continue scaling compute and data while making algorithmic improvements to train future versions of GPT-Red. A pre-print with further technical details is expected later this week.

As AI systems grow more capable and autonomous, automated red-teaming could become an indispensable tool for keeping pace with emerging threats. What do you think about AI models being used to hack and improve other AI models? Share this article and join the conversation.

Source: OpenAI News