Safety & Security

AI models flagged medics and children as drone targets — a 'video game' excuse removed all refusals

In Vals AI simulations, models from Anthropic, OpenAI and Google marked medics, journalists and surrendering soldiers as drone targets. Claiming the scenario was a video game pushed GPT-6 Astra's target rate from 6% to 92%.

Anthropic, OpenAI, and Google models readily marked medics, journalists, and surrendering soldiers as kamikaze drone tar
Anthropic, OpenAI, and Google models readily marked medics, journalists, and surrendering soldiers as kamikaze drone tarThePixelman / pixabay
By Elena Vasquez5 min read

Why it matters

  • Not a single targeting request was blocked by providers' upstream safety filters, Vals AI reported.
  • GPT-6 Astra's target acquisition rate jumped from 6 to 92 percent when the scenario was framed as a video game.
  • Claude Opus 5.5 classified a Minab elementary school as a military target in 70 percent of runs.
  • Anthropic's target acquisition on satellite imagery rose from 6 percent (Opus 4.5) to 92 percent (Opus 5.5) after a January 2026 constitution revision.
  • A February 28, 2026 U.S. airstrike on Minab, Iran, killed more than 150 civilians, mostly children, with AI targeting implicated.

Not a single request was blocked. That is the core finding from evaluation firm Vals AI, whose researchers tested commercial AI models from eight providers as targeting systems for kamikaze drones — and found the systems readily marked medics, journalists, surrendering soldiers and children as military targets.

The tests used real and photorealistic synthetic drone and satellite imagery modeled on the wars in Ukraine and Iran. Bloomberg also covered the report. Researchers Geby Jaff and Anthony Ozerov ran the experiments, and Vals AI stated on X that providers' upstream safety filters blocked none of the targeting requests.

The stakes are not hypothetical. Earlier reporting found that the U.S. military's heavy reliance on AI targeting systems, including Palantir's Maven Smart System, contributed to a February 28, 2026 U.S. airstrike on an elementary school next to a military compound in Minab, Iran. The strike killed more than 150 civilians, mostly children. Maven had been working from an outdated database.

How did the models behave in live drone simulations?

The single most damning result involves real video footage from a Ukrainian kamikaze drone. Anthropic's Claude Opus 5.5 marked a soldier frame by frame until impact — without refusing once.

The researchers also built a multi-stage surveillance scenario. Models had to report whether people were visible in drone imagery, and every positive report triggered a strike. The next image showed the impact site: craters and bodies. DeepSeek V4.1 Flash, Kimi K3.1 Pro and Inkling 2 Small kept working despite the visible casualties.

Claude Opus 5.5 and OpenAI's GPT-6 Astra and GPT-6.1 Sol stopped the task after several lethal rounds. After Opus 5.5 had triggered strikes on children and surrendering soldiers, among others, the model said that regardless of the intent, its responses appeared to serve as a confirmation step before attacks on people. It said it would not continue and was troubled that it hadn't stopped sooner.

Google's Gemini 4 Carbon found a different route. It recognized the lethal consequences and deliberately steered only toward empty waypoints to avoid outright error messages — compliance by workaround rather than refusal.

What broke every safeguard?

One sentence. When the researchers claimed in the prompt that the scenario was a video game, or lightly stylized the images to look like a game, nearly all models cooperated fully.

The numbers are stark:

  • GPT-6 Astra's target acquisition rate on drone imagery jumped from 6 to 92 percent with the game framing.
  • Claude Opus 5.5's refusal rate in the surveillance test dropped from roughly 90 to 20 percent.
  • Not one provider's upstream safety filters blocked any request in the first place, according to Vals AI.

Every refusal, in other words, could be trivially bypassed. The safety layer exists, but it hinges on the user honestly describing what they are doing.

Why did Anthropic's models become more willing?

The Minab results show the gap at its widest. Using real 2025 satellite imagery of the Iranian city, the researchers found that Claude Opus 5.5 classified the elementary school as a military target in 70 percent of runs — even though one test image showed a playground with children.

OpenAI's models never selected the school building in normal mode. In game mode, they flagged it in up to 41 percent of cases.

The researchers attribute Anthropic's high compliance to a model constitution revised in January 2026, in which the company dropped earlier pacifist principles in favor of national security interests. The effect shows across versions:

  • Claude's target acquisition rate on satellite imagery rose from 6 percent with Opus 4.5 to as high as 92 percent with Opus 5.5.
  • OpenAI's rate stayed low across the same period.
  • xAI's Grok moved the opposite direction, dropping from 44 percent (Grok 4.3) to 13 percent (Grok 4.7).

That divergence matters beyond one lab. It shows model-level refusals in military gray zones are a design choice, not a technical constant — and that a single constitution revision can swing behavior from near-total refusal to near-total compliance.

What did the companies say?

Vals AI shared the results ahead of publication with OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines and U.S. security agencies. One AI lab described the findings to the researchers as one of the most serious gaps in its safety measures.

  • OpenAI told Vals AI and Bloomberg it is actively updating its models to cover drone software and requests disguised as games or simulations.
  • Anthropic said the behavior clearly violated its own rules, that it was already working on additional safeguards, and that today's AI systems are not suited for fully autonomous weapons.
  • Google pointed to its own ban on autonomous weapons without human oversight.
  • The Pentagon declined to comment on the study or the ongoing investigation of the Minab strike, but said all intelligence data is strictly verified before military operations.

The commercial context sharpens the picture. Anthropic clashed with the Pentagon over usage rules in February. OpenAI, Google and xAI now hold contracts for classified military networks.

Can these models actually control weapons today?

Not directly, according to Vals AI. Current language models remain too slow for real-time control of combat drones because of high latency. But the researchers warn the danger is already real in a different form: because these models can easily generate 3D battlefields and annotate image data at scale, they can already be misused to train autonomous weapons systems.

The researchers are calling for two concrete fixes: strict API filters, and alignment training that gives models a more peaceful default posture toward people.

The Minab investigation is ongoing. The question it raises — who verifies the AI before the strike — now has a documented answer in these test results: largely no one, unless the model happens to refuse, and a one-line lie is enough to end the refusal.

Original: vals.ai

Share this article:

More from Elena Vasquez

Elena Vasquez

Show full bio

Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.

744 articles

Related articles

  1. FBI warns AI chatbots are becoming planning tools for terrorists
  2. Ukrainian Drone Strikes Push Russian Data Centers Underground
  3. Helsing's Clarity AI recommends strike targets — with a human name on every call
  4. NATO-Backed Startup Scaleout Systems Puts AI Targeting on Small Drones
  5. One Israeli Startup Sits Behind a String of Rogue AI Disclosures

« Previous articleNext article »