Safety & Security

Cheaper AI coding models sometimes produced safer code

Secure Code Warrior tested more than 30 AI models across 41.3 million lines of code. Curated security rules cut measured risk by as much as 70%, while some budget models produced less risky code than premium systems.

AI coding benchmarks show cheaper models can be safer
AI coding benchmarks show cheaper models can be saferPexels / pixabay
By Sophie Lindqvist6 min read

Why it matters

  • November 18, 2026: Secure Code Warrior released updated SCW AI Trust Index findings covering more than 30 AI models.
  • The study built more than 15,000 applications containing 41.3 million lines of code across 11 programming languages.
  • Measured output ranged from 0.24 to 3.66 vulnerabilities per 1,000 lines, a 15-fold gap.
  • Curated rules cut measured risk by as much as 70%; average reductions were 54.15% and 34.05% for the lower- and higher-risk model halves.
  • Without rules, GPT 6 Luna scored 19.4 at USD $0.10 a run, versus 43.6 for Claude Opus 5.5 at USD $8.17.

Secure Code Warrior's test used more than 30 AI coding models to build more than 15,000 applications and 41.3 million lines of code, producing a 15-fold gap between the lowest and highest vulnerability rates. The updated SCW AI Trust Index found that curated security rule sets cut measured risk by as much as 70%.

Some lower-cost models also produced less risky code than flagship systems. Secure Code Warrior cited external survey data expecting AI to account for 65% of committed code by 2027. That makes security evidence central to software procurement as teams rely more on large language models for coding work.

What did Secure Code Warrior test?

Secure Code Warrior released the findings on November 18, 2026. Its test covered models from:

  • Providers: OpenAI, Anthropic, Google, Alibaba, Mistral and others
  • Scale: More than 15,000 applications, 41.3 million lines of code and 11 programming languages

Secure Code Warrior updated its approach to assess models inside a harness designed to better reflect real software development use. It also added functional completeness comparisons alongside security results, so the index could show both the risk and utility of generated code across software development tasks.

The reported results therefore describe models operating inside a specified harness. The index also treats security as one measure of usefulness, rather than a score detached from whether the code completed the task. Secure Code Warrior measures risk and utility together.

What did the security results show?

Across the models tested, output ranged from 0.24 to 3.66 vulnerabilities per 1,000 lines of code. Secure Code Warrior called that a 15-fold gap between the benchmark's best and worst performers.

The benchmark did not produce one model that led across every framework. Secure Code Warrior's baseline results were fragmented:

  • Fifteen different models ranked first or tied for first in at least one of the 11 frameworks at baseline.
  • GPT-6-Astra-High led or tied for first in four.

A model that led one framework did not necessarily lead the others. Secure Code Warrior found winners across the tested field, and its price comparison complicates a rule that newer or pricier systems always land at one end of the table. For buyers, a single leaderboard cannot describe a model's measured security across different development stacks.

How much do prompts and rules change the risk?

Secure Code Warrior found that prompts and rule sets had a material effect on the security of generated code. Stronger models benefited more from curated rules than weaker models, while vague prompts produced less reliable results.

The report broke out the effect of Secure Code Warrior's own rules and third-party rules:

  • Maximum reduction: Curated security rule sets cut measured risk by as much as 70%.
  • Lower-risk half: The rules cut measured risk by an average of 54.15%.
  • Higher-risk half: The rules cut measured risk by an average of 34.05%.

Even the higher-risk half cut measured risk with rules, though its average gain was smaller than the lower-risk half's. The gap between those averages shows why a raw model result needs context.

Secure Code Warrior says the model's strength, the harness and the instructions all shape the measured outcome. Branding alone did not determine the measured risk.

Does a higher price guarantee safer code?

Some budget models recorded lower measured risk than premium models even when the models ran without additional rules. One comparison, using both models alone without rules, showed:

  • GPT 6 Luna: A mean risk score of 19.4 at USD $0.10 a run.
  • Claude Opus 5.5: A mean risk score of 43.6 at USD $8.17 a run.

The company presented those figures as mean risk scores, not as the vulnerabilities-per-1,000-lines measure used across the benchmark. Secure Code Warrior said decisions based only on branding, headline performance or price may not reflect actual security outcomes in coding tasks.

The pairing does not establish one model as the overall safest or most capable. Secure Code Warrior's framework results and functional completeness comparisons remain separate evidence in the index.

The result also underlines the challenge of standardizing AI coding tools across large engineering teams. Price alone was not a dependable guide to measured security risk in this benchmark.

How does AI-generated code compare with human-written code?

Separate academic work points to fewer vulnerabilities in human-only commits, though Secure Code Warrior warned that the methods differ. The company cited Stanford University research that found human-only commits introduced roughly 0.08 vulnerabilities per 1,000 lines.

That figure sits below the AI benchmark's 0.24-to-3.66 range. Secure Code Warrior said the methodologies differ, so the comparison remains limited.

The forecast that AI will produce 65% of committed code by 2027 gives the comparison practical weight. Secure Code Warrior cited it as pressure on security teams to understand how model choice and prompting affect risk. Human-only commits still had a lower reported vulnerability rate, but the comparison does not identify the safest model or setup.

What should engineering and security teams do next?

Secure Code Warrior Co-Founder and Chief Executive Officer Pieter Danhieux framed the results as a challenge to common assumptions about model age and cost.

"It's easy to assume every new AI model writes safer code than the last, or a frontier model is more secure than a budget model. Our updated findings show this isn't always the case," Danhieux said.

He tied the findings to governance rather than a permanent model ranking. His argument was that teams need repeatable evidence about the risks carried into their own development environments.

"Today's models know approaches to secure coding, but they don't do it by default.

The harness, prompts and rules they're given decide how secure the output is, and that behaviour can shift with every new release.

A one-time benchmark can't keep pace, which is why the SCW AI Trust Index evolves alongside the models themselves.

Security leaders and developers need a current, evidence-based view of the risks they're inheriting that is verifiable by the languages and frameworks in use within the development team."

In practice, that means teams need to revisit the same models when releases, prompts or rules change. A benchmark result can date quickly if the behavior behind it shifts.

Secure Code Warrior presents that as an ongoing governance task for security leaders and developers. The evidence must stay current and tied to the languages and frameworks teams use, because model behavior can change with every release.

Secure Code Warrior says the SCW AI Trust Index evolves alongside the models. As releases change model behavior, teams will need current, stack-specific security evidence as they standardize coding tools across engineering organizations.

Source: itbrief.com.au

Share this article:

More from Sophie Lindqvist

Sophie Lindqvist

Show full bio

Staff writer covering regulation, policy and their impact on artificial intelligence and machine learning at AI In Context.

749 articles

Related articles

  1. OpenAI commits $10M in API credits to Trusted Access for Cyber program
  2. 96% of Developers Don't Trust AI Code. Teams Are Rewriting Review.
  3. OpenAI and Paradigm Launch EVMbench for Smart Contract Security
  4. OpenAI Rolls Out GPT-5.4-Cyber to Vetted Defenders
  5. GitHub and Microsoft's AI bug scanner produced 99 verified reports across six projects

« Previous articleNext article »