Echo reports three AI models agree on 1 of 121 code flaws
Infosecurity Magazine reports Echo says three AI models flagged 121 code issues but all agreed on only one; treat single-model audits with caution.
- Published
- Ecosystem
- AI developer toolsnpmOpen source
- Exploitation
- No, the source reports no exploitation; the audited projects are unnamed and Echo’s findings are not confirmed by maintainers.
What happened
According to Infosecurity Magazine, software security firm Echo published a comparison on November 18 in which three frontier AI models audited the same open source code and produced findings that barely overlap. Infosecurity Magazine reports that Echo says that, of 121 distinct potential issues, Anthropic’s Claude Mythos 5.1, OpenAI’s GPT-5.6 Sol and Z.ai’s GLM-5.3 all flagged just one.
Per the same report, Echo ran each model against two Maven libraries and one npm package using its own hunting framework and the same validation process. Echo reached GPT-5.6 Sol through Daybreak, the program OpenAI runs for vetted security teams. Echo says 83% (100) of the 121 issues came from a single model and 20 from the other two.
Other details Infosecurity Magazine attributes to Echo:
- Claude Mythos logged 25 validation passes, against nine for GLM-5.3 and eight for GPT-5.6 Sol. GPT-5.6 Sol accounted for six of the seven passes on one Maven library, and GLM-5.3 found six validated issues on the other that its rivals missed.
- Severity ratings split on one flaw in an npm package with more than 350 million monthly downloads. GLM-5.3 rated it Critical, Claude Mythos rated it High, and GPT-5.6 Sol treated it as documented behavior rather than a bug. Automated validation rated it Low.
- GPT-5.6 Sol finished its hunts in about 24 minutes, against roughly 198 for Claude Mythos and 174 for GLM-5.3. GLM-5.3’s three hunts cost about $22, against $303 for Claude Mythos and $30 for GPT-5.6 Sol. Echo said the costs reflected its own access terms.
- The nine runs produced 446 initial findings, cut to 152 after cleanup. Of those, 74 were rated High or Critical, and 55 went through automated proof-of-concept testing. Forty-two passed, though Echo cautioned these were not necessarily distinct bugs.
- Findings rated Medium or Low were never tested.
Echo researcher Yuval Saggie said the race to find the best model was “the wrong question,” and that if an attacker hunts with one model and a defender relies on another, “the defender might never even see the vulnerability the attacker is about to exploit.” That is Echo’s view, not an observed event.
Not confirmed: Echo called the work “a controlled comparison rather than a comprehensive benchmark,” with each model run only once. It said automated validation did not amount to confirmation from the projects’ maintainers. The source does not name the projects, give CVE identifiers, or report any exploitation.
Am I affected?
Not confirmed. Echo said the projects would stay unnamed, and technical details unpublished, until their maintainers had been notified. Infosecurity Magazine therefore gives no way to tell whether a specific Maven library or npm package you use is involved. The source names no affected versions and no fixes.
This is a research comparison, not a disclosed vulnerability, so the main relevance is to teams that use AI models to review code.
What to do now
- Do not treat any single model’s clean or flagged result as complete. Echo reports that only one of 121 potential issues was flagged by all three models.
- Treat AI severity ratings as provisional. In Echo’s test, one flaw was rated Critical, High, not a bug, and Low by different parties.
- Remember that Echo says its automated validation is not maintainer confirmation. Wait for maintainers to confirm anything before acting on it.
- Watch for maintainer notices from the affected projects, which Echo says will be notified before details are published.