Three AI Engines, One Wrong Answer: Consensus Fails 300-Query Test
Scento tested 300 perfume-note queries across ChatGPT, Gemini and Perplexity. All three matched official lists exactly for only 39 of 99 perfumes — and agreed on the same wrong answers.

Why it matters
- Scento ran 300 queries on 20 October 2026: 100 perfumes from 45 houses, one prompt each to ChatGPT, Gemini and Perplexity.
- All three engines matched the official note list exactly for only 39 of 99 perfumes; for 20 perfumes none was exact.
- Gemini added at least one unique off-list note in 32% of answers, versus 10% for ChatGPT and 11% for Perplexity.
- ChatGPT's Montblanc Explorer answer placed only two of six matched notes in the correct tier, a 33% placement accuracy.
- Designer answers carried off-list notes more often than niche ones on all three engines, with gaps of 22 to 32 percentage points.
ChatGPT, Gemini and Perplexity returned identical, wrong seven-note pyramids for Jean Paul Gaultier Le Male Elixir — three notes short of nothing, but four notes more than any official source lists. That single case anchors Scento's October 2026 study of 300 product-fact queries, and it exposes a flaw at the heart of a common AI-agent design pattern: when several models trained on the same web text agree, agreement itself proves nothing.
Scento asked the three engines for the top, heart and base notes of 100 perfumes from 45 houses. All calls ran once, in English, from a US location on 20 October 2026. Gemini returned 99 answers; one call ended in a service error. The reference answers came from the houses' own sites for 91 perfumes, and from two authorised retailers in agreement for the other 9. A deterministic script — no language model involved — mapped every note onto a 411-note glossary and graded four metrics: recall, precision, exact match and placement. A manual check of 20 gradings found 17 exact agreements and 19 within one note, though the reviewer could see the script's result, so the check was not blind.
Why does this matter beyond perfume?
The study tests a safety net that AI builders lean on heavily. Self-consistency — sampling several reasoning paths and keeping the majority answer — genuinely improves accuracy on reasoning benchmarks. LLM-as-a-judge setups, where one model grades another, reach high agreement with human raters. But Scento's results show that for product facts, consensus has a hard ceiling: it catches one engine's outlier, and it is blind to errors all engines share. For any agent that turns product facts into shopping decisions, that blindness is a business risk.
How well did the engines do on their own?
On recall — how many official notes an answer contains — the three engines looked almost interchangeable. Median recall was 1.00 for each. ChatGPT returned every official note in 82% of answers, Perplexity in 78% and Gemini in 69%.
Full agreement was much rarer. Of the 99 perfumes all three engines answered:
- All three returned every official note for 59 perfumes
- All three matched the official list exactly for only 39
- At least one engine was exact for 79
- For 20 perfumes, no engine was exact
In roughly one case in five — 20 of 99 perfumes — all three engines returned every official note, yet at least one also invented notes absent from the official list. A recall-only check would have passed all of them.
What happens when the engines agree on the wrong thing?
The Le Male Elixir case shows the pattern at its cleanest. The two agreeing retailers list three notes: lavender on top, tonka bean in the heart, benzoin in the base. All three engines produced the same seven-note pyramid. Each included the three reference notes, swapped tonka bean and benzoin into each other's tiers, and added the same four extra notes. Three separate systems, one identical answer, departing from the reference in the same places.
Viktor&Rolf Flowerbomb shows a second variant. The house lists four notes. Gemini and Perplexity both returned the same 11-note pyramid, missed one official note and added eight. Perplexity even noted that the brand's current description highlights the four official notes — it held both versions and led with the older one.
Scento's data cannot explain why the answers match. The practical point survives regardless. When systems draw on the same pool of web text, a majority vote measures how widely a version has spread, not whether it is current.
Are one-engine outliers the real problem?
No — and this is the asymmetry the study wants builders to internalise. Single-engine additions are exactly what cross-checking catches. Gemini added at least one note that neither rival named in 32% of its answers, against 10% for ChatGPT and 11% for Perplexity.
The examples are stark. For Prada Infusion de Vétiver, the house lists four notes; ChatGPT and Perplexity returned exactly those four, correctly placed, while Gemini returned twelve — eight of them named by no other engine, for a precision of 0.33. For Rochas Eau de Rochas, Perplexity matched the ten-note list exactly; Gemini returned 18 notes, ten of them unique to it.
"Disagreement is a useful alarm," as the study puts it. "Agreement is not a certificate."
Can an answer have the right facts and still be wrong?
Yes, and standard evaluations miss it. For Montblanc Explorer, ChatGPT matched six of nine official notes, but only two of those six sat in the correct tier — a placement accuracy of 33%. A check asking only whether a note appears somewhere in the answer would not see that four of the six are misplaced. Structure carries meaning for any agent acting on product facts, whether the domain is sizes, compatibility, dosage or ingredients.
Did accuracy vary by product segment?
It did, and uniformly. Designer perfumes carried off-list notes more often than niche ones on all three engines, with gaps of 22 to 32 percentage points. ChatGPT showed the widest split: four in ten of its designer answers included at least one invented note, versus fewer than one in ten niche answers.
Because a short official list gives an engine more room to add notes, Scento controlled for list length by pooling all engines into three bands. Designer answers still carried extras more often in every band: 70% versus 31% for lists of five notes or fewer (27 and 29 answers), 61% versus 21% for six to eight notes (51 and 42 answers), and 26% versus 13% for nine or more (72 and 77 answers). The samples are small and drawn from different sets of houses, so the cause is not isolated. The implication stands: accuracy varies by segment, and a single catalogue-wide figure can hide that.
What should teams building shopping agents take from it?
The study closes with concrete guidance:
- Use consensus to find disagreement, not to confirm truth. Comparing models cheaply flags one-off additions, but cannot detect an answer all models share.
- Ground owned facts in an owned source. For manufacturer specifications, retrieve from a structured record carrying its source page — schema.org Product markup on the page, and the Model Context Protocol for direct agent queries. Google's guidance on AI features says structured data should match visible page text, and that no special schema.org markup is needed to appear in those features.
- Score precision, exact match and structure, not just recall. An answer containing every fact plus three invented ones passes a recall test and still misleads.
- Grade deterministically where reference data exists. A rule-based grader against structured data is repeatable and auditable, and it does not share a model's blind spots.
- Run more than once. Answers vary run to run — a point Andrew Lissimore made on the same site about measuring AI shopping recommendations. Scento's single call per perfume per engine leaves run-to-run variation unknown.
What are the study's limits?
Each perfume was asked once per engine, on one day, in English, from one country. Claude and engines without web search were not tested. A note missing from an official list is not automatically wrong, since houses sometimes publish only key notes, making precision a lower bound. The grader was tuned on the same 100 perfumes, and the sample skews toward Western designer and niche houses with verifiable lists.
Within those limits, the pattern held. Three engines agreeing tells you a version of the facts is widely repeated; only the source tells you whether it is current. The question the study leaves with anyone whose agent checks itself against a second model: what does your agent do when two models give the same wrong answer?
Source: unite.ai
More from Elena Vasquez
Show full bio
Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.
708 articles
Related articles
- Graphite finds 13,000 phrases that flag AI writing, including 'this matters'
- Google Messages gets cloud-based Gemini AI to catch scam texts
- Gemini Will Screen Your Texts: Google Rolls Out AI Scam Warnings for Android
- Meta Claims Millions Shop With Muse. Executives See 'Hundreds' of Orders
- Anthropic Will Watermark All Future Claude Model Output