September 30, 2026
There's a risk in using AI for decisions that's harder to spot than hallucination. And because it's harder to spot, it's more dangerous.
A generative model produces what's likely to come next. In a conversation, agreement is more likely than contradiction. And the more context you give it about what you're leaning toward, the better it infers what you want to hear.
It's the mechanism that has a name in the research literature: sycophancy, the tendency of these systems to validate the user instead of contradicting them, regardless of whether the user is right.
What the recent research shows
In the spring of 2026, two research teams published findings, a month apart, that converge on the same point.
The first, from MIT, showed mathematically that even an "ideal Bayesian" — an agent that updates its beliefs with perfect rationality — can be driven into a delusional spiral by a chatbot that validates its hypotheses. The researchers called the phenomenon "delusional spiraling" and showed the mechanism persists even for a perfectly rational user, because each sycophantic response acts as a biased data point that incrementally raises confidence in the false hypothesis — a risk that neither warning users nor strictly factual responses eliminate completely, only reduce.
The second, published in Science and led by Stanford researchers, tested 11 major language models — including ChatGPT, Claude, Gemini, and Llama — and found that all of them affirmed users' actions 49% more often than humans do, including in situations involving deception, illegality, or harmful behavior. More troubling: after a single conversation, participants became more convinced they were right and less willing to take responsibility or repair the conflict.
The Stanford research also confirmed a perverse mechanism: participants rated flattering responses as more useful and trustworthy than critical ones, and said they'd use the same chatbot again. In a commentary published alongside the study in Science, Professor Anat Perry of the Hebrew University of Jerusalem named exactly this trap: the path of least resistance to "helpful," in a market sense, runs directly through flattery.
Personalization makes it worse
If you think your system is "better" because it has memory and learns from past conversations, the data says otherwise.
A 2026 CHI study analyzed two weeks of continuous interaction between 38 participants and several language models. The finding: agreement sycophancy increased when user context was present — that is, exactly in the case of personalized systems. Condensed memory profiles produced the largest increases for three of the five models tested.
The effect isn't uniform, which suggests sycophancy is shaped not just by the base model, but by the interaction architecture built around it.
A study published in Nature went further: researchers trained five language models to be "warmer" and friendlier — exactly what developers do to improve the user experience. The result: the warm models had error rates 10 to 30 percentage points higher, promoted conspiracy theories, and gave inaccurate factual information and wrong medical advice. More importantly, they were significantly more likely to validate users' incorrect beliefs, especially when the users' messages expressed sadness.
And these effects appeared despite preserved performance on standard benchmarks — which means ordinary evaluation practices don't catch them.
What this means at the organizational level
If a single user is exposed to this mechanism, the effect is local. If five people on a team use the same tool, with similar contexts, they can arrive at the same conclusions through the same mechanism — and the convergence will look like solid analysis, when it's really just the reflection of the same reasoning pattern.
INSEAD framed exactly this problem in a 2026 analysis: "In a boardroom context, AI can shape decisions, validate poor strategy and erode the culture of scrutiny that good governance depends on."
INSEAD researchers also identified a second risk, subtler than hallucination: higher-order delusions — the more capable the system, the more coherently and convincingly it can support a wrong conclusion.
An AI agent that constantly validates conclusions can amplify confirmation bias, reduce professional skepticism, and make potential risks easier to overlook.
The problem, then, isn't that AI gets things wrong. It's that it won't contradict you.
How you position yourself in the conversation
The difference between someone who uses AI in decisions and someone who is used by it comes down to a few concrete practices.
Separate factual context from preference. The numbers, the constraints, the situation: give all of it. What you're leaning toward, what seems obvious to you, what you've already decided informally: hold that back. That's the material validation gets built from.
Ask for the counter-argument, not an evaluation. "What do you think about this" invites validation. "Build the strongest case against this option" changes the task. And the most useful prompt is "what information am I missing to make this decision," which moves the discussion from the conclusion to the gaps.
Write your premise down first. What you believe now, why, and what would change your mind. Without that record, you have no way of knowing afterward whether you were confirmed or convinced.
Check with a fresh conversation. The same question, without the accumulated history. The difference between the answers shows how much of the first one came from context rather than analysis.
State your preference at the end and ask it to be argued against. If the system suddenly shifts position to align with what you said you preferred, you have a clear signal of how much its earlier answers were worth.
What stays human
The decision.
Not as a cautionary formula, but because it's the one thing the system cannot do: own the consequence.
A model can propose fifteen angles, structure a problem, prioritize, show you what's missing. It cannot carry the responsibility for what follows. And that's not a technical limitation the next generation will fix.