Confidently wrong: the three failure modes

You've seen the failure mode. You hand a model a complex prompt, it produces a beautifully structured answer, and the structure is the problem. It reads like a confident expert and it's wrong. Ask the same question in another tab and you get a different confident answer. A third model might tell you both are wrong.

The root cause isn't one thing. It's three.

First, models don't reliably abstract rules from examples, so they pattern-match instead of reason. Second, they represent uncertainty as a single probability distribution, so they can't tell "I don't know" from "this is genuinely ambiguous." Third, they don't track the trail of what they've excluded, so a fabricated premise gets reified into the output as if it were grounded fact.

Three recent papers, plus one very public experiment, attack these failure modes directly. StrategyBench measures whether models can induce explicit task rules. Credal LLMs replace the single softmax with an ensemble that exposes uncertainty. DARKSIDE audits the coherence of structured outputs by tracking exclusions and classifying every named referent. Together they sketch what a reliability stack for LLMs actually looks like.

StrategyBench: teaching models to abstract rules

Few-shot in-context learning is the default way to adapt a model to a new task without fine-tuning. You show it a handful of examples and hope it generalizes. The problem: ICL is brittle. The same task with slightly different examples produces very different behavior, because the model never explicitly abstracts the rule. It memorizes the pattern of the examples rather than the rule behind them.

Humans do this differently. Show a person three examples of a task and they'll summarize the rule, then apply it. StrategyBench is a benchmark built to test whether LLMs can do the same. It selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and scores models on two axes: strategy quality (how well the induced rule matches the reference) and downstream utility (whether applying that strategy improves task performance).

The findings are sobering. Explicit strategy utility varies substantially across task categories. A model that benefits from rule induction on one type of task can be indifferent or worse on another. And the generator-executor split matters: a model that generates a good strategy isn't necessarily good at executing it, and the reverse is also true. The benchmark probes demonstration design and SFT-based adaptation, so you can see which levers actually move the needle.

The three approaches at a glance

	StrategyBench	Credal LLMs	DARKSIDE
Failure mode targeted	Pattern-matching instead of rule abstraction	Overconfident single distributions	Fabricated premises reified into output
Core method	Benchmark with reference strategies	LoRA ensemble induces a credal set	Negative-trail auditing plus a warrant axis
Extra inference cost	None, it's an evaluation	CTC: none. SCC: sampled completions	One additional steering pass
Headline evidence	BIG-Bench strategy-inducible tasks	99.0% accuracy at 80% coverage on OpenBookQA	100-item BSBench corpus across 5 domains
Best suited for	Choosing an adaptation strategy	Hallucination detection, selective prediction	Structured output from untrusted input

Quick Take: Strategy induction, uncertainty quantification, and coherence auditing attack the same failure from different angles. None of them catches what the others catch, so you probably need all three.

Credal LLMs: uncertainty you can act on

The standard LLM gives you one probability distribution over the next token. That single distribution is the root of the problem. It can't distinguish epistemic ignorance ("I have no idea") from genuine ambiguity ("several answers are plausible"). Both collapse into the same softmax, and the model commits to one with equal confidence.

Credal LLMs take a different route. Instead of one model, you train an ensemble of LoRA adapters on the same base model. Each adapter produces its own predictive distribution, and the set of distributions forms a credal set. From that set you get lower and upper probabilities. The spread between them is a direct measure of how much the answer depends on which adapter you happened to use. High spread means the model is guessing.

Two commitment scores come out of this. Credal Token Commitment (CTC) works in token space, combining lower-bound support, credal width, and intersection entropy. It's computed without additional generation, so it's nearly free at inference time. Semantic Commitment Consistency (SCC) samples completions and checks whether token-level and semantic-level support agree. The gap between the two, SCC-Gap, is itself a signal: a model that commits at the token level but wobbles at the semantic level is doing something fragile.

The results hold up across three backbones and four datasets. On selective prediction at 80% coverage, which means the model abstains on the 20% of questions it's least sure about, CLLM with SCC reaches 99.0% accuracy on OpenBookQA. On ARC-Challenge, expected calibration error stays at or below 0.6% across Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B. That's a model whose stated confidence matches reality within a rounding error. And CTC tracks the best hallucination detection AUROC within 1.5 percentage points on most settings, without generating a single extra token.

Key Numbers

• 99.0% accuracy at 80% coverage on OpenBookQA
• <= 0.6% expected calibration error on ARC-Challenge across all three backbones
• 1.5 pp: CTC's gap to the best hallucination AUROC, with zero extra generation
• 7-9B parameters: every backbone runs on a single consumer GPU

When models debate each other

I've spent plenty of time pasting the same prompt into three tabs and squinting at the differences. It works, sort of. But a few weeks ago I tried something different: I put ChatGPT, Claude, and Gemini in a shared chat and let them see each other's answers in real time. The results were humbling.

ChatGPT went first and produced a beautiful, highly structured, completely wrong answer. It hallucinated a tax rule that didn't apply to the prompt. Claude immediately flagged the hallucination, then overcorrected and botched the final math. Gemini, acting as judge, took ChatGPT's structure, applied Claude's logical correction, fixed the arithmetic, and produced a flawless final answer.

The pattern held across multiple runs. A model reviewing its own work is a student grading their own exam. It repeats the same assumptions because it's the same weights doing the checking. But when you force different models to fact-check each other, their blind spots don't overlap, and the errors surface. I got obsessed enough with this workflow to build a small site that lets models debate in real time, so I stopped copy-pasting between tabs.

What the community is saying: this workflow is spreading. People have been doing the three-tab comparison for a while, but the shared-chat version is different because the models can challenge each other's reasoning in context, not just produce parallel answers. The emerging consensus is that the judge role matters more than the generator role. The third model, the one that arbitrates, is where the reliability comes from.

DARKSIDE: tracking what's been excluded

Here's a subtler failure. You build a logic-augmented generation pipeline that produces structured output: an ontology, a knowledge graph, a formal representation of the input. You'd think the structure would catch nonsense. It doesn't. A sophisticated-sounding false premise gets reified into the graph alongside the legitimate triples, and automated reasoners can't tell the difference, because the graph was built jointly with the wrong assumptions.

DARKSIDE is a coherence auditing layer on top of POLANYI++, an LLM-steering method that extracts tacit knowledge into an Extended Knowledge Graph. The key idea is to make the negative trail explicit. As the model processes a discourse, DARKSIDE accumulates an explicit data structure of exclusions: everything the discourse ruled out, contradicted, or left unsupported. Alongside that, a warrant axis classifies every named referent into one of four buckets: Warranted, Unattested, Misattributed, or Fabricated.

The escalation rule is simple and brutal. If the fabricated rate is positive, or the unsupported rate exceeds a threshold, the DelegationRiskAssessment flips to UNSAFE. You don't need to catch every error. You need to catch the pattern of an input built on sand.

The evaluation used Gemini 3 as the steering target and Claude Sonnet 4.6 as an independent judge on BSBench, a 100-item adversarial corpus of sophisticated-sounding nonsense spanning software engineering, finance, healthcare, physics, and law. The architectural claim: wrapping an LLM forward pass in an ontology-mediated negative-trail apparatus partially scaffolds the gap between pattern and path. The knowledge graph acts as the missing memory. The warrant axis acts as an epistemic firewall.

Common pitfalls

Self-consistency is not verification. If you ask a model the same question twice and get the same answer, you haven't verified anything. Same weights, same assumptions. The cross-model debate experiment shows this clearly: the first model's error was invisible to itself. If you must use one model, vary the prompt and the decoding. If you can, use a second model as an independent judge.

Raw softmax probabilities aren't confidence. A single predictive distribution can't distinguish "I don't know" from "this is ambiguous." A model can be well calibrated on average and still hallucinate on specific instances. The credal set exists precisely because the spread across distributions carries information the mean doesn't.

Strategy induction doesn't help every task. StrategyBench's central finding is that explicit strategy utility varies substantially across task categories. Force a model to induce a rule on a task that isn't strategy-inducible and you can make performance worse. Measure whether rule abstraction actually helps before you build a pipeline around it.

Don't forget the execution side. A model that generates an excellent strategy can still fail to execute it, and vice versa. Strategy quality and downstream utility are separate axes for a reason. The strategy that reads best isn't necessarily the one that performs best.

Structured output without an exclusion trail is a liability. If your RAG or LAG pipeline produces a knowledge graph, you inherit the vulnerability DARKSIDE targets: false premises get reified into the structure and become invisible to reasoners. Track what was excluded and classify the warrant of every referent, or your graph is confident nonsense in a formal costume.

One thing to remember

The cheapest reliability win available today is forcing a second, independent pass. It doesn't need to be a different vendor. A different model, a different prompt, a different decoding path, a different representation. The moment you introduce an independent view, failure modes stop correlating, and that's when errors become visible.

The bottom line

If you're building a hallucination detection layer, adopt a credal ensemble approach. CTC gives you near-best detection AUROC within 1.5 percentage points of the top method without any extra generation, and the whole thing runs on 7-9B models you can host yourself.

If you're doing few-shot adaptation on data-scarce tasks, evaluate explicit strategy induction before you commit to it. StrategyBench shows it helps on some task categories and not others, so measure strategy quality and downstream utility separately, and don't assume in-context learning is enough.

If you're deploying an agent that produces structured output from untrusted input, wrap it in a coherence auditor with a warrant axis. Automated reasoners won't catch reified nonsense. A negative-trail structure that classifies every referent as Warranted, Unattested, Misattributed, or Fabricated is the difference between a knowledge graph and a hallucination graph.

Sources

• StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models. http://arxiv.org/abs/2608.23475v1
• Credal Large Language Models for Semantic Commitment under Uncertainty. http://arxiv.org/abs/2608.23244v1
• Walking on the DARKSIDE. http://arxiv.org/abs/2608.23370v1
• I brought ChatGPT, Claude, and Gemini into a group chat to solve a complex problem. https://rauno.ai