AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Instruction Tuning Makes Language Models More Confident Without Making Them More Accurate

Large language models are increasingly used in medicine, finance, and law, where knowing when to trust a model is as important as getting the answer right. A preprint examines a counterintuitive side effect of “instruction tuning”—the extra training that teaches models to follow natural-language instructions and turns a raw base model into a chatbot or reasoning assistant. The researchers find that instruction tuning consistently makes models sound more confident, but not more accurate, and that the explanations they generate become more uniform in some ways while varying unpredictably in others.

The study compared three matched pairs of base and instruction-tuned models: Qwen2.5-7B, Mistral-7B-v0.3, and Llama-3.1-8B. Each pair was evaluated on three multiple-choice reasoning benchmarks: ARC-Easy, covering grade-school science; MMLU, covering academic and professional subjects; and CommonsenseQA. For every question, the authors measured answer uncertainty using the entropy of the model’s probability scores across choices, and elicited “verbalized confidence” by prompting the model to state the probability that its chosen answer was correct. They then sampled five chain-of-thought rationales per question and measured lexical diversity two ways: Unique-2, the share of distinct two-word sequences, and 1-SelfBLEU, a similarity score where higher values mean less overlap across rationales.

The confidence results were consistent. Answer entropy fell in every model-benchmark pair—for example, Qwen’s entropy on MMLU dropped from 0.430 to 0.131, and Mistral’s on CommonsenseQA from 0.736 to 0.268—while verbalized confidence rose in every setting. Llama’s stated confidence on ARC-Easy jumped from 49.2% to 90.4%, a 41.2-percentage-point increase, while its accuracy stayed at 82.2%. Accuracy changes elsewhere were uneven, and likelihood-based calibration grew worse in every setting: Qwen’s expected calibration error on MMLU, a measure of the gap between stated confidence and actual accuracy, rose from 4.4% to 21.3%.

The lexical-diversity picture was more varied. Cross-rationale diversity, measured by 1-SelfBLEU, dropped across all models and benchmarks, with Mistral on ARC-Easy falling from 0.813 to 0.626. Surface-level bigram diversity, however, moved in different directions depending on the model and benchmark; Mistral’s Unique-2 on CommonsenseQA rose from 0.719 to 0.750. The authors illustrate the pattern with Mistral’s rationales for a science question about perfume spreading through a classroom: after instruction tuning, the five explanations converge on similar phrasing such as “This process is known as diffusion,” whereas the base model’s versions vary more, including one that begins “I can solve this by eliminating the alternatives.” The lower cross-rationale diversity persisted in controlled comparisons that matched answer choices and rationale lengths: on CommonsenseQA, Mistral’s 1-SelfBLEU still fell by 0.069 and Llama’s Unique-2 rose by 0.053.

The authors caution that the study covers only English multiple-choice benchmarks and lexical, not semantic, diversity, and that it did not test safety-sensitive or demographic-sensitive prompts, so the results should not be read as evidence about harmful or biased outputs. They also warn that a confident, uniform-sounding model can encourage overreliance on wrong answers in high-stakes settings. The result, they argue, suggests confidence and rationale diversity are distinct effects of instruction tuning—and that trustworthy deployment will require tracking both.