AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Fast AI Decision Models Show Uneven Cognitive Profiles, Study Finds

Small AI models built to return a single structured decision are increasingly placed inside automated systems that route requests, apply safety checks, and trigger actions. A well-formed answer, however, does not show that the model used the evidence it was given. A new benchmark from researchers at Tohoku University, led by Dengzhe Hou and Kazunori D Yamada, tests what these models can and cannot do using established psychology experiments.

The authors call these tools “System One” decision models, after the fast, intuitive mode of thinking in dual-process theory. Their benchmark, S1CogBench, contains 24 cognitive tasks and 27 evaluation modes. Each task presents a visible state and a list of legal responses, and the model returns a probability distribution over them. The authors score each model in several ways: how well it predicts the choices real participants made, how well it performs when acting on its own, and how accurately it reports probabilities. They also manipulate task conditions, such as memory load or future opportunities, to see how behavior shifts.

The study covers seven open models and one API model, Jev. Four open models, Laya, Intern-Decision, Nox, and CLM, are System One models, while others serve as general-purpose comparisons.

One example illustrates the gap between acting well and predicting people. In the Wisconsin Card Sorting Test, Nox had the highest autonomous accuracy among the open models, at 32.1 percent. Yet its predictions of human choices were worse than uniform guessing, with a negative log-likelihood (NLL, where lower is better) of 1.552 against 1.386. Minitaur, fine-tuned on human behavior, predicted human choices best, at 1.344. In Mouselab, an information-search task, Qwen earned the highest autonomous reward among open models, 17.40, but its human-choice NLL was 2.464, compared with 1.634 for uniform choice. The authors note that the autonomous version removed the observation cost participants faced.

Learning also varied. In a working-memory task, Intern-Decision gained 61.49 percentage points from the first to the tenth stimulus presentation at set size six, compared with 46.35 for humans. Laya and CLM stayed nearly flat. In a foraging task, humans chose the less-observed option 10.00 percentage points more often when more future choices remained. The models shifted by between −0.49 and 2.62 points, and every model’s gap from humans had a confidence interval below zero.

The choice-extraction method also changed rankings. In Horizon, Minitaur’s optimal-choice rate was 84.0 percent under legal argmax and 70.9 percent under sampling. Latency was also substantial. In Horizon, Llama’s free-text generation had a median of 2.958 seconds, roughly 80 times the 36.97 milliseconds of a single forward pass. Generating one token added only 0.24 to 2.02 percent overhead.

The authors acknowledge several constraints. Tasks were converted to text, histories were supplied, and some models may have seen related human data during training. Architecture, training data, and model size vary together, so the authors do not attribute differences to any single design choice. The authors also cannot rule out a bias toward stopping in Mouselab, since STOP was always listed first.

The work offers a common framework for examining where fast decision models fail. It does not establish that these models are unsuitable for deployment, and the authors present cognitive profiles as a complement to accuracy and latency, not a replacement.