AI Papers Reader

Personalized digests of latest AI research

View on GitHub

LLMs Under the Microscope: New Tool Visualizes Reasoning Processes, Reveals Unexpected Behaviors

Large language models (LLMs) have become increasingly powerful, but their internal reasoning remains largely a mystery. Now, researchers at Hong Kong Baptist University, Stanford University, Mila, HEC Montréal, and Intel AI Lab have developed “Landscape of Thoughts,” a novel visualization tool that offers unprecedented insight into the step-by-step reasoning of these models. Their work, published in a recent arXiv preprint, could significantly impact the development, safety, and understanding of LLMs.

The core idea behind Landscape of Thoughts is to represent the intermediate states of an LLM’s reasoning process as feature vectors, which are then projected onto a two-dimensional plot. Imagine asking an LLM a multiple-choice question like: “A class of 35 students has an average height of 180 cm. Calculate the new average height if one student leaves?”. The LLM might then follow a chain-of-thought process, first establishing “let’s calculate the sum of all the students”, then “we must find the sum after a student leaves”, and finally, “then divide by the number of students”. Landscape of Thoughts captures the state of the model at each of these intermediate “thought” steps. Each potential answer is also represented as a “landmark” in the space.

The tool then visualizes how the LLM’s reasoning path converges towards a particular answer. The visualization is created using t-SNE, a dimensionality reduction technique that preserves the relationships between data points. By observing the “landscape” of these paths, researchers can uncover a host of interesting and previously unknown patterns.

For instance, the tool reveals that LLMs often exhibit low consistency and high uncertainty in their intermediate reasoning steps. “This implies that the reasoning process can be quite unstable”, explains Zhanke Zhou, the study’s first author. While some algorithms, like Chain-of-Thought (CoT), are designed to solve problems directly, the generated thoughts may not consistently lead towards the correct solution.

The visualization also reveals insights into model accuracy. The convergence speed of reasoning paths toward correct answers reflects the overall accuracy, regardless of the specific model, decoding algorithm, or dataset used. A correct path is slower, whereas an incorrect path converges rapidly to a wrong answer.

But Landscape of Thoughts isn’t just a passive observer. The researchers have also shown how it can be adapted to build a lightweight verifier that evaluates the correctness of reasoning paths. By training a simple model to predict the success or failure of a reasoning path based on its visualized features, they were able to consistently improve reasoning performance across various models and datasets. The lightweight verifier is more computationally efficient compared to other methods in LLM reasoning. This feature is also a lightweight tool that can be adapted in other domains.

While the current implementation focuses on multiple-choice questions, the researchers envision extending the tool to more open-ended reasoning tasks. This could provide a powerful new lens through which to study the inner workings of these complex systems, ultimately leading to more reliable, understandable, and safe LLMs.