AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Models 'Erase' Token History to Learn Words, Revealing an Implicit Vocabulary

Large Language Models (LLMs) like Llama-2 and Llama-3 possess a remarkable ability to understand complex language, but the process by which they turn arbitrary sequences of subword units into meaningful concepts remains a key mystery. New research proposes that models solve this challenge by developing an “implicit vocabulary,” and the mechanism for forming these higher-level concepts leaves a distinctive signature: a rapid “erasure” of token information in the model’s early processing layers.

The paper, penned by researchers at Northeastern University, tackles the “detokenization” problem—the gap between the model’s token vocabulary and human semantic units. LLM tokenizers often break down words and phrases into constituent pieces that hold no semantic meaning on their own. For instance, the word “northeastern” might be split into [_n, ort, he, astern]. Likewise, named entities like “Neil Young” or idioms such as “break a leg” are composed of individual tokens whose meanings cannot predict the composite concept.

To investigate how LLMs fuse these arbitrary pieces, the team used linear probes—diagnostic tools that test what information is retained in a token’s hidden state as it moves through the model’s layers. They tested Llama-2-7b and Llama-3-8b on multi-token words and named entities.

The results showed a pronounced “erasure” effect, specifically on the last token of a sequence.

In the case of the entity “Star Wars,” the hidden state corresponding to the final token, _Wars, quickly loses the ability to predict the identity of the preceding token, Star, within the first few layers. This forgetting pattern is not observed for tokens that are not at the end of a multi-token entity. The researchers hypothesize that this dramatic loss of low-level information is a “footprint” of the model consolidating the sequence into a non-compositional, higher-level lexical representation. The information isn’t truly gone, but it has been abstracted.

Using this observation, the team developed a novel “erasure score” ($\psi$) to identify token sequences that consistently exhibit this consolidation pattern. By running this score across large datasets of natural text, they were able to “read out” segments of the LLM’s hidden, implicit vocabulary.

The results uncovered many sequences that the model treats as single units of meaning. Beyond expected named entities, the system identified non-compositional phrases associated with Thelonious Monk’s music—like “dram.atic,” “sil.ences,” and “tw.ists”—suggesting the model has lexicalized these descriptive units.

This erasure-based method offers a new tool for interpreting the opaque internal workings of LLMs, providing the first systematic attempt to map out the hidden lexicon that enables these models to understand words they were never explicitly taught. The findings suggest that LLMs are not just processing tokens sequentially but actively orchestrating the conversion of arbitrary subword units into meaningful lexical concepts early in the inference process.