AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Hybrid AI Models Build Massive Activations Right Before Full Attention Layers

Hybrid linear attention language models—systems that alternate cheap recurrent layers with a smaller number of standard full-attention layers—have become a common design for handling long contexts efficiently. But how that interleaving reshapes the network’s internal computations has been largely unexplored. A new study examines massive activations, a sparse set of hidden-state values that can exceed typical activations by orders of magnitude, and finds a clear architectural pattern: they surge immediately before full-attention layers and, when full attention is dense enough, persist through the intervening linear layers as plateaus.

The researchers call these morphologies pre-attention spikes (PAS) and inter-spike plateaus (ISP). Their method was to trace the maximum absolute activation of a fixed token across model depth, using “attention sink” tokens—positions that receive a disproportionate share of attention—as stable anchors. This was necessary because pure magnitude-based tracking proved unstable in hybrid models.

In a representative 1.3B-parameter Gated DeltaNet model with a 12:1 hybridization ratio, meaning one full-attention layer for every 12 sequence-mixing layers, the first token of the sentence “Summer is warm. Winter is cold.” behaved as an attention sink. Just before the full-attention layer, the network wrote an extreme value into the residual stream; during full attention, that token attracted disproportionate attention; afterward, an opposite-signed update at the same coordinate canceled the outlier, causing the spike to dissipate. The authors interpret this as a localized “write–sink–cancel” lifecycle, with ISP arising when cancellation is delayed across multiple layers.

The pattern recurred widely. Across five linear-attention backbones—RetNet, HGRN, GLA, DeltaNet, and Gated DeltaNet—at 340M and 1.3B scales, the alignment between sink tokens and pre-attention spikes was between 99.4% and 100% under a 12:1 configuration. The same organization appeared in open-source hybrids Qwen3.5, Kimi Linear, Nemotron-H, and Zamba2, spanning 1.2B to 397B total parameters. Inter-spike retention also rose with full-attention density: in 1.3B Gated DeltaNet models, the retention score increased from 18.4% at 12:1 to 26.6% at 6:1, then to 77.8% at 3:1, where full attention is much denser.

Controlled pretraining of GDN-based models showed both morphologies emerge early in training and strengthen as optimization proceeds. Gating experiments produced an asymmetry: adding output gates to full-attention layers sharply reduced the absolute magnitude of spikes and plateaus without eliminating their layerwise organization, while removing the native gates from GDN layers caused only modest amplification. Qwen3.5, which natively uses output-gated full attention, still exhibits both morphologies.

The authors caution that the mechanism regulating cancellation timing remains unresolved, and whether transient PAS and persistent ISP serve distinct computational roles—or are largely byproducts of the hybrid architecture—is still an open question. The study is observational and interpretive on this point, not a demonstration of causal function.

Still, the work gives researchers a new way to read hybrid architectures: massive activations are not random noise but track where full attention sits in the network. That could inform future model design and low-precision inference, even if the precise computational role of these spikes is not yet settled.