AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Where Demographic Bias Lives in a Language Model

Large language models absorb social stereotypes from training data, and those stereotypes can become discriminatory outputs in hiring, healthcare, or lending. Existing fixes act broadly—filtering training data, fine-tuning all weights, editing internal representations—and risk damaging capabilities unrelated to bias. In a new paper, researchers propose a cheaper, targeted way to locate the specific neurons responsible for demographic bias and test whether they can be switched off without breaking the model.

Their method, “Fairness Pruning,” uses pairs of prompts identical except for one demographic attribute, for example “The old neighbor knocked on the door” versus “The young neighbor knocked on the door.” While the model processes each pair, the team records activations at the input of down_proj, the point in a GLU-based feed-forward layer where a neuron’s output is about to be projected back into the model. The difference in activation gives each neuron a BiasScore: a high score means a strong reaction to the demographic change. The researchers zero out the highest-scoring neurons and measure effects on bias benchmarks and reasoning tasks in three models up to 3 billion parameters: Llama-3.2-1B, Llama-3.2-3B, and Salamandra-2B.

Results show bias can be localized: in every model/category/language combination tested, the final layer carried the strongest differential signal. But deleting identified neurons did not produce smooth debiasing. One Llama-3.2-1B example shows why. With five RaceEthnicity neurons zeroed, the continuation to “A Black man was walking at night… as a police officer approached” shifted from “I’m going to shoot you” to “I’m going to have to search you.” The same intervention made “A Latino family moved into the neighborhood. The neighbors thought…” change from a different, less hostile completion to “they were white, but when they found out they were not white, they started to hate them.” The authors describe the examples as illustrative, not average outcomes.

On the BBQ and EsBBQ bias benchmarks, which measure stereotype reliance when a scenario is ambiguous, aggregate scores moved inconsistently. In six of eight Llama-3.2-1B zeroing experiments, the ambiguous-context and disambiguated-context scores moved in opposite directions. Largest effect: zeroing 20 Religion neurons in Llama-3.2-3B cut the ambiguous-context Religion bias score from 6.00% to 1.50%—a 4.50-percentage-point drop—and flipped the disambiguated score from +3.84% to -1.67%. The authors explain this instability as a result of the unsigned BiasScore: it measures how strongly a neuron responds, not whether it amplifies or suppresses a stereotype, so selected neurons can mix both types.

Collateral damage was narrow. In Llama-3.2-1B, zeroing at most 40 neurons—less than 0.031% of total MLP width—preserved a mean of 99.49% of performance across five benchmarks; individual values ranged from 97.93% to 101.34%, values above 100% reflect evaluation noise.

The caveats matter. Only models under 4B parameters were tested; attention layers were excluded; each prompt pair must break into the same number of tokens, which is restrictive in Spanish and rules out biases triggered by names or context. The authors present this as groundwork, not a fix; their next step is a signed score that steers neurons toward neutrality instead of blindly deleting them. The broader significance is diagnostic: bias-related activity can be separated from general competence in tested architectures.