Claims Browser

Transformer models remain resilient to performance drops when low-importance attention heads are removed in 72.5% of tabular dataset experiments

"In 72.5\% of experimental examples, the model remains most resilient to performance drops when heads with the lowest importance scores are gradually removed."

Removing the most important attention head first causes the greatest reduction in classification performance

"removing the most important attention head first results in the greatest reduction in classification performance"

Important attention heads are scattered across layers without consistent layer-specific patterns in tabular transformers

"A closer look at individual head importance scores across six attention layers reveals that important heads are scattered across layers, with no consistent layer-specific trends."

Attention head importance varies considerably across different tabular datasets, unlike in image and language domains

"In contrast to the image and language domains, the importance of individual attention heads varies considerably across tabular datasets with different schemas and feature spaces."

Head importance scoring can improve efficiency and reduce redundancy in transformer architectures

"The proposed importance score can improve efficiency and redundancy within transformer architectures."

Circuit Condensation can reduce mechanistic interpretability circuits by 8.1x on average and up to 316x compared to frozen baselines across four behaviors and eight models

"Across four behaviors and eight models, condensed circuits are smaller than the strongest frozen baseline in 30 of 32 settings, by $8.1\times$ on average and up to $316\times$."

Weight updates during Circuit Condensation, rather than search alone, drive the reduction in circuit size

"Repeating the search without weight updates produces larger circuits in 29 of 32 settings, showing that weight updates, rather than search alone, drive the reduction."

Circuit Condensation produces more focused circuits with documented roles: 24 heads with 17 documented roles versus 61 heads with 36 undocumented roles for frozen circuits on indirect object identification

"On indirect object identification, condensation isolates 24 heads, 17 of them with documented roles, against 61 heads and 36 undocumented ones for the matched frozen circuit: a sufficient sub-circuit of the published mechanism rather than a reconstruction of it."

Edges in condensed circuits have dependencies and their effects cannot be understood independently

"Pair ablations expose dependencies between edges, showing that their effects cannot be understood independently."

Testing subsets of 19 circuits found 11 that cannot be reduced further and revealed removable edges in the rest

"Testing every subset of 19 circuits finds 11 that cannot be reduced and reveals removable edges in the rest."

Latent chain-of-thought models improve compactness by moving intermediate reasoning from emitted text into continuous states, but this hides the causal object.

"Latent chain-of-thought models move intermediate reasoning from emitted text into continuous states, improving compactness but hiding the causal object."

SCIT (Suffix Cache Interchange Test) can identify which transformer object carries counterfactual computation through causal protocols that construct exact source-recipient counterfactuals and patch cache segments.

"We introduce SCIT, the Suffix Cache Interchange Test, a causal protocol that constructs exact source-recipient counterfactuals, patches declared cache segments, and identifies which transformer object carries the counterfactual computation."

In CODI-GPT2 and Sim-CoT-style GPT-2 models, counterfactual arithmetic transfers primarily through value-cache suffix trajectories rather than through hidden states, keys, reusable answer slots, or single-token triggers.

"On CODI-GPT2 and a Sim-CoT-style GPT-2 reproduction, counterfactual arithmetic transfers primarily through value-cache suffix trajectories rather than hidden states, keys, reusable answer slots, or single-token triggers."

SCIT reveals carrier-regime shifts where arithmetic-like GPT-2/1B cells preserve latent-tail value/KV transfer, while competent 8B and repaired non-arithmetic cells route through prompt-prefix or full-cache K/V instead.

"SCIT reveals carrier-regime shifts: arithmetic-like GPT-2/1B cells preserve latent-tail value/KV transfer, whereas competent 8B and repaired non-arithmetic cells route through prompt-prefix or full-cache K/V"

SCIT contributes a cache-level diagnostic and a competence-gated carrier map rather than a universal latent-tail claim, showing that mechanism behavior is checkpoint-specific.

"SCIT therefore contributes a cache-level diagnostic, a checkpoint-specific GPT-2 arithmetic mechanism, and a competence-gated carrier map rather than a universal latent-tail claim."

Hidden coordinates in language models are not uniquely determined by the input-output function, making representation measurements non-invariant to function-preserving basis changes

"Hidden coordinates are not uniquely determined by a language model's input--output function, so representation-derived measurements should be invariant to function-preserving changes of basis."

Column-permutation parallel analysis violates function-preserving reparameterization invariance because its reference distribution and component count can change while the model function remains fixed

"column-permutation parallel analysis violates function-preserving reparameterization invariance because its reference distribution and selected component count can change while the model function and observed covariance spectrum remain fixed"

Data-internal reference procedures face fundamental limitations: they cannot simultaneously preserve coordinate marginals, remain orthogonally equivariant, and remove cross-coordinate covariance

"a data-internal reference procedure cannot simultaneously preserve every coordinate marginal, remain orthogonally equivariant, and remove cross-coordinate covariance"