Transformer models remain resilient to performance drops when low-importance attention heads are removed in 72.5% of tabular dataset experiments

In 72.5\% of experimental examples, the model remains most resilient to performance drops when heads with the lowest importance scores are gradually removed.

http://arxiv.org/abs/2608.27241v1