Masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points across eleven VLMs and five benchmarks, demonstrating their causal importance

Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect

http://arxiv.org/abs/2608.27417v1