Tight clusters can lead to larger errors than random clusters in sparse attention, but when excluded clusters are compensated through their centroids, the error shrinks with tighter clusters

We also derive an expression for the output error in sparse attention, that explains the counterintuitive experimental finding that tight clusters can lead to larger errors than random clusters. We then derive the error when excluded clusters are compensated through their centroids, and show that this error shrinks with tighter clusters.

http://arxiv.org/abs/2608.26965v1