Long-context inference is bottlenecked by KV cache memory footprint, especially for small models

Long-context inference is bottlenecked by the memory footprint of the key-value (KV) cache, especially for small models under tight resource budgets.

http://arxiv.org/abs/2608.27128v1