Optimal model size depends strongly on both the data budget and the downstream target task

We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target.

http://arxiv.org/abs/2608.26973v1