JEPA world model encoders with fixed-size Vision Transformer encoders are over-provisioned for simple tasks and under-provisioned for complex ones, with significant redundancy across attention heads

Joint-Embedding Predictive Architectures (JEPAs) for world modeling typically employ fixed-size Vision Transformer encoders that are over-provisioned for simple tasks and under-provisioned for complex ones, with significant redundancy across attention heads

http://arxiv.org/abs/2608.27367v1