The orthogonality among principal subspaces of the Fisher information matrix's K-FAC approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks in pre-trained models

a key observation about the Fisher information matrix (FIM) of pre-trained models: the orthogonality among the principal subspaces of its Kronecker-Factored Approximate Curvature (K-FAC) approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks.

http://arxiv.org/abs/2608.27070v1