Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
A study of premature upper-layer attention specialization during language model pretraining.
jinchang-zhu
A study of premature upper-layer attention specialization during language model pretraining.