Language Model Pretraining

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

A study of premature upper-layer attention specialization during language model pretraining.

jinchang-zhu
•