Fantastic Pretraining Optimizers and Where to Find Them
Descripción general
Resumen del artículo
This paper benchmarks 11 optimizers for large language model pretraining and finds that while some like Muon and Soap do offer a speedup over AdamW, it is smaller (up to 1.4x) than previously claimed and diminishes as model size increases. Furthermore, they find that optimal hyperparameters vary significantly between optimizers, making comparisons using shared hyperparameters unfair, and early checkpoints can be misleading as optimizer rankings can shift during training.
Explícamelo como si tuviera cinco años
Some new, fancy ways to train AI models work faster than the old way, but the gains are less than hyped and disappear as models get huge.
Posibles conflictos de intereses
The authors acknowledge support from Google, which has a vested interest in efficient large language model training, but this seems appropriately disclosed and does not obviously bias the research.
Limitaciones identificadas
Explicación de la calificación
This is a strong study with rigorous methodology addressing a relevant problem. The hyperparameter tuning, scaling analysis, and identification of misleading evaluation practices are valuable. The limited model size is a notable weakness preventing a 5, but the findings are important for current-scale models and motivate important further research at larger scales.
Conviene saber
Este es el análisis de Starter. Paperzilla Pro verifica cada cita, investiga los antecedentes de los autores y las fuentes de financiación, y utiliza razonamiento avanzado con IA para ofrecer información más exhaustiva.
Explorar Pro →