Abstract: In this talk, I will give an overview about scaling laws, including their application in sample size planning and learning curve extrapolation, with an emphasis on how they have been recently used to optimize the model size (e.g. in Chinchilla). After that, we extend those methods to optimize the full model shape (e.g. depth and width). We demonstrate that scaled-down architectures, trained at their optimal shape for the right amount of compute, are comparable to (or even better than) fully-scaled models.

Bio: Ibrahim Alabdulmohsin is a senior research scientist at Google Deepmind, focusing on deep learning and ethical AI. Previously, he founded and led the Advanced Analytics at Saudi Aramco, managed the company's Enterprise Analytics program, and was a technical lead at its Digital Transformation program. He obtained his M.S. in Electrical Engineering from Stanford University and his Ph.D. in Computer Science from KAUST. More details are at: https://ibomohsin.github.io/