Scaling laws

Are scaling laws over? A report from the Information announced that OpenAI is seeing diminishing returns from scaling up the next GPT models.

📊 What are scaling laws? These are empiric laws that say “Every time you increase compute spent in training 10-fold, your LLM’s performance will go up by a predictable tick”. Of course, they apply only if you train your model with the right methods.

The image below illustrates it: they’re from a paper by Google, “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation”, and they show how quality and instruction following of models improve when you scale the model up (which is equivalent to scaling up the compute spent in training).

➡️ These scaling laws have immense impact: they triggered the largest gold rush ever, with companies pouring billions into scaling up theiur training. Microsoft and OpenAI spent 100B into their “Startgate” mega training cluster, due to start running in 2028.

🤔 So, what about these reports of scaling laws slowing down?

If they are true, they would mean a gigantic paradigm shift, as the hundreds of billions poured by AI companies into scaling could be a dead-end. ⛔️

But I doubt it: until the most recent publications, scaling laws showed no signs of weakness, and the researchers at the higher end of the scale-up seems to imply the scaling up continues.

Wait and see!