How Much Runway Do Scaling Laws Have Left? A 2026 Deep Dive

How Much Runway Do Scaling Laws Have Left?

In 2020, OpenAI published the now-famous Scaling Laws paper. The conclusion was clean: as you scale model parameters, data, and compute following a power law, loss drops predictably. The entire industry placed a bet: more parameters means more capability.

Four years later, does that bet still hold?

The Power Law Curve Is Bending

From GPT-3 (175B) to GPT-4 (estimated ~1.8T MoE), parameter count grew roughly 10x. MMLU scores climbed from about 70% to around 87%. But from GPT-4 to today's frontier models, the same compute investment is buying smaller and smaller MMLU gains. This isn't Scaling failing — it's us measuring the wrong thing. MMLU is fill-in-the-blank at heart, and the ceiling for LLMs on fill-in-the-blank is lower than we thought.

Emergent Abilities: Real or Artifact?

The 2022 emergent abilities paper sparked major debate. The claim: past certain parameter thresholds, capabilities appear suddenly. The counter-argument (Schaeffer et al., 2023): this emergence is likely a measurement artifact. Use continuous metrics and capability gains look smooth all along. The practical takeaway: don't wait for the next emergence moment — today's models are already capable enough, the question is how to use them well.

Compute Efficiency: The Real Battlefield

We run a local inference cluster — 680GB unified memory, Qwen3.5-122B. Key insight: for the same 1T tokens of training, different data compositions can produce 10-15% performance differences. Llama 3 70B outperforms GPT-4-turbo on certain tasks not because of more parameters, but because of better data quality control. This is the core shift in Scaling Laws 2.0: from stack scale to stack efficiency.

Inference-Time Compute: A New Scaling Dimension

OpenAI o1, DeepSeek R1, Claude 3.7 Extended Thinking — all exploring the same direction: shift compute from training time to inference time. Our measured data: Qwen3.5-122B with thinking mode enabled vs disabled on complex coding tasks — accuracy jumps from 62% to 81%, at the cost of inference time going from 3s to 25s. Worth it in many scenarios.

The Truth About the Scaling Ceiling

Marginal returns on the 2020 definition of Scaling are diminishing. But inference-time Scaling, data efficiency Scaling, and architectural innovations are far from exhausted. The 2026 landscape: large labs using MoE to compress inference costs; open source building stronger smaller models with better data; inference-time Scaling becoming the mainstream direction.

SFD Lab Notes

Running Qwen3.5-122B locally day-to-day, we've found: raw model size is no longer the decisive factor. Prompt engineering, toggling thinking modes, task decomposition — these usage-layer optimizations now have more impact on real output quality than parameter count alone. Scaling Laws aren't dead. The battlefield just moved.