We’ve long equated LLM quality with scale—from GPT-1 (117 million) to GPT-2 (1.5 billion), GPT-3 (175 billion), and GPT-4 (estimated 1.8 trillion). Size was the headline, backed by virtually unlimited budgets at equally large tech companies. But that assumption is starting to break down. Inference compute (the extra tokens and passes at query time) now yields greater performance gains than adding parameters. We’ve entered an era where runtime compute, not model size, is the more efficient path to accuracy.

The Pattern Emerging Across the Board

Simple trick to extend inference computer - just append wait to the generation.

In the past year, a growing body of research has challenged the assumption that performance scales primarily with model size. The evidence points elsewhere. Smaller models, when paired with smarter inference strategies, often match or surpass larger ones under equal compute. On GSM8K and MATH500, a 7B model using selective tree search outperforms a 34B model by spending compute more efficiently, not by knowing more (4).

These results are consistent across domains. For coding, increasing sample count from one to 250 pushes solve rates on SWE-bench Lite from 15.9% to 56%, outperforming any larger model using single-shot inference (3). In math, adding a simple delay before answering leads to a seven-point accuracy gain on AIME24, without changing the model weights (2). These aren’t marginal tricks—they’re repeatable improvements that come from using inference as an active process.

Tools to extend compute such as reranking, tree search, and self-revision are becoming standard tools. While others focus on the opposite: stopping early once the answer is known. Large models often continue generating well past the point of correctness. Teaching them to stop cuts token usage nearly in half, with no accuracy loss (5).

What Does This Actually Mean?

Fingers, apples, distance… sometimes an LLM takes the scenic route just to get to 5.

If this shift feels abstract, it’s not. The implications are practical, and they’re already shaping how leading teams design, deploy, and scale LLM applications. Here’s what that looks like when translated into engineering and product decisions.

1. Inference compute is now your primary optimization surface

For years, model performance meant upgrading to the next size up. But that logic no longer holds. Increasingly, the best returns come not from scaling parameters, but from how you run the model. Whether it’s a few more samples, a re-ranking pass, or a self-revision loop, test-time compute now moves the needle further than adding weights. It’s cheaper, faster to iterate on, and often more effective. A bigger model is no longer the best implementation path, and you instead need a better inference strategy.

2. Adaptive pipelines outperform static ones

LLM applications often treat inference like a fixed routine—one prompt in, one response out, same sampling budget every time. But not all prompts are equal. Some require careful reasoning. Most don’t. Treating them the same is inefficient.

A dynamic pipeline adjusts. It allocates minimal compute to straightforward tasks and scales up only when needed. That might mean triggering additional samples, reranking, or revision passes only when confidence is low. This is how you keep latency low without sacrificing quality—and how smaller models close the gap with far larger ones. Static inference burns tokens indiscriminately. Adaptive pipelines use them with intent.

3. Overthinking isn’t just a metaphor, it’s a real cost

LLMs don’t just get things wrong—they often get them right, inefficiently. A model might solve “2+3=?” in five tokens, then continue generating another hundred just to explain itself. That surplus isn’t helping. It’s waste.

The same pattern appears in more complex tasks: multiple redundant reasoning steps, repetitive justification, and re-deriving answers already reached. Without guardrails, models will burn compute on outputs that don’t meaningfully improve quality.

You need to watch for this. Track how many tokens were spent before the first correct answer appeared. Measure how much new information each reasoning step adds. Use those signals to cut the generation short when the job is done. Overthinking isn’t just a design flaw—it’s a budget issue. Left unchecked, it becomes the silent majority of your inference cost.

Where This Leaves Us

For most teams, scaling model size is no longer the highest leverage move. The returns have slowed, the costs have not. What the research shows is that the real gains now come from how the model is run. Not in how many parameters it has, but in how each token is spent.

This is a shift in mindset. Inference is not an afterthought. It is the part of the system you control the most, and the one with the most headroom. Smarter inference strategies are already outperforming larger models. They’re cheaper, easier to deploy, and available now.

If you're still thinking in terms of which model to upgrade to next, think again.

References

1. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
   Demonstrates that adaptive test-time compute strategies outperform static sampling and often beat scaling model parameters, with compute-optimal policies delivering 4× efficiency gains.

2. s1: Simple Test-Time Scaling
   Introduces a minimal and effective test-time scaling method using budget forcing, showing strong performance on competition-level math with open-sourced models and data.

3. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
   Establishes that repeated sampling yields predictable, power-law improvements in problem-solving tasks, especially when paired with automatic verifiers.

4. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
   Provides a formal framework for understanding how to best allocate fixed inference budgets between model size and reasoning depth, introducing efficient tree-search methods like Rebase.

5. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
   Analyzes inefficiencies in over-reasoning, introduces outcome and process efficiency metrics, and proposes training strategies to reduce redundant generation without sacrificing accuracy.