The Bitter Truth: Is Raw Compute Enough for AI, or Do We Need Smarter Algorithms?

Richard Sutton, the AI researcher behind the famous "bitter lesson," recently stirred the AI community by stating that raw compute alone won't lead to intelligence, according to The Decoder. He believes that AI science is "far from finished" and needs clever new methods beyond simply scaling up existing models.

What happened

In a recent post that sparked widespread discussion, Turing Award winner Richard Sutton argued that raw compute is insufficient for intelligence and that AI needs new methods, The Decoder reports. This stance echoes his famous 2019 essay, "The Bitter Lesson," where he contended that general methods that scale with compute eventually outperform sophisticated human-engineered approaches. However, Sutton now clarifies that while scaling is important, it must be paired with the right ideas, not just brute force.

Sutton has previously criticized the limitations of merely scaling large language models, calling synthetic data a "big mistake" because the real world is infinitely complex and any simulation is only "microscopic," The Decoder notes. He argues that human expertise acts as a bottleneck, preventing true scaling. Instead, Sutton advocates for agents with "world models" that continuously learn from their own experience, arguing that today's language models stop learning after training and the real world is far more complex than internet text can capture. He emphasizes the need for continuous learning without losing old knowledge.

Jerry Tworek, an AI researcher who left OpenAI to explore new approaches, contextualizes Sutton's statement, writing that the bitter lesson still holds, but the algorithm itself becomes more crucial as compute increases, according to The Decoder. Tworek suggests that the temporary success of transformer scaling, combined with a misreading of the bitter lesson, has "reward-hacked the brains of a whole generation of scientists." Jayden Teoh, another AI researcher, adds that a misinterpretation of Sutton's message has led to promising academic ideas being dismissed for not adhering to the bitter lesson's spirit, even if they enabled progress at the time.

This tension between Sutton's call for new methods and Tworek's critique of blind scaling is evident in a growing wave of startups. The Decoder, citing its Frontier Radar #5, tracked 45 new AI labs, including Tworek's Core Automation and Sutton's Oak Lab, that are betting on these gaps, seeking learning algorithms beyond current pretraining and reinforcement learning.

However, not everyone agrees. OpenAI CEO Sam Altman believes LLM scaling is far from exhausted, accusing critics of slowing down research. He points to an OpenAI model that recently disproved a long-standing mathematical conjecture as evidence that data clearly favors further LLM scaling, though he sees world models as useful in specific areas like robotics, The Decoder reports. Anthropic CEO Dario Amodei similarly argues that scaling progress is "a rainbow with no end." The Decoder points out that both Altman and Amodei, as CEOs of companies built on LLM scaling, are inherently biased in its favor.

Why it matters for small companies

This isn't just academic AI squabbling; it's a fundamental debate that will shape the AI tools you'll be using in the next few years. If Sutton and his allies are right, we're moving beyond bigger, dumber LLMs to more sophisticated, continuously learning agents that adapt to your business's unique context. Imagine an AI assistant that actually gets smarter about your specific workflows the more you use it, rather than just giving you a slightly better generic answer with each new model generation. This shift could mean a new wave of highly specialized, adaptive AI tools that truly understand your niche, potentially leveling the playing field against larger competitors who rely on massive, general datasets. The current crop of startups focusing on these "gaps" signals a future with a wider array of specialized, intelligent tools, not just a few monolithic AI giants.

What to do this week

1. Experiment with agentic frameworks (like AutoGen or CrewAI) to see how chaining LLM calls can simulate continuous learning and adapt to specific tasks within your business.
2. Keep an eye on emerging AI tools from the new labs mentioned by The Decoder's Frontier Radar, particularly those focused on continuous learning or world models, as they might offer specialized solutions beyond generic LLMs.
3. Identify specific business processes where current static LLM responses fall short due to a lack of continuous learning or real-world understanding, and consider how a more agentic or adaptive AI tool could address these gaps.

Sources

• The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts — The Decoder