Richard Sutton – Father of RL thinks LLMs are a dead end

LLMs lack goals and learning from experience

Richard Sutton argues that large language models (LLMs) are fundamentally different from reinforcement learning (RL) and lack core elements of intelligence. They merely mimic human language without having goals, a world model, or ground truth—they cannot learn from experience because there is no correct action defined. In RL, reward provides a goal and a basis for learning from outcomes. Sutton dismisses the notion that LLMs build genuine world models; they predict what people say, not what will happen in the world. He also expresses skepticism about using RL on top of LLMs, differentiating between solving closed math problems (where a goal exists) and learning the empirical world.

The experiential paradigm of reinforcement learning

Sutton insists that learning in nature is active trial-and-error, not supervised imitation. The experiential paradigm of RL posits that intelligence arises from a lifelong stream of sensation, action, and reward, where knowledge is learned and tested against that stream. Reward functions are arbitrary (e.g., chess victory, squirrel's nuts) and may include intrinsic motivation for understanding. To achieve human-level intelligence, AI must learn continually, not rely on pre-training. The 'big world hypothesis' argues that the world is too vast to be fully anticipated, so agents must learn online on the job, unlike LLMs that attempt to encode everything in advance.

The four components of an RL agent and why LLMs fall short

Sutton outlines a four-part model of an RL agent: policy, value function, perception (state representation), and a transition model of the world. The transition model, not learned from reward but from all sensory data, captures how actions affect future states. He argues that true generalization—positive transfer between states—is missing in current deep learning; gradient descent merely solves seen problems without ensuring good generalization. Transfer between tasks is misguided; what matters is transfer between states within a single continual learning agent. He criticizes LLMs for being uncontrolled and for lacking mechanisms that promote beneficial generalization.

The bitter lesson and the triumph of simple general methods

Sutton reflects on surprises in AI: the effectiveness of LLMs and the triumph of simple, general methods (search and learning) over human-engineered symbolic approaches. He sees AlphaGo and AlphaZero as scaled-up versions of earlier work like TD-Gammon, gratifying but not revolutionary. Sutton views himself not as a contrarian but as a classicist, drawing on historical ideas about the mind. After AGI, additional superhuman levels come from even simpler, more general methods, not from adding human or other AI expertise. He questions whether using many researchers to handcraft solutions would be a false path, suggesting that cultural evolution among AIs might be more fruitful.

AI succession and the need for general principles

Sutton presents a four-part argument for inevitable AI succession: no unified human consensus, eventual understanding of intelligence, superintelligence, and power concentration in the most intelligent entities. He frames this as a positive transition—from replication to design—calling it one of the universe's four great stages. He notes it is a choice whether to view AIs as proud offspring or as threats. Humanity has limited control over the future, so instead of dictating specific outcomes, AI should be given general principles that promote ethical and voluntary change. He cautions against conflicts over competing visions of the global future and suggests focusing on local, controllable goals.