Why Prompting is Just Math: Schmidhuber on World Models and the Illusion of Chain of Thought
Our read
True artificial intelligence reasoning is not about English-language prompt engineering, but a mathematical protocol where an active controller network injects raw vector queries into the latent layers of a passive world model.
What happened
Deep learning pioneer Jürgen Schmidhuber traces the mathematical lineage of modern LLM prompting back to his 1990 and 2015 architectures. He argues that the industry's obsession with step-by-step planning and board games is a distraction from building hierarchical world models that compress messy, partially observable physical realities.
The brief
The AI industry's obsession with prompt engineering is a marketing cope designed to hide the fact that we are still running on Jürgen Schmidhuber's decades-old architecture while pretending we discovered fire.
Key findings
Prompting is a raw mathematical interface where a controller network injects number vectors to wake up latent representations inside a prediction machine.
Chess is a trivial 8x8 pixel thumbnail of perfect information, whereas the real world is massive, noisy, and mostly hidden from view.
Real-world agents survive by decomposing plans into sparse macro-steps rather than calculating muscle micro-movements.
The sides
- The Failure of Millisecond Planning 00:30
Millisecond-by-millisecond action planning is useless for real-world environments.
Evidence: Humans traveling from New York to Paris decompose plans into sparse macro-steps (taxi, airport, flight) rather than calculating muscle micro-movements.
- Partially Observable World Models 02:40
High-dimensional, partially observable environments force neural networks to construct internal latent representations rather than direct pixel-to-pixel predictions.
Evidence: A world model that attempts to predict every pixel in a video stream fails, forcing it to develop hidden units that model typical conditional expectations of the environment.
- Prompting as Controller-Model Communication 04:30
Actionable intelligence is achieved when a reinforcement learning controller learns to actively prompt the hidden units of a pre-trained world model.
Evidence: Schmidhuber's 2015 Learning to Think paper outlining controller networks injecting query vectors into the latent layers of a world model.
Quotes
“When you go from New York to Paris... you don't plan millisecond by millisecond... You don't say, 'Let me lift my pinky a little bit like this and grab my phone...'”
Jürgen Schmidhuber · 01:03
“A board game like chess... all the information you need about the current state of the world is in an eight-by-eight pixel field... It's a thumbnail image basically.”
Jürgen Schmidhuber · 02:06
“This controller has to learn to inject prompts... into the hidden units of the model network. It has to inject sequences of queries, and the queries are just number vectors.”
Jürgen Schmidhuber · 04:30
Why now
Jürgen Schmidhuber demystifies the current generative AI wave by reframing it through the lens of historical neural network architectures.
The industry's current fascination with prompt engineering and chain of thought prompting is simply a user-friendly wrapper around a fundamental mathematical mechanism where an active controller injects vector queries into a passive world model.
By shifting the focus away from perfect-information toy environments like chess, Schmidhuber explains how neural networks must build compressed, latent representations of our partially observable reality to perform meaningful, hierarchical planning.