Expected Free Energy: How We Replaced the Hand-Coded Controller with Pure Prediction
Keywords: Expected Free Energy, Active Inference, Predictive Coding, World Model, Artificial General Intelligence, AGI, Neuro-inspired AI, Machine Learning, Cognitive Science
If you walk into a dark, unfamiliar room, your brain does not wait for a reward signal to act. It does not calculate a point-score for moving your left foot. Instead, you feel a subtle tension—an epistemic drive. You reach out your hand to feel for a light switch or a wall. You want to resolve the uncertainty of the darkness. At the same time, you tread carefully to avoid stubbing your toe or falling over an unseen object.
You are balancing two forces: the desire to resolve uncertainty (exploration) and the need to avoid harm (exploitation).
In traditional artificial intelligence, this balance is achieved by writing a complex reward function with hand-tuned weights (e.g., $Reward = 2.0 \times Novelty - 1.5 \times Hazard$). But this is an engineering cheat. If the environment changes, those weights collapse, and the agent's behavior breaks.
We wanted to know: Can we delete the hand-coded controller entirely? Can we let these two drives emerge mathematically from a single prediction-error minimization process?
To find out, we entered Volume 2: Prediction & Active Inference.
The Core Concept: Expected Free Energy (EFE)
Active Inference, a theory popularized by neuroscientist Karl Friston, suggests that all biological action is driven by a single objective: minimizing free energy (or prediction error).
When applied to decision-making, this becomes Expected Free Energy (G). The agent imagines the future consequences of its actions and evaluates each path based on two terms:
$$G(action) = Epistemic\ Value - Pragmatic\ Value$$
- Epistemic Value (Curiosity): The expected reduction in uncertainty. The agent is drawn to actions that yield the most learnable information (reducing prediction error).
- Pragmatic Value (Caution): The expected alignment with survival. The agent avoids actions that are predicted to lead to self-damaging or lethal states.
The beauty of this equation is that it requires no hand-tuned weights. Epistemic value and pragmatic value are measured in the same unit: information bits. They naturally balance each other out based on the agent's current state of knowledge.
Building the Recurrent World Model
To implement Expected Free Energy, our agent needed a brain that could imagine the future. We built a from-scratch, online-recurrent neural network: the Predictor (brain.py).
The Predictor took the agent's current visual observation, its working memory hidden state, and a proposed action, and predicted the next visual frame and the next viability state.
[Senses + Action + Hidden State] ──> [Predictor (RNN)] ──> [Predicted Senses + Predicted Harm]
At every step, the agent used its world model to imagine all 5 possible actions (up, down, left, right, stay). For each action, it calculated the Expected Free Energy: 1. It measured the entropy of the predicted visual frame (how uncertain it was about the outcome). This was the epistemic value. 2. It measured the probability of predicted harm. This was the pragmatic value.
It selected the action that minimized Expected Free Energy. There were no hand-coded scores, no custom weights, and no heuristics. The model decided.
The Gated Results: The Controller Dies
We ran comparative evaluations (exp_prediction.py) over 4 seeds, comparing our EFE model-derived policy against the old hand-coded controller.
| Controller Type | Final Prediction Error | Safe Terrain Coverage | Hazard Hits |
|---|---|---|---|
| Hand-Coded Controller | $0.0274$ | $69\%$ | $13.8$ |
| Model-Derived (EFE) | $0.0186$ | $59\%$ | $12.8$ |
The results validated our active inference thesis: * Deepened Understanding: The EFE agent achieved a $32\%$ lower prediction error than the hand-coded controller. Because it was driven to resolve uncertainty, it systematically explored the world's dynamics, building a far more accurate world model. * Comparable Safety: With no hand-tuned weights, the agent was naturally cautious, suffering fewer hazard hits ($12.8$) than the hand-tuned baseline. * The Coverage Trade-Off: The EFE agent covered slightly less terrain ($59\%$ vs. $69\%$). This was an honest trade-off: the agent was drawn to resolve learnable uncertainty and avoid danger, rather than blindly touring every cell.
The Honest Limit: The Noisy-TV Trap
While Expected Free Energy worked, it exposed a classic cognitive bottleneck: the noisy-TV problem.
Because our epistemic value was calculated from the entropy of predictions, the agent was drawn to states that were highly unpredictable. In our world, the moving orb was highly unpredictable. The agent would find the orb and stare at it, trapped by the irreducible randomness of its motion, ignoring its foraging duties until it starved.
To solve this, we needed to refine our curiosity. We needed to chase learning progress—surprise that is actively shrinking—rather than raw surprise.
In the next chapter, we will walk through Episode 4: how we built a shared harm-prediction head to generalize caution to unseen worlds.