Generalizing Caution: How We Taught AI to Recognize Danger in Unseen Worlds
Keywords: Generalization, Model-Based Caution, Active Inference, World Model, Artificial General Intelligence, AGI, Neuro-inspired AI, Machine Learning, Cognitive Science
Imagine walking through a forest. You spot a snake you have never seen before—it has a unique pattern and a vibrant color you don't recognize. Even though you have zero specific memories of this exact snake, you do not step on it. You step back. Your brain generalizes the concept of "danger" based on features: the shape, the movement, the context.
You don't need to get bitten by every individual species of snake to learn caution.
In Volume 1, our homeostatic agent learned caution using an exact-match memory. If it stepped on a red hazard cell, it recorded that exact visual configuration as dangerous. But this memory was brittle. If we dropped the agent into a brand-new grid layout, its memories were useless. It had to walk onto hazards and get hurt all over again to learn caution in the new world.
For a system to scale toward AGI, caution must generalize. The agent must learn the pattern of danger, not just memorize the coordinates of past injuries.
To solve this, we upgraded our online recurrent world model with a shared harm-prediction head.
The Shared Harm Head
We modified our neural Predictor (brain.py). Instead of just predicting the next visual frame, we added a second output head dedicated to predicting integrity damage (harm).
Crucially, this harm head shared its hidden representation layer with the main sensory predictor:
[Senses + Action + Hidden State] ──> [Shared Hidden Layer] ──> [Predicted Senses]
│
└──> [Predicted Harm]
Because the hidden layer was shared, the features learned to predict visual movement and layout structures were also used to predict harm. This meant the agent's caution was no longer a flat list of memorized states; it was a model-based generalization. The agent could evaluate a state it had never seen before, run it through the predictor, and see if the network projected harm.
The Over-Generalization Trap: False Fears
When we first booted this pure model-based caution agent, we encountered a severe cognitive regression: false fears.
Neural networks are prone to over-generalization, especially in online, low-data regimes. The agent stepped on a hazard cell next to a wall. The network generalized this as: "all states next to walls are dangerous."
Since our gridworld was bounded by walls, the agent became terrified of the entire perimeter. It retreated to the center of the grid, dithering back and forth, refusing to explore safe corridors because it predicted harm everywhere. It was trapped by its own generalized anxiety, eventually starving to death.
This was a classic machine learning failure, but we documented it as an honest research finding. Pure model-based generalization is too loose; it creates false fears that paralyze the agent.
The Solution: The Hybrid Caution Engine
To resolve this, we built a hybrid caution engine that balances the precision of memory with the flexibility of generalization:
- Precise Memory Lookup: If the agent has encountered the exact situation before, it uses its precise experience memory (preventing false fears on safe, familiar cells).
- Model-Based Fallback: If the situation is novel, the agent queries the Predictor's harm head to generalize caution to the unseen state.
This hybrid approach ensures the agent remains optimal in familiar terrain while staying safe in novel environments.
The Gated Results: Transfer to Unseen Worlds
We tested this hybrid caution engine by training our agent in one world, freezing its mind (disabling further learning), and dropping it into a completely unseen random layout (exp_generalize_caution.py). We measured the total hazard hits during the run:
| Caution Configuration | Hits in Familiar Training World | Hits in Brand-New Unseen World |
|---|---|---|
| Memorized Caution (v0.3) | 39 | $111$ (Failed - walks onto hazards freely) |
| Pure Model-Based Caution | 27 | $43$ (Generalizes, but suffers from false fears) |
| Hybrid Caution Engine | 44 | 10 (Passed - optimal safety) |
The data was decisive: * Memorized Caution collapsed in the new world, suffering $111$ hazard hits because it couldn't recognize danger in the new layout. * Hybrid Caution kept the agent safe, holding hits to just $10$ in a world it had never seen before—a $11\times$ improvement over the memorized baseline.
We had successfully built a caution drive that generalizes across environments, moving us closer to a transferable, independent mind.
The Road Ahead
With caution generalized, our agent was safe. But the moving orb remained an unlearned mystery. In the next chapter, we will look at Episode 5: our direct, head-on attempt to track the moving orb using deep recurrence, and the honest negatives that reshaped our entire roadmap.