The Orb Wall: What a Moving Object Taught Us About the Limits of Online Recurrence
Keywords: Recurrent Neural Networks, RNN, Backpropagation Through Time, BPTT, Online Machine Learning, Representation Learning, Cognitive Science, Artificial General Intelligence, AGI, Predictive Coding
Science is not a straight line of clean victories. The most valuable moments in research are often the failures—the walls that refuse to fall, forcing you to dismantle your assumptions and rebuild your worldview from scratch.
In the history of Project Halo, that wall was a simple, bouncing red dot: the orb.
In our 12x12 gridworld substrate, we placed an orb that moved with a hidden constant velocity, bouncing off walls in a deterministic pattern. Because the agent only possessed a local 5x5 egocentric window, the orb was only visible when it was close. When it left the window, the agent was blind to it.
To predict when the orb would reappear, the agent had to remember the orb's prior position and velocity, compute its trajectory, and compensate for the agent's own movements. This is the definition of object permanence and temporal representation.
We believed that a recurrent neural network (RNN) with a working memory state would easily learn this dynamic through backpropagation. We were wrong.
The First Attempt: 1-Step Backpropagation Through Time (BPTT)
Our Predictor was an online recurrent neural network trained via gradient descent on its own prediction error. We set the default temporal window to 1-step backpropagation.
When we analyzed the prediction error on the orb channel specifically, the results were flat. The network's Mean Squared Error (MSE) on the orb was approximately $0.040$.
Why is $0.040$ significant? In a 25-cell egocentric window, predicting that no orb exists at all yields an error of exactly $1/25 = 0.040$.
In other words: the recurrent memory had learned nothing. It simply predicted the orb's absence, treating it as random noise.
Deepening the Window: Multi-Step BPTT
We hypothesized that 1-step gradient updates were too short to carry the temporal credit assignment needed to track velocity. We refactored our training engine to implement true multi-step truncated BPTT, sweeping temporal windows of 4, 8, and 16 steps.
We ran the evaluations over 8,000 steps (exp_prediction.py), measuring overall MSE and orb-channel MSE:
| BPTT Window (k) | Overall MSE | Orb-Channel MSE | (Predict-Absent Baseline) |
|---|---|---|---|
| k=1 | $0.0068$ | $0.0402$ | $0.0400$ |
| k=4 | $0.0106$ | $0.0402$ | $0.0400$ |
| k=8 | $0.0135$ | $0.0394$ | $0.0400$ |
| k=16 | $0.0377$ | $0.0386$ | $0.0400$ |
The results were a stark negation: * Even at $k=16$, the orb-channel error barely budged ($0.0402 \rightarrow 0.0386$), meaning the model was still essentially predicting the orb was absent. * Meanwhile, deepening the BPTT window degraded the overall prediction error ($0.0068 \rightarrow 0.0377$). The gradients backpropagating through 16 steps of noisy online weights corrupted the stable, static terrain representations the network had previously mastered.
Frame-Stacking: The Shortcut Fails
We tried one last classic reinforcement learning shortcut: frame-stacking. We fed the previous and current frames directly into the sensory input, bypassing the need for recurrent state integration entirely.
| Configuration | Overall MSE | Orb-Channel MSE | (Predict-Absent Baseline) |
|---|---|---|---|
| Single Frame | $0.0068$ | $0.0402$ | $0.0400$ |
| Frame-Stacked | $0.0168$ | $0.0386$ | $0.0400$ |
Like BPTT, frame-stacking failed to crack the orb and doubled the overall prediction error.
Why did it fail? The Diagnostics
Before giving up, we wrote a diagnostic tool (diag_orb.py) to dissect the statistics of the task. We found three coupled causes for the failure:
- Sparsity: The orb was only in view $15\%$ of the time. The training signal was incredibly sparse.
- Egocentric Entanglement: Because the visual input was egocentric, the orb's apparent motion was a complex mixture of the orb's actual velocity and the agent's own step actions.
- The Online Regime: Unlike offline training where a model sees the same trajectories millions of times across epochs, our agent is learning online in a single pass. It only got a few dozen steps of orb-view experience per life before starving or dying. A small recurrent net cannot disentangle complex ego-motion from sparse online streams.
The Pivot: The Developmental Ladder
Instead of trying to patch the RNN with more hyperparameters, we chose to be boundary-honest. We accepted that tracking the moving orb in this toy's sparse, online, egocentric regime was a structural limit of our simple network architecture.
This failure changed the course of Project Halo. It taught us that we were asking our model to solve complex temporal integration before we had solved abstraction and spatial mapping.
We froze the v1 RNN and pivoted. We wrote ARCHITECTURE.md and defined a systematic developmental ladder. We resolved to build the cognitive structures one step at a time: transfer, version spaces, allocentric maps, and MDL libraries. We would earn our way up to temporal dynamics.
In the next Volume, we climb the ladder. In Volume 3, Episode 6, we look at the reframe: how we defined our developmental curriculum and proved cross-world transfer for the first time.