FF-JEPA: Long-Horizon Planning in World Models with Latent Planners
RSS Workshop on Robot World Models • 2026
FF-JEPA extends JEPA-style world models with an action-free latent planner that predicts subgoals, enabling long-horizon, goal-image-free planning where flat world models collapse.
Joint Embedding Predictive Architectures (JEPAs) have shown promising world modeling capabilities, enabling planning in latent space by optimizing action trajectories with methods like the Cross-Entropy Method (CEM). These methods are, however, too computationally expensive and ineffective for long-horizon planning, and typically require an explicit image of the goal state.
Method
We tackle these limitations by proposing Forward-Forward-JEPA (FF-JEPA), a hierarchical approach leveraging two forward dynamics models. Alongside a standard action-conditioned forward model, we introduce an action-free latent planner that predicts the next subgoal given the current state. This removes the need for goal images and enables long-horizon planning by decomposing complex trajectories into a sequence of tractable, short-term optimization problems.
We evaluate two architectures for the latent planner: a lightweight deterministic transformer, and a diffusion-based planner built on a DiT backbone.
Results
Preliminary results on Push-T show that FF-JEPA overcomes the long-horizon collapse of flat world models:
| Model | Short (t=25) | Long (t=75) | Random Init. |
|---|---|---|---|
| LeWM (baseline) | 94.53% | 3.52% | 0.00% |
| FF-JEPA (Det) | 76.95% | 88.67% | 81.25% |
| FF-JEPA (DM) | 96.09% | 91.80% | 82.42% |
The flat world-model baseline collapses on long-horizon tasks (3.52% at t=75) and fails completely under random initialization. FF-JEPA's diffusion planner reaches 91.80% and 82.42% on those same settings without ever requiring a goal image.
The deterministic planner adds only 9.5M parameters and ~2ms of inference overhead per planning cycle, versus ~927ms for CEM alone, making it a lightweight option for long-horizon tasks where the diffusion planner's extra cost isn't justified.
Citation
@misc{masip2026ffjepa,
title = {FF-JEPA: Long-Horizon Planning in World Models with Latent Planners},
author = {Sergi Masip and Jonathan Swinnen and Yutong Hu and Renaud Detry and Tinne Tuytelaars},
year = {2026},
eprint = {2606.09311},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2606.09311}
}