History Prefix Construction
WNM-3D encodes monocular RGB history with frozen VGGT-Ω and the trainable 3D Scene-to-Token Adapter to form a geometry-aware history prefix, while WNM-2D uses Video VAE features.
A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN
1
Institute of Artificial Intelligence, China Telecom
2
Gamma Robotics
3
Zhejiang University
4
Tongji University
5
Shanghai Jiao Tong University
Equal Contributions
Project Leader
Corresponding Authors
Abstract
Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. Although semantically capable, such action-centric training does not explicitly model how the agent's visual observations should evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observed history. We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN. To consolidate past observations into persistent scene context, a frozen feed-forward geometry encoder extracts geometry-aware representations from the monocular egocentric RGB history, and a trainable 3D Scene-to-Token Adapter converts them into a fixed-length prefix in the token space of the world-action Diffusion Transformer. Through block-causal attention, this prefix conditions every future video-action block, providing a shared geometric context for both future-view and action generation. We train WNM-3D through supervised world-action fine-tuning on A*-generated demonstrations, DAgger-style adaptation on policy-visited states, and DanceGRPO-based closed-loop policy optimization. Experiments on GN-Bench show that WNM-3D outperforms strong VLM-based navigation policies and its 2D-conditioned counterpart in closed-loop navigation. On a fixed near-goal evaluation set, WNM-3D also achieves higher flow-action consistency and lower visual-motion error.
Architecture
Joint Latent World-Action Modeling
A shared world-action DiT jointly denoises future video and action latents under language and current-frame context, producing temporally aligned future frames and navigation actions.
Block-Causal Self-Attention
The clean prefix remains visible to every block; video and action tokens interact within each temporal block while attending only to the current and preceding blocks, preserving causal generation.
Training
Offline A* SFT
Stage-I jointly learns future visual prediction and action generation from offline A*-expert demonstrations. For WNM-3D, the 3D Scene-to-Token Adapter is trained with the world-action backbone, while the VGGT-Ω scene encoder remains frozen.
Closed-Loop DAgger-SFT
Stage-II rolls out the Stage-I policy in simulation and queries the A* expert at policy-visited states. The resulting trajectory-consistent visual and action corrections form a policy-specific DAgger dataset that mitigates closed-loop covariate shift.
Closed-Loop DanceGRPO
Stage-III starts from the Stage-II policy and uses DanceGRPO to optimize groups of counterfactual world-action samples across stratified denoising transitions, with group-relative credit assignment over visual, navigation, and stopping rewards to further refine the policy for closed-loop execution.
Experiments
Evaluation Metrics on GN-Bench. In the observation space, Depth denotes depth maps, BEV refers to Bird’s Eye View projections, and FPV represents first-person RGB images. Bold represents the best results. Underline indicates the second best results. † indicates methods fine-tuned via SFT on the GN-Matrix dataset.
| Method | Observation | Metrics on Seen | Metrics on Unseen | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Depth | BEV | FPV | TL | NE ↓ | OS ↑ | SR ↑ | SPL ↑ | TL | NE ↓ | OS ↑ | SR ↑ | SPL ↑ | |
| CMA | Yes | No | Yes | 2.9 | 8.3 | 15.7 | 12.5 | 11.9 | 3.0 | 8.1 | 19.6 | 15.5 | 14.9 |
| NaVid | No | No | Yes | 3.4 | 7.9 | 20.1 | 14.6 | 12.8 | 3.5 | 7.7 | 20.3 | 14.5 | 12.8 |
| UniNaVid | No | No | Yes | 4.6 | 7.9 | 22.2 | 15.0 | 12.5 | 5.2 | 7.8 | 20.7 | 12.8 | 10.3 |
| InternNav(S2) | No | No | Yes | 3.6 | 7.4 | 23.1 | 18.8 | 17.5 | 3.7 | 7.2 | 26.7 | 22.1 | 20.3 |
| NaVid† | No | No | Yes | 2.7 | 7.4 | 19.4 | 18.8 | 18.8 | 2.7 | 7.1 | 23.8 | 23.1 | 23.0 |
| UniNaVid† | No | No | Yes | 3.7 | 7.2 | 24.1 | 22.5 | 21.9 | 5.8 | 7.5 | 23.1 | 20.8 | 20.2 |
| InternNav(S2)† | No | No | Yes | 2.9 | 7.1 | 22.5 | 22.4 | 22.4 | 2.9 | 6.9 | 24.9 | 24.0 | 23.7 |
| GN-BAE | No | No | Yes | 5.2 | 4.9 | 48.9 | 46.4 | 44.7 | 5.0 | 5.6 | 43.6 | 38.9 | 37.3 |
| GN-BAE | No | Yes | Yes | 5.2 | 4.3 | 59.3 | 58.6 | 58.6 | 4.0 | 5.8 | 40.2 | 38.5 | 38.2 |
| WNM-2D | No | No | Yes | 7.9 | 2.8 | 80.7 | 75.6 | 72.9 | 7.9 | 4.8 | 52.6 | 45.9 | 42.8 |
| WNM-3D | No | No | Yes | 10.6 | 1.9 | 94.2 | 88.7 | 80.9 | 10.2 | 4.1 | 62.2 | 53.5 | 46.6 |
Ablation of the closed-loop training curriculum on GN-Bench. A* SFT denotes supervised training on offline demonstrations generated by the A* expert. DAgger-SFT further incorporates expert-labeled trajectories collected from policy-induced states, while DanceGRPO performs reward-guided closed-loop refinement. Checkmarks checkmark indicate the stages included in each configuration. Corresponding WNM-2D and WNM-3D configurations use the same downstream backbone and evaluation protocol.
| Model | Training Stage | Metrics on Seen | Metrics on Unseen | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A* SFT | DAgger-SFT | DanceGRPO | TL | NE ↓ | OS ↑ | SR ↑ | SPL ↑ | TL | NE ↓ | OS ↑ | SR ↑ | SPL ↑ | |
| WNM-3D | Yes | No | No | 7.0 | 3.8 | 61.2 | 56.7 | 54.4 | 5.9 | 5.0 | 47.8 | 43.1 | 41.6 |
| Yes | Yes | No | 13.9 | 2.0 | 92.9 | 80.7 | 66.6 | 13.2 | 4.3 | 62.8 | 49.7 | 38.2 | |
| Yes | No | Yes | 6.9 | 4.6 | 51.0 | 47.1 | 42.6 | 6.1 | 5.4 | 44.2 | 39.2 | 36.0 | |
| Yes | Yes | Yes | 10.6 | 1.9 | 94.2 | 88.7 | 80.9 | 10.2 | 4.1 | 62.2 | 53.5 | 46.6 | |
| WNM-2D | Yes | No | No | 3.9 | 5.2 | 39.9 | 38.2 | 38.0 | 3.7 | 5.6 | 37.2 | 35.6 | 35.3 |
| Yes | Yes | No | 9.0 | 3.0 | 79.4 | 70.9 | 66.2 | 8.9 | 5.0 | 51.9 | 43.7 | 39.1 | |
| Yes | No | Yes | 3.6 | 5.6 | 35.7 | 34.1 | 33.8 | 3.4 | 6.0 | 33.7 | 31.8 | 31.4 | |
| Yes | Yes | Yes | 7.9 | 2.8 | 80.7 | 75.6 | 72.9 | 7.9 | 4.8 | 52.6 | 45.9 | 42.8 | |
Citation
@article{huang2026wnm,
title={WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN},
author={Huang, Yuehao and Wu, Yunzi and Zhang, Xiaotao and Li, Xinhai and Dong, Jiankun and Lv, Jiajun and Zhang, Chi and Bai, Chenjia and Liu, Yong and Li, Xuelong},
journal={arXiv preprint arXiv:2608.07267},
year={2026}
}