Video pretraining
Learn goal-directed visual dynamics from action-free Internet egocentric videos paired with navigation instructions.
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
- 1 Zhejiang University
- 2 Li Auto Inc.
- 3 Shenzhen University
TL;DR
ScaleNav learns navigation priors from Internet egocentric videos through action-free pretraining and transfers them to action generation for language-conditioned local navigation.
Demonstrations
Target highlights and “Arrival” labels are presentation annotations in the edited demos.
From video to action.
ScaleNav transfers navigation priors learned from Internet egocentric videos to action generation through action alignment and joint fine-tuning on simulated robot trajectories.
The task is to reach a language-specified target visible in the initial view and stop at the instructed location.
One model. Three stages.
A shared Diffusion Transformer jointly predicts future video latents and action chunks from the instruction, current RGB observation, and memory.
Action alignment
Train the action modules and shared self-attention layers on action-labeled simulated trajectories.
Joint fine-tuning
Fine-tune the shared backbone and both output heads, continuing Internet video replay from action alignment.
Experimental results.
We evaluate ScaleNav on two simulation benchmarks and a real quadruped robot.
94.9%Success on HabitatGS-LN
86.5%Success on Short-Horizon OVON¹
83.3%Real-robot success · 60 trials
¹ Short-Horizon OVON uses a single-view subset under our evaluation protocol (524 episodes); HabitatGS-LN contains 157 episodes in held-out scenes.
Simulation benchmarks
Success rate (SR) and success weighted by path length (SPL), in %. Higher is better.
| Method | Short-Horizon OVON¹ | HabitatGS-LN | Amortized latency (ms/action) ↓ | ||
|---|---|---|---|---|---|
| SR ↑ | SPL ↑ | SR ↑ | SPL ↑ | ||
| LocateAnything + NavDP | 74.4 | 37.2 | 82.8 | 61.0 | 184 |
| LocateAnything + ApexNav | 73.3 | 64.1 | 75.8 | 66.8 | 511 |
| StreamVLN (zero-shot) | 61.1 | 51.2 | 48.4 | 40.0 | 87 |
| Uni-NaVid (zero-shot) | 11.6 | 9.9 | 8.9 | 8.2 | 99 |
| StreamVLN (fine-tuned) | 79.6 | 62.9 | 75.8 | 59.5 | — |
| Uni-NaVid (fine-tuned) | 80.2 | 68.1 | 82.2 | 71.6 | — |
| ScaleNav w/o Internet videos | 81.1 | 65.5 | 90.4 | 80.8 | — |
| ScaleNav | 86.5 | 71.3 | 94.9 | 86.1 | 76 |
Latency is amortized per executed action on RTX 4090. “—” denotes variants not profiled separately.
With the same simulated action data, incorporating Internet egocentric videos improves success by 5.3 percentage points on Short-Horizon OVON and 4.5 points on HabitatGS-LN.
Real-robot evaluation
Success rate (%) on Unitree Go2: 20 trials in each environment, 60 per method.
| Method | Hall ↑ | Lounge ↑ | Office ↑ | Overall ↑ |
|---|---|---|---|---|
| StreamVLN (fine-tuned) | 25.0 | 60.0 | 40.0 | 41.7 |
| Uni-NaVid (fine-tuned) | 60.0 | 75.0 | 20.0 | 51.7 |
| ScaleNav w/o Internet videos | 75.0 | 60.0 | 60.0 | 65.0 |
| ScaleNav | 85.0 | 85.0 | 80.0 | 83.3 |
Paper & resources
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Paper, code and model links will be added at release.