WorldVLM

A vision-language model decides what the vehicle should do; a world model predicts how the scene will unfold. WorldVLM joins the two: the VLM’s behaviour commands condition the driving world model, so the actions stay interpretable and aware of context.

Abstract

Autonomous driving systems depend on on models that can reason about high-level scene contexts and accurately predict the dynamics of their surrounding environment. Vision- Language Models (VLMs) have recently emerged as promising tools for decision-making and scene understanding, offering strong capabilities in contextual reasoning. However, their limited spatial comprehension constrains their effectiveness as end-to-end driving models. World Models (WM) internalize environmental dynamics to predict future scene evolution. Recently explored as ego-motion predictors and foundation models for autonomous driving, they represent a promising direction for addressing key challenges in the field, particularly enhancing generalization while maintaining dynamic prediction. To leverage the complementary strengths of context-based decision making and prediction, we propose WorldVLM: A hybrid architecture that unifies VLMs and WMs. In our design, the high-level VLM generates behavior commands to guide the driving WM, enabling interpretable and context-aware actions. We evaluate conditioning strategies and provide insights into the hybrid design challenges.

Citation

@inproceedings{Englmeier2026WorldVLM,
  title     = {WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning},
  author    = {Stefan Englmeier and Katharina Winter and Fabian B. Flohr},
  booktitle = {2026 IEEE International Conference on Intelligent Transportation Systems (ITSC)},
  year      = {2026},
  note      = {arXiv:2603.14497}
}

Copy BibTeX