BEVDriver
A language-model planner that is grounded in a bird’s-eye-view map rather than in raw pixels, and is evaluated in closed loop instead of on logged data.
What it does
Language models can describe a traffic situation convincingly and still plan badly, because their input never contained the geometry. BEVDriver feeds the planner a BEV map, which puts distances and lane structure into the representation the model reasons over.
Abstract
Autonomous driving has the potential to set the stage for more efficient future mobility, requiring the research domain to establish trust through safe, reliable and transparent driving. Large Language Models (LLMs) possess reasoning capabilities and natural language understanding, presenting the potential to serve as generalized decision-makers for ego-motion planning that can interact with humans and navigate environments designed for human drivers. While this research avenue is promising, current autonomous driving approaches are challenged by combining 3D spatial grounding and the reasoning and language capabilities of LLMs. We introduce BEVDriver, an LLM-based model for end-to-end closed-loop driving in CARLA that utilizes latent BEV features as perception input. BEVDriver includes a BEV encoder to efficiently process multi-view images and 3D LiDAR point clouds. Within a common latent space, the BEV features are propagated through a Q-Former to align with natural language instructions and passed to the LLM that predicts and plans precise future trajectories while considering navigation instructions and critical scenarios. On the LangAuto benchmark, our model reaches up to 18.9% higher performance on the Driving Score compared to SoTA methods.
Result
Citation
@conference{Winter2025BEVDriver,
title = {BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving},
author = {Katharina Winter and Mark Azer and Fabian B. Flohr},
url = {https://iv.ee.hm.edu/bevdriver/
https://ieeexplore.ieee.org/document/11247237},
doi = {10.1109/IROS60139.2025.11247237},
year = {2025},
date = {2025-11-27},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
pages = {20379-20385},
publisher = {IEEE},
abstract = {Autonomous driving has the potential to set the stage for more efficient future mobility, requiring the research domain to establish trust through safe, reliable and transparent driving. Large Language Models (LLMs) possess reasoning capabilities and natural language understanding, presenting the potential to serve as generalized decision-makers for ego-motion planning that can interact with humans and navigate environments designed for human drivers. While this research avenue is promising, current autonomous driving approaches are challenged by combining 3D spatial grounding and the reasoning and language capabilities of LLMs. We introduce BEVDriver, an LLM-based model for end-to-end closed-loop driving in CARLA that utilizes latent BEV features as perception input. BEVDriver includes a BEV encoder to efficiently process multi-view images and 3D LiDAR point clouds. Within a common latent space, the BEV features are propagated through a Q-Former to align with natural language instructions and passed to the LLM that predicts and plans precise future trajectories while considering navigation instructions and critical scenarios. On the LangAuto benchmark, our model reaches up to 18.9% higher performance on the Driving Score compared to SoTA methods.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}