ADRIVE-GPT

Joint project of HM and Hochschule Esslingen with Mercedes-Benz and Spleenlab on large language models for automated driving – from camera and lidar perception to language-based action plans.

About the project

ADRIVE-GPT (Autonomous Driving with Generative Pre-trained Transformers) investigates how grounded large language models (LLMs) can be used in automated driving. Previous studies had applied LLMs only to simple tasks in stationary robotics. The project examines how multimodal perception from camera and lidar sensors can be combined with natural language to generate action plans for accident-free, human-like autonomous driving. It also studies whether learning can be supported by already processed knowledge about static and dynamic objects, and whether traffic rules or detailed map information must be provided explicitly or can be learned implicitly – the latter would make the model more independent of fixed rules and data and thus more scalable. A further focus is on self-prompting, agent-based and reasoning methods: the model should generate further prompts and action sequences on its own and, through feedback loops and reflection, correct itself, identify alternatives and monitor and improve its performance in new scenarios. The innovations are to be integrated in at least one vehicle demonstrator and evaluated on large datasets from the industry partners. ADRIVE-GPT is a joint project of Hochschule München and Hochschule Esslingen with Mercedes-Benz and Spleenlab, funded by the BMFTR in the FH-Kooperativ programme from 2025 to 2029.

Our contribution

At HM, ADRIVE-GPT is led by Prof. Dr. Fabian Flohr and Prof. Dr. habil. Alfred Schöttl at the Institute for Applications of Machine Learning and Intelligent Systems (IAMLIS) in the Faculty of Electrical Engineering and Information Technology, and the project is presented on the Intelligent Vehicles Lab website. HM’s grant (FKZ 13FH544KA2) is administered by the faculty. On the Esslingen side, Prof. Dr. Markus Enzweiler’s Institute for Intelligent Systems is involved. The published sources describe the joint research agenda – language models for camera- and lidar-based driving, implicit learning of rules and maps, self-prompting and reasoning, validation in a vehicle demonstrator – but do not specify which tasks HM carries out.