HABIT: Human Action Benchmark for Interactive Traffic in CARLA

HABIT brings real human behaviour into simulation, offering thousands of lifelike pedestrian motions for testing autonomous vehicles. It provides a realistic, interactive, and reproducible benchmark for next-generation driving AI.

Abstract

Current autonomous driving (AD) simulations are critically limited by their inadequate representation of realistic and diverse human behavior, which is essential for ensuring safety and reliability. Existing benchmarks often simplify pedestrian interactions, failing to capture complex, dynamic intentions and varied responses critical for robust system deployment. To overcome this, we introduce HABIT (Human Action Benchmark for Interactive Traffic), a high-fidelity simulation benchmark. HABIT integrates real-world human motion, sourced from mocap and videos, into CARLA (Car Learning to Act, a full autonomous driving simulator) via a modular, extensible, and physically consistent motion retargeting pipeline. From an initial pool of approximately 30,000 retargeted motions, we curate 4,730 traffic-compatible pedestrian motions, standardized in SMPL format for physically consistent trajectories. HABIT seamlessly integrates with CARLA’s Leaderboard, enabling automated scenario generation and rigorous agent evaluation. Our safety metrics, including Abbreviated Injury Scale (AIS) and False Positive Braking Rate (FPBR), reveal critical failure modes in state-of-the-art AD agents missed by prior evaluations. Evaluating three state-of-the-art autonomous driving agents, InterFuser, TransFuser, and BEVDriver, demonstrates how HABIT exposes planner weaknesses that remain hidden in scripted simulations. Despite achieving close or equal to zero collisions per kilometer on the CARLA Leaderboard, the autonomous agents perform notably worse on HABIT, with up to 7.43 collisions/km and a 12.94% AIS 3+ injury risk, and they brake unnecessarily in up to 33% of cases. All components are publicly released to support reproducible, pedestrian-aware AI research.

Pipeline

@misc{ramesh2026habit,
  title        = {HABIT: Human Action Benchmark for Interactive Traffic in CARLA},
  author       = {Ramesh, Mohan and Azer, Mark and Flohr, Fabian}, 
  note         = {To appear in WACV 2026},
  year         = {2026}
}

This work is a result of the joint research projects “STADT:up (19A22006N)” and “NXT GEN AI METHODS – Generative Methoden für Perzeption, Prädiktion und Planung”. The projects are supported by the German Federal Ministry for Economic Affairs and Energy (BMWE), based on a decision of the German Bundestag. The authors are solely responsible for the content of this publication.