Walk-the-Talk
Simulated pedestrians usually walk in straight lines. Walk-the-Talk generates their motion from a text description, so a scenario can be written rather than scripted.
What it does
If the people in your simulator do not behave like people, closed-loop evaluation flatters the planner. Describing a behaviour in words and getting plausible motion back makes hard scenarios cheap to build.
Abstract
In the field of autonomous driving, a key challenge is the “reality gap”: transferring knowledge gained in simulation to real-world settings. Despite various approaches to mitigate this gap, there’s a notable absence of solutions targeting agent behavior generation which are crucial for mimicking spontaneous, erratic, and realistic actions of traffic participants. Recent advancements in Generative AI have enabled the representation of human activities in semantic space and generate real human motion from textual descriptions. Despite current limitations such as modality constraints, motion sequence length, resource demands, and data specificity, there’s an opportunity to innovate and use these techniques in the intelligent vehicles domain. We propose Walk-the-Talk, a motion generator utilizing Large Language Models (LLMs) to produce reliable pedestrian motions for high-fidelity simulators like CARLA. Thus, we contribute to autonomous driving simulations by aiming to scale realistic, diverse long-tail agent motion data – currently a gap in training datasets. We employ Motion Capture (MoCap) techniques to develop the Walk-the-Talk dataset, which illustrates a broad spectrum of pedestrian behaviors in street-crossing scenarios, ranging from standard walking patterns to extreme behaviors such as drunk walking and near-crash incidents. By utilizing this new dataset within a LLM, we facilitate the creation of realistic pedestrian motion sequences, a capability previously unattainable (cf. Figure 1). Additionally, our findings demonstrate that leveraging the Walk-the-Talk dataset enhances cross-domain generalization and significantly improves the Fréchet Inception Distance (FID) score by approximately 15% on the HumanML3D dataset. https://iv.ee.hm.edu/publications/w-the-t/
Result
Citation
@conference{Ramesh2024WalktheTalk,
title = {Walk-the-Talk: LLM driven pedestrian motion generation},
author = {Mohan Ramesh and Fabian B. Flohr},
url = {iv.ee.hm.edu/publications/w-the-t/},
doi = {10.1109/IV55156.2024.10588860},
issn = {2642-7214},
year = {2024},
date = {2024-08-01},
urldate = {2024-08-01},
booktitle = {2024 IEEE Intelligent Vehicles Symposium (IV)},
pages = {3057-3062},
publisher = {IEEE},
address = {Jeju Island, Korea, Republic of},
abstract = {In the field of autonomous driving, a key challenge is the “reality gap”: transferring knowledge gained in simulation to real-world settings. Despite various approaches to mitigate this gap, there’s a notable absence of solutions targeting agent behavior generation which are crucial for mimicking spontaneous, erratic, and realistic actions of traffic participants. Recent advancements in Generative AI have enabled the representation of human activities in semantic space and generate real human motion from textual descriptions. Despite current limitations such as modality constraints, motion sequence length, resource demands, and data specificity, there’s an opportunity to innovate and use these techniques in the intelligent vehicles domain. We propose Walk-the-Talk, a motion generator utilizing Large Language Models (LLMs) to produce reliable pedestrian motions for high-fidelity simulators like CARLA. Thus, we contribute to autonomous driving simulations by aiming to scale realistic, diverse long-tail agent motion data – currently a gap in training datasets. We employ Motion Capture (MoCap) techniques to develop the Walk-the-Talk dataset, which illustrates a broad spectrum of pedestrian behaviors in street-crossing scenarios, ranging from standard walking patterns to extreme behaviors such as drunk walking and near-crash incidents. By utilizing this new dataset within a LLM, we facilitate the creation of realistic pedestrian motion sequences, a capability previously unattainable (cf. Figure 1). Additionally, our findings demonstrate that leveraging the Walk-the-Talk dataset enhances cross-domain generalization and significantly improves the Fréchet Inception Distance (FID) score by approximately 15% on the HumanML3D dataset.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}