LIF-Net: LiDAR–camera fusion for 3D human pose in urban scenes (IEEE IV 2026 2026)

Where a detector gives a box, LIF-Net gives a body: seventeen 3D joints per pedestrian and cyclist, fused from LiDAR geometry and camera appearance.

What it does

A box says a person is there. A pose says which way they face, whether they have seen the vehicle, whether they are stepping off the kerb. LIF-Net lifts sparse LiDAR returns and image keypoints into one skeleton per person, which is what the prediction models downstream actually need.

Abstract

Estimation of human pose and shape (3DHPS) in 3D is crucial to ensure the safety of vulnerable road users (VRUs) in autonomous driving (AD) scenarios, as it can serve as an additional feature for trajectory prediction and ego-motion planning in complex urban environments. To tackle this problem, we propose a novel 3DHPS model called LIF-Net, which utilizes RGB and LiDAR data to estimate VRUs as complete 3D bodies in the world coordinate system. LIF-Net utilizes a two-branch approach to encode both modalities separately into a latent representation. A cross-attention-based intermediate-fusion module learns to combine the two representations into a joint latent feature space, which is used by a Transformer decoder module to predict the parameters of a Skinned Multi-Person Linear Model (SMPL). We train and evaluate LIF-Net on the Waymo Open Dataset (WOD), containing challenging real-world scenarios with 2D and 3D keypoint annotations. Experimental results demonstrate the effectiveness and robustness of our approach compared to single-modality methods in challenging real-world AD scenarios, including poor lighting, occlusions, and varying bounding box and detection quality, while achieving an MPJPE of 122 mm and a PA-MPJPE of 76 mm, which is an improvement in MPJPE of 35.5% over image-based methods and 9.1% over the LiDAR-based method. Code and model will be made publicly available at lif-net.

Result

Pose recovered at distances and occlusion levels where camera-only estimation breaks down.

Citation

@inproceedings{Buettner2026LIFNet,
title = {LIF-Net: LiDAR-Camera Fusion for 3D Human Pose in Urban Scenes},
author = {Max A. Buettner and Erik Schuetz and Fabian B. Flohr},
url = {https://iv.ee.hm.edu/wp-content/uploads/2026/01/iv2025_buettner_fused3dhps-2.pdf},
year  = {2026},
date = {2026-01-01},
urldate = {2026-01-01},
booktitle = {2026 IEEE Intelligent Vehicles Symposium (IV)},
organization = {IEEE},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}