Bridging the MoCap-to-Visual domain gap in human mesh recovery from 2D keypoints

Date

2023-7-31

Author

UĞUZ, BEDİRHAN

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

92
views

73
downloads

Cite This

This study tackles the problem of recovering human meshes from 2D keypoint keypoints. We develop a model that utilizes MoCap datasets (Motion Capture domain) during training without the need for corresponding RGB images. This approach allows us to leverage existing large MoCap datasets and overcome the scarcity of paired RGB-3D datasets, which are difficult to collect. After training, we switch domain and apply our models to real visual data (visual domain) by using an off-the-shelf 2D detector to obtain 2D keypoints as inputs to our model. To minimize the impact of the domain shift from the MoCap domain to the visual domain, we introduce an adversarial domain adaptation framework and adapt our models to the visual domain. Our contributions are as follows: (i) we introduce a direct regression method, i.e. without any iterations or recurrent connections, to recover 3D human body mesh from 2D keypoint detections, (ii) we bridge the domain gap between the MoCap domain and the visual domain without relying on labeled visual data, (iii) our experimental results on widely used datasets, H3.6M and 3DPW, demonstrate that our approach outperforms existing methods in terms of PA-MPJPE on both datasets, as well as MPJPE and PVE on 3DPW for the task of recovering human meshes from 2D keypoints under unpaired 3D training conditions and (iv) due to the single-stage approach of our method, it is up to 33x faster than its closest competitor (LGD).

Subject Keywords

Deep learning, Human pose and shape estimation, Domain adaptation

URI

https://hdl.handle.net/11511/104847

Collections

Graduate School of Natural and Applied Sciences, Thesis