Computer Graphics
TU Braunschweig

Learning to Reconstruct People in Clothing from a Single RGB Camera


Learning to Reconstruct People in Clothing from a Single RGB Camera

We present Octopus, a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving with a reconstruction accuracy of 4 to 5mm, while being orders of magnitude faster than previous methods. From semantic segmentation images, our Octopus model reconstructs a 3D shape, including the parameters of SMPL plus clothing and hair in 10 seconds or less. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into pose-invariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, Octopus can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 5mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach.

 

Code & Dataset

Download code and dataset here.


Author(s):Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll
Published:to appear
Type:Article in conference proceedings
Book:IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Presented at:Conference on Computer Vision and Pattern Recognition (CVPR) 2019
Project(s): Comprehensive Human Performance Capture from Monocular Video Footage 


@inproceedings{alldieck2019learning,
  title = {Learning to Reconstruct People in Clothing from a Single {RGB} Camera},
  author = {Alldieck, Thiemo and Magnor, Marcus and Bhatnagar, Bharat Lal and Theobalt, Christian and Pons-Moll, Gerard},
  booktitle = {{IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year = {2019}
}

Authors