Human Performance Capture from Monocular Video in the Wild
Paper | Video | Project Page
Official code release for 3DV 2021 paper Human Performance Capture from Monocular Video in the Wild. We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.
If you find our code or paper useful, please cite as
@inproceedings{guo2021human,
title={Human Performance Capture from Monocular Video in the Wild},
author={Guo, Chen and Chen, Xu and Song, Jie and Hilliges, Otmar},
booktitle={2021 International Conference on 3D Vision (3DV)},
pages={889--898},
year={2021},
organization={IEEE}
}
Quick Start
CLone this repo:
git clone https://github.com/MoyGcc/hpcwild.git
cd hpcwild
conda env create -f environment.yml
conda activate hpcwild
Additional Dependencies:
- Kaolin 0.1.0 (https://github.com/NVIDIAGameWorks/kaolin)
- MPI mesh library (https://github.com/MPI-IS/mesh)
- torch-mesh-isect (https://github.com/vchoutas/torch-mesh-isect)
Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mkdir lib/smpl/smpl_model/
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_MALE.pkl
Download checkpoints for external modules:
wget https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth
mv /path/to/checkpoint_iter_370000.pth external/lightweight-human-pose-estimation.pytorch/checkpoint_iter_370000.pth
wget https://dl.fbaipublicfiles.com/pifuhd/checkpoints/pifuhd.pt pifuhd.pt
mv /path/to/pifuhd.pt external/pifuhd/checkpoints/pifuhd.pt
Download IPNet weights: https://datasets.d2.mpi-inf.mpg.de/IPNet2020/IPNet_p5000_01_exp_id01.zip
unzip IPNet_p5000_01_exp_id01.zip
mv /path/to/IPNet_p5000_01_exp_id01 registration/experiments/IPNet_p5000_01_exp_id01
gdown --id 1mcr7ALciuAsHCpLnrtG_eop5-EYhbCmz -O modnet_photographic_portrait_matting.ckpt
mv /path/to/modnet_photographic_portrait_matting.ckpt external/MODNet/pretrained/modnet_photographic_portrait_matting.ckpt
Test on 3DPW dataset
Download 3DPW dataset
- modify the
dataset_pathintest.conf. - run
bash mesh_recon.shto obtain the rigid body shape. - run
bash registration.shto register a SMPL+D model to the rigid human body. - run
bash tracking.shto capture the human performance temporally.
Test on your own video
- run OpenPose to obtain the 2D keypoints.
- run LGD to acquire the initial 3D poses.
- run MODNet to extract sihouettes.
Acknowledgement
We use the code in PIFuHD for the rigid body construction and adapt IPNet for human model registration. We use off-the-shelf methods OpenPose and MODNet for the extraction of 2D keypoints and sihouettes. We sincerely thank these authors for their awesome work.
25 Dec 21, 2022
120 Dec 15, 2022
215 Dec 07, 2022
71 Dec 22, 2022
2 Mar 12, 2022
1 Nov 04, 2022
92 Dec 25, 2022
1.1k Dec 30, 2022
13 Nov 07, 2022
1 Dec 22, 2021
13 Dec 27, 2022
1 Jan 06, 2022
28 Nov 06, 2022
18 Dec 30, 2022
1k Dec 30, 2022
259 Dec 05, 2022
15 Sep 11, 2022
656 Dec 29, 2022
23 Sep 21, 2022
4 Feb 22, 2022