Code for ICCV2021 paper SPEC: Seeing People in the Wild with an Estimated Camera

Related tags

Deep Learningcomputer-visioncamera-calibrationcomputer-graphicshuman-pose-estimation3d-human-pose3d-human-mesh3d-human-shape-and-pose-estimation3d-human-reconstruction
Overview

SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021]

Open In Colab report report

SPEC: Seeing People in the Wild with an Estimated Camera,
Muhammed Kocabas, Chun-Hao Paul Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, Michael J. Black,
International Conference on Computer Vision (ICCV), 2021

Features

SPEC is a camera-aware human body pose and shape estimation method. It both predicts the camera parameters and SMPL body model for a given image. CamCalib predicts the camera parameters. SPEC uses these parameters to predict SMPL body model parameters.

This implementation:

  • has the demo code for SPEC and CamCalib implemented in PyTorch.
  • achieves SOTA results in SPEC-SYN and SPEC-MTP datasets.
  • shows how to perform evaluation on SPEC-SYN and SPEC-MTP datasets.

Updates

  • 13/10/2021: Demo and evaluation code is released.

Getting Started

SPEC has been implemented and tested on Ubuntu 18.04 with python >= 3.7. If you don't have a suitable device, try running our Colab demo.

Clone the repo:

git clone https://github.com/mkocabas/SPEC.git

Install the requirements using virtualenv or conda:

# pip
source scripts/install_pip.sh

# conda
source scripts/install_conda.sh

Running the Demo

SPEC

First, you need to download the required data (i.e our trained model and SMPL model parameters). It is approximately 1GB. To do this you can just run:

source scripts/prepare_data.sh

Then, running the demo is as simple as:

python scripts/spec_demo.py \
  --image_folder data/sample_images \
  --output_folder logs/spec/sample_images

Sample demo output:

Here the green line is the horizon obtained using estimated camera parameters. On the right, the ground plane is visualized to show how accurate the global translation is.

CamCalib

If you are only interested in estimating the camera parameters of an image, run the CamCalib demo:

python scripts/camcalib_demo.py \
  --img_folder <input image folder> \
  --out_folder <output folder> \
  --show # visualize the raw network predictions

This script outputs a pickle file which contains the predicted camera parameters for each input image along with an output image which visualizes the camera parameters as a horizon line. Pickle file contains:

'vfov' : vertical field of view in radians
'f_pix': focal length in pixels
'pitch': pitch in radians
'roll' : roll in radians

Google Colab

Training

Training instructions will follow soon.

Datasets

Pano360, SPEC-MTP, and SPEC-SYN are new datasets introduced in our paper. You can download them from the Downloads section of our project page.

For Pano360 dataset, we have released the Flickr image ids which can be used to download images using FlickrAPI. We have provided a download script in this repo. Some of the images will be missing due to users deleting their photos. In this case, you can also use scrape_and_download function provided in the script to find and download more photos.

After downloading the SPEC-SYN, SPEC-MTP, Pano360, and 3DPW datasets, the data folder should look like:

data/
├── body_models
│   └── smpl
├── camcalib
│   └── checkpoints
├── dataset_extras
├── dataset_folders
│   ├── 3dpw
│   ├── pano360
│   ├── spec-mtp
│   └── spec-syn
├── sample_images
└── spec
    └── checkpoints

Evaluation

You can evaluate SPEC on SPEC-SYN, SPEC-MTP, and 3DPW datasets by running:

python scripts/spec_eval.py \
  --cfg data/spec/checkpoints/spec_config.yaml \
  --opts DATASET.VAL_DS spec-syn_spec-mtp_3dpw-test-cam

Running this script should give results reported in this table:

W-MPJPE PA-MPJPE W-PVE
SPEC-MTP 124.3 71.8 147.1
SPEC-SYN 74.9 54.5 90.5
3DPW 106.7 53.3 124.7

Citation

@inproceedings{SPEC:ICCV:2021,
  title = {{SPEC}: Seeing People in the Wild with an Estimated Camera},
  author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Tesch, Joachim and M\"uller, Lea and Hilliges, Otmar and Black, Michael J.},
  booktitle = {Proc. International Conference on Computer Vision (ICCV)},
  pages = {11035--11045},
  month = oct,
  year = {2021},
  doi = {},
  month_numeric = {10}
}

License

This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party datasets and software are subject to their respective licenses.

References

We indicate if a function or script is borrowed externally inside each file. Here are some great resources we benefit:

Consider citing these works if you use them in your project.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Comments
  • Translation of the camera

    Translation of the camera

    I'd like to know if it's possible to get a translation of the camera. I've found the code just for rotation and focal length estimation.

    On this image https://github.com/mkocabas/SPEC/blob/master/docs/assets/spec_gif.gif the camera is placed in the world space somehow. How did you do this? Thanks.

    opened by Dene33 4
  • Segmentation fault (core dumped)

    Segmentation fault (core dumped)

    Hi~ When I run the command python scripts/spec_demo.py --batch_size 1 --image_folder data/sample_images --output_folder logs/spec/sample_images, I got the error as this: Segmentation fault (core dumped) I have no idea about it, could you help me, please?

    opened by JinkaiZheng 2
  • There is no flickr_photo_ids.npy

    There is no flickr_photo_ids.npy

    Hello author:

    After unzipping the spec-github-data.zip, there is no data/dataset_folders/pano360/flickr_photo_ids.npy. I don't know whether I find the right place, could you help me?

    Best regards.

    opened by songxujay 1
  • video demo

    video demo

    It's a great job. Is the video demo not available now? `def main(args):

    demo_mode = args.mode

if demo_mode == 'video':
    raise NotImplementedError
elif demo_mode == 'webcam':
    raise NotImplementedError`
    opened by FatherPrime 1
  • Fix PARE requirement in requirement.txt

    Fix PARE requirement in requirement.txt

    Thanks for sharing this work! Please fix the requirement in requirement.txt file of PARE. It should be (currently missing "git+" prefix): git+https://github.com/mkocabas/PARE.git

    opened by Omri-CG 1
  • get error horizon line by camcalib_demo.py

    get error horizon line by camcalib_demo.py

    all example_image seem get error result by camcalib_demo.py, look like this: COCO_val2014_000000326555 it seems the predicted horizon line shift by model,how should I solve this problem？

    opened by hhhlllyyy 0
  • feet fit accuracy

    feet fit accuracy

    I've encountered frequent problems that SPEC is having to fit the model to feet correctly:

    image image

    Is this the same problem as mentioned in the VIBE discussion: https://github.com/mkocabas/VIBE/issues/24#issuecomment-596714558 ? Have you tried to include the OpenPose feet keypoint predictions to the loss?

    Thanks for the great work!

    opened by smidm 0
  • Dataset npz interpretation

    Dataset npz interpretation

    Dear Muhammed,

    Thank you for this work and the datasets! I'm trying to interpret the data in the SPEC-SYN npz data files, but I'm not sure what each key means. Is there documentation? They are the following: ['imgname', 'center', 'scale', 'pose', 'shape', 'part', 'mmpose_keypoints', 'openpose', 'openpose_gt', 'S', 'focal_length', 'cam_rotmat', 'cam_trans', 'cam_center', 'cam_pitch', 'cam_roll', 'cam_hfov', 'cam_int', 'camcalib_pitch', 'camcalib_roll', 'camcalib_vfov', 'camcalib_f_pix']

    At this point, I'd just like to plot the 24 SMPL joints on the image. Based on the array shape, I assume 'S' contains the joints. Is cam_rotmat the rotation from world space to camera space? Is the cam_trans the position of the camera or the top right part of the extrinsic matrix? I assume for this plotting exercise I can ignore everything except S, cam_rotmat, cam_trans and cam_int. Still for some reason the points end up at at very wrong places. Maybe 'S' is something else? Or am I using the camera params wrong?

    Thanks! Istvan

    opened by isarandi 5
  • Colab notebook inaccesible

    Colab notebook inaccesible

    Hi,I tried accesing the colab notebook but wasn't able to access it.Would you be able to kindly suggest another way to run inference as I am currently working on a windows based CPU system.

    opened by sparshgarg23 0
  • Error during single dataset evaluation

    Error during single dataset evaluation

    I tried running the evaluation only for the 3DPW dataset with the following command:

    python scripts/spec_eval.py --cfg data/spec/checkpoints/spec_config.yaml --opts DATASET.VAL_DS 3dpw-test-cam

    But it gives the following error:

    TypeError: test_step() missing 1 required positional argument: 'dataloader_nb'

    Fixed it by giving a default value for dataloader_nb for the test_step function in trainer.py:

    def test_step(self, batch, batch_nb, dataloader_nb=0):
    return self.validation_step(batch, batch_nb, dataloader_nb)

    The error occurs because we don't append dataloader_idx to args in evaluation_loop.py:

    if multiple_test_loaders or multiple_val_loaders:
    args.append(dataloader_idx)
    opened by umariqb 1
Owner
Muhammed Kocabas
Muhammed Kocabas
GitHub Repository
Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

Annoy Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given quer

Spotify 10.6k Jan 04, 2023
GraphLily: A Graph Linear Algebra Overlay on HBM-Equipped FPGAs

GraphLily: A Graph Linear Algebra Overlay on HBM-Equipped FPGAs GraphLily is the first FPGA overlay for graph processing. GraphLily supports a rich se

Cornell Zhang Research Group 39 Dec 13, 2022
A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms

Denoising Diffusion Probabilistic Model for Proteins Implementation of Denoising Diffusion Probabilistic Model in Pytorch. It is a new approach to gen

Phil Wang 108 Nov 23, 2022
Code for 2021 NeurIPS --- Towards Multi-Grained Explainability for Graph Neural Networks

ReFine: Multi-Grained Explainability for GNNs This is the official code for Towards Multi-Grained Explainability for Graph Neural Networks (NeurIPS 20

Shirley (Ying-Xin) Wu 47 Dec 16, 2022
[CVPR 2021 Oral] Variational Relational Point Completion Network

VRCNet: Variational Relational Point Completion Network This repository contains the PyTorch implementation of the paper: Variational Relational Point

PL 121 Dec 12, 2022
Exploration of some patients clinical variables.

Answer_ALS_clinical_data Exploration of some patients clinical variables. All the clinical / metadata data is available here: https://data.answerals.o

1 Jan 20, 2022
Starter kit for getting started in the Music Demixing Challenge.

Music Demixing Challenge - Starter Kit 👉 Challenge page This repository is the Music Demixing Challenge Submission template and Starter kit! Clone th

AIcrowd 106 Dec 20, 2022
MetaBalance: High-Performance Neural Networks for Class-Imbalanced Data

This repository is the official PyTorch implementation of Meta-Balance. Find the paper on arxiv MetaBalance: High-Performance Neural Networks for Clas

Arpit Bansal 20 Oct 18, 2021
Powerful and efficient Computer Vision Annotation Tool (CVAT)

Computer Vision Annotation Tool (CVAT) CVAT is free, online, interactive video and image annotation tool for computer vision. It is being used by our

OpenVINO Toolkit 8.6k Jan 01, 2023
Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction

Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction. arxiv This repository contains python scripts for tr

12 Dec 12, 2022
Code for NAACL 2021 full paper "Efficient Attentions for Long Document Summarization"

LongDocSum Code for NAACL 2021 paper "Efficient Attentions for Long Document Summarization" This repository contains data and models needed to reprodu

56 Jan 02, 2023
PyTorch implementation of the ExORL: Exploratory Data for Offline Reinforcement Learning

ExORL: Exploratory Data for Offline Reinforcement Learning This is an original PyTorch implementation of the ExORL framework from Don't Change the Alg

Denis Yarats 52 Jan 01, 2023
A simple baseline for the 2022 IEEE GRSS Data Fusion Contest (DFC2022)

DFC2022 Baseline A simple baseline for the 2022 IEEE GRSS Data Fusion Contest (DFC2022) This repository uses TorchGeo, PyTorch Lightning, and Segmenta

isaac 24 Nov 28, 2022
VolumeGAN - 3D-aware Image Synthesis via Learning Structural and Textural Representations

VolumeGAN - 3D-aware Image Synthesis via Learning Structural and Textural Representations 3D-aware Image Synthesis via Learning Structural and Textura

GenForce: May Generative Force Be with You 116 Dec 26, 2022
A Python library for common tasks on 3D point clouds

Point Cloud Utils (pcu) - A Python library for common tasks on 3D point clouds Point Cloud Utils (pcu) is a utility library providing the following fu

Francis Williams 622 Dec 27, 2022
A voice recognition assistant similar to amazon alexa, siri and google assistant.

kenyan-Siri Build an Artificial Assistant Full tutorial (video) To watch the tutorial, click on the image below Installation For windows users (run th

Alison Parker 3 Aug 19, 2022
Generate Cartoon Images using Generative Adversarial Network

AvatarGAN ✨ Generate Cartoon Images using DC-GAN Deep Convolutional GAN is a generative adversarial network architecture. It uses a couple of guidelin

Aakash Jhawar 50 Dec 29, 2022
An Efficient Implementation of Analytic Mesh Algorithm for 3D Iso-surface Extraction from Neural Networks

AnalyticMesh Analytic Marching is an exact meshing solution from neural networks. Compared to standard methods, it completely avoids geometric and top

Karbo 45 Dec 21, 2022
A Python Package For System Identification Using NARMAX Models

SysIdentPy is a Python module for System Identification using NARMAX models built on top of numpy and is distributed under the 3-Clause BSD license. N

Wilson Rocha 175 Dec 25, 2022
Elucidating Robust Learning with Uncertainty-Aware Corruption Pattern Estimation

Elucidating Robust Learning with Uncertainty-Aware Corruption Pattern Estimation Introduction 📋 Official implementation of Explainable Robust Learnin

JeongEun Park 6 Apr 19, 2022