Forward Propagation, Backward Regression and Pose Association for Hand Tracking in the Wild (CVPR 2022)

Overview

HandLer

This repository contains the code and data for the following paper:

Forward Propagation, Backward Regression, and Pose Association for Hand Tracking in the Wild (CVPR 2022).

We develop a method to detect and track multiple hands in videos.

Installation

Install Torch 1.7.1

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Install other requirements

pip install -r requirements.txt

Install Detectron2 (version 0.4)

For CUDA 11.0 and Torch 1.7.x

python -m pip install detectron2==0.4 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html

Youtube-Hand Dataset

Please find the Youtube-Hand dataset in the project page.

Go to main folder

cd ./projects/HandLer

Model Weights

Please find the pretraind model weights here:

https://drive.google.com/file/d/1eaZn9E7lHvXY3Fh90f2eIK1P2yZwNfnP/view?usp=sharing

Please find the weights trained on Youtube-Hand training set here:

https://drive.google.com/file/d/1sK5bRTHt5zLOVPruUt-zyZ_AOrBvGQ9-/view?usp=sharing

Hand Tracking Demo: Run HandLer on custom videos

python demo.py --cfg configs/yt_trk.yaml --output output_vid.mp4 --weights /path/to/weights --input input_vid.mp4

Training on Youtube-Hand Dataset

For training on single GPU:

python train_trk.py --config-file configs/yt_trk.yaml

For training on multiple GPUs:

python train_trk.py --config-file configs/yt_trk.yaml --num_gpu 4

Testing on Youtube-Hand Test set

python tracking.py --cfg /path/to/cfg --out_dir /path/to/results/folders --weights /path/to/weights --root_path /path/to/dataset

Evaluating on Youtube-Hand Test set

python evaluations.py /path/to/results/folders

Improving Hand Tracking using Hand-Body Association

By associating hands with human bodies and linking human bodies across frames, we can establish correspondence between detected instances of the same hand across different frames, reducing identity switches in tracking. To know more about this, please check our work Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild

Citation

If you find our code or dataset useful, please cite:

@inproceedings{handler_2022,
      title={Forward Propagation, Backward Regression and Pose Association for Hand Tracking in the Wild},
      author={Mingzhen Huang and Supreeth Narasimhaswamy and Saif Vazir and Haibin Ling and Minh Hoai},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2022},
  }

@inproceedings{bodyhands_2022,
      title={Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild},
      author={Supreeth Narasimhaswamy and Thanh Nguyen and Mingzhen Huang and Minh Hoai},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2022},
  }
Owner
[email protected]
Computer Vision Lab at Stony Brook University
<a href=[email protected]">
Resources for the AMLD 2022 workshop "DevOps on AWS"

MLOPS on AWS | AMLD 2022 This repository contains all the resources necessary to follow along and reproduce the workshop "MLOps on AWS: a Hands-On Tut

xtream 8 Jun 16, 2022
A Discord bot coded in Python

Perseverance-Bot By Toricane Replit Code | GitHub Code | Discord Server | Website Perseverance is a multi-purpose bot coded in Python. It has moderati

4 Mar 30, 2022
A simple Discord Bot created for basic functionality and fun chat commands for use in a private server.

LoveAndChaos-Bot v0.1.0 LoveAndChaos-Bot is a Discord Bot specifically designed for a private server; this bot is merely a test and a method to expose

Morgan Rose 1 Dec 12, 2021
Userbot untuk memutar video dan lagu di vcg/os

Userbot untuk memutar video dan lagu di vcg/os

FJ_GAMING 2 Nov 13, 2021
The most versatile torrent leecher and youtube-dl bot for telegram

TorToolkit Telegram So basically Tortoolkit is aimed to be the most versatile torrent leecher and youtube-dl bot for telegram. This bot is highly cust

αвιנтн 1 Nov 11, 2021
Yes, it's true :purple_heart: This repository has 353 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serverless! If

510 Dec 28, 2022
An EmbedBuilder for Discord bots in Python.

An EmbedBuilder for Discord bots in Python. You need discord.py to use this module.

6 Jan 13, 2022
The official Python client library for the Kite Connect trading APIs

The Kite Connect API Python client - v3 The official Python client for communicating with the Kite Connect API. Kite Connect is a set of REST-like API

Zerodha Technology 756 Jan 06, 2023
A Telegram bot for combining emojis.

combimoji combimoji is a Telegram bot for combining emojis. How can I use it? You can find combimoji at @combimoji_bot, however it is not up (as of No

Yarema Mishchenko 2 Dec 02, 2021
A simple library for interacting with Amazon SQS.

qoo is a very simple Amazon SQS client, written in Python. It aims to be much more straight-forward to use than boto3, and specializes only in Amazon

Jacobi Petrucciani 2 Oct 30, 2020
Bot for tenhou.net riichi mahjong server written in Python

[Tests coverage] Bot was tested with Python 3.7+ and PyPy3, we are not supporting Python 2. What do we have here? Mahjong hands calculation You can fi

190 Jan 06, 2023
Adriano's Diets Consulting Bot - Parses and extracts informations about your diet (files in the Adriano's format).

Adriano's Diets Consulting Bot - Parses and extracts informations about your diet (files in the Adriano's format).

Marco A. 2 Feb 07, 2022
A Discord/Xenforo bot!

telathbot A Discord/Xenforo bot! Pre-requisites pyenv (via installer) poetry Docker (with Go version of docker compose enabled) Local development Crea

Telath 4 Mar 09, 2022
WhatsApp Web API client with multi-device support

Tauros WhatsApp Web client for multi-device in python Free software: MIT Documentation: https://tauros.readthedocs.io Features TODO Credits This packa

Manjit Pardeshi 0 Jan 20, 2022
GBSLocalLauncher - A script to compose ENV file for Local Compose

GBSLocalLauncher This is a script to compose ENV file for Local Compose. It crea

2 Jan 27, 2022
Telegram music & video bot direct play music

⚡ NOINOI MUSIC PLAYER 🎵 SUPERFAST MUSIC BOT WHO CAN DIRECT PLAY SONG ON TELEGRAM VOICE CHAT ALSO CAN PLAY VIDEO ON VOICE CHATS ✨ Heroku Deploy YOU CA

noinoi-X 1 Dec 28, 2021
Video Stream is a telegram bot project that's allow you to play video on telegram group video chat

Video Stream is a telegram bot project that's allow you to play video on telegram group video chat 🚀 Get SESSION_NAME from below: Pyrogram ## ✨ Featu

1 Nov 10, 2021
TG-Streaming-bot - TG Simple Streaming bot

TG Simple Streaming bot telegram video straming bot 🎚️ Features Play youtube li

HyDrix 4 May 05, 2022
Git Plan - a better workflow for git

git plan A better workflow for git. Git plan inverts the git workflow so that you can write your commit message first, before you start writing code.

Rory Byrne 178 Dec 11, 2022
A Python app which retrieves the rank and players' equipped skins during a match

VALORANT rank yoinker About The Project Usage Contributing Contact Acknowledgements Disclaimer About The Project Their Queue Current Skin Current Rank

Isaac Kenyon 270 Jan 04, 2023