PVDNet: Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes

This repository contains the official PyTorch implementation of the following paper:

Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes
Hyeongseok Son, Junyong Lee, Jonghyeop Lee, Sunghyun Cho, Seungyong Lee, TOG 2021 (presented at SIGGRAPH 2021)

About the Research

Getting Started

Prerequisites

Tested environment

1. Environment setup

   $ git clone https://github.com/codeslake/PVDNet.git
   $ cd PVDNet
   
   $ conda create -y --name PVDNet python=3.8 && conda activate PVDNet
   # for CUDA10.2
   $ sh install_CUDA10.2.sh
   # for CUDA11.1
   $ sh install_CUDA11.1.sh

2. Datasets

   • Download and unzip Su et al.'s dataset and Nah et al.'s dataset under [DATASET_ROOT]:

     ├── [DATASET_ROOT]
     │   ├── train_DVD
     │   ├── test_DVD
     │   ├── train_nah
     │   ├── test_nah
     

     Note:

     • [DATASET_ROOT] is currently set to ./datasets/video_deblur. It can be specified by modifying config.data_offset in ./configs/config.py.

3. Pre-trained models

   • Download and unzip pretrained weights under ./ckpt/:

     ├── ./ckpt
     │   ├── BIMNet.pytorch
     │   ├── PVDNet_DVD.pytorch
     │   ├── PVDNet_nah.pytorch
     │   ├── PVDNet_large_nah.pytorch
     

Testing models of TOG2021

For PSNRs and SSIMs reported in the paper, we use the approach of Koehler et al. following Su et al., that first aligns two images using global translation to represent the ambiguity in the pixel location caused by blur.
Refer here for the evaluation code.

## Table 4 in the main paper (Evaluation on Su etal's dataset)
# Our final model 
CUDA_VISIBLE_DEVICES=0 python run.py --mode PVDNet_DVD --config config_PVDNet --data DVD --ckpt_abs_name ckpt/PVDNet_DVD.pytorch

## Table 5 in the main paper (Evaluation on Nah etal's dataset)
# Our final model 
CUDA_VISIBLE_DEVICES=0 python run.py --mode PVDNet_nah --config config_PVDNet --data nah --ckpt_abs_name ckpt/PVDNet_nah.pytorch

# Larger model
CUDA_VISIBLE_DEVICES=0 python run.py --mode PVDNet_large_nah --config config_PVDNet_large --data nah --ckpt_abs_name ckpt/PVDNet_large_nah.pytorch

Note:

• Testing results will be saved in [LOG_ROOT]/PVDNet_TOG2021/[mode]/result/quanti_quali/[mode]_[epoch]/[data]/.
• [LOG_ROOT] is set to ./logs/ by default. Refer here for more details about the logging.

• options
  • --data: The name of a dataset to evaluate: DVD | nah | random. Default: DVD
    • The data structure can be modified in the function set_eval_path(..) in ./configs/config.py.
    • random is for testing models with any video frames, which should be placed as [DATASET_ROOT]/random/[video_name]/*.[jpg|png].

Wiki

• Logging
• Training and testing details

Citation

If you find this code useful, please consider citing:

@artical{Son_2021_TOG,
    author = {Son, Hyeongseok and Lee, Junyong and Lee, Jonghyeop and Cho, Sunghyun and Lee, Seungyong},
    title = {Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes},
    journal = {ACM Transactions on Graphics},
    year = {2021}
}

Contact

Open an issue for any inquiries. You may also have contact with [email protected] or [email protected]

Resources

All material related to our paper is available by following links:

License

This software is being made available under the terms in the LICENSE file.

Any exemptions to these terms require a license from the Pohang University of Science and Technology.

About Coupe Project

Project ‘COUPE’ aims to develop software that evaluates and improves the quality of images and videos based on big visual data. To achieve the goal, we extract sharpness, color, composition features from images and develop technologies for restoring and improving by using them. In addition, personalization technology through user reference analysis is under study.

Please check out other Coupe repositories in our Posgraph github organization.

Useful Links

• Coupe Library
• POSTECH CG Lab.