Neural Radiance Fields Using PyTorch

NeRF (Neural Radiance Fields) is a method for achieving outcomes for synthesizing novel views of complex scenes. Posted below are a few videos generated by this project.

This project is a PyTorch implementation of NeRF that reproduces the results while running approximately 1.3x faster. The code is based on the TensorFlow implementation here of the author. PyTorch has been used for numerical testing and validation of the same.

Installation

git clone https://github.com/yenchenlin/nerf-pytorch.git
cd nerf-pytorch
pip install -r requirements.txt
cd torchsearchsorted
pip install .
cd ../

How To Run?

Quick Start

Download data for two example datasets: lego and fern

bash download_example_data.sh

To train a low-res lego NeRF:

python run_nerf.py --config configs/config_lego.txt

After training for 100,000 iterations (~4 hours on a single 2080 Ti), you can find the following video at logs/lego_test/lego_test_spiral_100000_rgb.mp4.

To train a low-res fern NeRF:

python run_nerf.py --config configs/config_fern.txt

After training for 200k iterations (~8 hours on a single 2080 Ti), you can find the following video at logs/fern_test/fern_test_spiral_200000_rgb.mp4 and logs/fern_test/fern_test_spiral_200000_disp.mp4

More Datasets

To test out the other scenes presented in the paper, you can download the data here. Please place the downloaded dataset according to the following directory structure:

├── configs                                                                                                       
│   ├── ...                                                                                     
│                                                                                               
├── data                                                                                                                                                                                                       
│   ├── nerf_llff_data                                                                                                  
│   │   └── fern                                                                                                                             
│   │   └── flower  # downloaded llff dataset                                                                                  
│   │   └── horns   # downloaded llff dataset
|   |   └── ...
|   ├── nerf_synthetic
|   |   └── lego
|   |   └── ship    # downloaded synthetic dataset
|   |   └── ...

To train NeRF on different datasets:

python run_nerf.py --config configs/config_{DATASET}.txt

replace {DATASET} with trex | horns | flower | fortress | lego | etc.

To test NeRF trained on different datasets:

python run_nerf.py --config configs/config_{DATASET}.txt --render_only

replace {DATASET} with trex | horns | flower | fortress | lego | etc.

Pre-trained Models

You can download the pre-trained models here. Please place the downloaded directory in ./logs in order to test it later. Check the following directory structure for an example:

├── logs 
│   ├── fern_test
│   ├── flower_test  # downloaded logs
│   ├── trex_test    # downloaded logs

Reproducibility

The tests that ensure the results of all functions and training loop match the official implentation are contained in a different branch reproduce. One can check it out and run the tests:

git checkout reproduce
py.test

Method

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall*¹, Pratul P. Srinivasan*¹, Matthew Tancik*¹, Jonathan T. Barron², Ravi Ramamoorthi³, Ren Ng^1
1UC Berkeley, ²Google Research, ³UC San Diego
*denotes equal contribution

A Neural Radiance Field is a simple fully connected network (weights are ~5MB) trained to reproduce input views of a single scene using a rendering loss. The network directly maps from spatial location and viewing direction (5D input) to color and opacity (4D output), acting as the "volume" so we can use volume rendering to differentiably render new views

Citation

A big thank-you to the authors (below) for their amazing work and results:

@misc{mildenhall2020nerf,
    title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
    author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
    year={2020},
    eprint={2003.08934},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}