Learning to Adapt Structured Output Space for Semantic Segmentation

Pytorch implementation of our method for adapting semantic segmentation from the synthetic dataset (source domain) to the real dataset (target domain). Based on this implementation, our result is ranked 3rd in the VisDA Challenge.

Contact: Yi-Hsuan Tsai (wasidennis at gmail dot com) and Wei-Chih Hung (whung8 at ucmerced dot edu)

Paper

Learning to Adapt Structured Output Space for Semantic Segmentation
Yi-Hsuan Tsai*, Wei-Chih Hung*, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang and Manmohan Chandraker
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (spotlight) (* indicates equal contribution).

Please cite our paper if you find it useful for your research.

@inproceedings{Tsai_adaptseg_2018,
  author = {Y.-H. Tsai and W.-C. Hung and S. Schulter and K. Sohn and M.-H. Yang and M. Chandraker},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  title = {Learning to Adapt Structured Output Space for Semantic Segmentation},
  year = {2018}
}

Example Results

Quantitative Reuslts

Installation

• Install PyTorch from http://pytorch.org with Python 2 and CUDA 8.0

• NEW Add the LS-GAN objective to improve the performance

  • Usage: add --gan LS option during training (see below for more details)

• PyTorch 0.4 with Python 3 and CUDA 8.0

  • Usage: replace the training and evaluation codes with the ones in the pytorch_0.4 folder
  • Update: tensorboard is provided by adding --tensorboard in the command
  • Note: the single-level model works as expected, while the multi-level model requires smaller weights, e.g., --lambda-adv-target1 0.00005 --lambda-adv-target2 0.0005. We will investigate this issue soon.

• Clone this repo

git clone https://github.com/wasidennis/AdaptSegNet
cd AdaptSegNet

Dataset

• Download the GTA5 Dataset as the source domain, and put it in the data/GTA5 folder

• Download the Cityscapes Dataset as the target domain, and put it in the data/Cityscapes folder

Pre-trained Models

• Please find our-pretrained models using ResNet-101 on three benchmark settings here

• They include baselines (without adaptation and with feature adaptation) and our models (single-level and multi-level)

Testing

• NEW Update results using LS-GAN and using Synscapes as the source domain

  • Performance: check the appendix of the updated arXiv paper (updated on 10/17/2019)
  • Pre-trained models

• Download the pre-trained multi-level GTA5-to-Cityscapes model and put it in the model folder

• Test the model and results will be saved in the result folder

python evaluate_cityscapes.py --restore-from ./model/GTA2Cityscapes_multi-ed35151c.pth

• Or, test the VGG-16 based model Model Link

python evaluate_cityscapes.py --model DeeplabVGG --restore-from ./model/GTA2Cityscapes_vgg-ac4ac9f6.pth

• Compute the IoU on Cityscapes (thanks to the code from VisDA Challenge)

python compute_iou.py ./data/Cityscapes/data/gtFine/val result/cityscapes

Training Examples

• NEW Train the GTA5-to-Cityscapes model (single-level with LS-GAN)

python train_gta2cityscapes_multi.py --snapshot-dir ./snapshots/GTA2Cityscapes_single_lsgan \
                                     --lambda-seg 0.0 \
                                     --lambda-adv-target1 0.0 --lambda-adv-target2 0.01 \
                                     --gan LS

• Train the GTA5-to-Cityscapes model (multi-level)

python train_gta2cityscapes_multi.py --snapshot-dir ./snapshots/GTA2Cityscapes_multi \
                                     --lambda-seg 0.1 \
                                     --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001

• Train the GTA5-to-Cityscapes model (single-level)

python train_gta2cityscapes_multi.py --snapshot-dir ./snapshots/GTA2Cityscapes_single \
                                     --lambda-seg 0.0 \
                                     --lambda-adv-target1 0.0 --lambda-adv-target2 0.001

Related Implementation and Dataset

• Y.-H. Tsai, K. Sohn, S. Schulter, and M. Chandraker. Domain Adaptation for Structured Output via Discriminative Patch Representations. In ICCV, 2019. (Oral) [paper] [project] [Implementation Guidance]
• W.-C. Hung, Y.-H Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang. Adversarial Learning for Semi-supervised Semantic Segmentation. In BMVC, 2018. [paper] [code]
• Y.-H. Chen, W.-Y. Chen, Y.-T. Chen, B.-C. Tsai, Y.-C. Frank Wang, and M. Sun. No More Discrimination: Cross City Adaptation of Road Scene Segmenters. In ICCV 2017. [paper] [project]

Acknowledgment

This code is heavily borrowed from Pytorch-Deeplab.

Note

The model and code are available for non-commercial research purposes only.

• 10/2019: update performance and training/evaluation codes for using LS-GAN and Synscapes (especially thanks to Yan-Ting Liu for helping experiments)
• 01/2019: upate the training code for PyTorch 0.4
• 07/23/2018: update evaluation code for PyTorch 0.4
• 06/04/2018: update pretrained VGG-16 model
• 02/2018: code released