High-Resolution Transformer for Dense Prediction
Introduction
This is the official implementation of High-Resolution Transformer (HRT). We present a High-Resolution Transformer (HRT) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformeron human pose estimation and semantic segmentation tasks.
- The High-Resolution Transformer architecture:
Pose estimation
2d Human Pose Estimation
Results on COCO val2017 with detector having human AP of 56.4 on COCO val2017 dataset
| Backbone | Input Size | AP | AP50 | AP75 | ARM | ARL | AR | ckpt | log | script |
|---|---|---|---|---|---|---|---|---|---|---|
| HRT-S | 256x192 | 74.0% | 90.2% | 81.2% | 70.4% | 80.7% | 79.4% | ckpt | log | script |
| HRT-S | 384x288 | 75.6% | 90.3% | 82.2% | 71.6% | 82.5% | 80.7% | ckpt | log | script |
| HRT-B | 256x192 | 75.6% | 90.8% | 82.8% | 71.7% | 82.6% | 80.8% | ckpt | log | script |
| HRT-B | 384x288 | 77.2% | 91.0% | 83.6% | 73.2% | 84.2% | 82.0% | ckpt | log | script |
Results on COCO test-dev with detector having human AP of 56.4 on COCO val2017 dataset
| Backbone | Input Size | AP | AP50 | AP75 | ARM | ARL | AR | ckpt | log | script |
|---|---|---|---|---|---|---|---|---|---|---|
| HRT-S | 384x288 | 74.5% | 92.3% | 82.1% | 70.7% | 80.6% | 79.8% | ckpt | log | script |
| HRT-B | 384x288 | 76.2% | 92.7% | 83.8% | 72.5% | 82.3% | 81.2% | ckpt | log | script |
The models are first pre-trained on ImageNet-1K dataset, and then fine-tuned on COCO val2017 dataset.
Semantic segmentation
Cityscapes
Performance on the Cityscapes dataset. The models are trained and tested with input size of 512x1024 and 1024x2048 respectively.
| Methods | Backbone | Window Size | Train Set | Test Set | Iterations | Batch Size | OHEM | mIoU | mIoU (Multi-Scale) | Log | ckpt | script |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OCRNet | HRT-S | 7x7 | Train | Val | 80000 | 8 | Yes | 80.0 | 81.0 | log | ckpt | script |
| OCRNet | HRT-B | 7x7 | Train | Val | 80000 | 8 | Yes | 81.4 | 82.0 | log | ckpt | script |
| OCRNet | HRT-B | 15x15 | Train | Val | 80000 | 8 | Yes | 81.9 | 82.6 | log | ckpt | script |
PASCAL-Context
The models are trained with the input size of 520x520, and tested with original size.
| Methods | Backbone | Window Size | Train Set | Test Set | Iterations | Batch Size | OHEM | mIoU | mIoU (Multi-Scale) | Log | ckpt | script |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OCRNet | HRT-S | 7x7 | Train | Val | 60000 | 16 | Yes | 53.8 | 54.6 | log | ckpt | script |
| OCRNet | HRT-B | 7x7 | Train | Val | 60000 | 16 | Yes | 56.3 | 57.1 | log | ckpt | script |
| OCRNet | HRT-B | 15x15 | Train | Val | 60000 | 16 | Yes | 57.6 | 58.5 | log | ckpt | script |
COCO-Stuff
The models are trained with input size of 520x520, and tested with original size.
| Methods | Backbone | Window Size | Train Set | Test Set | Iterations | Batch Size | OHEM | mIoU | mIoU (Multi-Scale) | Log | ckpt | script |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OCRNet | HRT-S | 7x7 | Train | Val | 60000 | 16 | Yes | 37.9 | 38.9 | log | ckpt | script |
| OCRNet | HRT-B | 7x7 | Train | Val | 60000 | 16 | Yes | 41.6 | 42.5 | log | ckpt | script |
| OCRNet | HRT-B | 15x15 | Train | Val | 60000 | 16 | Yes | 42.4 | 43.3 | log | ckpt | script |
ADE20K
The models are trained with input size of 520x520, and tested with original size. The results with window size 15x15 will be updated latter.
| Methods | Backbone | Window Size | Train Set | Test Set | Iterations | Batch Size | OHEM | mIoU | mIoU (Multi-Scale) | Log | ckpt | script |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OCRNet | HRT-S | 7x7 | Train | Val | 150000 | 8 | Yes | 44.0 | 45.1 | log | ckpt | script |
| OCRNet | HRT-B | 7x7 | Train | Val | 150000 | 8 | Yes | 46.3 | 47.6 | log | ckpt | script |
| OCRNet | HRT-B | 13x13 | Train | Val | 150000 | 8 | Yes | 48.7 | 50.0 | log | ckpt | script |
| OCRNet | HRT-B | 15x15 | Train | Val | 150000 | 8 | Yes | - | - | - | - | - |
Classification
Results on ImageNet-1K
| Backbone | [email protected] | [email protected] | #params | FLOPs | ckpt | log | script |
|---|---|---|---|---|---|---|---|
| HRT-T | 78.6% | 94.2% | 8.0M | 1.83G | ckpt | log | script |
| HRT-S | 81.2% | 95.6% | 13.5M | 3.56G | ckpt | log | script |
| HRT-B | 82.8% | 96.3% | 50.3M | 13.71G | ckpt | log | script |
Citation
If you find this project useful in your research, please consider cite:
@article{YuanFHZCW21,
title={HRT: High-Resolution Transformer for Dense Prediction},
author={Yuhui Yuan and Rao Fu and Lang Huang and Chao Zhang and Xilin Chen and Jingdong Wang},
booktitle={arXiv},
year={2021}
}
Acknowledgment
This project is developed based on the Swin-Transformer, openseg.pytorch, and mmpose.
git diff-index HEAD
git subtree add -P pose <url to sub-repo> <sub-repo branch>
3.2k Dec 29, 2022
84 Dec 27, 2022
1.8k Dec 30, 2022
25 Dec 13, 2022
2.3k Jan 03, 2023
25 Aug 11, 2021
74 Jan 03, 2023
26 Nov 02, 2022
67 Dec 03, 2022
11.5k Dec 30, 2022
19 Dec 07, 2022
127 Dec 31, 2022
1.8k Jan 08, 2023
184 Jan 04, 2023
4 Apr 17, 2021
4 Dec 02, 2021
199 Jan 03, 2023
1 Feb 01, 2022
6 Nov 24, 2022
86 Dec 28, 2022