By Byungseok Roh*, Jaewoong Shin*, Wuhyun Shin*, and Saehoon Kim at Kakao Brain. (*: Equal contribution)
TL; DR. Sparse DETR is an efficient end-to-end object detector that sparsifies encoder tokens by using the learnable DAM(Decoder Attention Map) predictor. It achieves better performance than Deformable DETR even with only 10% encoder queries on the COCO dataset.
Abstract. DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency on high resolution feature maps. The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10x faster convergence and improved performance. Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck. In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated. Inspired by this observation, we propose Sparse DETR that selectively updates only the tokens expected to be referenced by the decoder, thus help the model effectively detect objects. In addition, we show that applying an auxiliary detection loss on the selected tokens in the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens on the COCO dataset. Albeit only the encoder tokens are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR.
We have tested the code on the following environments:
Run the following command to install dependencies:
pip install -r requirements.txt
cd ./models/ops sh ./make.sh # unit test (should see all checking is True) python test.py
Please download COCO 2017 dataset and organize them as follows:
code_root/
└── data/
└── coco/
├── train2017/
├── val2017/
└── annotations/
├── instances_train2017.json
└── instances_val2017.json
For example, the command for training Sparse DETR with the keeping ratio of 10% on 8 GPUs is as follows:
$ GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/swint_sparse_detr_rho_0.1.shTraining on multiple nodes
For example, the command Sparse DETR with the keeping ratio of 10% on 2 nodes of each with 8 GPUs is as follows:
On node 1:
$ MASTER_ADDR=<IP address of node 1> NODE_RANK=0 GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 16 ./configs/swint_sparse_detr_rho_0.1.sh
On node 2:
$ MASTER_ADDR=<IP address of node 2> NODE_RANK=1 GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 16 ./configs/swint_sparse_detr_rho_0.1.sh
# Deformable DETR (with bounding-box-refinement and two-stage argument, if wanted) $ GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 python main.py --with_box_refine --two_stage # Efficient DETR (with the class-specific head as describe in their paper) $ GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 python main.py --with_box_refine --two_stage --eff_query_init --eff_specific_head # Sparse DETR (with the keeping ratio of 10% and encoder auxiliary loss) $ GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 python main.py --with_box_refine --two_stage --eff_query_init --eff_specific_head --rho 0.1 --use_enc_aux_lossSome tips to speed-up training
You can get the pre-trained model of Sparse DETR (the link is in "Main Results" session), then run the following command to evaluate it on COCO 2017 validation set:
# Note that you should run the command with the corresponding configuration. $ ./configs/swint_sparse_detr_rho_0.1.sh --resume <path to pre-trained model> --eval
You can also run distributed evaluation by using ./tools/run_dist_launch.sh.
The tables below demonstrate the detection performance of Sparse DETR on the COCO 2017 validation set when using different backbones.
The performance of Sparse DETR can be further improved when the backbone network is initialized with the SCRL(Spatially Consistent Representation Learning) that aims to learn dense representations in a self-supervised way, compared to the default initialization with the ImageNet pre-trained one, denoted as IN-sup in the table below.
IN-sup, and the SCRL GitHub repository for SCRL.SCRL results, add --scrl_pretrained_path <downloaded_filepath> to the training command.If you find Sparse DETR useful in your research, please consider citing:
@inproceedings{roh2022sparse,
title={Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity},
author={Roh, Byungseok and Shin, JaeWoong and Shin, Wuhyun and Kim, Saehoon},
booktitle={ICLR},
year={2022}
}
This project is released under the Apache 2.0 license. Copyright 2021 Kakao Brain Corp. All Rights Reserved.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4