1. BaseInfo
| Title | UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces |
| Adress | https://arxiv.org/abs/2312.15715 |
| Journal/Time | ICCV2023 |
| Author | 港大、字节 |
| Code | https://github.com/FoundationVision/UniRef |
| Read | 20241002 |
2. Creative Q&A
- referring image segmentation (RIS), few-shot image segmentation (FSS), referring video object segmentation (RVOS), and video object segmentation (VOS) 将四个任务结合起来的大一统框架。是 UniRef 的升级版。(VOS / FSS 是 图+掩码，RVOS 是 图+掩码+文本)
- UniFusion 处理不同任务的模块。
- a unified Transformer architecture 实例级分割
3. Concrete
用分层的方式融合视觉和参考信息。
3.1. Model
Unifusion 用视觉做 Q，参考特征做 K 和 V
对参考特征做池化和回归。有 scale 、shift 和 gate 参数。
对多头交叉注意力出来的特征利用参数再残差。
使用 FlashAttention 执行跨注意力操作，当计算稠密的特征图时效率更高且内存消耗更小；受 adaLN-zero 块的启发，偏移和门的参数都是 zero-initialized 的。
3.1.1. Input
图 + 文 + 掩码
3.1.2. Backbone
Resnet 50 / Swin-L
Bert-base / RoBERT-base
3.1.3. Neck
3.1.4. Decoder
DeformableDETR（mask 的动态卷积核参数） + DynamicConv 类似 FPN 结构的，只使用了 2 3 4层的特征。最终要得到 H/4 x W/4 x C
3.1.5. Loss
set as λcls = 2.0, λcls = 2.0, λL1 = 5.0,λmask = 2.0 and λdice = 5.0, respectively
3.2. Training
3.2.1. Resource
NVIDIA A100 GPUs.
4 × 8 A100 GPUs for the objects365 pretraining and 2 × 8 GPUs for the following image-level and video-level training.
3.2.2 Dataset
(i) RIS: RefCOCO 122 consists of 142,209 language descriptions for 50,000 objects in 19,994 images. RefCOCO+ 122 has 141,564 expressions for 49,856 objects in 19,992 images. RefCOCOg 67 includes 85,474 referring expressions for 54,822 objects in 26,711 images. And we use the UMD split for RefCOCOg 67.
(ii) FSS: FSS-1000 50 is a large-scale dataset for FSS task. It contains 10,000 images from 1,000 classes.
(iii) RVOS: Ref-Youtube-VOS 82 is a large-scale referring video object segmentation dataset which contains 3,978 videos with around 15k langauge descriptions. Ref-DAVIS17 42 provides the referring expressions for each object in DAVIS17 75. It contains 90 videos in total.
(iv) VOS: Youtube-VOS1 109 is the popular benchmark for video object segmentation. There are 474 and 507 videos in the validation set for 2018 and 2019 version, respectively. LVOS 31 is a long-term video object segmentation benchmark consisting of 220 videos.videos in LVOS have an average duration of 1.59 minutes, and the videos in Youtube-VOS last 6 seconds on average. MOSE 21 is a newly proposed dataset for evaluating VOS algorithms in complex scenes, such as occlusion and disappearance. It have 2,149 videos clips and 5,200 objects from 36 categories, with a total of 431,725 annotated masks.
3.3. Eval
Table 1: 感觉不算公平对比阿，用的 encoder 都不一样。
3.4. Ablation
UniFusion 在 SAM 中即插即用。
4. Reference
5. Additional
含附录
附件有实验具体配置。