[VL|Ref]UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces

1. BaseInfo

Title UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
Adress https://arxiv.org/abs/2312.15715
Journal/Time ICCV2023
Author 港大、字节
Code https://github.com/FoundationVision/UniRef
Read 20241002

2. Creative Q&A

  1. referring image segmentation (RIS), few-shot image segmentation (FSS), referring video object segmentation (RVOS), and video object segmentation (VOS) 将四个任务结合起来的大一统框架。是 UniRef 的升级版。(VOS / FSS 是 图+掩码,RVOS 是 图+掩码+文本)
  2. UniFusion 处理不同任务的模块。
  3. a unified Transformer architecture 实例级分割

3. Concrete

用分层的方式融合视觉和参考信息。

3.1. Model

Unifusion 用视觉做 Q,参考特征做 K 和 V

对参考特征做池化和回归。有 scale 、shift 和 gate 参数。

对多头交叉注意力出来的特征利用参数再残差。

使用 FlashAttention 执行跨注意力操作,当计算稠密的特征图时效率更高且内存消耗更小;受 adaLN-zero 块的启发,偏移和门的参数都是 zero-initialized 的。

3.1.1. Input

图 + 文 + 掩码

3.1.2. Backbone

Resnet 50 / Swin-L

Bert-base / RoBERT-base

3.1.3. Neck

3.1.4. Decoder

DeformableDETR(mask 的动态卷积核参数) + DynamicConv 类似 FPN 结构的,只使用了 2 3 4层的特征。最终要得到 H/4 x W/4 x C

3.1.5. Loss

set as λcls = 2.0, λcls = 2.0, λL1 = 5.0,λmask = 2.0 and λdice = 5.0, respectively

3.2. Training

3.2.1. Resource

NVIDIA A100 GPUs.

4 × 8 A100 GPUs for the objects365 pretraining and 2 × 8 GPUs for the following image-level and video-level training.

3.2.2 Dataset

(i) RIS: RefCOCO 122 consists of 142,209 language descriptions for 50,000 objects in 19,994 images. RefCOCO+ 122 has 141,564 expressions for 49,856 objects in 19,992 images. RefCOCOg 67 includes 85,474 referring expressions for 54,822 objects in 26,711 images. And we use the UMD split for RefCOCOg 67.

(ii) FSS: FSS-1000 50 is a large-scale dataset for FSS task. It contains 10,000 images from 1,000 classes.

(iii) RVOS: Ref-Youtube-VOS 82 is a large-scale referring video object segmentation dataset which contains 3,978 videos with around 15k langauge descriptions. Ref-DAVIS17 42 provides the referring expressions for each object in DAVIS17 75. It contains 90 videos in total.

(iv) VOS: Youtube-VOS1 109 is the popular benchmark for video object segmentation. There are 474 and 507 videos in the validation set for 2018 and 2019 version, respectively. LVOS 31 is a long-term video object segmentation benchmark consisting of 220 videos.videos in LVOS have an average duration of 1.59 minutes, and the videos in Youtube-VOS last 6 seconds on average. MOSE 21 is a newly proposed dataset for evaluating VOS algorithms in complex scenes, such as occlusion and disappearance. It have 2,149 videos clips and 5,200 objects from 36 categories, with a total of 431,725 annotated masks.

3.3. Eval

Table 1: 感觉不算公平对比阿,用的 encoder 都不一样。

3.4. Ablation

UniFusion 在 SAM 中即插即用。

4. Reference

5. Additional

含附录

附件有实验具体配置。

相关推荐
databook15 分钟前
处理一批 PDF,怎么算真的处理完了
后端·python
土豆125022 分钟前
你只说要什么,它负责拆活交付:任务编排插件上手指南
人工智能·llm
承渊政道2 小时前
【Python编程快速上手—让繁琐工作自动化】(列表:把数据排成队,从下标取值到深拷贝)
开发语言·python·pycharm·自动化·列表
GoldenSpider.AI2 小时前
讲透 Claude Code 的 Code Mode:原理、流程、用法与 24 轮实测
人工智能·ai编程·claude code
oak隔壁找我2 小时前
一款运行在 DSH(DeepSeek Harness)中的工作区文件管理插件
人工智能
大胖爱学习_MJ2 小时前
Agent 如何安全地拿数据库密码?基于 HashiCorp Vault 的动态凭证下发
经验分享·python
江屿风2 小时前
AI Agent架构全解析:从规划到记忆的智能跃迁
人工智能·笔记·架构·agent·c/s·mcp工作流
ZONGYUAN-ROOT2 小时前
【API配额】多源 LLM API 配额池化实战:轮换调度、变化量监控、故障自愈一套方案(Python 完整实现)
人工智能
syso_稻草人3 小时前
qwen-image-2.1 高清放大拆解:Qwen21LatentUpscale 潜空间 2x + denoise 0.35 修细节,12 组场景实测
人工智能·stable diffusion·prompt·comfyui·qwen-image2.1