arXiv-2019
https://github.com/GOATmessi7/ASFF
文章目录
- [1 Background and Motivation](#1 Background and Motivation)
- [2 Related Work](#2 Related Work)
- [3 Advantages / Contributions](#3 Advantages / Contributions)
- [4 Method](#4 Method)
-
- [4.1 Strong Baseline](#4.1 Strong Baseline)
- [4.2 Adaptively Spatial Feature Fusion](#4.2 Adaptively Spatial Feature Fusion)
-
- [4.2.1 Feature Resizing](#4.2.1 Feature Resizing)
- [4.2.2 Adaptive Fusion](#4.2.2 Adaptive Fusion)
- [4.3 Consistency Property](#4.3 Consistency Property)
- [5 Experiments](#5 Experiments)
-
- [5.1 Datasets and Metrics](#5.1 Datasets and Metrics)
- [5.2 Ablation Study](#5.2 Ablation Study)
- [5.3 Evaluation on Other Single-Shot Detector](#5.3 Evaluation on Other Single-Shot Detector)
- [5.4 Comparison to State of the Art](#5.4 Comparison to State of the Art)
- [6 Conclusion（own） / Future work](#6 Conclusion（own） / Future work)
1 Background and Motivation
目标检测任务中，特征金字塔技术可以缓解目标的 scale variation（同一类物体，物体的尺寸可能不一样）
the inconsistency across different feature scales is a primary limitation for the single-shot detectors based on feature pyramid（同一类物体，特征最好一样，但是由于尺寸原因，会分布在特征金字塔的不同 level 上，不同 level 的特征也没有强制协同，可能会影响效果）
if an image contains both small and large objects, the conflict among features at different levels tends to occupy the major part of the feature pyramid
作者提出 adaptively spatial feature fusion (ASFF)， improving the scale-invariance of features，nearly free inference overhead
2 Related Work
Feature pyramid representations or multi-level feature
still suffer from the inconsistency across different scales
作者的方法
adaptively learns the import degrees（入度） for different levels of features on each location to avoid spatial contradiction
3 Advantages / Contributions
提出 ASFF 模块，即插即用且基本 cost free，强化特征金字塔能力，to address the inconsistency in feature pyramids of single-shot detector
在 COCO 数据集上验证了其有效性
4 Method
4.1 Strong Baseline
开源的 yolov3 基础上，引入了一些比较好的 trick，效果提升明显
BoF 是 Bag of freebies
Zhang Z, He T, Zhang H, et al. Bag of freebies for training object detection neural networksJ. arXiv preprint arXiv:1902.04103, 2019.
GA 是 guided anchoring
Wang J, Chen K, Yang S, et al. Region proposal by guided anchoringC//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2965-2974.
细节可以跳转到本博客最后总结部分
IoU 指的是额外引入了 IoU loss
4.2 Adaptively Spatial Feature Fusion
adaptively learn the spatial weight of fusion for feature maps at each scale
本文的核心
4.2.1 Feature Resizing
下采样 2 倍时，2-stride 1x1 convolution
下采样 4 倍时，add a 2-stride max pooling layer before the 2-stride convolution
上采样用的插值
4.2.2 Adaptive Fusion
核心公式
Let x i j n → l x_{ij}^{n →l} xijn→ldenote the feature vector at the position (i, j) on the feature maps resized from level n n n to level l l l.
特征金字塔 resize 到同一尺寸，然后加权在一起，只不过加权的系数是 learning 出来的，权重 shared across all the channels，有点类似于空间注意力
α i j l + β i j l + γ i j l = 1 \alpha_{ij}^{l} + \beta_{ij}^{l} + \gamma_{ij}^{l} = 1 αijl+βijl+γijl=1
加权系数约束到了和为1，实现的话就是 softmax
λ \lambda λ 为 control parameters------代码中好像没有体现
看看代码
python
class ASFF(nn.Module):
def __init__(self, level, rfb=False, vis=False):
super(ASFF, self).__init__()
self.level = level
self.dim = [512, 256, 256]
self.inter_dim = self.dim[self.level]
if level==0:
self.stride_level_1 = add_conv(256, self.inter_dim, 3, 2)
self.stride_level_2 = add_conv(256, self.inter_dim, 3, 2)
self.expand = add_conv(self.inter_dim, 1024, 3, 1)
elif level==1:
self.compress_level_0 = add_conv(512, self.inter_dim, 1, 1)
self.stride_level_2 = add_conv(256, self.inter_dim, 3, 2)
self.expand = add_conv(self.inter_dim, 512, 3, 1)
elif level==2:
self.compress_level_0 = add_conv(512, self.inter_dim, 1, 1)
self.expand = add_conv(self.inter_dim, 256, 3, 1)
compress_c = 8 if rfb else 16 #when adding rfb, we use half number of channels to save memory
self.weight_level_0 = add_conv(self.inter_dim, compress_c, 1, 1)
self.weight_level_1 = add_conv(self.inter_dim, compress_c, 1, 1)
self.weight_level_2 = add_conv(self.inter_dim, compress_c, 1, 1)
self.weight_levels = nn.Conv2d(compress_c*3, 3, kernel_size=1, stride=1, padding=0)
self.vis= vis
def forward(self, x_level_0, x_level_1, x_level_2):
if self.level==0:
level_0_resized = x_level_0
level_1_resized = self.stride_level_1(x_level_1)
level_2_downsampled_inter =F.max_pool2d(x_level_2, 3, stride=2, padding=1)
level_2_resized = self.stride_level_2(level_2_downsampled_inter)
elif self.level==1:
level_0_compressed = self.compress_level_0(x_level_0)
level_0_resized =F.interpolate(level_0_compressed, scale_factor=2, mode='nearest')
level_1_resized =x_level_1
level_2_resized =self.stride_level_2(x_level_2)
elif self.level==2:
level_0_compressed = self.compress_level_0(x_level_0)
level_0_resized =F.interpolate(level_0_compressed, scale_factor=4, mode='nearest')
level_1_resized =F.interpolate(x_level_1, scale_factor=2, mode='nearest')
level_2_resized =x_level_2
level_0_weight_v = self.weight_level_0(level_0_resized) # 缩放后的特征图压缩成 16 通道
level_1_weight_v = self.weight_level_1(level_1_resized) # 缩放后的特征图压缩成 16 通道
level_2_weight_v = self.weight_level_2(level_2_resized) # 缩放后的特征图压缩成 16 通道
levels_weight_v = torch.cat((level_0_weight_v, level_1_weight_v, level_2_weight_v),1) # concat 在一起
levels_weight = self.weight_levels(levels_weight_v) # 变成 3 通道
levels_weight = F.softmax(levels_weight, dim=1) # 沿通道做 softmax
fused_out_reduced = level_0_resized * levels_weight[:,0:1,:,:]+\
level_1_resized * levels_weight[:,1:2,:,:]+\
level_2_resized * levels_weight[:,2:,:,:] # 与缩放后的特征图加权在一起
out = self.expand(fused_out_reduced) # 扩充通道数
if self.vis:
return out, levels_weight, fused_out_reduced.sum(dim=1)
else:
return out其中 add_conv 定义如下
python
def add_conv(in_ch, out_ch, ksize, stride, leaky=True):
"""
Add a conv2d / batchnorm / leaky ReLU block.
Args:
in_ch (int): number of input channels of the convolution layer.
out_ch (int): number of output channels of the convolution layer.
ksize (int): kernel size of the convolution layer.
stride (int): stride of the convolution layer.
Returns:
stage (Sequential) : Sequential layers composing a convolution block.
"""
stage = nn.Sequential()
pad = (ksize - 1) // 2
stage.add_module('conv', nn.Conv2d(in_channels=in_ch,
out_channels=out_ch, kernel_size=ksize, stride=stride,
padding=pad, bias=False))
stage.add_module('batch_norm', nn.BatchNorm2d(out_ch))
if leaky:
stage.add_module('leaky', nn.LeakyReLU(0.1))
else:
stage.add_module('relu6', nn.ReLU6(inplace=True))
return stagelevel = 0
level = 1
level = 2
4.3 Consistency Property
反向传播推导推导
简化一下
感觉 resize 的时候如果涉及到了 conv + activation 的话，不太能简化吧，哈哈
进一步简化，当多个特征图融合的方式为 add 或者 concat 的时候
结果为
作者方法的反向传播公式为
这样通过设置 α \alpha α 就可以避免各 level 梯度的影响
比如目标由 level 1 负责预测， α i j 1 = 1 \alpha_{ij}^1 = 1 αij1=1, α i j 2 = 0 \alpha_{ij}^2 = 0 αij2=0， α i j 3 = 1 \alpha_{ij}^3 = 1 αij3=1
5 Experiments
5.1 Datasets and Metrics
MS COCO 2017
AP
5.2 Ablation Study
（1）Solid Baseline
Table1，前面第四小节已介绍过了
（2） Effectiveness of Adjacent Ignore Regions
前面说梯度的时候说 ignor 不好，这里又是 ignore area，可能我还没有理解到精髓，需看看参考文献和代码加深下理解
（3）Adaptively Spatial Feature Fusion
exhibit the images that have several objects of different sizes
5.3 Evaluation on Other Single-Shot Detector
体现了其即插即用
5.4 Comparison to State of the Art
6 Conclusion（own） / Future work
- trained to find the optimal fusion
- fusion is differential（可微分的，也即可以反向传播）
Wang J, Chen K, Yang S, et al. Region proposal by guided anchoringC//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2965-2974.
