groupnorm_backward反向公式推导

前向
- 均值
  μ n g = ∑ i = 1 M ( X i ) M (1) {\large \mathit{\color{Blue} \mu_{ng} = \frac{\sum_{i=1}^M(X^{i})}{M}} } \tag{1} μng=M∑i=1M(Xi)(1)
- 方差
  σ n g 2 = ∑ i = 1 M ( X i − μ n g ) M (2) {\large \mathit{\color{Blue} \sigma_{ng}^2 = \frac{\sum_{i = 1}^{M}(X^i - \mu_{ng})}{M}}} \tag{2} σng2=M∑i=1M(Xi−μng)(2)
- 归一化：
  
  令
  r s i g = 1 σ n g 2 + ε (3) {\large \mathit{\color{Blue}{rsig = \frac{1}{\sqrt{\sigma_{ng}^2 + \varepsilon}}}}} \tag{3} rsig=σng2+ε 1(3)
  
  则：
  Y = γ ∗ ( X − μ ) ∗ r s i g + β = γ ∗ X ∗ r s i g + β − γ ∗ μ ∗ r s i g (4) {\large \mathit{\color{Blue} Y = \gamma * (X - \mu) * rsig + \beta = \gamma * X * rsig + \beta - \gamma * \mu * rsig}} \tag{4} Y=γ∗(X−μ)∗rsig+β=γ∗X∗rsig+β−γ∗μ∗rsig(4)
反向

令
S = γ ∗ r s i g (5) {\large \mathit{\color{Blue} S = \gamma * rsig}} \tag{5} S=γ∗rsig(5)

B = β − γ ∗ μ ∗ r s i g (6) {\large \mathit{\color{Blue} B = \beta - \gamma * \mu * rsig}} \tag{6} B=β−γ∗μ∗rsig(6)

则
Y = S ∗ X + B {\large \mathit{\color{Blue}Y = S * X + B}} Y=S∗X+B

令
M = K × H × W （ K = C / G r o u p ） (7) {\large \mathit{\color{Blue} M = K × H × W （K = C / Group）}} \tag{7} M=K×H×W（K=C/Group）(7)

由链式法则：
d L d X = d L d Y ∗ d Y d X = d L d Y ∗ ( d ( S ∗ X ) d X + d B d X ) (8) {\large \mathit{\color{Blue} \frac{dL}{dX} = \frac{dL}{dY} * \frac{dY}{dX} = \frac{dL}{dY} * (\frac{d(S * X)}{dX} + \frac{dB}{dX})}} \tag{8} dXdL=dYdL∗dXdY=dYdL∗(dXd(S∗X)+dXdB)(8)

其中：

d ( S ∗ X ) d X = S + X ∗ d S d X = S + X ∗ γ ∗ d r s i g d X (9) {\large \mathit{\color{Blue} \frac{d(S * X)}{dX} = S + X * \frac{dS}{dX} = S + X * \gamma * \frac{drsig}{dX}}} \tag{9} dXd(S∗X)=S+X∗dXdS=S+X∗γ∗dXdrsig(9)

d B d X = − γ ∗ μ ∗ d r s i g d X − γ ∗ r s i g ∗ d μ d X {\large \mathit{\color{Blue} \frac{dB}{dX} = -\gamma * \mu * \frac{drsig}{dX} - \gamma * rsig * \frac{d\mu}{dX}}} dXdB=−γ∗μ∗dXdrsig−γ∗rsig∗dXdμ

d r s i g d X = − r s i g 3 ∗ ( X − μ ) M (10) {\large \mathit{\color{Blue} \frac{drsig}{dX} = -rsig^3 * \frac{(X -\mu)}{M}}} \tag{10} dXdrsig=−rsig3∗M(X−μ)(10)

d μ d X = 1 M (11) {\large \mathit{\color{Blue}\frac{d\mu}{dX} = \frac{1}{M}}} \tag{11} dXdμ=M1(11)

由(5),(8)(9)(10)(11)得：
d L d X = d y ∗ ( S + X ∗ γ ∗ r s i g 3 ∗ ( μ − X ) M + γ ∗ μ ∗ r s i g 3 ∗ ( X − μ ) M − γ ∗ r s i g M ) = d y ∗ S + d y ∗ γ ∗ r s i g 3 ∗ ( u − X ) M ∗ ( X − μ ) − d y ∗ γ ∗ r s i g M (12) {\large \mathit{\color{Blue} \frac{dL}{dX} = dy * (S + X * \gamma * rsig^3 * \frac{(\mu - X)}{M} + \gamma * \mu * rsig^3 * \frac{(X - \mu)}{M} - \frac{\gamma * rsig}{M})}} \\ {\large \mathit{\color{Blue} = dy * S + dy * \gamma * rsig^3 * \frac{(u - X)}{M} * (X - \mu) - dy * \frac{\gamma * rsig}{M}}}\tag{12} dXdL=dy∗(S+X∗γ∗rsig3∗M(μ−X)+γ∗μ∗rsig3∗M(X−μ)−Mγ∗rsig)=dy∗S+dy∗γ∗rsig3∗M(u−X)∗(X−μ)−dy∗Mγ∗rsig(12)

令
C 1 = S = γ ∗ r s i g C 2 = d y ∗ γ ∗ r s i g 3 ∗ μ − X M C 3 = − C 2 ∗ μ − d y ∗ γ ∗ r s i g M (13) {\large \mathit{\color{Blue} C_1 = S = \gamma * rsig}} \\ {\large \mathit{\color{Blue} C_2 = dy * \gamma * rsig^3 * \frac{\mu - X}{M}}} \\ {\large \mathit{\color{Blue} C_3 = -C_2 * \mu - \frac{dy * \gamma * rsig}{M}}} \tag{13} C1=S=γ∗rsigC2=dy∗γ∗rsig3∗Mμ−XC3=−C2∗μ−Mdy∗γ∗rsig(13)

得：
d x = C 1 ∗ d y + C 2 ∗ X + C 3 (14) {\large \mathit{\color{Blue} dx = C_1 * dy + C_2 * X + C_3}} \tag{14} dx=C1∗dy+C2∗X+C3(14)