groupnorm_backward反向公式推导

  1. 前向

    • 均值
      μ n g = ∑ i = 1 M ( X i ) M (1) {\large \mathit{\color{Blue} \mu_{ng} = \frac{\sum_{i=1}^M(X^{i})}{M}} } \tag{1} μng=M∑i=1M(Xi)(1)

    • 方差
      σ n g 2 = ∑ i = 1 M ( X i − μ n g ) M (2) {\large \mathit{\color{Blue} \sigma_{ng}^2 = \frac{\sum_{i = 1}^{M}(X^i - \mu_{ng})}{M}}} \tag{2} σng2=M∑i=1M(Xi−μng)(2)

    • 归一化:


      r s i g = 1 σ n g 2 + ε (3) {\large \mathit{\color{Blue}{rsig = \frac{1}{\sqrt{\sigma_{ng}^2 + \varepsilon}}}}} \tag{3} rsig=σng2+ε 1(3)

      则:
      Y = γ ∗ ( X − μ ) ∗ r s i g + β = γ ∗ X ∗ r s i g + β − γ ∗ μ ∗ r s i g (4) {\large \mathit{\color{Blue} Y = \gamma * (X - \mu) * rsig + \beta = \gamma * X * rsig + \beta - \gamma * \mu * rsig}} \tag{4} Y=γ∗(X−μ)∗rsig+β=γ∗X∗rsig+β−γ∗μ∗rsig(4)

  2. 反向


    S = γ ∗ r s i g (5) {\large \mathit{\color{Blue} S = \gamma * rsig}} \tag{5} S=γ∗rsig(5)

    B = β − γ ∗ μ ∗ r s i g (6) {\large \mathit{\color{Blue} B = \beta - \gamma * \mu * rsig}} \tag{6} B=β−γ∗μ∗rsig(6)


    Y = S ∗ X + B {\large \mathit{\color{Blue}Y = S * X + B}} Y=S∗X+B


    M = K × H × W ( K = C / G r o u p ) (7) {\large \mathit{\color{Blue} M = K × H × W (K = C / Group)}} \tag{7} M=K×H×W(K=C/Group)(7)

    由链式法则:
    d L d X = d L d Y ∗ d Y d X = d L d Y ∗ ( d ( S ∗ X ) d X + d B d X ) (8) {\large \mathit{\color{Blue} \frac{dL}{dX} = \frac{dL}{dY} * \frac{dY}{dX} = \frac{dL}{dY} * (\frac{d(S * X)}{dX} + \frac{dB}{dX})}} \tag{8} dXdL=dYdL∗dXdY=dYdL∗(dXd(S∗X)+dXdB)(8)

    其中:

    d ( S ∗ X ) d X = S + X ∗ d S d X = S + X ∗ γ ∗ d r s i g d X (9) {\large \mathit{\color{Blue} \frac{d(S * X)}{dX} = S + X * \frac{dS}{dX} = S + X * \gamma * \frac{drsig}{dX}}} \tag{9} dXd(S∗X)=S+X∗dXdS=S+X∗γ∗dXdrsig(9)

    d B d X = − γ ∗ μ ∗ d r s i g d X − γ ∗ r s i g ∗ d μ d X {\large \mathit{\color{Blue} \frac{dB}{dX} = -\gamma * \mu * \frac{drsig}{dX} - \gamma * rsig * \frac{d\mu}{dX}}} dXdB=−γ∗μ∗dXdrsig−γ∗rsig∗dXdμ

    d r s i g d X = − r s i g 3 ∗ ( X − μ ) M (10) {\large \mathit{\color{Blue} \frac{drsig}{dX} = -rsig^3 * \frac{(X -\mu)}{M}}} \tag{10} dXdrsig=−rsig3∗M(X−μ)(10)

    d μ d X = 1 M (11) {\large \mathit{\color{Blue}\frac{d\mu}{dX} = \frac{1}{M}}} \tag{11} dXdμ=M1(11)

    由(5),(8)(9)(10)(11)得:
    d L d X = d y ∗ ( S + X ∗ γ ∗ r s i g 3 ∗ ( μ − X ) M + γ ∗ μ ∗ r s i g 3 ∗ ( X − μ ) M − γ ∗ r s i g M ) = d y ∗ S + d y ∗ γ ∗ r s i g 3 ∗ ( u − X ) M ∗ ( X − μ ) − d y ∗ γ ∗ r s i g M (12) {\large \mathit{\color{Blue} \frac{dL}{dX} = dy * (S + X * \gamma * rsig^3 * \frac{(\mu - X)}{M} + \gamma * \mu * rsig^3 * \frac{(X - \mu)}{M} - \frac{\gamma * rsig}{M})}} \\ {\large \mathit{\color{Blue} = dy * S + dy * \gamma * rsig^3 * \frac{(u - X)}{M} * (X - \mu) - dy * \frac{\gamma * rsig}{M}}}\tag{12} dXdL=dy∗(S+X∗γ∗rsig3∗M(μ−X)+γ∗μ∗rsig3∗M(X−μ)−Mγ∗rsig)=dy∗S+dy∗γ∗rsig3∗M(u−X)∗(X−μ)−dy∗Mγ∗rsig(12)


    C 1 = S = γ ∗ r s i g C 2 = d y ∗ γ ∗ r s i g 3 ∗ μ − X M C 3 = − C 2 ∗ μ − d y ∗ γ ∗ r s i g M (13) {\large \mathit{\color{Blue} C_1 = S = \gamma * rsig}} \\ {\large \mathit{\color{Blue} C_2 = dy * \gamma * rsig^3 * \frac{\mu - X}{M}}} \\ {\large \mathit{\color{Blue} C_3 = -C_2 * \mu - \frac{dy * \gamma * rsig}{M}}} \tag{13} C1=S=γ∗rsigC2=dy∗γ∗rsig3∗Mμ−XC3=−C2∗μ−Mdy∗γ∗rsig(13)

    得:
    d x = C 1 ∗ d y + C 2 ∗ X + C 3 (14) {\large \mathit{\color{Blue} dx = C_1 * dy + C_2 * X + C_3}} \tag{14} dx=C1∗dy+C2∗X+C3(14)

相关推荐
eason_fan6 分钟前
前端面试手撕代码(字节)
前端·算法·面试
今天_也很困13 分钟前
牛客2025年愚人节比赛
c++·算法
Joe_Wang515 分钟前
[图论]拓扑排序
数据结构·c++·算法·leetcode·图论·拓扑排序
2401_8582861135 分钟前
CD21.【C++ Dev】类和对象(12) 流插入运算符的重载
开发语言·c++·算法·类和对象·运算符重载
梭七y1 小时前
【力扣hot100题】(033)合并K个升序链表
算法·leetcode·链表
月亮被咬碎成星星1 小时前
LeetCode[383]赎金信
算法·leetcode
无难事者若执1 小时前
新手村:逻辑回归-理解03:逻辑回归中的最大似然函数
算法·机器学习·逻辑回归
写代码写到手抽筋1 小时前
C++多线程的性能优化
java·c++·性能优化
IT从业者张某某2 小时前
机器学习-04-分类算法-03KNN算法案例
算法·机器学习·分类
chen_song_2 小时前
WebRTC的ICE之TURN协议的交互流程中继转发Relay媒体数据的turnserver的测试
算法·音视频·webrtc·交互·媒体