单个样本点特征向量使用列向量表示，样本点数据 x i x_i xi表示如下，共有n个特征
x i = ( x i 0 x i 1 ⋮ x i n ) x_i = \begin{pmatrix} x_{i0} \\ x_{i1} \\ \vdots \\ x_{in} \\ \end{pmatrix} xi= xi0xi1⋮xin
全部样本点特征向量使用矩阵进行表示，样本点数据 X X X表示如下，共有m个样本点，每行代表一个样本点
X = x 1 T x 2 T ⋮ x m T = x 11 x 12 ⋯ x 1 n x 21 x 22 ⋯ x 2 n ⋮ ⋮ ⋮ ⋮ x m 1 x m 2 ⋯ x m n X=\begin{bmatrix} x_1^{T} \\ x_2^{T} \\ \vdots \\ x_m^{T} \end{bmatrix}= \begin{bmatrix} x_{11} &x_{12} &\cdots &x_{1n}\\ x_{21} &x_{22} &\cdots &x_{2n}\\ \vdots &\vdots &\vdots &\vdots \\ x_{m1} &x_{m2} &\cdots &x_{mn} \end{bmatrix} X= x1Tx2T⋮xmT = x11x21⋮xm1x12x22⋮xm2⋯⋯⋮⋯x1nx2n⋮xmn
因为共有n个特征，权重向量使用列向量表示，全部样本点共用一个权重向量 w w w表示如下
w = ( w 1 w 2 ⋮ w n ) w = \begin{pmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{pmatrix} w= w1w2⋮wn
预测值 y ^ \hat{y} y^表示方式如下，每个分量代表每行样本的预测值
y ^ = X w = x 11 x 12 ⋯ x 1 n x 21 x 22 ⋯ x 2 n ⋮ ⋮ ⋮ ⋮ x m 1 x m 2 ⋯ x m n ( w 1 w 2 ⋮ w n ) = x 11 w 1 + x 12 w 2 + . . . + x 1 n w n x 21 w 1 + x 22 w 2 + . . . + x 2 n w n ⋮ x m 1 w 1 + x m 2 w 2 + . . . + x m n w n = ( y ^ 1 y ^ 2 ⋮ y ^ m ) \scriptsize{ \begin{split} \hat y &= Xw =\begin{bmatrix} x_{11} &x_{12} &\cdots &x_{1n} \\ x_{21} &x_{22} &\cdots &x_{2n} \\ \vdots &\vdots &\vdots &\vdots \\ x_{m1} &x_{m2} &\cdots &x_{mn} \end{bmatrix} \begin{pmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{pmatrix} = \begin{bmatrix} x_{11}w_1+x_{12}w_2+...+x_{1n}w_n \\ x_{21}w_1+x_{22}w_2+...+x_{2n}w_n \\ \vdots \\ x_{m1}w_1+x_{m2}w_2+...+x_{mn}w_n \\ \end{bmatrix} = \begin{pmatrix} \hat{y}{1} \\ \hat{y}{2} \\ \vdots \\ \hat{y}_{m} \\ \end{pmatrix} \end{split}} y^=Xw= x11x21⋮xm1x12x22⋮xm2⋯⋯⋮⋯x1nx2n⋮xmn w1w2⋮wn = x11w1+x12w2+...+x1nwnx21w1+x22w2+...+x2nwn⋮xm1w1+xm2w2+...+xmnwn = y^1y^2⋮y^m
将预测值 y ^ \hat{y} y^换一种写法，添加上截距项
y ^ = X w = 1 x 11 x 12 ⋯ x 1 n 1 x 21 x 22 ⋯ x 2 n ⋮ ⋮ ⋮ ⋮ ⋮ 1 x m 1 x m 2 ⋯ x m n ( w 0 w 1 ⋮ w n ) = w 0 + x 11 w 1 + x 12 w 2 + ⋯ + x 1 n w n w 0 + x 21 w 1 + x 22 w 2 + ⋯ + x 2 n w n ⋮ w 0 + x m 1 w 1 + x m 2 w 2 + ⋯ + x m n w n = ( y ^ 1 y ^ 2 ⋮ y ^ m ) \scriptsize{ \begin{split} \hat y &= Xw =\begin{bmatrix} 1 & x_{11} &x_{12} &\cdots &x_{1n} \\ 1 & x_{21} &x_{22} &\cdots &x_{2n} \\ \vdots &\vdots &\vdots &\vdots &\vdots \\ 1 & x_{m1} &x_{m2} &\cdots &x_{mn} \end{bmatrix} \begin{pmatrix} w_0 \\ w_1 \\ \vdots \\ w_n \end{pmatrix} = \begin{bmatrix} w_0 + x_{11}w_1+ x_{12}w_2 + \cdots + x_{1n}w_n \\ w_0 + x_{21}w_1+x_{22}w_2 + \cdots + x_{2n}w_n \\ \vdots \\ w_0 + x_{m1}w_1+x_{m2}w_2 + \cdots + x_{mn}w_n \\ \end{bmatrix} = \begin{pmatrix} \hat{y}{1} \\ \hat{y}{2} \\ \vdots \\ \hat{y}_{m} \\ \end{pmatrix} \end{split}} y^=Xw= 11⋮1x11x21⋮xm1x12x22⋮xm2⋯⋯⋮⋯x1nx2n⋮xmn w0w1⋮wn = w0+x11w1+x12w2+⋯+x1nwnw0+x21w1+x22w2+⋯+x2nwn⋮w0+xm1w1+xm2w2+⋯+xmnwn = y^1y^2⋮y^m
此时单个样本点 y ^ i \hat{y}_{i} y^i表达式如下，即多元线性回归表达式
y ^ i = w 0 + x i 1 w 1 + x i 2 w 2 + ⋯ + x i n w n = ( w 0 w 1 ⋯ w n ) ⋅ ( 1 x i 1 ⋮ x i n ) = w T x i \begin{split} \hat{y}{i} &= w_0 + x{i1}w_1+ x_{i2}w_2 + \cdots + x_{in}w_n \\ &= \begin{pmatrix} w_0 & w_1 &\cdots &w_n \end{pmatrix} \cdot \begin{pmatrix} 1 \\ x_{i1} \\ \vdots \\ x_{in} \\ \end{pmatrix}\\ &= w^{T}x_{i} \end{split} y^i=w0+xi1w1+xi2w2+⋯+xinwn=(w0w1⋯wn)⋅ 1xi1⋮xin =wTxi
定义如下损失函数
L ( w ) = 1 m ( y 1 \^ − y 1 ) 2 + ( y 2 \^ − y 2 ) 2 + ⋯ + ( y m \^ − y m ) 2 = 1 m ∑ i m ( y i ^ − y i ) 2 = 1 m ∑ i m ( w T x i − y i ) 2 \begin{split} L(w) &= \cfrac{1}{m}\begin{bmatrix} (\hat{y_1}-y_1)^2 + (\hat{y_2}-y_2)^2 + \cdots + (\hat{y_m}-y_m)^2 \end{bmatrix} \\ &= \cfrac{1}{m} \sum_{i}^{m} (\hat{y_i}-y_i)^2 \\ &= \cfrac{1}{m} \sum_{i}^{m} (w^{T}x_i-y_i)^2 \end{split} L(w)=m1(y1\^−y1)2+(y2\^−y2)2+⋯+(ym\^−ym)2=m1i∑m(yi^−yi)2=m1i∑m(wTxi−yi)2
损失函数格式转换
( X w − y ) T ( X w − y ) = ( y ^ 1 − y 1 y ^ 2 − y 2 ⋯ y ^ m − y m ) ( y ^ 1 − y 1 y ^ 2 − y 2 ⋮ y ^ m − y m ) = ∑ i m ( y ^ i − y i ) 2 = ∑ i m ( w T x i − y i ) 2 \begin{split} (Xw-y)^{T}(Xw-y) &= \begin{pmatrix} \hat{y}{1} - y{1} & \hat{y}{2} - y{2} &\cdots &\hat{y}{m} - y{m} \end{pmatrix} \begin{pmatrix} \hat{y}{1} - y{1} \\ \hat{y}{2} - y{2} \\ \vdots \\ \hat{y}{m} - y{m} \\ \end{pmatrix} \\ &= \sum_{i}^{m}(\hat{y}{i} - y{i})^2 \\ &= \sum_{i}^{m} (w^{T}x_i-y_i)^2 \end{split} (Xw−y)T(Xw−y)=(y^1−y1y^2−y2⋯y^m−ym) y^1−y1y^2−y2⋮y^m−ym =i∑m(y^i−yi)2=i∑m(wTxi−yi)2
损失函数向量化表示
L ( w ) = 1 m ∑ i m ( w T x i − y i ) 2 = 1 m ( X w − y ) T ( X w − y ) L(w) = \cfrac{1}{m} \sum_{i}^{m} (w^{T}x_i-y_i)^2 = \cfrac{1}{m} (Xw-y)^{T}(Xw-y) L(w)=m1i∑m(wTxi−yi)2=m1(Xw−y)T(Xw−y)
定义线性回归的目标函数，即当 L ( w ) L(w) L(w)取最小值时 w w w是多少
w ∗ = arg min w L ( w ) = arg min w 1 m ∑ i m ( w T x i − y i ) 2 = arg min w 1 m ( X w − y ) T ( X w − y ) w^{*} = \mathop{\arg\min}\limits_{w} L(w) = \mathop{\arg\min}\limits_{w} \cfrac{1}{m} \sum_{i}^{m} (w^{T}x_i-y_i)^2 = \mathop{\arg\min}\limits_{w} \cfrac{1}{m} (Xw-y)^{T}(Xw-y) w∗=wargminL(w)=wargminm1i∑m(wTxi−yi)2=wargminm1(Xw−y)T(Xw−y)
损失函数 L ( w ) L(w) L(w)格式转换，下式中 X w = y ^ Xw = \hat{y} Xw=y^，所以 y T X w = y T y ^ = 标量 y^{T}Xw = y^{T}\hat{y}=标量 yTXw=yTy^=标量，标量的转置还是它自己
L ( w ) = 1 m ( X w − y ) T ( X w − y ) = 1 m ( ( X w ) T − y T ) ( X w − y ) = 1 m ( w T X T − y T ) ( X w − y ) = 1 m ( w T X T X w − y T X w − w T X T y − y T y ) = 1 m ( w T X T X w − ( y T X w ) T − w T X T y − y T y ) = 1 m ( w T X T X w − w T X T y − w T X T y − y T y ) = 1 m ( w T X T X w − 2 w T X T y − y T y ) \begin{split} L(w) &= \cfrac{1}{m} (Xw-y)^{T}(Xw-y) \\ &=\cfrac{1}{m}((Xw)^\Tau-y^\Tau)(Xw-y) \\ &=\cfrac{1}{m}(w^\Tau X^\Tau-y^\Tau)(Xw-y) \\ &=\cfrac{1}{m}(w^\Tau X^\Tau Xw-y^\Tau Xw-w^\Tau X^\Tau y-y^\Tau y) \\ &=\cfrac{1}{m}(w^\Tau X^\Tau Xw-(y^\Tau Xw)^\Tau-w^\Tau X^\Tau y-y^\Tau y) \\ &=\cfrac{1}{m}(w^\Tau X^\Tau Xw-w^\Tau X^\Tau y-w^\Tau X^\Tau y-y^\Tau y) \\ &=\cfrac{1}{m}(w^\Tau X^\Tau Xw-2w^\Tau X^\Tau y-y^\Tau y) \end{split} L(w)=m1(Xw−y)T(Xw−y)=m1((Xw)T−yT)(Xw−y)=m1(wTXT−yT)(Xw−y)=m1(wTXTXw−yTXw−wTXTy−yTy)=m1(wTXTXw−(yTXw)T−wTXTy−yTy)=m1(wTXTXw−wTXTy−wTXTy−yTy)=m1(wTXTXw−2wTXTy−yTy)
由于损失函数 L ( w ) L(w) L(w)是凸函数，对 w w w求导，并使得导数等于0时的 w w w就是闭式解
L ( w ) ′ = ( 1 m ( w T X T X w − 2 w T X T y − y T y ) ) ′ = 1 m ( ( w T X T X w ) ′ − ( 2 w T X T y ) ′ ) = 1 m ( 2 X T X w − 2 X T y ) = 2 m ( X T X w − X T y ) \begin{split} L(w)' &= \begin{pmatrix} \cfrac{1}{m}(w^\Tau X^\Tau Xw-2w^\Tau X^\Tau y-y^\Tau y) \end{pmatrix}' \\ &=\cfrac{1}{m}\begin{pmatrix} (w^\Tau X^\Tau Xw)'-(2w^\Tau X^\Tau y)' \end{pmatrix} \\ &=\cfrac{1}{m}(2X^\Tau Xw-2X^\Tau y) \\ &=\cfrac{2}{m}(X^\Tau Xw-X^\Tau y) \\ \end{split} L(w)′=(m1(wTXTXw−2wTXTy−yTy))′=m1((wTXTXw)′−(2wTXTy)′)=m1(2XTXw−2XTy)=m2(XTXw−XTy)
令 L ( w ) ′ = 0 L(w)' = 0 L(w)′=0，求得 w w w的闭式解
L ( w ) ′ = 2 m ( X T X w − X T y ) = 0 X T X w − X T y = 0 X T X w = X T y ( X T X ) − 1 ( X T X w ) = ( X T X ) − 1 X T y w = ( X T X ) − 1 X T y \begin{split} L(w)' = \cfrac{2}{m}(X^\Tau Xw-X^\Tau y) &= 0\\ X^\Tau Xw-X^\Tau y &= 0 \\ X^\Tau Xw&= X^\Tau y \\ (X^\Tau X)^{-1}(X^\Tau Xw)&=(X^\Tau X)^{-1}X^\Tau y \\ w&=(X^\Tau X)^{-1}X^\Tau y \end{split} L(w)′=m2(XTXw−XTy)XTXw−XTyXTXw(XTX)−1(XTXw)w=0=0=XTy=(XTX)−1XTy=(XTX)−1XTy