快速上手大模型：机器学习6（过拟合、正则化）

目录

[1 过拟合](#1 过拟合)

[2 解决过拟合](#2 解决过拟合)

[2.1 获取更多训练数据集](#2.1 获取更多训练数据集)

[2.2 使用较少特征](#2.2 使用较少特征)

[2.3 正则化(Regularization)](#2.3 正则化(Regularization))

[3 正则化](#3 正则化)

[3.1 定义](#3.1 定义)

[3.2 正则化线性回归](#3.2 正则化线性回归)

[3.3 正则化逻辑回归](#3.3 正则化逻辑回归)

[3.4 核心代码](#3.4 核心代码)

1 过拟合

以房屋价格预测为例，线性回归中，图1对训练数据为欠拟合(underfit)，即预测曲线与数据样本偏差较大，这种情况也称高度偏差(high bias)；图2模型拟合情况良好，具有良好泛化性(generalization)；图3模型虽然能够准确拟合真实样本数据、做到误差为0，但出现波动，与实际场景不符，称为过拟合(overfit)，也称高方差(high variance)。

逻辑回归中同理，图1欠拟合，图2泛化性良好，图3过拟合。

2 解决过拟合

2.1 获取更多训练数据集

通过训练更多的数据集，将过拟合曲线变为泛化性良好的状态。

2.2 使用较少特征

使用关键特征进行模型建立，进而进行预测。

2.3 正则化(Regularization)

正则化通过鼓励学习算法缩小参数值，而不要求参数直接为0。正则化只改变w大小，修改b对曲线无太大影响。

3 正则化

3.1 定义

正则化后代价函数转变为：，影响较小可忽略。

选择：

为0，正则化失效，过拟合；过大，欠拟合。

正则化的思想是平衡拟合，使其适中，满足实际场景；正则化的作用是在每次迭代中使wi变小一点。

3.2 正则化线性回归

梯度下降：

此处回归模型为

3.3 正则化逻辑回归

梯度下降：

梯度下降公式与正则化的一致，唯一区别在于f函数，

3.4 核心代码

计算带正则化项的线性回归总损失函数：

复制代码

def compute_cost_linear_reg(X, y, w, b, lambda_ = 1):
    """
    Computes the cost over all examples
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns:
      total_cost (scalar):  cost 
    """

    m  = X.shape[0]
    n  = len(w)
    cost = 0.
    for i in range(m):
        f_wb_i = np.dot(X[i], w) + b                                   #(n,)(n,)=scalar, see np.dot
        cost = cost + (f_wb_i - y[i])**2                               #scalar             
    cost = cost / (2 * m)                                              #scalar  
 
    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)                                          #scalar
    reg_cost = (lambda_/(2*m)) * reg_cost                              #scalar
    
    total_cost = cost + reg_cost                                       #scalar
    return total_cost

计算逻辑回归模型的损失函数:

复制代码

def compute_cost_logistic_reg(X, y, w, b, lambda_ = 1):
    """
    Computes the cost over all examples
    Args:
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns:
      total_cost (scalar):  cost 
    """

    m,n  = X.shape
    cost = 0.
    for i in range(m):
        z_i = np.dot(X[i], w) + b                                      #(n,)(n,)=scalar, see np.dot
        f_wb_i = sigmoid(z_i)                                          #scalar
        cost +=  -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)      #scalar
             
    cost = cost/m                                                      #scalar

    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)                                          #scalar
    reg_cost = (lambda_/(2*m)) * reg_cost                              #scalar
    
    total_cost = cost + reg_cost                                       #scalar
    return total_cost                                                  #scalar

计算线性回归代价函数（含正则化）的梯度,

复制代码

def compute_gradient_linear_reg(X, y, w, b, lambda_): 
    """
    Computes the gradient for linear regression 
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
      
    Returns:
      dj_dw (ndarray (n,)): The gradient of the cost w.r.t. the parameters w. 
      dj_db (scalar):       The gradient of the cost w.r.t. the parameter b. 
    """
    m,n = X.shape           #(number of examples, number of features)
    dj_dw = np.zeros((n,))
    dj_db = 0.

    for i in range(m):                             
        err = (np.dot(X[i], w) + b) - y[i]                 
        for j in range(n):                         
            dj_dw[j] = dj_dw[j] + err * X[i, j]               
        dj_db = dj_db + err                        
    dj_dw = dj_dw / m                                
    dj_db = dj_db / m   
    
    for j in range(n):
        dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]

    return dj_db, dj_dw

计算逻辑回归代价函数（含正则化）的梯度

复制代码

def compute_gradient_logistic_reg(X, y, w, b, lambda_): 
    """
    Computes the gradient for linear regression 
 
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns
      dj_dw (ndarray Shape (n,)): The gradient of the cost w.r.t. the parameters w. 
      dj_db (scalar)            : The gradient of the cost w.r.t. the parameter b. 
    """
    m,n = X.shape
    dj_dw = np.zeros((n,))                            #(n,)
    dj_db = 0.0                                       #scalar

    for i in range(m):
        f_wb_i = sigmoid(np.dot(X[i],w) + b)          #(n,)(n,)=scalar
        err_i  = f_wb_i  - y[i]                       #scalar
        for j in range(n):
            dj_dw[j] = dj_dw[j] + err_i * X[i,j]      #scalar
        dj_db = dj_db + err_i
    dj_dw = dj_dw/m                                   #(n,)
    dj_db = dj_db/m                                   #scalar

    for j in range(n):
        dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]

    return dj_db, dj_dw