Kaggle Intermediate ML Part Four——Cross-Validation

What is it?

Cross-validation is a technique used to evaluate the generalizability of a machine learning model. In simpler terms, it helps you understand how well your model will perform on unseen data, which is crucial for real-world applications.

Here's how it works:

  1. Split the data: Your original dataset is divided into folds (usually equally sized).
  2. Train-Test Split: In each fold, one fold is kept for testing (hold-out set), while the remaining folds are used for training the model.
  3. Evaluate and Repeat: The model is trained on the training data and evaluated on the hold-out set. This process is repeated for each fold, ensuring every data point is used for both training and testing.
  4. Combine and Analyze: The performance metrics (e.g., accuracy, precision, recall) from each fold are combined to get an overall estimate of the model's performance on unseen data.

Common Cross-Validation Techniques:

  • K-Fold Cross-validation: The data is split into k folds, and the training-testing process is repeated k times.
  • Stratified K-Fold: Similar to k-fold, but ensures each fold has a similar distribution of class labels (important for imbalanced datasets).
  • Leave-One-Out Cross-validation (LOOCV): Each data point is used as the testing set once, while all other points are used for training. This is computationally expensive for large datasets.

Production Use and Examples:

  • Model Selection: Compare different models and choose the one with the best cross-validation performance.
  • Hyperparameter Tuning: Optimize hyperparameters (model settings) by evaluating their impact on cross-validation performance.
  • Feature Selection: Identify and remove irrelevant or redundant features that may lead to overfitting.

python 复制代码
from sklearn.model_selection import KFold
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# Load the Iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Define the model
model = LogisticRegression()

# Define the K-Fold cross-validation strategy
kfold = KFold(n_splits=5, shuffle=True, random_state=42)

# Track performance metrics
auc_scores = []

# Iterate through each fold
for train_index, test_index in kfold.split(X):
    # Split data into training and testing sets
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    # Train the model on the training data
    model.fit(X_train, y_train)

    # Make predictions on the testing data
    y_proba = model.predict_proba(X_test)[:, 1]  # Probability of positive class

    # Calculate AUC
    auc = roc_auc_score(y_test, y_proba)
    auc_scores.append(auc)

# Print the average AUC across all folds
print(f"Average AUC: {sum(auc_scores) / len(auc_scores):.2f}")
相关推荐
想知道哇1 小时前
机器学习入门:决策树的欠拟合与过拟合
人工智能·机器学习
go54631584652 小时前
修改Spatial-MLLM项目,使其专注于无人机航拍视频的空间理解
人工智能·算法·机器学习·架构·音视频·无人机
还有糕手2 小时前
西南交通大学【机器学习实验2】
人工智能·机器学习
小白狮ww3 小时前
VASP 教程:VASP 机器学习力场微调
人工智能·深度学习·机器学习
阿水实证通3 小时前
Stata如何做机器学习?——SHAP解释框架下的足球运动员价值驱动因素识别:基于H2O集成学习模型
人工智能·机器学习·集成学习
呆头鹅AI工作室4 小时前
[2025CVPR]SEEN-DA:基于语义熵引导的领域感知注意力机制
人工智能·深度学习·机器学习
大千AI助手4 小时前
蒙特卡洛方法:随机抽样的艺术与科学
人工智能·机器学习·贝叶斯·概率·蒙特卡洛·随机
山顶望月川4 小时前
并行科技MaaS平台支持文心4.5系列开源模型调用
人工智能·机器学习·编辑器
FF-Studio5 小时前
【硬核数学】3. AI如何应对不确定性?概率论为模型注入“灵魂”《从零构建机器学习、深度学习到LLM的数学认知》
大数据·人工智能·深度学习·机器学习·数学建模·自然语言处理·概率论