集成学习实战：Bagging/Boosting/Stacking

1. 集成学习原理

复制代码

集成学习（Ensemble Learning）：
├── 核心思想：多个弱学习器组合成强学习器
├── 三大方法：
│   ├── Bagging：并行训练，投票/平均（随机森林）
│   ├── Boosting：串行训练，逐步纠错（XGBoost）
│   └── Stacking：多层模型，元学习器组合
└── 优势：降低方差、降低偏差、提高泛化能力

2. Bagging

python 复制代码

from sklearn.ensemble import BaggingClassifier, BaggingRegressor
from sklearn.tree import DecisionTreeClassifier

# Bagging 分类
bagging = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=100,
    max_samples=0.8,
    max_features=0.8,
    bootstrap=True,
    random_state=42,
    n_jobs=-1
)
bagging.fit(X_train, y_train)

3. Boosting

python 复制代码

# AdaBoost
from sklearn.ensemble import AdaBoostClassifier
ada = AdaBoostClassifier(
    n_estimators=100,
    learning_rate=0.1,
    random_state=42
)
ada.fit(X_train, y_train)

# Gradient Boosting
from sklearn.ensemble import GradientBoostingClassifier
gb = GradientBoostingClassifier(
    n_estimators=100,
    max_depth=3,
    learning_rate=0.1,
    subsample=0.8,
    random_state=42
)
gb.fit(X_train, y_train)

# XGBoost
import xgboost as xgb
xgb_clf = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    random_state=42
)
xgb_clf.fit(X_train, y_train)

# LightGBM
import lightgbm as lgb
lgb_clf = lgb.LGBMClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    random_state=42
)
lgb_clf.fit(X_train, y_train)

4. Stacking

python 复制代码

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression

# 定义基学习器
estimators = [
    ('rf', RandomForestClassifier(n_estimators=100)),
    ('svm', SVC(probability=True)),
    ('xgb', xgb.XGBClassifier(n_estimators=100))
]

# Stacking
stacking = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(),
    cv=5,
    n_jobs=-1
)
stacking.fit(X_train, y_train)

5. 投票集成

python 复制代码

from sklearn.ensemble import VotingClassifier

# 硬投票
voting_hard = VotingClassifier(
    estimators=estimators,
    voting='hard'
)

# 软投票（概率平均）
voting_soft = VotingClassifier(
    estimators=estimators,
    voting='soft'
)

总结

方法	代表算法	优势	适用场景
Bagging	随机森林	降低方差	高方差模型
Boosting	XGBoost	降低偏差	高偏差模型
Stacking	多模型组合	综合优势	竞赛/复杂场景