算法解析：
这份 catboost.ipynb 用 Adult 收入数据做二分类，演示的是 CatBoost （Categorical Boosting） ：在梯度提升树（GBDT）框架上，专门把类别特征处理好的 Boosting 算法。
任务在代码里长什么样
- 数据：adult.data，预测 income（是否 >50K）
- 特征里大量类别列：workclass / education / occupation / sex ...
- 关键：CatBoostClassifier，并用
cat_features=1,3,5,6,7,8,9,13
明确告诉模型哪些是类别特征
整体算法思想
1. 属于 Boosting 家族
和 AdaBoost / GBDT / XGBoost 一样：
F M x = F 0 x + m=1 M η h m x
- 每轮加一棵树 hm
，拟合当前残差/梯度 - iterations=500：最多加 500 棵树
- learning_rate=0.1：每棵树贡献打个折扣，防过拟合
- depth=4：每棵树较浅，靠多棵树叠加
- l2_leaf_reg：叶节点 L2 正则
思想一句话：很多弱决策树串起来，后一棵专修前几棵还没学好的部分。
2. CatBoost 的核心卖点：原生吃类别特征
传统做法常把类别 One-Hot / Label Encode，容易：
- 维度爆炸
- 或引入虚假数值顺序
CatBoost 则在训练中对类别特征做有序目标统计编码（Ordered Target Statistics）之类处理：
用"该类别对应标签的统计信息"把类别变成树可分的数值，同时尽量减轻目标泄漏（target leakage）。
代码里的关键一句就是：
clf.fit(X_train, y_train, cat_features=cat_features_index)
不必先手工 One-Hot，把类别列索引交给模型即可。
3. 另一点：有序提升，减轻预测偏移
普通 GBDT 里，某样本可能既参与建树又被同一批信息编码，容易偏乐观。
CatBoost 用 Ordered Boosting ：编码/梯度估计时尽量只用"排在前面"的样本信息，降低这种偏差。
这是它名字里 Cat + Boost 之外，另一个重要设计点。
对应到 notebook 流程
读 Adult 数据
→ 标签编码成 0/1
→ 划分训练/测试
→（可选）GridSearch 调 depth / lr / l2
→ CatBoost 在指定类别特征上训练
→ predict，看 F1 / Accuracy
网格搜索得到较优参数大致是：
depth=4, iterations=500, l2_leaf_reg=1, learning_rate=0.1
GridSearchCV 的原理很直接：把你给出的参数组合全部试一遍，用交叉验证打分，选分最高的那组。
它在搜什么
代码里大致是：
params = {
'depth': 4, 6,
'learning_rate': 0.01, 0.1,
'l2_leaf_reg': 1, 4, 9,
'iterations': 500
}
cb_model = GridSearchCV(cb, params, scoring="roc_auc", cv=3)
这是网格搜索：每个参数取给定候选值，做笛卡尔积。
组合数 = 2 × 2 × 3 × 1 = 12 组参数。
怎么判断哪组最好
对每一组参数：
- 3 折交叉验证 （cv=3）
把训练集切成 3 份：轮流用 2 份训练、1 份验证，共 3 次。 - 每次算验证集上的 roc_auc。
- 把 3 次分数平均，作为这组参数的得分。
- 12 组都试完后，取平均分最高的一组 → best_params_
对应分数 → best_score_
参数组合1 ──► 折1/折2/折3 AUC ──► 平均分
参数组合2 ──► 折1/折2/折3 AUC ──► 平均分
...
参数组合12 ─► ... ──► 平均分
↓
选平均分最高者
为什么这样能"找到较佳参数"
- 不是理论推导最优，而是暴力试探 + 验证集评估
- 交叉验证比单次划分更稳，减少"碰巧分得好"的运气成分
- scoring="roc_auc" 表示优化目标是 AUC，不是准确率
注意：它只能在你给的候选里找最好的，搜不到网格外的参数。候选太少可能漏掉更好点；候选太多会很慢（CatBoost × 12 组 × 3 折 = 要训很多遍）。
和随机搜索的区别
| GridSearchCV | RandomizedSearchCV | |
|---|---|---|
| 方式 | 网格全试 | 随机抽若干组 |
| 优点 | 给定范围内不漏 | 大空间更省时 |
| 本代码 | 用的是网格全试 | --- |
一句话：穷举你指定的参数组合，用交叉验证的 AUC 比较，分数最高的就是"最佳参数"。
每组参数都要完整训练，而且交叉验证下还不止一遍，代价确实不小。
代价怎么算
对本 notebook：
- 参数组合：2 × 2 × 3 × 1 = 12 组
- cv=3：每组训 3 次
所以大约：
12×3=36 次 CatBoost 完整训练
搜完后，GridSearchCV 通常还会用最优参数在全部训练集上再拟合一次，所以实际还会再多一轮。
为什么还要这么做
因为没法靠公式直接算出"最优 depth/learning_rate"，只能靠验证集表现比较。
交叉验证是用时间换更稳的估计，避免某次划分碰巧分得好。
实际中怎么降成本
| 做法 | 作用 |
|---|---|
| 缩小搜索网格 | 少试几组（本例只搜了 12 组，已经算克制） |
| 减少 cv（如 3→2）或改用 hold-out | 每组少训几次 |
| RandomizedSearchCV | 大空间里随机抽，不穷举 |
| 先粗搜再细搜 | 先大步找区间，再小范围精搜 |
| 减少 iterations / 子采样数据做搜参 | 搜参阶段用更轻模型 |
| 早停（early stopping） | 无效轮次提前停 |
一句话：网格搜索本质就是"用算力换参数"；参数空间一大，代价会线性甚至成倍涨。教学 demo 还能接受；真实大模型/大数据里通常会改用随机搜索、贝叶斯优化，或先小数据粗调再全量精调。
和其他树模型怎么区分
| AdaBoost | XGBoost/LightGBM | CatBoost | |
|---|---|---|---|
| 框架 | Boosting | GBDT + 工程优化 | GBDT + 类别特征特化 |
| 类别特征 | 一般要自己编码 | 可处理，但常需预处理 | 原生强项 |
| 本 notebook 重点 | --- | --- | cat_features + 表格混合特征分类 |
一句话：CatBoost = 梯度提升树 + 更稳妥的类别特征处理（及有序提升）；这份代码就是在"收入预测"这种类别特征很多的表格任务上，展示它怎么直接用。
代码实战：
! wget https://docker-76009.sz.gfp.tencent-cloud.com/github/cube-studio/aihub/ml/catboost/adult.data--2022-10-10 19:55:28-- https://docker-76009.sz.gfp.tencent-cloud.com/github/cube-studio/aihub/ml/catboost/adult.data
Resolving docker-76009.sz.gfp.tencent-cloud.com (docker-76009.sz.gfp.tencent-cloud.com)... 183.47.104.64, 183.47.104.100, 183.47.114.103, ...
Connecting to docker-76009.sz.gfp.tencent-cloud.com (docker-76009.sz.gfp.tencent-cloud.com)|183.47.104.64|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3974305 (3.8M) application/octet-stream
Saving to: 'adult.data'
adult.data 0% 0 --.-KB/s
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
import catboost as cb
from sklearn.metrics import f1_score
# 读取数据
data = pd.read_csv('./adult.data', header=None)
# 变量重命名
data.columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num',
'marital-status', 'occupation', 'relationship', 'race', 'sex',
'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income']
# 标签转换
data['income'] = data['income'].astype("category").cat.codes
# 划分数据集
X_train, X_test, y_train, y_test = train_test_split(data.drop(['income'], axis=1), data['income'],
random_state=10, test_size=0.3)
# 配置训练参数
clf = cb.CatBoostClassifier(eval_metric="AUC", depth=4, iterations=500, l2_leaf_reg=1,
learning_rate=0.1)
# 类别特征索引
cat_features_index = [1, 3, 5, 6, 7, 8, 9, 13]
# 训练
clf.fit(X_train, y_train, cat_features=cat_features_index)
# 预测
y_pred = clf.predict(X_test)
# 测试集f1得分
print(f1_score(y_test, y_pred))(32561, 15)
data.head()| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 39 | State-gov | 77516 | Bachelors | 13 | Never-married | Adm-clerical | Not-in-family | White | Male | 2174 | 0 | 40 | United-States | <=50K |
| 1 | 50 | Self-emp-not-inc | 83311 | Bachelors | 13 | Married-civ-spouse | Exec-managerial | Husband | White | Male | 0 | 0 | 13 | United-States | <=50K |
| 2 | 38 | Private | 215646 | HS-grad | 9 | Divorced | Handlers-cleaners | Not-in-family | White | Male | 0 | 0 | 40 | United-States | <=50K |
| 3 | 53 | Private | 234721 | 11th | 7 | Married-civ-spouse | Handlers-cleaners | Husband | Black | Male | 0 | 0 | 40 | United-States | <=50K |
| 4 | 28 | Private | 338409 | Bachelors | 13 | Married-civ-spouse | Prof-specialty | Wife | Black | Female | 0 | 0 | 40 | Cuba | <=50K |
data.columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation',
'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income']
data.head()| age | workclass | fnlwgt | education | education-num | marital-status | occupation | relationship | race | sex | capital-gain | capital-loss | hours-per-week | native-country | income | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 39 | State-gov | 77516 | Bachelors | 13 | Never-married | Adm-clerical | Not-in-family | White | Male | 2174 | 0 | 40 | United-States | <=50K |
| 1 | 50 | Self-emp-not-inc | 83311 | Bachelors | 13 | Married-civ-spouse | Exec-managerial | Husband | White | Male | 0 | 0 | 13 | United-States | <=50K |
| 2 | 38 | Private | 215646 | HS-grad | 9 | Divorced | Handlers-cleaners | Not-in-family | White | Male | 0 | 0 | 40 | United-States | <=50K |
| 3 | 53 | Private | 234721 | 11th | 7 | Married-civ-spouse | Handlers-cleaners | Husband | Black | Male | 0 | 0 | 40 | United-States | <=50K |
| 4 | 28 | Private | 338409 | Bachelors | 13 | Married-civ-spouse | Prof-specialty | Wife | Black | Female | 0 | 0 | 40 | Cuba | <=50K |
data['income'] = data['income'].astype("category").cat.codes
data['income'].value_counts()0 24720
1 7841
Name: income, dtype: int64
from sklearn.model_selection import train_test_split, GridSearchCV
X_train, X_test, y_train, y_test = train_test_split(data.drop(['income'], axis=1), data['income'],
random_state=10, test_size=0.3)
print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)(22792, 14) (22792,) (9769, 14) (9769,)
import catboost as cb
cat_features_index = [1, 3, 5, 6, 7, 8, 9, 13]
params = {'depth': [4, 6],
'learning_rate': [0.01, 0.1],
'l2_leaf_reg': [1, 4, 9],
'iterations': [500]}
cb = cb.CatBoostClassifier()
cb_model = GridSearchCV(cb, params, scoring="roc_auc", cv=3)
cb_model.fit(X_train, y_train, cat_features=cat_features_index)0: learn: 0.6835058 total: 61.7ms remaining: 30.8s
1: learn: 0.6742830 total: 90.4ms remaining: 22.5s
2: learn: 0.6654358 total: 139ms remaining: 23.1s
3: learn: 0.6571546 total: 170ms remaining: 21.1s
4: learn: 0.6492061 total: 188ms remaining: 18.6s
5: learn: 0.6411828 total: 216ms remaining: 17.8s
6: learn: 0.6333846 total: 271ms remaining: 19.1s
7: learn: 0.6256674 total: 311ms remaining: 19.1s
8: learn: 0.6185847 total: 339ms remaining: 18.5s
9: learn: 0.6111679 total: 365ms remaining: 17.9s
10: learn: 0.6039260 total: 392ms remaining: 17.4s
11: learn: 0.5970827 total: 410ms remaining: 16.7s
12: learn: 0.5906176 total: 437ms remaining: 16.4s
13: learn: 0.5842135 total: 455ms remaining: 15.8s
14: learn: 0.5776900 total: 478ms remaining: 15.5s
15: learn: 0.5716740 total: 526ms remaining: 15.9s
16: learn: 0.5653441 total: 555ms remaining: 15.8s
17: learn: 0.5596729 total: 574ms remaining: 15.4s
18: learn: 0.5538479 total: 610ms remaining: 15.4s
19: learn: 0.5484339 total: 627ms remaining: 15s
20: learn: 0.5428251 total: 645ms remaining: 14.7s
21: learn: 0.5373363 total: 675ms remaining: 14.7s
22: learn: 0.5314705 total: 683ms remaining: 14.2s
23: learn: 0.5263212 total: 700ms remaining: 13.9s
24: learn: 0.5214608 total: 717ms remaining: 13.6s
...
496: learn: 0.2546030 total: 21.1s remaining: 127ms
497: learn: 0.2544729 total: 21.1s remaining: 84.8ms
498: learn: 0.2543877 total: 21.2s remaining: 42.4ms
499: learn: 0.2543846 total: 21.2s remaining: 0us
Output is truncated. View as a scrollable element**or open in a text editor*. Adjust cell output* settings*...*
GridSearchCV(cv=3,
estimator=<catboost.core.CatBoostClassifier object at 0x000001D05BB49308>,
param_grid={'depth': [4, 6], 'iterations': [500],
'l2_leaf_reg': [1, 4, 9],
'learning_rate': [0.01, 0.1]},
scoring='roc_auc')
print(cb_model.best_score_)
print(cb_model.best_params_)0.9275055301975431
{'depth': 4, 'iterations': 500, 'l2_leaf_reg': 1, 'learning_rate': 0.1}
import catboost as cb
clf = cb.CatBoostClassifier(eval_metric="AUC", depth=4, iterations=500, l2_leaf_reg=1,
learning_rate=0.1)
clf.fit(X_train, y_train, cat_features=cat_features_index)0: total: 77.2ms remaining: 38.5s
1: total: 132ms remaining: 32.9s
2: total: 164ms remaining: 27.2s
3: total: 195ms remaining: 24.2s
4: total: 228ms remaining: 22.6s
5: total: 288ms remaining: 23.7s
6: total: 310ms remaining: 21.8s
7: total: 353ms remaining: 21.7s
8: total: 395ms remaining: 21.5s
9: total: 435ms remaining: 21.3s
10: total: 477ms remaining: 21.2s
11: total: 515ms remaining: 20.9s
12: total: 557ms remaining: 20.9s
13: total: 587ms remaining: 20.4s
14: total: 629ms remaining: 20.3s
15: total: 671ms remaining: 20.3s
16: total: 720ms remaining: 20.5s
17: total: 762ms remaining: 20.4s
18: total: 804ms remaining: 20.3s
19: total: 855ms remaining: 20.5s
20: total: 913ms remaining: 20.8s
21: total: 978ms remaining: 21.3s
22: total: 1.02s remaining: 21.2s
23: total: 1.1s remaining: 21.9s
24: total: 1.17s remaining: 22.1s
...
496: total: 24s remaining: 145ms
497: total: 24.1s remaining: 96.8ms
498: total: 24.2s remaining: 48.4ms
499: total: 24.2s remaining: 0us
Output is truncated. View as a scrollable element**or open in a text editor*. Adjust cell output* settings*...*
<catboost.core.CatBoostClassifier at 0x1d05ce33e48>
y_pred = clf.predict(X_test)
y_pred.shape(9769,)
from sklearn.metrics import f1_score
print(f1_score(y_test, y_pred))0.7112659698025551
from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred)0.8727607738765483
from sklearn.metrics import classification_report
classification_report(y_test, y_pred)' precision recall f1-score support\n\n 0 0.90 0.94 0.92 7423\n 1 0.78 0.65 0.71 2346\n\n accuracy 0.87 9769\n macro avg 0.84 0.80 0.81 9769\nweighted avg 0.87 0.87 0.87 9769\n'