机器学习 - 集成学习

1. bagging

bagging的核心思想： bootstrap sample自主采样；即从m个样本的数据集中有放回的随机采样n(n ≤ m)个样本；这样重复k次，将得到k个不同的数据集作为训练数据。

随机森林：随机森林是bagging的经典应用；随机森林相对于bagging更加高级，不但可以对样本进行bootstrap sampling，而且可以对特征进行bootstrap；从而形成拥有多个决策树的森林。

优点：并行集成化容易； 降低模型方差； 抗过拟合能力强。

# -*- coding: utf-8 -*-
""" 利用sklearn来实现bagging """
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier

def load_data():
pass

def bagging(train_x, train_y, test):
""" bagging """
clf = KNeighborsClassifier(n_neighbors=15)
bagging_clf = BaggingClassifier(clf, n_estimators=50, max_samples=0.9, max_features=0.9,
bootstrap=True, bootstrap_features=True, n_jobs=1, random_state=1)
bagging_clf.fit(train_x, train_y)
prediction = bagging_clf.predict(test)
return prediction

def main():
(train_x, train_y, test) = load_data()
prediction = bagging(train_x, train_y, test)
return prediction

if __name__ == "__main__":
prediction = main()

2. boosting

boosting的核心思想：调整样本分布。

boosting的过程：预测，改变样本分布；预测，改变样本分布；...；加权融合。



左边的条形图象征着样本分布的改变，三角连接为分类结果融合的权值。

优点 & 缺点： 深度学习之前的屠龙刀，精度高；但是容易过拟合，并行化难以实现。

3. stacking

stacking的核心：在训练集上进行预测，从而构建更高层的学习器。

3.1 stacking训练过程

1） 拆解训练集。将训练数据随机且大致均匀的拆为m份

2）在拆解后的训练集上训练模型，同时在测试集上预测。利用m-1份训练数据进行训练，预测剩余一份；在此过程进行的同时，利用相同的m-1份数据训练，在真正的测试集上预测；如此重复m次，将训练集上m次结果叠加为1列，将测试集上m次结果取均值融合为1列。

3）使用k个分类器重复2过程。将分别得到k列训练集的预测结果，k列测试集预测结果。

4）训练3过程得到的数据。将k列训练集预测结果和训练集真实label进行训练，将k列测试集预测结果作为测试集。

# -*- coding: utf-8 -*-
import numpy as np
from sklearn.model_selection import StratifiedKFold
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
import xgboost as xgb
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.linear_model import LogisticRegression

def load_data():
pass

def stacking(train_x, train_y, test):
""" stacking
input: train_x, train_y, test
output: test的预测值
clfs: 5个一级分类器
dataset_blend_train: 一级分类器的prediction, 二级分类器的train_x
dataset_blend_test: 二级分类器的test
"""
# 5个一级分类器
clfs = [SVC(C = 3, kernel="rbf"),
RandomForestClassifier(n_estimators=100, max_features="log2", max_depth=10, min_samples_leaf=1, bootstrap=True, n_jobs=-1, random_state=1),
KNeighborsClassifier(n_neighbors=15, n_jobs=-1),
xgb.XGBClassifier(n_estimators=100, objective="binary:logistic", gamma=1, max_depth=10, subsample=0.8, nthread=-1, seed=1),
ExtraTreesClassifier(n_estimators=100, criterion="gini", max_features="log2", max_depth=10, min_samples_split=2, min_samples_leaf=1,bootstrap=True, n_jobs=-1, random_state=1)]

# 二级分类器的train_x, test
dataset_blend_train = np.zeros((train_x.shape[0], len(clfs)), dtype=np.int)
dataset_blend_test = np.zeros((test.shape[0], len(clfs)), dtype=np.int)

# 5个分类器进行8_folds预测
n_folds = 8
skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=1)
for i,clf in enumerate(clfs):
dataset_blend_test_j = np.zeros((test.shape[0], n_folds))  # 每个分类器的单次fold预测结果
for j,(train_index,test_index) in enumerate(skf.split(train_x, train_y)):
tr_x = train_x[train_index]
tr_y = train_y[train_index]
clf.fit(tr_x, tr_y)
dataset_blend_train[test_index, i] = clf.predict(train_x[test_index])
dataset_blend_test_j[:, j] = clf.predict(test)
dataset_blend_test[:, i] = dataset_blend_test_j.sum(axis=1) // (n_folds//2 + 1)

# 二级分类器进行预测
clf = LogisticRegression(penalty="l1", tol=1e-6, C=1.0, random_state=1, n_jobs=-1)
clf.fit(dataset_blend_train, train_y)
prediction = clf.predict(dataset_blend_test)
return prediction

def main():
(train_x, train_y, test) = load_data()
prediction = stacking(train_x, train_y, test)
return prediction

if __name__ == "__main__":
prediction = main()