Learning Best Practices for Model Evaluation and Hyperparameter Tuning
2017-04-18 22:41
666 查看
1. Streamlining workflows with pipelines
1.1 Loading the Breast Cancer Wisconsin dataset
1.2 Combining transformers and estimators in a pipeline
2 Using k-fold cross-validation to assess model performance
2.1 K-fold cross-validation
2.2 scikit-learn also implements a k-fold cross-validation scorer, which allows us to evaluate our model using stratifed k-fold cross-validation more effciently
3. Debugging algorithms with learning and validation curves
3.1 Diagnosing bias and variance problems with learning curves
3.2 Addressing overftting and underftting with validation curves
4 Fine-tuning machine learning models via grid search
4.1 Tuning hyperparameters via grid search
4.2 Algorithm selection with nested cross-validation
Reference:《Python Machine Learning》
1.1 Loading the Breast Cancer Wisconsin dataset
from distutils.version import LooseVersion as Version from sklearn import __version__ as sklearn_version import numpy as np import pandas as pd from sklearn.preprocessing import LabelEncoder from sklearn.model_selection import train_test_split df = pd.read_csv('./datasets/wdbc/wdbc.data', header=None) # https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data X = df.loc[:, 2:].values y = df.loc[:, 1].values le = LabelEncoder() y = le.fit_transform(y) y_enc = le.transform(['M', 'B']) X_train, X_test, y_train, y_test = \ train_test_split(X, y, test_size=0.20, random_state=1)
1.2 Combining transformers and estimators in a pipeline
from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline pipe_lr = Pipeline([('scl', StandardScaler()), ('pca', PCA(n_components=2)), ('clf', LogisticRegression(random_state=1))]) pipe_lr.fit(X_train, y_train) print('Test Accuracy: %.3f' % pipe_lr.score(X_test, y_test))
2 Using k-fold cross-validation to assess model performance
2.1 K-fold cross-validation
from sklearn.model_selection import StratifiedKFold kfold = StratifiedKFold(n_splits=10, random_state=1).split(X_train, y_train) scores = [] for k, (train, test) in enumerate(kfold): pipe_lr.fit(X_train[train], y_train[train]) score = pipe_lr.score(X_train[test], y_train[test]) scores.append(score) print('Fold: %s, Class dist.: %s, Acc: %.3f' % (k + 1, np.bincount(y_train[train]), score))
2.2 scikit-learn also implements a k-fold cross-validation scorer, which allows us to evaluate our model using stratifed k-fold cross-validation more effciently
from sklearn.model_selection import cross_val_scorescores = cross_val_score(estimator=pipe_lr, X=X_train, y=y_train, cv=10, n_jobs=1)print('CV accuracy scores: %s' % scores)print('CV accuracy: %.3f +/- %.3f' % (np.mean(scores), np.std(scores)))
3. Debugging algorithms with learning and validation curves
3.1 Diagnosing bias and variance problems with learning curves
import matplotlib.pyplot as plt from sklearn.model_selection import learning_curve pipe_lr = Pipeline([('scl', StandardScaler()), ('clf', LogisticRegression(penalty='l2', random_state=0))]) train_sizes, train_scores, test_scores =\ learning_curve(estimator=pipe_lr, X=X_train, y=y_train, train_sizes=np.linspace(0.1, 1.0, 10), cv=10, n_jobs=1) train_mean = np.mean(train_scores, axis=1) train_std = np.std(train_scores, axis=1) test_mean = np.mean(test_scores, axis=1) test_std = np.std(test_scores, axis=1) plt.plot(train_sizes, train_mean, color='blue', marker='o', markersize=5, label='training accuracy') plt.fill_between(train_sizes, train_mean + train_std, train_mean - train_std, alpha=0.15, color='blue') plt.plot(train_sizes, test_mean, color='green', linestyle='--', marker='s', markersize=5, label='validation accuracy') plt.fill_between(train_sizes, test_mean + test_std, test_mean - test_std, alpha=0.15, color='green') plt.grid() plt.xlabel('Number of training samples') plt.ylabel('Accuracy') plt.legend(loc='lower right') plt.ylim([0.8, 1.0]) plt.show()
3.2 Addressing overftting and underftting with validation curves
param_range = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0] train_scores, test_scores = validation_curve( estimator=pipe_lr, X=X_train, y=y_train, param_name='clf__C', param_range=param_range, cv=10) train_mean = np.mean(train_scores, axis=1) train_std = np.std(train_scores, axis=1) test_mean = np.mean(test_scores, axis=1) test_std = np.std(test_scores, axis=1) plt.plot(param_range, train_mean, color='blue', marker='o', markersize=5, label='training accuracy') plt.fill_between(param_range, train_mean + train_std, train_mean - train_std, alpha=0.15, color='blue') plt.plot(param_range, test_mean, color='green', linestyle='--', marker='s', markersize=5, label='validation accuracy') plt.fill_between(param_range, test_mean + test_std, test_mean - test_std, alpha=0.15, color='green') plt.grid() plt.xscale('log') plt.legend(loc='lower right') plt.xlabel('Parameter C') plt.ylabel('Accuracy') plt.ylim([0.8, 1.0]) plt.show()
4 Fine-tuning machine learning models via grid search
4.1 Tuning hyperparameters via grid search
from sklearn.model_selection import GridSearchCV from sklearn.svm import SVC pipe_svc = Pipeline([('scl', StandardScaler()), ('clf', SVC(random_state=1))]) param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0] param_grid = [{'clf__C': param_range, 'clf__kernel': ['linear']}, {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['rbf']}] gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring='accuracy', cv=10, n_jobs=-1) gs = gs.fit(X_train, y_train) print('Validation accuracy', gs.best_score_) print('Best parameters', gs.best_params_)
clf = gs.best_estimator_ clf.fit(X_train, y_train) print('Test accuracy: %.3f' % clf.score(X_test, y_test))
4.2 Algorithm selection with nested cross-validation
from sklearn.tree import DecisionTreeClassifier gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring='accuracy', cv=2) # Note: Optionally, you could use cv=2 # in the GridSearchCV above to produce # the 5 x 2 nested CV that is shown in the figure. scores = cross_val_score(gs, X_train, y_train, scoring='accuracy', cv=5) print('CV accuracy: %.3f +/- %.3f' % (np.mean(scores), np.std(scores))) gs = Gri af51 dSearchCV(estimator=DecisionTreeClassifier(random_state=0), param_grid=[{'max_depth': [1, 2, 3, 4, 5, 6, 7, None]}], scoring='accuracy', cv=2) scores = cross_val_score(gs, X_train, y_train, scoring='accuracy', cv=5) print('CV accuracy: %.3f +/- %.3f' % (np.mean(scores), np.std(scores)))
Reference:《Python Machine Learning》
相关文章推荐
- caffe image recognization model and prototxt for transfer learning (fine-tuning)
- Coursera deeplearning.ai 深度学习笔记2-3-Hyperparameter tuning, Batch Normalization and Programming Framew
- Enterprise dashboards : design and best practices for IT
- Mastering BEA WebLogic Server: Best Practices for Building and Deploying J2EE Applications
- Client API - Best Practices for performance tuning
- Best Practices and Requirements for OpenSolaris development
- Best Practices for Creating and Deploying HTTP Live Streaming Media for the iPhone and iPad
- Core Security Patterns : Best Practices and Strategies for J2EE(TM), Web Services, and Identity Mana
- Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices
- Secure Coding Best Practices for Memory Allocation in C and C++
- Optimizing Data Access and Messaging - Best Practices for Maximizing Scalability and Cost Effectiven
- 30 Java Programming Tips and Best Practices for Beginners
- IT Compliance and Controls: Best Practices for Implementation
- Secure Coding Best Practices for Memory Allocation in C and C++
- Best Practices for Managing Your Linux/Unix Performance and Availability @ JDJ
- Coding Standards and Best Practices for "appobjects" Files
- [收藏] Best Practices To Develop Perfect Websites for iPhone and Mobile Devices
- [收藏] Best Practices To Develop Perfect Websites for iPhone and Mobile Devices
- Note for video Machine Learning and Data Mining——Linear Model
- Secure Coding Best Practices for Memory Allocation in C and C++