机器学习工程实践方法论

Andrew Ng 斯坦福机器学习公开课听课笔记（11）

Key ideas:

1. Diagnostics for debugging learning algorithms

2. Error analyses and ablative analysis

3 . How to get started on a machine learning problem

- premature (statistical) optimization. 过早统计优化

Debugging Learning Algorithms

Motivating example:

. Anti-spam You carefully choose a small set of 100 words to use as features.

. Bayesian logistic regression, implemented with gradient descent, gets 20% test error, which is unacceptably high.

. What to do next?

. Bayesian logistic regression:

. Common approach: Try improving the algorithm in different ways.

- Try getting more training examples

- Try a smaller set of features

- Try a larger set of features

- Try changing the features: Email header vs. email body features

- Run gradient descent for more iterations

- Try Newton’s method

- Use a different value for lambda

- Try using an SVM

Diagnostic for bias vs. variance

Better approach:

- Run diagnostics to figure out what the problem is.

- Fix whatever the problem is

Suppose you suspect the problem is either”

- Overfiting (high bias, 高偏差-过拟合)

- Too few features to classify spam(high variance，高方差-欠拟合).

Diagnostic:

- Variance: Training error will the much lower than test error.

- Bias: Training error will also be high.

方法：检查训练误差与测试误差的差异，判断是高偏差还是高方差。（图形或数值）

Diagnostics tell you what to try next

Bayesian logistic regression, implemented with gradient descent

Fixes to try:

- Try getting more training examples.      Fixes high variance.

- Try a smaller set of features                    Fixes high variance

- Try a larger set of features                       Fixes high bias

- Try changing the features: Try Email header vs. email body features  
 Fixes high bias

- Run gradient descent for more iterations          
  Fixes optimization algorithm

- Try Newton’s method                                              
 Fixes optimization algorithm

- Use a different value for lambda                            Fixes optimization objective

- Try using an SVM                                              
       Fixes optimization objective

Optimization algorithm diagnostics

. Bias vs. variance is one common diagnostic

. For other problems. it’s usually up to your own ingenuity to construct your own diagnostics to figure out what’s wrong.

. Another example:

- Bayesian logistic regression gets 2% error on spam, and 2% spam error on non-spam. (Unacceptably high error on non-spam.)

- SVM using a linear kernel gets 10% error on spam, and 0.01% error on non-spam. (Acceptable performance)

- But you want to use logistic regression because of computation efficiency. etc

. What to do next?

More diagnostics

. Other common questions:

- Is the algorithm(gradient descent for logistic regression) converging?

- Are you optimizing the right function?

- I.e. what you care about (weights w higher or non-spam than for spam, 准确率的加权值)

- Bayesian logistic regression? Correct value for lambda?

- SVM? Correct value for C?