ET4248E Chap8.2 Boosting
ET4248E Chap8.2 Boosting
ET4248E Chap8.2 Boosting
Boosting
1
- Simple (weak) classifiers are good
2
- Finding a classifier that’s
just right
3
Boosting question
4
Ensemble classifier
- A single classifier
5
- Ensemble method: each classifier “votes” on prediction
6
- Ensemble classifier in general
7
Boosting
Training a classifier
8
Boosting
Focus learning on
“hard” points
9
Learning on weighted data
More weight on “hard” or more important points
10
Learning a decision stump on weighted data
11
Boosting
Greedy learning ensembles
from data
12
AdaBoost algorithm
13
Computing coefficient wt of classifier ft(x)
14
15
Updating weights αi
16
Normalize weights αi
17
AdaBoost
18
AdaBoost example
19
Updating weights
20
t=2: learn classifier on weighted data
21
Ensemble becomes weighted sum of learned classifiers
22
Decision boundary of ensemble classifier
after 30 iterations
23
Boosting convergence and overfitting
24
AdaBoost Theorem
25
Condition of AdaBoost Theorem
26
27
Boosting tends to be robust to overfitting
28
But boosting will eventually overfit, so must choose max number of
components T
29
30
Threshold split selection algorithm
31
Visualizing the threshold split
32
Split on Age >=38
33
Depth 2: split on Income >=$60k
34
Each split partitions the 2-D space
35
Decision trees vs logistic regression
Logistic regression
36
Decision tree
- Deep 1: Split on
x[1]
37
- Deep 2: split on x[2]
38
39
Decision boundaries
40
Comparing decision boundaries
41