model 4 --- decision tree
1 decision tree
1. component
usage: classification
- root node
- decision node
2. choose feature on each node
maximize purity (minimize inpurity)
3. stop splitting
- a node is 100% on class
- splitting a node will result in the tree exceeding a maximum depth
- improvement in purity score are below a threshold
- number of examples in a node is below a threshold
2 meature of impurity
use entropy( H H H) as a meature of impurity
H ( p ) = − p l o g 2 ( p ) − ( 1 − p ) l o g 2 ( 1 − p ) n o t e : 0 l o g 0 = 0 H(p) = -plog_2(p) - (1-p)log_2(1-p)\\ note: 0log0 = 0 H(p)=−plog2(p)−(1−p)log2(1−p)note:0log0=0
3 information gain
1. definition
i n f o m a t i o n _ g a i n = H ( p r o o t ) − ( w l e f t H ( p l e f t ) + w r i g h t H ( p r i g h t ) ) infomation\_gain = H(p^{root}) - (w^{left}H(p^{left}) + w^{right}H(p^{right})) infomation_gain=H(proot)−(wleftH(pleft)+wrightH(pright))
2. usage
- meature the reduction in entropy
- a signal of stopping splitting
3. continuous
find the threshold that has the most infomation gain
4 random forest
- generating a tree sample
given training set of size m
for b = 1 to B:
use sampling with replacement to create a new training set of size m
train a decision tree on the training set
- randomizing the feature choice: at each node, when choosing a feature to use to split, if n features is available, pick a random subset of k < n(usually k = n k = \sqrt{n} k=n ) features and alow the algorithm to only choose from that subset of features