Machine Learning 1 (CM50264)

Week 2, Lecture 1: Ensemble learning, random forests, and Bayesian decision rule

Subhadip Mukherjee
Acknowledgment: Tom S. F. Haines (for many images and other contents)
11 Oct. 2022
University of Bath, UK

1
Summary of decision tree learning – 1

• Binary classification, two classes each with 4 examples (total of 8 data points)
• Entropy of parent node: p = 4
8 = 12 , where p = prob. of dot
• H(parent node) = −p log2 p − (1 − p) log2 (1 − p) = 1 bit

2
Summary of decision tree learning – 2

• Split-1:
• left child: p = 34 ; H(left child) = −p log2 p − (1 − p) log2 (1 − p) = 0.8113 bit
• right child: p = 14 ; H(right child) = −p log2 p − (1 − p) log2 (1 − p) = 0.8113 bit
nleft nright
• H(parent node|split-1) = H(left child) + H(right child) = 0.8113 bit
n n
• info gain(split-1) = H(parent node) − H(parent node|split-1) = 0.1887 bit
• info gain(split-2) = 0
• Choose split-1, and apply this process recursively until stopping criterion is met
3
Ensemble learning

4
Ensemble learning: motivating example

• Which class (red or blue) should the

magneta dot be?

5
Ensemble learning: motivating example

• Which class (red or blue) should the

magneta dot be?
• The ‘obvious’ classification boundary

5
Ensemble learning: motivating example

• Which class (red or blue) should the

magneta dot be?
• The ‘obvious’ classification boundary
• But this is just as good (suggesting blue)

5
Ensemble learning: motivating example

• Which class (red or blue) should the

magneta dot be?
• The ‘obvious’ classification boundary
• But this is just as good (suggesting blue)
• As are all of these!

5
Ensemble learning: motivating example

• Which class (red or blue) should the

magneta dot be?
• The ‘obvious’ classification boundary
• But this is just as good (suggesting blue)
• As are all of these!

• Models can be fit in many ways due to:

• Insufficient data
• Noisy data
• Model not complex enough

5
Ensemble learning: things to remember

• Ensembles need diversity (not well defined!)

• Classifiers must make different mistakes

i.e. if all make same mistake =⇒ ensemble will repeat it
• Increasing estimator diversity at expense of individual performance → better
ensemble!
(up to a limit)

6
Ensemble learning: mathematical motivation – 1

• Consider a binary classification problem

• Suppose you have m independent classifiers, each predicts correctly with
probability p
• If the i-th classifier predicts correctly, ŷi = 1, else ŷi = 0

1 with prob. p
ŷi =
0 with prob. 1 − p

• Bernoulli distribution
• Suppose, your final classification does majority voting
m
X jmk
• The final predicted class is correct if ŷ = ŷi ≥ +1
2
i=1

7
Ensemble learning: mathematical motivation – 2

• ŷ has binomial distribution (sum of m independent Bernoulli random variables)

• For any r ∈ {0, 1, 2, · · · , m}, we have
 
m r
P(ŷ = r) = p (1 − p)m−r
r
• Probability of correct prediction is given by
m  
X m r
pfinal = p (1 − p)m−r
r
2 ⌋
r=⌊ m +1

• Let’s plug in some values for m in this formula for p = 0.60 (verify yourself)
• m = 10 =⇒ pfinal = 0.6331
• m = 20 =⇒ pfinal = 0.7553
• m = 40 =⇒ pfinal = 0.8702
8
Ensemble learning: mathematical motivation – 3

• pfinal = 1 − P ŷ ≤
 m 

2
⌊ m2 ⌋
• Let p ≥ 12 + ϵ, then for m large enough, p > m
• In this case, you can show the following bound on pfinal using the Hoeffding’s
inequality (left as an exercise):
  m  !2 
2
pfinal ≥ 1 − exp −2 p − m
m
m
1
• As m → ∞, 2
→ , we have that pfinal → 1
m 2
• So, if we could learn a large number of weak independent classifiers, each with an
error rate slightly better than that of random guess, we could combine them
together to get a very strong classifier!
9
What about regression problems?

• Suppose you have m independent predictors with predicted value ŷi

• Suppose each ŷi has mean y0 (the true value of the target) and variance σ
m
1 X
• Your prediction: ŷ = ŷi
m
i=1
• Variance of ensemble prediction:
 !2 
m m
h i 1 X 1 X
var(ŷ) = E (ŷ − E[ŷ])2 = E  ŷi − E[ŷi ] 
m m
i=1 i=1
 
m
! 2 m
1 X why? 1 X
= 2E  (ŷi − E[ŷi ])  = E (ŷi − E[ŷi ])2
m m2
i=1 i=1

1 σ2
= m · σ2 = (variance reduces by a factor of m)
m2 m
10
Bootstrapping

• Instead of collecting more data. . .

. . . fake it from available data

• Given data set of size n:

Create new data set by drawing
samples with replacement
(will be repetitions)

Sampling with replacement

11
Bagging

• An ensemble technique!

• Short for “Bootstrap AGGregatING”

• Algorithm:
1. Select m, size of ensemble
2. Create m bootstrap draws of original
data set
3. Train estimator on each
4. Combine outputs of all
estimators/classifiers for each query

12
Boosting

• Another ensemble learning algorithm where the models are learned sequentially.
AdaBoost algorithm
• input: Training dataset D = (xi , yi )ni=1 of size n
init t ← 0, πt (i) = n1 , for all i
for t = 1, 2, · · · , T :
– Sample (with replacement) a training subset Dt of size k according to πt
– Learn a classifier (or, a discriminant function) ft
– Apply ft on D and update the sampling distribution to πt+1 based on the
errors ft makes (increase the weights for misclassified samples, decrease for
correctly classified ones)
– Assign a ‘strength’ of αt to ft (based
 P on accuracy)
T 
αt ft (x)
return final classifier: hfinal (x) = sign t=1
PT
t=1 αt
13
Random subspace method

• Another technique to make the classifiers learned on different subsets

uncorrelated.

• Bootstrap applied to features

i.e. fit each estimator with a random subset of features

• How does it help?

• Some features might be highly predictive for the training dataset (due to some bias
in sampling the dataset). Random feature selection helps avoid focusing too much
on such features.
• One might remove repeated features (makes no difference, saves computation)

14
Random Forests

15
Random forests

• Random forest = decision trees + bagging + random subspace method

• Many decision trees (hence the name random forest)

• Algorithm:
1. Select m, number of trees (more is better, up to a limit)
2. Create m bootstrap draws of original data set
3. Train decision tree on each, with random subspace method
4. Combine outputs of all trees for each input feature vector

16
Combine outputs

• Classification:
• The decision trees vote – ensemble outputs winner

• Regression:
• Take the mean/median of all of the estimates

• Can generate probabilities!

• Classification: Create histogram
• Regression: Fit Gaussian distribution

17
Example: red wine – 1

• 1599 exemplars, split: 1199 train, 400 test.

• Input: 11 measurable features:
• fixed acidity
• volatile acidity
• citric acid
• residual sugar
• chlorides
• ···

• Output: 1–10 human rating

(reduced to 1–4 here, to fit on screen)

• Can be used for classification or regression!

18
Example: red wine – 2

Classification tree, max depth 3:

alcohol <= 10.55

entropy = 1.138
samples = 100.0%
value = [0.007, 0.467, 0.513, 0.013]
True False

sulphates <= 0.575 volatile acidity <= 0.87

entropy = 1.025 entropy = 0.986
samples = 61.3% samples = 38.7%
value = [0.007, 0.627, 0.363, 0.003] value = [0.006, 0.213, 0.75, 0.03]

alcohol <= 9.75 total sulfur dioxide <= 105.5 sulphates <= 0.585 density <= 0.994
entropy = 0.815 entropy = 1.07 entropy = 0.892 entropy = 1.054
samples = 23.9% samples = 37.4% samples = 37.0% samples = 1.7%
value = [0.01, 0.783, 0.206, 0.0] value = [0.004, 0.528, 0.463, 0.004] value = [0.0, 0.189, 0.779, 0.032] value = [0.15, 0.75, 0.1, 0.0]

entropy = 0.65 entropy = 0.99 entropy = 1.078 entropy = 0.0 entropy = 0.938 entropy = 0.809 entropy = 1.0 entropy = 0.696
samples = 15.1% samples = 8.8% samples = 34.5% samples = 2.9% samples = 9.4% samples = 27.6% samples = 0.3% samples = 1.3%
value = [0.011, 0.856, 0.133, 0.0] value = [0.01, 0.657, 0.333, 0.0] value = [0.005, 0.488, 0.502, 0.005] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.354, 0.646, 0.0] value = [0.0, 0.133, 0.825, 0.042] value = [0.0, 0.5, 0.5, 0.0] value = [0.188, 0.812, 0.0, 0.0]

Accuracy = 69% Explainable?

19
Example: red wine – 3

Classification tree, unlimited depth:

alcohol <= 10.55
entropy = 1.138
samples = 100.0%
value = [0.007, 0.467, 0.513, 0.013]

sulphates <= 0.575 True False volatile acidity <= 0.87

entropy = 1.025 entropy = 0.986
samples = 61.3% samples = 38.7%
value = [0.007, 0.627, 0.363, 0.003] value = [0.006, 0.213, 0.75, 0.03]

alcohol <= 9.75 total sulfur dioxide <= 105.5 sulphates <= 0.585 alcohol <= 11.45
entropy = 0.815 entropy = 1.07 entropy = 0.892 entropy = 1.054
samples = 23.9% samples = 37.4% samples = 37.0% samples = 1.7%
value = [0.01, 0.783, 0.206, 0.0] value = [0.004, 0.528, 0.463, 0.004] value = [0.0, 0.189, 0.779, 0.032] value = [0.15, 0.75, 0.1, 0.0]

density <= 1.0 fixed acidity <= 7.45 volatile acidity <= 0.365 alcohol <= 11.45 alcohol <= 11.25 chlorides <= 0.083 fixed acidity <= 5.4
entropy = 0.0
entropy = 0.65 entropy = 0.99 entropy = 1.078 entropy = 0.938 entropy = 0.809 entropy = 0.696 entropy = 1.0
samples = 2.9%
samples = 15.1% samples = 8.8% samples = 34.5% samples = 9.4% samples = 27.6% samples = 1.3% samples = 0.3%
value = [0.0, 1.0, 0.0, 0.0]
value = [0.011, 0.856, 0.133, 0.0] value = [0.01, 0.657, 0.333, 0.0] value = [0.005, 0.488, 0.502, 0.005] value = [0.0, 0.354, 0.646, 0.0] value = [0.0, 0.133, 0.825, 0.042] value = [0.188, 0.812, 0.0, 0.0] value = [0.0, 0.5, 0.5, 0.0]

fixed acidity <= 9.15 pH <= 3.175 volatile acidity <= 0.465 sulphates <= 0.535 chlorides <= 0.072 total sulfur dioxide <= 50.5 pH <= 3.355 fixed acidity <= 6.75 total sulfur dioxide <= 99.0 sulphates <= 0.635 volatile acidity <= 0.952 density <= 0.997
entropy = 0.0 entropy = 0.0
entropy = 0.589 entropy = 0.918 entropy = 0.745 entropy = 0.99 entropy = 0.944 entropy = 1.041 entropy = 0.99 entropy = 0.681 entropy = 0.831 entropy = 0.702 entropy = 0.985 entropy = 0.918
samples = 0.8% samples = 0.1%
samples = 14.8% samples = 0.3% samples = 3.1% samples = 5.7% samples = 5.1% samples = 29.4% samples = 4.3% samples = 5.1% samples = 11.4% samples = 16.2% samples = 0.6% samples = 0.3%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.006, 0.871, 0.124, 0.0] value = [0.333, 0.0, 0.667, 0.0] value = [0.027, 0.838, 0.135, 0.0] value = [0.0, 0.559, 0.441, 0.0] value = [0.0, 0.213, 0.754, 0.033] value = [0.006, 0.535, 0.459, 0.0] value = [0.0, 0.558, 0.442, 0.0] value = [0.0, 0.18, 0.82, 0.0] value = [0.0, 0.226, 0.766, 0.007] value = [0.0, 0.067, 0.866, 0.067] value = [0.429, 0.571, 0.0, 0.0] value = [0.0, 0.333, 0.667, 0.0]

volatile acidity <= 0.972 density <= 0.995 density <= 0.998 volatile acidity <= 0.775 chlorides <= 0.089 free sulfur dioxide <= 6.5 fixed acidity <= 10.9 volatile acidity <= 0.605 alcohol <= 10.35 citric acid <= 0.41 residual sugar <= 2.55 volatile acidity <= 0.515 residual sugar <= 6.25 chlorides <= 0.094 chlorides <= 0.054 chlorides <= 0.08 sulphates <= 0.57
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.666 entropy = 0.863 entropy = 0.211 entropy = 0.878 entropy = 0.963 entropy = 0.619 entropy = 0.843 entropy = 1.062 entropy = 0.931 entropy = 0.954 entropy = 0.863 entropy = 0.94 entropy = 0.196 entropy = 0.778 entropy = 0.761 entropy = 0.63 entropy = 0.811
samples = 2.6% samples = 0.2% samples = 0.1% samples = 0.4% samples = 0.3% samples = 0.2% samples = 0.1%
samples = 12.3% samples = 0.6% samples = 2.5% samples = 3.1% samples = 2.6% samples = 1.1% samples = 4.0% samples = 18.8% samples = 10.6% samples = 2.0% samples = 2.3% samples = 2.3% samples = 2.8% samples = 11.0% samples = 4.3% samples = 11.9% samples = 0.3%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.007, 0.844, 0.15, 0.0] value = [0.0, 0.286, 0.714, 0.0] value = [0.033, 0.967, 0.0, 0.0] value = [0.0, 0.703, 0.297, 0.0] value = [0.0, 0.387, 0.613, 0.0] value = [0.0, 0.0, 0.846, 0.154] value = [0.0, 0.271, 0.729, 0.0] value = [0.009, 0.469, 0.522, 0.0] value = [0.0, 0.654, 0.346, 0.0] value = [0.0, 0.375, 0.625, 0.0] value = [0.0, 0.714, 0.286, 0.0] value = [0.0, 0.357, 0.643, 0.0] value = [0.0, 0.03, 0.97, 0.0] value = [0.0, 0.197, 0.795, 0.008] value = [0.0, 0.157, 0.824, 0.02] value = [0.0, 0.035, 0.881, 0.084] value = [0.75, 0.25, 0.0, 0.0]

pH <= 3.54 pH <= 3.305 fixed acidity <= 6.85 residual sugar <= 1.85 residual sugar <= 2.225 alcohol <= 10.1 citric acid <= 0.575 free sulfur dioxide <= 4.5 fixed acidity <= 6.75 sulphates <= 1.305 residual sugar <= 5.5 density <= 0.996 pH <= 3.28 citric acid <= 0.005 total sulfur dioxide <= 106.5 volatile acidity <= 0.82 density <= 0.994 total sulfur dioxide <= 51.5 sulphates <= 0.75 chlorides <= 0.047 pH <= 3.06 volatile acidity <= 0.317 residual sugar <= 4.45
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.599 entropy = 1.585 entropy = 1.0 entropy = 0.948 entropy = 0.84 entropy = 0.971 entropy = 0.952 entropy = 0.976 entropy = 1.118 entropy = 0.868 entropy = 0.934 entropy = 0.672 entropy = 0.592 entropy = 1.0 entropy = 0.414 entropy = 0.989 entropy = 0.918 entropy = 0.644 entropy = 0.998 entropy = 0.954 entropy = 0.522 entropy = 0.69 entropy = 0.337
samples = 0.2% samples = 0.4% samples = 2.3% samples = 0.6% samples = 0.4% samples = 0.7% samples = 1.1% samples = 1.0% samples = 2.5% samples = 0.3% samples = 0.1%
samples = 12.0% samples = 0.3% samples = 0.2% samples = 2.5% samples = 2.2% samples = 0.4% samples = 2.9% samples = 12.3% samples = 6.6% samples = 8.9% samples = 1.7% samples = 1.4% samples = 0.6% samples = 1.3% samples = 1.0% samples = 1.3% samples = 0.3% samples = 9.3% samples = 1.8% samples = 0.7% samples = 3.6% samples = 7.9% samples = 4.0%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.0, 0.854, 0.146, 0.0] value = [0.333, 0.333, 0.333, 0.0] value = [0.5, 0.5, 0.0, 0.0] value = [0.0, 0.633, 0.367, 0.0] value = [0.0, 0.269, 0.731, 0.0] value = [0.0, 0.0, 0.6, 0.4] value = [0.0, 0.371, 0.629, 0.0] value = [0.0, 0.408, 0.592, 0.0] value = [0.025, 0.582, 0.392, 0.0] value = [0.0, 0.71, 0.29, 0.0] value = [0.0, 0.35, 0.65, 0.0] value = [0.0, 0.176, 0.824, 0.0] value = [0.0, 0.857, 0.143, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.083, 0.917, 0.0] value = [0.0, 0.562, 0.438, 0.0] value = [0.0, 0.333, 0.667, 0.0] value = [0.0, 0.135, 0.856, 0.009] value = [0.0, 0.524, 0.476, 0.0] value = [0.0, 0.625, 0.375, 0.0] value = [0.0, 0.07, 0.907, 0.023] value = [0.0, 0.021, 0.853, 0.126] value = [0.0, 0.062, 0.938, 0.0]

citric acid <= 0.075 volatile acidity <= 0.99 residual sugar <= 1.65 residual sugar <= 3.65 sulphates <= 0.565 residual sugar <= 2.7 density <= 0.996 free sulfur dioxide <= 2.0 total sulfur dioxide <= 35.5 fixed acidity <= 6.65 alcohol <= 9.05 chlorides <= 0.097 total sulfur dioxide <= 52.5 alcohol <= 10.85 pH <= 3.505 density <= 0.992 chlorides <= 0.067 free sulfur dioxide <= 30.0 total sulfur dioxide <= 40.5 chlorides <= 0.08 density <= 0.996 sulphates <= 0.685 fixed acidity <= 10.1
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.568 entropy = 1.0 entropy = 0.863 entropy = 0.828 entropy = 0.323 entropy = 0.918 entropy = 0.869 entropy = 0.469 entropy = 0.952 entropy = 0.391 entropy = 1.097 entropy = 0.84 entropy = 0.787 entropy = 0.954 entropy = 0.961 entropy = 0.89 entropy = 0.455 entropy = 0.943 entropy = 0.937 entropy = 0.371 entropy = 0.678 entropy = 0.617 entropy = 1.0
samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.3% samples = 0.3% samples = 0.3% samples = 0.3% samples = 0.8% samples = 0.5% samples = 0.1% samples = 0.3% samples = 0.9% samples = 0.1% samples = 0.3% samples = 0.2% samples = 0.1% samples = 0.3% samples = 0.3% samples = 0.4% samples = 0.1% samples = 3.5%
samples = 11.8% samples = 0.2% samples = 0.6% samples = 1.9% samples = 1.4% samples = 0.8% samples = 2.6% samples = 0.8% samples = 11.4% samples = 1.1% samples = 5.5% samples = 8.7% samples = 1.4% samples = 0.7% samples = 1.1% samples = 1.1% samples = 7.2% samples = 2.1% samples = 1.4% samples = 3.5% samples = 1.9% samples = 6.0% samples = 0.5%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.866, 0.134, 0.0] value = [0.5, 0.0, 0.5, 0.0] value = [0.0, 0.286, 0.714, 0.0] value = [0.0, 0.739, 0.261, 0.0] value = [0.0, 0.059, 0.941, 0.0] value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.29, 0.71, 0.0] value = [0.0, 0.9, 0.1, 0.0] value = [0.0, 0.372, 0.628, 0.0] value = [0.077, 0.923, 0.0, 0.0] value = [0.015, 0.515, 0.47, 0.0] value = [0.0, 0.731, 0.269, 0.0] value = [0.0, 0.235, 0.765, 0.0] value = [0.0, 0.375, 0.625, 0.0] value = [0.0, 0.615, 0.385, 0.0] value = [0.0, 0.692, 0.308, 0.0] value = [0.0, 0.07, 0.919, 0.012] value = [0.0, 0.36, 0.64, 0.0] value = [0.0, 0.647, 0.353, 0.0] value = [0.0, 0.071, 0.929, 0.0] value = [0.0, 0.087, 0.87, 0.043] value = [0.0, 0.0, 0.847, 0.153] value = [0.0, 0.5, 0.5, 0.0]

residual sugar <= 4.9 chlorides <= 0.098 volatile acidity <= 0.34 citric acid <= 0.225 free sulfur dioxide <= 7.5 fixed acidity <= 8.4 volatile acidity <= 0.548 alcohol <= 9.55 fixed acidity <= 9.55 alcohol <= 9.85 density <= 0.996 density <= 0.996 citric acid <= 0.575 residual sugar <= 2.35 pH <= 3.58 free sulfur dioxide <= 14.5 total sulfur dioxide <= 16.5 free sulfur dioxide <= 27.5 volatile acidity <= 0.345 chlorides <= 0.14 citric acid <= 0.605 total sulfur dioxide <= 17.5 sulphates <= 0.71 sulphates <= 0.915
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.258 entropy = 0.674 entropy = 0.702 entropy = 1.0 entropy = 0.811 entropy = 0.797 entropy = 0.842 entropy = 1.0 entropy = 0.65 entropy = 0.987 entropy = 0.944 entropy = 0.337 entropy = 0.567 entropy = 0.991 entropy = 0.684 entropy = 0.663 entropy = 0.254 entropy = 1.0 entropy = 0.65 entropy = 0.722 entropy = 0.672 entropy = 1.379 entropy = 0.694 entropy = 0.811
samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.4% samples = 0.2% samples = 1.3% samples = 0.4% samples = 0.2% samples = 0.1% samples = 0.8% samples = 1.0% samples = 0.1% samples = 0.2% samples = 0.4% samples = 0.3% samples = 0.3% samples = 0.2% samples = 0.6% samples = 2.1% samples = 1.3% samples = 1.1% samples = 0.2%
samples = 3.8% samples = 8.0% samples = 1.8% samples = 0.2% samples = 0.3% samples = 2.4% samples = 6.2% samples = 5.3% samples = 0.5% samples = 5.0% samples = 6.0% samples = 2.7% samples = 1.3% samples = 0.8% samples = 0.9% samples = 2.4% samples = 4.8% samples = 1.5% samples = 1.0% samples = 0.4% samples = 1.4% samples = 0.6% samples = 4.9% samples = 0.3%
value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.957, 0.043, 0.0] value = [0.0, 0.823, 0.177, 0.0] value = [0.0, 0.81, 0.19, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.25, 0.75, 0.0] value = [0.0, 0.241, 0.759, 0.0] value = [0.0, 0.27, 0.73, 0.0] value = [0.0, 0.492, 0.508, 0.0] value = [0.167, 0.0, 0.833, 0.0] value = [0.0, 0.567, 0.433, 0.0] value = [0.0, 0.639, 0.361, 0.0] value = [0.0, 0.938, 0.062, 0.0] value = [0.0, 0.133, 0.867, 0.0] value = [0.0, 0.444, 0.556, 0.0] value = [0.0, 0.818, 0.182, 0.0] value = [0.0, 0.172, 0.828, 0.0] value = [0.0, 0.018, 0.965, 0.018] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.833, 0.167, 0.0] value = [0.0, 0.2, 0.8, 0.0] value = [0.0, 0.176, 0.824, 0.0] value = [0.0, 0.286, 0.571, 0.143] value = [0.0, 0.0, 0.814, 0.186] value = [0.0, 0.75, 0.25, 0.0]

sulphates <= 0.565 free sulfur dioxide <= 33.5 volatile acidity <= 0.73 citric acid <= 0.23 chlorides <= 0.089 citric acid <= 0.315 chlorides <= 0.087 residual sugar <= 1.85 sulphates <= 0.675 residual sugar <= 1.85 total sulfur dioxide <= 29.5 pH <= 3.535 density <= 0.995 total sulfur dioxide <= 9.5 alcohol <= 12.35 residual sugar <= 2.45 sulphates <= 0.67 volatile acidity <= 0.685 volatile acidity <= 0.34 citric acid <= 0.53 pH <= 3.505 chlorides <= 0.065 residual sugar <= 1.95 alcohol <= 12.55
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.154 entropy = 0.736 entropy = 0.485 entropy = 0.371 entropy = 0.971 entropy = 0.665 entropy = 1.0 entropy = 0.755 entropy = 0.91 entropy = 0.863 entropy = 0.943 entropy = 0.974 entropy = 0.371 entropy = 0.65 entropy = 0.918 entropy = 0.896 entropy = 0.918 entropy = 0.133 entropy = 0.971 entropy = 1.0 entropy = 0.544 entropy = 0.918 entropy = 0.971 entropy = 0.605
samples = 0.1% samples = 1.2% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.4% samples = 0.1% samples = 0.7% samples = 0.2% samples = 2.5% samples = 0.1% samples = 0.3% samples = 0.7% samples = 1.1% samples = 0.3% samples = 0.7% samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1%
samples = 3.8% samples = 6.8% samples = 1.6% samples = 1.2% samples = 1.3% samples = 4.3% samples = 1.8% samples = 1.9% samples = 3.3% samples = 2.9% samples = 2.1% samples = 5.3% samples = 1.2% samples = 0.5% samples = 0.3% samples = 1.3% samples = 0.3% samples = 4.5% samples = 1.3% samples = 0.3% samples = 1.3% samples = 0.5% samples = 0.4% samples = 4.5%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [1.0, 0.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.978, 0.022, 0.0] value = [0.0, 0.793, 0.207, 0.0] value = [0.0, 0.895, 0.105, 0.0] value = [0.0, 0.071, 0.929, 0.0] value = [0.0, 0.4, 0.6, 0.0] value = [0.0, 0.173, 0.827, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.783, 0.217, 0.0] value = [0.0, 0.325, 0.675, 0.0] value = [0.0, 0.714, 0.286, 0.0] value = [0.0, 0.36, 0.64, 0.0] value = [0.0, 0.594, 0.406, 0.0] value = [0.0, 0.071, 0.929, 0.0] value = [0.0, 0.167, 0.833, 0.0] value = [0.0, 0.333, 0.667, 0.0] value = [0.0, 0.312, 0.688, 0.0] value = [0.0, 0.0, 0.667, 0.333] value = [0.0, 0.019, 0.981, 0.0] value = [0.0, 0.4, 0.6, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.125, 0.875, 0.0] value = [0.0, 0.333, 0.667, 0.0] value = [0.0, 0.0, 0.4, 0.6] value = [0.0, 0.0, 0.852, 0.148]

residual sugar <= 2.15 chlorides <= 0.063 chlorides <= 0.092 density <= 0.998 alcohol <= 10.45 fixed acidity <= 7.1 fixed acidity <= 8.25 volatile acidity <= 0.59 citric acid <= 0.355 chlorides <= 0.096 total sulfur dioxide <= 17.0 citric acid <= 0.035 pH <= 3.47 alcohol <= 10.45 total sulfur dioxide <= 12.5 volatile acidity <= 0.71 chlorides <= 0.061 fixed acidity <= 7.85 density <= 0.995 total sulfur dioxide <= 31.5 chlorides <= 0.057 pH <= 3.235
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.918 entropy = 0.783 entropy = 0.31 entropy = 0.503 entropy = 0.65 entropy = 0.381 entropy = 0.855 entropy = 0.964 entropy = 0.98 entropy = 0.999 entropy = 0.961 entropy = 0.971 entropy = 0.992 entropy = 1.0 entropy = 1.0 entropy = 0.811 entropy = 0.722 entropy = 0.722 entropy = 0.971 entropy = 0.918 entropy = 0.211 entropy = 0.871
samples = 3.5% samples = 0.8% samples = 0.1% samples = 0.1% samples = 1.1% samples = 0.3% samples = 0.9% samples = 1.3% samples = 0.8% samples = 0.8% samples = 0.5% samples = 1.0% samples = 0.1% samples = 0.4% samples = 0.2% samples = 0.1% samples = 0.5% samples = 0.1% samples = 0.2% samples = 4.2% samples = 0.2% samples = 0.2% samples = 0.9% samples = 0.3% samples = 0.3% samples = 0.2%
samples = 0.3% samples = 6.1% samples = 1.5% samples = 0.8% samples = 0.5% samples = 2.3% samples = 2.1% samples = 1.5% samples = 1.0% samples = 2.1% samples = 2.2% samples = 1.3% samples = 4.8% samples = 0.2% samples = 0.8% samples = 0.3% samples = 0.4% samples = 0.8% samples = 0.4% samples = 0.3% samples = 2.5% samples = 2.0%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.767, 0.233, 0.0] value = [0.0, 0.944, 0.056, 0.0] value = [0.0, 0.111, 0.889, 0.0] value = [0.0, 0.833, 0.167, 0.0] value = [0.0, 0.074, 0.926, 0.0] value = [0.0, 0.28, 0.72, 0.0] value = [0.0, 0.389, 0.611, 0.0] value = [0.0, 0.583, 0.417, 0.0] value = [0.0, 0.52, 0.48, 0.0] value = [0.0, 0.615, 0.385, 0.0] value = [0.0, 0.6, 0.4, 0.0] value = [0.0, 0.552, 0.448, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.25, 0.75, 0.0] value = [0.0, 0.8, 0.2, 0.0] value = [0.0, 0.2, 0.8, 0.0] value = [0.0, 0.4, 0.6, 0.0] value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.0, 0.967, 0.033] value = [0.0, 0.0, 0.708, 0.292]

free sulfur dioxide <= 5.5 citric acid <= 0.235 alcohol <= 9.95 alcohol <= 10.3 free sulfur dioxide <= 12.0 chlorides <= 0.089 free sulfur dioxide <= 17.5 alcohol <= 9.45 sulphates <= 0.595 sulphates <= 0.81 chlorides <= 0.125 volatile acidity <= 0.65 volatile acidity <= 0.41 free sulfur dioxide <= 4.5 pH <= 3.405 alcohol <= 11.5 pH <= 3.345
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.744 entropy = 0.918 entropy = 1.0 entropy = 0.722 entropy = 0.811 entropy = 0.702 entropy = 1.0 entropy = 0.65 entropy = 0.971 entropy = 0.887 entropy = 0.684 entropy = 0.966 entropy = 0.592 entropy = 0.65 entropy = 1.0 entropy = 0.918 entropy = 0.773
samples = 0.1% samples = 0.2% samples = 0.2% samples = 1.3% samples = 0.6% samples = 0.4% samples = 0.1% samples = 1.4% samples = 0.3% samples = 0.5% samples = 0.4% samples = 0.3% samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.3% samples = 0.5% samples = 0.3% samples = 0.2% samples = 0.1% samples = 0.2% samples = 2.3% samples = 0.2%
samples = 5.9% samples = 0.3% samples = 0.2% samples = 0.8% samples = 0.3% samples = 1.8% samples = 1.2% samples = 0.5% samples = 1.7% samples = 1.9% samples = 0.9% samples = 4.3% samples = 0.6% samples = 0.5% samples = 0.3% samples = 0.3% samples = 1.8%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 0.0, 1.0]
value = [0.0, 0.789, 0.211, 0.0] value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.2, 0.8, 0.0] value = [0.0, 0.75, 0.25, 0.0] value = [0.0, 0.19, 0.81, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.167, 0.833, 0.0] value = [0.0, 0.4, 0.6, 0.0] value = [0.0, 0.696, 0.304, 0.0] value = [0.0, 0.818, 0.182, 0.0] value = [0.0, 0.608, 0.392, 0.0] value = [0.0, 0.143, 0.857, 0.0] value = [0.0, 0.833, 0.167, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.0, 0.667, 0.333] value = [0.0, 0.0, 0.773, 0.227]

residual sugar <= 2.05 total sulfur dioxide <= 32.0 total sulfur dioxide <= 26.5 alcohol <= 10.1 pH <= 3.505 sulphates <= 0.585 volatile acidity <= 0.657 pH <= 3.385 fixed acidity <= 5.45
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.792 entropy = 0.503 entropy = 0.918 entropy = 0.98 entropy = 0.811 entropy = 0.722 entropy = 0.469 entropy = 0.996 entropy = 0.94
samples = 0.7% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.8% samples = 0.2% samples = 0.4% samples = 0.1% samples = 0.3% samples = 0.3% samples = 0.1% samples = 0.7% samples = 0.1% samples = 0.5% samples = 0.1% samples = 0.4% samples = 0.2% samples = 0.2% samples = 0.1% samples = 0.2% samples = 0.7%
samples = 5.3% samples = 0.8% samples = 1.0% samples = 1.0% samples = 1.3% samples = 1.7% samples = 0.8% samples = 3.6% samples = 1.2%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.762, 0.238, 0.0] value = [0.0, 0.111, 0.889, 0.0] value = [0.0, 0.333, 0.667, 0.0] value = [0.0, 0.583, 0.417, 0.0] value = [0.0, 0.25, 0.75, 0.0] value = [0.0, 0.8, 0.2, 0.0] value = [0.0, 0.9, 0.1, 0.0] value = [0.0, 0.535, 0.465, 0.0] value = [0.0, 0.0, 0.643, 0.357]

free sulfur dioxide <= 7.5 chlorides <= 0.093 total sulfur dioxide <= 16.5 alcohol <= 9.45 sulphates <= 0.63 sulphates <= 0.66 fixed acidity <= 10.3 density <= 1.0 residual sugar <= 1.85 citric acid <= 0.21 sulphates <= 0.87
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.966 entropy = 0.61 entropy = 0.722 entropy = 0.954 entropy = 0.918 entropy = 0.629 entropy = 1.0 entropy = 0.94 entropy = 0.567 entropy = 0.503 entropy = 0.722
samples = 0.7% samples = 0.1% samples = 0.2% samples = 0.3% samples = 0.8% samples = 0.1% samples = 0.7%
samples = 1.9% samples = 3.3% samples = 0.8% samples = 0.7% samples = 0.5% samples = 1.6% samples = 0.2% samples = 2.3% samples = 1.3% samples = 0.8% samples = 0.4%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.0, 0.609, 0.391, 0.0] value = [0.0, 0.85, 0.15, 0.0] value = [0.0, 0.2, 0.8, 0.0] value = [0.0, 0.375, 0.625, 0.0] value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.842, 0.158, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.357, 0.643, 0.0] value = [0.0, 0.867, 0.133, 0.0] value = [0.0, 0.0, 0.889, 0.111] value = [0.0, 0.0, 0.2, 0.8]

density <= 0.997 sulphates <= 0.475 density <= 0.998 volatile acidity <= 0.555 citric acid <= 0.065 volatile acidity <= 0.57 sulphates <= 0.715
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.918 entropy = 0.485 entropy = 0.918 entropy = 0.811 entropy = 0.881 entropy = 0.855 entropy = 1.0
samples = 0.2% samples = 0.2% samples = 0.6% samples = 0.3% samples = 0.2% samples = 0.3% samples = 0.8% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.9% samples = 0.7% samples = 0.1% samples = 0.3% samples = 0.1%
samples = 1.8% samples = 3.2% samples = 0.3% samples = 0.3% samples = 0.8% samples = 2.1% samples = 0.3%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 0.0, 1.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.895, 0.105, 0.0] value = [0.0, 0.667, 0.333, 0.0] value = [0.0, 0.75, 0.25, 0.0] value = [0.0, 0.7, 0.3, 0.0] value = [0.0, 0.28, 0.72, 0.0] value = [0.0, 0.5, 0.5, 0.0]

pH <= 3.26 chlorides <= 0.076 chlorides <= 0.067 chlorides <= 0.068 total sulfur dioxide <= 56.5
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.997 entropy = 0.985 entropy = 0.206 entropy = 0.544 entropy = 0.989
samples = 0.5% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.3% samples = 0.2% samples = 0.8% samples = 0.2% samples = 0.2%
samples = 1.3% samples = 0.6% samples = 2.6% samples = 0.7% samples = 1.3%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.0, 0.533, 0.467, 0.0] value = [0.0, 0.571, 0.429, 0.0] value = [0.0, 0.968, 0.032, 0.0] value = [0.0, 0.875, 0.125, 0.0] value = [0.0, 0.438, 0.562, 0.0]

free sulfur dioxide <= 13.5 fixed acidity <= 8.35 pH <= 3.315
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.946 entropy = 1.0 entropy = 0.946
samples = 0.3% samples = 0.3% samples = 0.3% samples = 2.4% samples = 0.1% samples = 0.6% samples = 0.4%
samples = 0.9% samples = 0.2% samples = 0.9%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.364, 0.636, 0.0] value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.636, 0.364, 0.0]

total sulfur dioxide <= 62.5 total sulfur dioxide <= 85.0
entropy = 0.0 entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.764 entropy = 0.918
samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.4%
samples = 0.8% samples = 0.5%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.0, 0.222, 0.778, 0.0] value = [0.0, 0.333, 0.667, 0.0]

alcohol <= 9.45 density <= 0.997

entropy = 0.0 entropy = 0.0
entropy = 1.0 entropy = 0.918
samples = 0.4% samples = 0.3%
samples = 0.3% samples = 0.3%
value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]
value = [0.0, 0.5, 0.5, 0.0] value = [0.0, 0.667, 0.333, 0.0]

total sulfur dioxide <= 77.0

entropy = 0.0 entropy = 0.0 entropy = 0.0
entropy = 0.918
samples = 0.1% samples = 0.1% samples = 0.2%
samples = 0.3%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0] value = [0.0, 1.0, 0.0, 0.0]
value = [0.0, 0.333, 0.667, 0.0]

entropy = 0.0 entropy = 0.0

samples = 0.1% samples = 0.2%
value = [0.0, 1.0, 0.0, 0.0] value = [0.0, 0.0, 1.0, 0.0]

Accuracy = 71% Explainable?

20
Example: red wine – 4

Regression tree, max depth 3:

alcohol <= 10.55

mse = 0.289
samples = 100.0%
value = 2.533
True False

sulphates <= 0.575 volatile acidity <= 0.87

mse = 0.25 mse = 0.231
samples = 61.3% samples = 38.7%
value = 2.362 value = 2.804

alcohol <= 9.75 total sulfur dioxide <= 82.5 sulphates <= 0.585 alcohol <= 11.45
mse = 0.178 mse = 0.267 mse = 0.196 mse = 0.248
samples = 23.9% samples = 37.4% samples = 37.0% samples = 1.7%
value = 2.196 value = 2.468 value = 2.842 value = 1.95

mse = 0.129 mse = 0.238 mse = 0.27 mse = 0.127 mse = 0.229 mse = 0.167 mse = 0.152 mse = 0.25
samples = 15.1% samples = 8.8% samples = 31.3% samples = 6.2% samples = 9.4% samples = 27.6% samples = 1.3% samples = 0.3%
value = 2.122 value = 2.324 value = 2.531 value = 2.149 value = 2.646 value = 2.909 value = 1.812 value = 2.5

Mean error = 0.36

21
Example :red wine – 5

Regression tree, unlimited depth:

alcohol <= 10.55
mse = 0.289
samples = 100.0%
value = 2.533

True
sulphates <= 0.575 False volatile acidity <= 0.87
mse = 0.25 mse = 0.231
samples = 61.3% samples = 38.7%
value = 2.362 value = 2.804

alcohol <= 9.75 total sulfur dioxide <= 82.5 sulphates <= 0.585 density <= 0.994
mse = 0.178 mse = 0.267 mse = 0.196 mse = 0.248
samples = 23.9% samples = 37.4% samples = 37.0% samples = 1.7%
value = 2.196 value = 2.468 value = 2.842 value = 1.95

pH <= 3.54 volatile acidity <= 0.582 volatile acidity <= 0.355 total sulfur dioxide <= 105.5 alcohol <= 11.45 alcohol <= 11.25 free sulfur dioxide <= 16.0 total sulfur dioxide <= 11.5
mse = 0.129 mse = 0.238 mse = 0.27 mse = 0.127 mse = 0.229 mse = 0.167 mse = 0.25 mse = 0.152
samples = 15.1% samples = 8.8% samples = 31.3% samples = 6.2% samples = 9.4% samples = 27.6% samples = 0.3% samples = 1.3%
value = 2.122 value = 2.324 value = 2.531 value = 2.149 value = 2.646 value = 2.909 value = 2.5 value = 1.812

citric acid <= 0.645 sulphates <= 0.475 fixed acidity <= 7.85 chlorides <= 0.075 volatile acidity <= 0.535 alcohol <= 9.45 pH <= 3.355 pH <= 3.485 total sulfur dioxide <= 99.0 fixed acidity <= 13.25 residual sugar <= 2.05 alcohol <= 10.75
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.122 mse = 0.25 mse = 0.176 mse = 0.182 mse = 0.262 mse = 0.202 mse = 0.247 mse = 0.148 mse = 0.186 mse = 0.134 mse = 0.222 mse = 0.071
samples = 0.2% samples = 2.9% samples = 0.2% samples = 0.2%
samples = 14.9% samples = 4.3% samples = 4.5% samples = 4.3% samples = 27.0% samples = 3.3% samples = 4.3% samples = 5.1% samples = 11.4% samples = 16.2% samples = 0.3% samples = 1.1%
value = 3.0 value = 2.0 value = 3.0 value = 2.0
value = 2.112 value = 2.49 value = 2.167 value = 2.882 value = 2.475 value = 2.282 value = 2.442 value = 2.82 value = 2.781 value = 3.0 value = 1.333 value = 1.923

density <= 1.0 residual sugar <= 2.225 residual sugar <= 4.35 residual sugar <= 2.7 volatile acidity <= 0.325 fixed acidity <= 10.35 free sulfur dioxide <= 4.5 fixed acidity <= 6.75 fixed acidity <= 9.25 citric acid <= 0.41 chlorides <= 0.103 residual sugar <= 3.275 density <= 0.994 chlorides <= 0.094 sulphates <= 0.635 total sulfur dioxide <= 43.0
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.115 mse = 0.243 mse = 0.033 mse = 0.24 mse = 0.122 mse = 0.169 mse = 0.243 mse = 0.26 mse = 0.239 mse = 0.234 mse = 0.204 mse = 0.079 mse = 0.249 mse = 0.169 mse = 0.126 mse = 0.222
samples = 0.1% samples = 0.7% samples = 0.9% samples = 0.4% samples = 0.1% samples = 0.2% samples = 0.1% samples = 1.0%
samples = 14.8% samples = 3.6% samples = 2.4% samples = 2.1% samples = 1.2% samples = 3.1% samples = 12.3% samples = 14.7% samples = 2.3% samples = 2.0% samples = 2.3% samples = 3.8% samples = 1.3% samples = 11.0% samples = 15.9% samples = 0.3%
value = 1.0 value = 2.0 value = 2.0 value = 2.0 value = 2.0 value = 1.0 value = 1.0 value = 2.0
value = 2.118 value = 2.581 value = 1.966 value = 2.4 value = 3.143 value = 2.784 value = 2.581 value = 2.386 value = 2.393 value = 2.625 value = 2.286 value = 2.913 value = 2.533 value = 2.811 value = 3.01 value = 2.333

density <= 0.997 citric acid <= 0.495 fixed acidity <= 7.05 pH <= 3.435 chlorides <= 0.08 alcohol <= 10.25 citric acid <= 0.62 total sulfur dioxide <= 26.5 free sulfur dioxide <= 2.0 total sulfur dioxide <= 33.0 density <= 0.995 chlorides <= 0.073 chlorides <= 0.075 pH <= 3.135 total sulfur dioxide <= 41.0 residual sugar <= 2.35 citric acid <= 0.425 residual sugar <= 5.5 total sulfur dioxide <= 51.5 sulphates <= 0.75 chlorides <= 0.054 alcohol <= 12.45
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.108 mse = 0.188 mse = 0.196 mse = 0.25 mse = 0.21 mse = 0.16 mse = 0.071 mse = 0.227 mse = 0.099 mse = 0.238 mse = 0.166 mse = 0.258 mse = 0.248 mse = 0.145 mse = 0.122 mse = 0.16 mse = 0.24 mse = 0.099 mse = 0.128 mse = 0.249 mse = 0.158 mse = 0.103
samples = 0.2% samples = 2.3% samples = 0.1% samples = 1.2% samples = 0.7% samples = 0.3% samples = 3.0% samples = 0.5% samples = 0.1% samples = 0.2%
samples = 14.7% samples = 2.3% samples = 1.3% samples = 0.2% samples = 1.7% samples = 0.4% samples = 1.1% samples = 1.9% samples = 0.8% samples = 11.6% samples = 1.9% samples = 12.8% samples = 1.7% samples = 1.4% samples = 0.6% samples = 2.1% samples = 0.8% samples = 0.8% samples = 9.3% samples = 1.8% samples = 4.3% samples = 11.7%
value = 3.0 value = 2.0 value = 4.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0
value = 2.108 value = 2.75 value = 2.267 value = 1.5 value = 2.3 value = 2.8 value = 3.077 value = 2.652 value = 2.111 value = 2.612 value = 2.087 value = 2.431 value = 2.55 value = 2.824 value = 2.143 value = 2.2 value = 2.6 value = 2.889 value = 2.874 value = 2.476 value = 2.863 value = 3.064

citric acid <= 0.075 residual sugar <= 5.0 total sulfur dioxide <= 58.5 density <= 0.996 citric acid <= 0.375 total sulfur dioxide <= 30.0 citric acid <= 0.385 residual sugar <= 1.75 pH <= 3.055 chlorides <= 0.097 volatile acidity <= 0.732 total sulfur dioxide <= 19.0 alcohol <= 8.7 density <= 0.996 citric acid <= 0.305 pH <= 3.505 free sulfur dioxide <= 19.5 volatile acidity <= 0.185 volatile acidity <= 0.34 total sulfur dioxide <= 40.5 density <= 0.993 free sulfur dioxide <= 41.25 residual sugar <= 2.025 residual sugar <= 6.3
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.156 mse = 0.037 mse = 0.155 mse = 0.076 mse = 0.234 mse = 0.076 mse = 0.139 mse = 0.145 mse = 0.16 mse = 0.25 mse = 0.095 mse = 0.134 mse = 0.265 mse = 0.168 mse = 0.109 mse = 0.25 mse = 0.16 mse = 0.078 mse = 0.23 mse = 0.228 mse = 0.234 mse = 0.091 mse = 0.071 mse = 0.154
samples = 0.2% samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1% samples = 1.0% samples = 0.1% samples = 0.1% samples = 0.7% samples = 0.2% samples = 0.5% samples = 0.1% samples = 0.5% samples = 0.1% samples = 1.3% samples = 0.4% samples = 0.7% samples = 0.1% samples = 0.3%
samples = 8.3% samples = 6.4% samples = 2.2% samples = 1.0% samples = 0.7% samples = 1.0% samples = 0.5% samples = 1.4% samples = 4.2% samples = 7.4% samples = 1.8% samples = 2.1% samples = 10.7% samples = 1.2% samples = 1.3% samples = 0.8% samples = 0.4% samples = 7.2% samples = 2.1% samples = 1.4% samples = 0.7% samples = 3.6% samples = 8.2% samples = 3.5%
value = 2.0 value = 3.0 value = 1.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 4.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0
value = 2.162 value = 2.039 value = 2.808 value = 2.083 value = 2.625 value = 2.083 value = 2.167 value = 2.824 value = 2.8 value = 2.506 value = 2.0 value = 2.16 value = 2.484 value = 2.786 value = 2.875 value = 2.5 value = 2.2 value = 2.942 value = 2.64 value = 2.353 value = 2.375 value = 2.953 value = 3.01 value = 3.19

sulphates <= 0.565 citric acid <= 0.105 volatile acidity <= 0.345 alcohol <= 9.5 sulphates <= 0.565 alcohol <= 9.85 fixed acidity <= 7.4 fixed acidity <= 10.15 sulphates <= 0.62 residual sugar <= 1.35 sulphates <= 0.675 density <= 0.996 total sulfur dioxide <= 55.5 chlorides <= 0.069 total sulfur dioxide <= 13.5 sulphates <= 0.815 pH <= 3.295 citric acid <= 0.015 total sulfur dioxide <= 22.0 citric acid <= 0.165 pH <= 3.085 total sulfur dioxide <= 55.0 sulphates <= 0.65 volatile acidity <= 0.275 fixed acidity <= 6.2 fixed acidity <= 10.6 sulphates <= 0.71 chlorides <= 0.058 volatile acidity <= 0.843
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.08 mse = 0.169 mse = 0.026 mse = 0.222 mse = 0.083 mse = 0.188 mse = 0.25 mse = 0.062 mse = 0.16 mse = 0.116 mse = 0.241 mse = 0.104 mse = 0.05 mse = 0.25 mse = 0.222 mse = 0.083 mse = 0.25 mse = 0.076 mse = 0.204 mse = 0.139 mse = 0.068 mse = 0.139 mse = 0.166 mse = 0.139 mse = 0.16 mse = 0.066 mse = 0.099 mse = 0.056 mse = 0.142
samples = 0.8% samples = 0.4% samples = 0.3% samples = 0.1% samples = 0.9% samples = 0.1% samples = 0.4% samples = 0.2% samples = 0.1% samples = 0.2% samples = 0.8% samples = 0.3% samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.4% samples = 0.1% samples = 0.1%
samples = 2.1% samples = 6.2% samples = 6.2% samples = 0.3% samples = 1.8% samples = 0.3% samples = 0.2% samples = 1.3% samples = 0.4% samples = 3.8% samples = 6.0% samples = 1.4% samples = 1.6% samples = 0.2% samples = 0.3% samples = 1.8% samples = 10.6% samples = 1.0% samples = 0.6% samples = 0.5% samples = 7.1% samples = 0.5% samples = 1.6% samples = 1.0% samples = 0.4% samples = 3.5% samples = 2.3% samples = 5.9% samples = 3.4%
value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 1.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 4.0 value = 4.0
value = 2.0 value = 2.216 value = 2.027 value = 2.333 value = 2.909 value = 2.25 value = 2.5 value = 2.933 value = 2.2 value = 2.867 value = 2.597 value = 2.118 value = 2.053 value = 1.5 value = 2.667 value = 2.091 value = 2.496 value = 2.917 value = 2.714 value = 2.833 value = 2.953 value = 2.167 value = 2.789 value = 2.167 value = 2.8 value = 2.929 value = 3.111 value = 2.972 value = 3.171

volatile acidity <= 0.968 chlorides <= 0.105 density <= 0.997 chlorides <= 0.081 chlorides <= 0.093 pH <= 3.155 free sulfur dioxide <= 12.0 pH <= 3.525 volatile acidity <= 0.455 alcohol <= 9.6 volatile acidity <= 0.405 density <= 0.997 pH <= 3.465 residual sugar <= 1.6 fixed acidity <= 9.95 free sulfur dioxide <= 16.5 density <= 0.997 chlorides <= 0.077 volatile acidity <= 0.637 total sulfur dioxide <= 61.0 fixed acidity <= 10.4 free sulfur dioxide <= 4.0 pH <= 3.565 free sulfur dioxide <= 8.5 chlorides <= 0.08 chlorides <= 0.057 density <= 0.995 residual sugar <= 4.45 pH <= 3.71
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.04 mse = 0.247 mse = 0.141 mse = 0.222 mse = 0.014 mse = 0.047 mse = 0.25 mse = 0.101 mse = 0.249 mse = 0.164 mse = 0.059 mse = 0.25 mse = 0.048 mse = 0.25 mse = 0.228 mse = 0.242 mse = 0.222 mse = 0.25 mse = 0.059 mse = 0.222 mse = 0.109 mse = 0.083 mse = 0.047 mse = 0.25 mse = 0.24 mse = 0.043 mse = 0.204 mse = 0.031 mse = 0.128
samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.1% samples = 1.2% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.1% samples = 1.4% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.9% samples = 0.3% samples = 0.4% samples = 0.1% samples = 0.1% samples = 0.4% samples = 0.1% samples = 0.1% samples = 0.3% samples = 0.1%
samples = 2.0% samples = 0.8% samples = 5.4% samples = 0.3% samples = 5.9% samples = 1.7% samples = 0.2% samples = 3.7% samples = 3.6% samples = 2.4% samples = 1.3% samples = 0.2% samples = 1.7% samples = 0.2% samples = 4.3% samples = 6.3% samples = 0.3% samples = 0.2% samples = 6.9% samples = 0.3% samples = 1.3% samples = 0.9% samples = 3.3% samples = 0.2% samples = 0.4% samples = 1.8% samples = 0.6% samples = 5.3% samples = 3.3%
value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 1.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 4.0
value = 1.958 value = 2.556 value = 2.169 value = 2.333 value = 2.014 value = 2.95 value = 2.5 value = 2.886 value = 2.465 value = 2.793 value = 2.062 value = 2.5 value = 2.05 value = 2.5 value = 2.353 value = 2.592 value = 2.333 value = 2.5 value = 2.964 value = 2.333 value = 2.875 value = 2.091 value = 2.95 value = 2.5 value = 3.4 value = 3.045 value = 2.714 value = 3.0 value = 3.15

fixed acidity <= 6.9 total sulfur dioxide <= 11.0 chlorides <= 0.093 density <= 0.996 citric acid <= 0.37 chlorides <= 0.075 sulphates <= 0.605 density <= 0.998 volatile acidity <= 0.395 residual sugar <= 2.4 residual sugar <= 2.0 alcohol <= 9.15 alcohol <= 9.25 alcohol <= 9.275 alcohol <= 10.3 chlorides <= 0.062 citric acid <= 0.035 free sulfur dioxide <= 16.5 density <= 0.995 free sulfur dioxide <= 10.5 volatile acidity <= 0.325 sulphates <= 0.905 residual sugar <= 4.55 free sulfur dioxide <= 27.5
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.204 mse = 0.132 mse = 0.083 mse = 0.222 mse = 0.084 mse = 0.226 mse = 0.204 mse = 0.24 mse = 0.109 mse = 0.222 mse = 0.222 mse = 0.193 mse = 0.173 mse = 0.216 mse = 0.231 mse = 0.04 mse = 0.188 mse = 0.062 mse = 0.204 mse = 0.188 mse = 0.222 mse = 0.017 mse = 0.16 mse = 0.112
samples = 1.9% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.2% samples = 5.0% samples = 1.4% samples = 0.1% samples = 0.1% samples = 0.1% samples = 1.1% samples = 0.1% samples = 0.1% samples = 1.4% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.8% samples = 2.8% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.3% samples = 1.5% samples = 0.3% samples = 0.1%
samples = 0.6% samples = 5.3% samples = 0.9% samples = 0.3% samples = 3.6% samples = 2.4% samples = 1.2% samples = 0.4% samples = 2.0% samples = 0.3% samples = 0.3% samples = 3.5% samples = 0.8% samples = 4.5% samples = 1.8% samples = 6.3% samples = 0.7% samples = 1.3% samples = 0.6% samples = 0.3% samples = 0.3% samples = 4.9% samples = 0.4% samples = 3.3%
value = 2.0 value = 1.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 4.0 value = 3.0 value = 3.0 value = 3.0 value = 4.0
value = 2.714 value = 2.156 value = 2.091 value = 2.667 value = 2.907 value = 2.345 value = 2.714 value = 2.4 value = 2.875 value = 2.333 value = 2.333 value = 2.262 value = 2.778 value = 2.685 value = 2.364 value = 2.987 value = 2.75 value = 2.933 value = 2.714 value = 3.25 value = 2.333 value = 3.017 value = 2.8 value = 3.128

free sulfur dioxide <= 22.5 pH <= 3.46 fixed acidity <= 8.3 volatile acidity <= 0.365 pH <= 3.265 fixed acidity <= 6.75 chlorides <= 0.082 total sulfur dioxide <= 46.0 sulphates <= 1.445 density <= 0.999 chlorides <= 0.085 free sulfur dioxide <= 23.5 citric acid <= 0.51 pH <= 3.165 fixed acidity <= 7.95 total sulfur dioxide <= 17.5 total sulfur dioxide <= 17.5 citric acid <= 0.205
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.222 mse = 0.122 mse = 0.16 mse = 0.182 mse = 0.188 mse = 0.09 mse = 0.245 mse = 0.222 mse = 0.157 mse = 0.109 mse = 0.237 mse = 0.194 mse = 0.109 mse = 0.017 mse = 0.222 mse = 0.139 mse = 0.08 mse = 0.24
samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.8% samples = 0.1% samples = 0.2% samples = 1.9% samples = 0.3% samples = 0.3% samples = 0.2% samples = 1.4% samples = 0.1% samples = 0.2% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.8% samples = 0.3% samples = 0.4% samples = 0.1% samples = 1.2% samples = 0.4% samples = 0.2% samples = 0.3% samples = 0.1% samples = 0.2% samples = 0.1% samples = 4.4% samples = 0.1% samples = 0.3%
samples = 0.3% samples = 5.3% samples = 1.7% samples = 2.1% samples = 0.3% samples = 0.8% samples = 0.6% samples = 0.5% samples = 3.0% samples = 0.7% samples = 3.7% samples = 1.6% samples = 1.3% samples = 4.9% samples = 0.3% samples = 0.5% samples = 2.8% samples = 0.4%
value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 4.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0
value = 2.333 value = 2.143 value = 2.8 value = 2.24 value = 2.25 value = 2.9 value = 2.571 value = 2.667 value = 2.194 value = 2.875 value = 2.614 value = 2.263 value = 2.875 value = 3.017 value = 2.333 value = 3.167 value = 3.088 value = 3.4

density <= 0.997 citric acid <= 0.58 volatile acidity <= 0.375 fixed acidity <= 6.85 fixed acidity <= 7.05 alcohol <= 10.45 chlorides <= 0.109 fixed acidity <= 7.4 alcohol <= 11.15 volatile acidity <= 0.365 free sulfur dioxide <= 6.5 residual sugar <= 3.25
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.112 mse = 0.188 mse = 0.059 mse = 0.144 mse = 0.142 mse = 0.182 mse = 0.244 mse = 0.237 mse = 0.062 mse = 0.139 mse = 0.222 mse = 0.034
samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.3% samples = 0.1% samples = 0.8% samples = 0.3% samples = 0.3% samples = 0.2% samples = 0.3% samples = 0.1% samples = 0.1% samples = 0.6% samples = 0.5% samples = 0.1% samples = 4.4% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.4% samples = 0.2% samples = 0.3%
samples = 5.2% samples = 0.3% samples = 1.3% samples = 1.9% samples = 2.9% samples = 2.1% samples = 1.6% samples = 1.1% samples = 1.3% samples = 0.5% samples = 0.5% samples = 2.3%
value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 4.0 value = 3.0 value = 4.0 value = 3.0
value = 2.129 value = 2.25 value = 2.938 value = 2.174 value = 2.171 value = 2.76 value = 2.421 value = 2.385 value = 2.933 value = 3.167 value = 3.333 value = 3.036

citric acid <= 0.235 chlorides <= 0.079 citric acid <= 0.47 volatile acidity <= 0.435 total sulfur dioxide <= 78.5 density <= 0.997 total sulfur dioxide <= 52.5 sulphates <= 0.74 residual sugar <= 1.8 free sulfur dioxide <= 18.0 residual sugar <= 2.25 residual sugar <= 3.85
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.051 mse = 0.182 mse = 0.25 mse = 0.222 mse = 0.09 mse = 0.125 mse = 0.144 mse = 0.168 mse = 0.109 mse = 0.16 mse = 0.222 mse = 0.16
samples = 0.3% samples = 0.1% samples = 1.2% samples = 0.1% samples = 0.2% samples = 0.4% samples = 1.0% samples = 0.1% samples = 0.4% samples = 0.3% samples = 0.2% samples = 1.9%
samples = 3.1% samples = 2.1% samples = 0.2% samples = 0.3% samples = 1.7% samples = 2.8% samples = 1.9% samples = 1.2% samples = 0.7% samples = 0.4% samples = 0.3% samples = 0.4%
value = 2.0 value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 4.0 value = 3.0 value = 3.0 value = 4.0 value = 3.0
value = 2.054 value = 2.24 value = 2.5 value = 2.667 value = 2.1 value = 2.147 value = 2.826 value = 2.214 value = 2.125 value = 2.8 value = 2.667 value = 3.2

sulphates <= 0.555 free sulfur dioxide <= 26.5 citric acid <= 0.145 fixed acidity <= 9.95 alcohol <= 9.75 volatile acidity <= 0.665 residual sugar <= 2.35
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.139 mse = 0.25 mse = 0.104 mse = 0.05 mse = 0.215 mse = 0.086 mse = 0.188
samples = 2.1% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.1% samples = 1.5% samples = 0.2% samples = 0.8% samples = 0.1% samples = 0.6% samples = 0.1% samples = 0.3% samples = 0.2% samples = 0.1% samples = 0.1% samples = 0.3%
samples = 1.0% samples = 0.7% samples = 1.4% samples = 1.6% samples = 1.3% samples = 1.8% samples = 0.3%
value = 2.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 4.0 value = 3.0
value = 2.167 value = 2.5 value = 2.118 value = 2.053 value = 2.312 value = 2.905 value = 2.75

citric acid <= 0.245 chlorides <= 0.067 volatile acidity <= 0.735 total sulfur dioxide <= 53.0 citric acid <= 0.255 pH <= 3.165 total sulfur dioxide <= 37.0 volatile acidity <= 0.675
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.083 mse = 0.222 mse = 0.059 mse = 0.25 mse = 0.09 mse = 0.222 mse = 0.05 mse = 0.25
samples = 0.1% samples = 0.2% samples = 0.1% samples = 1.4% samples = 0.1% samples = 0.3%
samples = 0.9% samples = 0.5% samples = 1.3% samples = 0.2% samples = 0.8% samples = 0.5% samples = 1.6% samples = 0.2%
value = 3.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0
value = 2.091 value = 2.333 value = 2.062 value = 2.5 value = 2.1 value = 2.667 value = 2.947 value = 2.5

residual sugar <= 2.1 chlorides <= 0.061 total sulfur dioxide <= 39.0
mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
mse = 0.25 mse = 0.222 mse = 0.222
samples = 0.8% samples = 0.3% samples = 1.3% samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.8% samples = 0.1% samples = 0.2% samples = 0.3% samples = 1.3% samples = 0.1% samples = 0.1%
samples = 0.2% samples = 0.3% samples = 0.3%
value = 2.0 value = 2.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0 value = 3.0 value = 2.0 value = 3.0
value = 2.5 value = 2.667 value = 2.667

mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0 mse = 0.0
samples = 0.1% samples = 0.1% samples = 0.1% samples = 0.2% samples = 0.1% samples = 0.2%
value = 3.0 value = 2.0 value = 2.0 value = 3.0 value = 2.0 value = 3.0

Mean error = 0.33

22
Example: red wine – 6

• Random forest with 32 trees

• Classification: Accuracy = 79% (better than 71%)

• Regression: Mean error = 0.30 (better than 0.33)

• Basically impossible to visualize/understand/interpret!

23
A Bayesian view of classification

24
Bayesian decision rule – 1

Statistical foundation of classification:

• Let the target y ∈ Y = {1, 2, · · · , c}

• Input feature x ∈ X ⊆ Rd and the target y are viewed as samples of two jointly
distributed random variables x and y, respectively
• Given x = x, we want to design a decision rule ŷ : X 7→ Y that predicts y
Optimal Bayes decision rule:
Z
• Bayes risk R(ŷ) = Ex,y [L(y, ŷ(x))] = Ey|x=x [L(y, ŷ(x)] p(x) dx
X | {z }
conditional risk
• Observation: A decision rule that minimizes the conditional risk for any given
x ∈ X also minimizes the overall risk.
25
Bayesian decision rule – 2

The conditional risk corresponding to ŷ = i for some i ∈ {1, 2, · · · , c} is given by:

R(ŷ = i|x) = Ey|x=x [L(y, ŷ = i)]
c
L(y = j, ŷ = i)P(y = j|x = x)
X
=
j=1

Now, consider the 0-1 loss defined as:


0 if i = j
L(y = j, ŷ = i) =
1 if i ̸= j

P(y = j|x = x) = 1 − P(y = i|x = x)

X
=⇒ R(ŷ = i|x) =
j̸=i

26
Optimal Bayes classifier – 1

• Predict ŷ = i∗ where

i∗ = arg min R(ŷ = i|x)

i∈{1,2,··· ,c}

= arg max P(y = i|x = x)

i∈{1,2,··· ,c}

• What does this classifier do?

• It predicts the class that has the highest posterior probability
p(x|y = i) · P(y = i)
• Bayes rule: P(y = i|x = x) =
p(x)

27
Optimal Bayes classifier – 2

Optimal Bayes classifier: Suppose you are given an input x.

• For each class i ∈ {1, 2, · · · , c}, compute the product of likelihood and prior
(which is proportional to the posterior probability of the class):

p(x|y = i) · P(y = i) ∝ P(y = i|x = x)

| {z } | {z }
pi (x) πi

• Decide in favor of the class for which this product is the largest.
How to estimate the prior and the likelihood from the data?
no. of examples from class-i
• Prior: π̂i = total no. of examples = ni
n
• What about the likelihood pi (x) = p(x|y = i)?
• Also called class conditional distributions
28
Naı̈ve Bayes classifier
d
Y  
• Feature independence: p(x|y = i) = p x(j) |y = i , i ∈ {1, 2, · · · , c}
j=1
• Assume a parametric form for p (j)


x |y = i , and estimate the parameters from
data (simplest being Gaussian)
(j) (j)
• Let p x(j) |y = i be Gaussian with mean µi and standard deviation σi

! 12
1 X (j) 1 X  (j) 2
(j) (j) (j)
• µ̂i = x , σ̂i = x − µ̂i
ni ni
class-i class-i

To put it simply, estimate the mean and standard deviation of all features from
samples belonging to each class i

• Use the estimated likelihoods and priors to determine the class with the highest
posterior, given a new input x
29