VISVESVARAYA TECHNOLOGICAL UNIVERSITY

“Jnana Sangama”, Belagavi-560014, Karnataka

SEMINAR REPORT ON

DATA SCIENCE AND ITS APPLICATIONS(21AD62)

TOPICS:
• K-Nearest Neighbors
• The Iris Dataset
• The Course of Dimensionality
• Naïve Bayes

Submitted By
Bhavana (1SV21AD003)

Under the guidance of

Mrs. Shruthi S B.E,M.Tech.

Assistant Professor, Dept. Of AI&DS.

Department of Artificial Intelligence and Data Science

SHRIDEVI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Affiliated To Visvesvaraya Technological University)
Sira Road, Tumakuru – 572106, Karnataka.
2023-2024
 ❑ k-Nearest Neighbors:

Nearest neighbors is one of the simplest predictive models there is. It makes no mathematical
assumptions, and it doesn’t require any sort of heavy machinery. The only things it requires are:

▪ Some notion of distance

▪ An assumption that points that are close to one another are similar Most of the techniques we’ll
see in this book look at the dataset as a whole
in order to learn patterns in the data. Nearest neighbors, on the other hand,quite consciously
neglects a lot of information, since the prediction for each new point depends only on the handful of
points closest to it.

What’s more, nearest neighbors is probably not going to help you

understand the drivers of whatever phenomenon you’re looking at.
Predicting my votes based on my neighbors’ votes doesn’t tell you muchabout what causes me to
vote the way I do, whereas some alternative model that predicted my vote based on (say) my
income and marital status very well might.

In the general situation, we have some data points and we have a

corresponding set of labels. The labels could be True and False
, indicating whether each input satisfies some condition like “is spam?” or “is poisonous?” or “would
be enjoyable to watch?” Or they could be categories, like movie ratings (G, PG, PG-13, R, NC-17).
Or they could be the names of presidential candidates. Or they could be favorite programming
languages

Let’s say we’ve picked a number k like 3 or 5. Then, when we want to classify some new data point,
we find the k nearest labeled points and let them vote on the new output. To do this, we’ll need a
function that counts votes. One possibility is:

from typing import List

from collections import Counter

def raw_majority_vote(labels: List[str]) -> str:

votes = Counter(labels)
winner, _ = votes.most_common(1)[0]
return winner

assert raw_majority_vote(['a', 'b', 'c', 'b']) == 'b’

But this doesn’t do anything intelligent with ties. For example, imagine we’re rating movies and the
five nearest movies are rated G, G, PG, PG, and R. Then G has two votes and PG also has two
votes. In that case, we
have several options:
▪ Pick one of the winners at random.
Weight the votes by distance and pick the weighted winner.
▪ Reduce k until we find a unique winner
We’ll implement the third:
def majority_vote(labels: List[str]) -> str:
"""Assumes that labels are ordered from nearest to farthest."""
vote_counts = Counter(labels)
winner, winner_count = vote_counts.most_common(1)[0]
num_winners = len([count
for count in vote_counts.values()
if count == winner_count])

if num_winners == 1:
return winner # unique winner, so return it
else:
return majority_vote(labels[:-1]) # try again without the farthest

# Tie, so look at first 4, then 'b'

assert majority_vote(['a', 'b', 'c', 'b', 'a']) == ‘b’

This approach is sure to work eventually, since in the worst case we go all
the way down to just one label, at which point that one label wins.
With this function it’s easy to create a classifier:
from typing import NamedTuple
from scratch.linear_algebra import Vector, distance

class LabeledPoint(NamedTuple):
point: Vector
label: str

def knn_classify(k: int,

labeled_points: List[LabeledPoint],
new_point: Vector) -> str:

# Order the labeled points from nearest to farthest.

by_distance = sorted(labeled_points,
key=lambda lp: distance(lp.point, new_point))

# Find the labels for the k closest

k_nearest_labels = [lp.label for lp in by_distance[:k]]

# and let them vote.

return majority_vote(k_nearest_labels)
 ❑ The Iris Dataset:
The Iris dataset is a staple of machine learning. It contains a bunch of measurements for 150 flowers
representing three species of iris. For each lower we have its petal length, petal width, sepal length,
and sepal width,
as well as its species. You can download it from
https://archive.ics.uci.edu/ml/datasets/iris:

import requests

data = requests.get(
"https://archive.ics.uci.edu/ml/machine-learningdatabases/iris/iris.data"

with open('iris.dat', 'w') as f:

f.write(data.text)

The data is comma-separated, with fields:

sepal_length, sepal_width, petal_length, petal_width, class

For example, the first row looks like:

5.1,3.5,1.4,0.2,Iris-setosa

In this section we’ll try to build a model that can predict the class (that is,
the species) from the first four measurements.

To start with, let’s load and explore the data. Our nearest neighbors function expects a LabeledPoint ,
so let’s represent our data that way

from typing import Dict

import csv
from collections import defaultdict

def parse_iris_row(row: List[str]) -> LabeledPoint:

sepal_length, sepal_width, petal_length, petal_width, class

"""
measurements = [float(value) for value in row[:-1]]
# class is e.g. "Iris-virginica"; we just want "virginica"
label = row[-1].split("-")[-1]

return LabeledPoint(measurements, label)

with open('iris.data') as f:
reader = csv.reader(f)
iris_data = [parse_iris_row(row) for row in reader]

# We'll also group just the points by species/label so we can plot them
points_by_species: Dict[str, List[Vector]] = defaultdict(list)
for iris in iris_data:
points_by_species[iris.label].append(iris.point)
 We’d like to plot the measurements so we can see how they vary by
species.Unfortunately, they are four-dimensional, which makes them tricky to plot. One
thing we can do is look at the scatterplots for each of the six pairs of measurements I won’t
explain all the details, but it’s a nice illustration of more complicated things you can do with
matplotlib, so it’s worth studying:

from matplotlib import pyplot as plt

metrics = ['sepal length', 'sepal width', 'petal length', 'petal width']
pairs = [(i, j) for i in range(4) for j in range(4) if i < j]
marks = ['+', '.', 'x'] # we have 3 classes, so 3 markers

fig, ax = plt.subplots(2, 3)

for row in range(2):

for col in range(3):
i, j = pairs[3 * row + col]
from matplotlib import pyplot as plt
metrics = ['sepal length', 'sepal width', 'petal length', 'petal width']
pairs = [(i, j) for i in range(4) for j in range(4) if i < j]
marks = ['+', '.', 'x'] # we have 3 classes, so 3 markers
fig, ax = plt.subplots(2, 3)

for row in range(2):

for col in range(3):
i, j = pairs[3 * row + col]
ax[row][col].set_title(f"{metrics[i]} vs {metrics[j]}", fontsize=8)

fig, ax = plt.subplots(2, 3)

for row in range(2):

for col in range(3):
i, j = pairs[3 * row + col]
ax[row][col].set_title(f"{metrics[i]} vs {metrics[j]}", fontsize=8)
ax[row][col].set_xticks([])
ax[row][col].set_yticks([])
for mark, (species, points)
in zip(marks, points_by_species.items()):
xs = [point[i]
for point in points]
ys = [point[j]
for point in points]
ax[row][col].scatter(xs, ys, marker=mark, label=species)
ax[-1][-1].legend(loc='lower right', prop={'size': 6})
If you look at those plots, it seems like the measurements really do cluster by species.
For example, looking at sepal length and sepal width alone, you probably couldn’t
distinguish between versicolor and virginica. But once you add petal length and width
into the mix, it seems like you should be able to predict the species based on the
nearest neighbors.

To start with, let’s split the data into a test set and a training set:
import
random
from
scratch.machine_learning
import split_data
random.seed(12)
iris_train, iris_test = split_data(iris_data, 0.70)
assert len(iris_train) == 0.7 * 150
assert len(iris_test) == 0.3 * 150

The training set will be the “neighbors” that we’ll use to classify the points in the test set.
We just need to choose a value for k, the number of neighbors who get to vote. Too small
(think k = 1), and we let outliers have too much influence; too large (think k = 105), and we
just predict the most common class in the dataset. In a real application (and with more
data), we might create a separate validation set and use it to choose k. Here we’ll just use
k = 5:

from
typing
import Tuple
# track how many times we see (predicted, actual)
confusion_matrix: Dict[Tuple[str, str], int] = defaultdict(int)
num_correct = 0
for iris in iris_test:
predicted = knn_classify(5, iris_train, iris.point)
actual = iris.label
if predicted == actual:
num_correct += 1
confusion_matrix[(predicted, actual)] += 1
pct_correct = num_correct / len(iris_test)
print(pct_correct, confusion_matrix)

On this simple dataset, the model predicts almost perfectly. There’s one versicolor for which it
predicts virginica, but otherwise it gets things exactly right.

Originally used as an example data set on which Fisher's linear discriminant analysis was
applied, it became a typical test case for many statistical classification techniques in machine
learning such as support vector machines.
The use of this data set in cluster analysis however is not common, since the data set only
contains two clusters with rather obvious separation. One of the clusters contains Iris setosa,
while the other cluster contains both Iris virginica and Iris versicolor and is not separable without
the species information Fisher used. This makes the data set a good example to explain the
difference between supervised and unsupervised techniques in data mining: Fisher's linear
discriminant model can only be obtained when the object species are known: class labels and
clusters are not necessarily the same.

Nevertheless, all three species of Iris are separable in the projection on the nonlinear and
branching principal component.[7] The data set is approximated by the closest tree with some
penalty for the excessive number of nodes, bending and stretching. Then the so-called "metro
map" is constructed.[4] The data points are projected into the closest node. For each node
the pie diagram of the projected points is prepared. The area of the pie is proportional to the
number of the projected points. It is clear from the diagram (left) that the absolute majority of
the samples of the different Iris species belong to the different nodes. Only a small fraction
of Iris-virginica is mixed with Iris-versicolor (the mixed blue-green nodes in the diagram).
Therefore, the three species of Iris (Iris setosa, Iris virginica and Iris versicolor) are separable
by the unsupervising procedures of nonlinear principal component analysis. To discriminate
them, it is sufficient just to select the corresponding nodes on the principal tree.
 ❑ The Curse of Dimensionality:
The k-nearest neighbors algorithm runs into trouble in higher dimensions thanks to the “curse of
dimensionality,” which boils down to the fact that high-dimensional spaces are vast. Points in high-
dimensional spaces tend not to be close to one another at all. One way to see this is randomly
generating pairs of points in the d-dimensional “unit cube” in a variety of dimensions, and
calculating the distances between them.Generating random points should be second nature by
now:

def random_point(dim: int) -> Vector:

return [random.random()
for _ in range(dim)]

as is writing a function to generate the distances:

def random_distances(dim: int, num_pairs: int) -> List[float]:

return [distance(random_point(dim), random_point(dim))
for _ in range(num_pairs)]

For every dimension from 1 to 100, we’ll compute 10,000 distances and use those to compute the
average distance between points and the minimum distance between points in each dimension

import
tqdm
dimensions = range(1, 101)
avg_distances = []
min_distances = []
random.seed(0)
for dim in tqdm.tqdm(dimensions, desc="Curse of Dimensionality"):
In low-dimensional datasets, the closest points tend to be much closer than average. But
two points are close only if they’re close in every dimension, and every extra dimension—
even if just noise—is another opportunity for each point to be farther away from every other
point. When you have a lot of dimensions,
It’s likely that the closest points aren’t much closer than average, so two points being close
doesn’t mean very much (unless there’s a lot of structure in your data that makes it behave
as if it were much lower
dimensional).

A different way of thinking about the problem involves the sparsity of higher-dimensional
spaces.
If you pick 50 random numbers between 0 and 1, you’ll probably get a pretty good sample of
the unit interval

As the number of dimensions increases, the average distance between points increases. But
what’s more problematic is the ratio between the closest distance and the average distance
(Figure 12-3): min_avg_ratio = [min_dist / avg_dist for min_dist, avg_dist in
zip(min_distances, avg_distances)]
❑ Naive Bayes:

naive Bayes is a conditional probability model: it assigns probabilities for each of the K
possible outcomes or classes given a problem instance to be classified, represented by a
vector encoding some n features (independent variables).

Naive Bayes is known to outperform even highly sophisticated classification methods.

Bayes theorem provides a way of computing posterior probability P (c|x) from P (c), P (x)
and P (x|c). Look at the equation below: Above, P ( c|x) is the posterior probability of class
(c, target) given predictor (x, attributes ).

The Naive Bayes classifier algorithm is a machine learning technique used for classification
tasks. It is based on Bayes’ theorem and assumes that features are conditionally
independent of each other given the class label.

Naive Bayes is a probabilistic algorithm that’s typically used for classification problems.
Naive Bayes is simple, intuitive, and yet performs surprisingly well in many cases. For
example, spam filters Email app uses are built on Naive Bayes. In this article, I’ll explain
the rationales behind Naive Bayes and build a spam filter in Python.

Imagine a “universe” that consists of receiving a message chosen randomly from all
possible messages. Let S be the event “the message is spam” and B be the event
“the message contains the word bitcoin.” Bayes’s theorem tells us that the
probability that the message is spam conditional on containing the word bitcoin is:

P(S|B) = [P(B|S)P(S)]/[P(B|S)P(S) + P(B|¬S)P(¬S)]

The numerator is the probability that a message is spam and contains bitcoin, while the
denominator is just the probability that a message contains bitcoin. Hence, youn can think
of this calculation as simply representing the proportion of bitcoin
messages that are spam.

If we have a large collection of messages we know are spam, and a large collection of
messages we know are not spam, then we can easily estimate P(B|S) and P(B|¬S).

If we further assume that any message is equally likely to be spam or not spam (so that
P(S) = P(¬S) = 0.5), then:

P(S|B) = P(B|S)/[P(B|S)+P(B|¬S)]

For example, if 50% of spam messages have the word bitcoin, but only 1% of nonspam
messages do, then the probability that any given bitcoin-containing email is spam is:
0. 5/(0.5 + 0.01) = 98%

Imagine now that we have a vocabulary of many words, w ..., w . To move this into the
realm of probability theory, we’ll write X for the event “a message contains the word w .”
Also imagine that (through some unspecified-at-this-point process) we’ve come up with an
estimate P(X |S) for the probability that a spam message contains the ith word, and a
similar estimate P(X |¬S) for the probability that a nonspam message contains the ith word.
The key to Naive Bayes is making the (big) assumption that the presences (or absences) of each
word are independent of one another, conditional on a message being spam or not. Intuitively,
this assumption means that knowing whether a certain spam message contains the word bitcoin
gives you no information about whether that same

message contains the word rolex. In math terms, this means that

P (X1 =x1,...,Xn = xn|S) = P (X1 = x1|S)×⋯×P (Xn = xn|S)

This is an extreme assumption. (There’s a reason the technique has naive in its name.) Imagine
that our vocabulary consists only of the words bitcoin and rolex, and that half of all spam
messages are for “earn bitcoin” and that the other half are for “authentic rolex.” In this case, the
Naive Bayes estimate that a spam message contains both bitcoin and rolex is

P (X1 =1,X2 =1|S) = P(X1 =1|S)P (X2 = 1|S) =.5×.5 =.25

since we’ve assumed away the knowledge that bitcoin and rolex actually never occur together.
Despite the unrealisticness of this assumption, this model often performs well and has historically
been used in actual spam filters. The same Bayes’s theorem reasoning we used for our “bitcoin-
only” spam filter tells us that we can calculate the probability a message is spam using the
equation:

P(S|X = x) =P(X =x|S)/[P(X = x|S)+P(X = x|¬S)]

The Naive Bayes assumption allows us to compute each of the probabilities on the right simply
by multiplying together the individual probability estimates for each vocabulary word.

exp(log(p1) + ⋯+log(pn))

The only challenge left is coming up with estimates for P(Xi|S) and P(Xi|¬S) , the probabilities
that a spam message (or nonspam message) contains the word wi . If we have a fair number of
“training” messages labeled as spam and not spam, an obvious first try is to estimate P(Xi|S)
simply as the fraction of spam messages containing the word wi .

In particular, we’ll choose a pseudocount—k—and estimate the probability of seeing the ith word
in a spam message as

P (Xi|S) = (k+number of spams containing wi)/(2k+number of spams)

We do similarly for P(Xi|¬S) . That is, when computing the spam probabilities for the ith word, we
assume we also saw k additional nonspams containing the word and k additional nonspams not
containing the word.

THANK YOU