ML | Naive Bayes Scratch Implementation using Python
Naive Bayes is a probabilistic machine learning algorithms based on the Bayes Theorem. It is a simple yet powerful algorithm because of its understanding, simplicity and ease of implementation. It is popular method for classification applications such as spam filtering and text classification. In this article we will learn about Naive Bayes Classifier From Scratch in Python.
Naive Bayes Scratch Implementation using Python
Here we are implementing a Naive Bayes Algorithm using Gaussian distributions. It performs all the necessary steps from data preparation and model training to testing and evaluation.
1. Importing Libraries
Importing necessary libraries:
math: for mathematical operationsrandom: for random number generationpandas: for data manipulationnumpy: for scientific computing
import math
import random
import pandas as pd
import numpy as np
2. Encode Class
The encode_class function converts class labels in the dataset into numeric values. It assigns a unique numeric identifier to each class.
def encode_class(mydata):
classes = []
for i in range(len(mydata)):
if mydata[i][-1] not in classes:
classes.append(mydata[i][-1])
for i in range(len(classes)):
for j in range(len(mydata)):
if mydata[j][-1] == classes[i]:
mydata[j][-1] = i
return mydata
3. Data Splitting
The splitting function is used to split the dataset into training and testing sets based on the given ratio.
def splitting(mydata, ratio):
train_num = int(len(mydata) * ratio)
train = []
test = list(mydata)
while len(train) < train_num:
index = random.randrange(len(test))
train.append(test.pop(index))
return train, test
4. Group Data by Class
The groupUnderClass function takes the data and returns a dictionary where each key is a class label and the value is a list of data points belonging to that class.
def groupUnderClass(mydata):
data_dict = {}
for i in range(len(mydata)):
if mydata[i][-1] not in data_dict:
data_dict[mydata[i][-1]] = []
data_dict[mydata[i][-1]].append(mydata[i])
return data_dict
5. Calculate Mean and Standard Deviation for Class
The MeanAndStdDev function takes a list of numbers and calculates the mean and standard deviation.
The MeanAndStdDevForClass function takes the data and returns a dictionary where each key is a class label and the value is a list of lists, where each inner list contains the mean and standard deviation for each attribute of the class.
def MeanAndStdDev(numbers):
avg = np.mean(numbers)
stddev = np.std(numbers)
return avg, stddev
def MeanAndStdDevForClass(mydata):
info = {}
data_dict = groupUnderClass(mydata)
for classValue, instances in data_dict.items():
info[classValue] = [MeanAndStdDev(attribute) for attribute in zip(*instances)]
return info
6. Calculate Gaussian and Class Probabilities
- The
calculateGaussianProbabilityfunction takes a value, mean, and standard deviation and calculates the probability of the value occurring under a Gaussian distribution with that mean and standard deviation. - The
calculateClassProbabilitiesfunction takes the information dictionary and a test data point as arguments. It iterates through each class and calculates the probability of the test data point belonging to that class based on the mean and standard deviation of each attribute for that class.
def calculateGaussianProbability(x, mean, stdev):
epsilon = 1e-10
expo = math.exp(-(math.pow(x - mean, 2) / (2 * math.pow(stdev + epsilon, 2))))
return (1 / (math.sqrt(2 * math.pi) * (stdev + epsilon))) * expo
def calculateClassProbabilities(info, test):
probabilities = {}
for classValue, classSummaries in info.items():
probabilities[classValue] = 1
for i in range(len(classSummaries)):
mean, std_dev = classSummaries[i]
x = test[i]
probabilities[classValue] *= calculateGaussianProbability(x, mean, std_dev)
return probabilities
7. Prediction for Test Set
- The
predictfunction takes the information dictionary and a test data point as arguments. It calculates the class probabilities and returns the class with the highest probability. - The
getPredictionsfunction takes the information dictionary and the test set as arguments. It iterates through each test data point and predicts its class using thepredictfunction.
def predict(info, test):
probabilities = calculateClassProbabilities(info, test)
bestLabel = max(probabilities, key=probabilities.get)
return bestLabel
def getPredictions(info, test):
predictions = [predict(info, instance) for instance in test]
return predictions
8. Calculate Accuracy
The accuracy_rate function takes the test set and the predictions as arguments. It compares the predicted classes with the actual classes and calculates the percentage of correctly predicted data points.
def accuracy_rate(test, predictions):
correct = sum(1 for i in range(len(test)) if test[i][-1] == predictions[i])
return (correct / float(len(test))) * 100.0
9. Load and Preprocess Data
The code then loads the data from a CSV file using pandas and converts it into a list of lists. It then encodes the class labels and converts all attributes to floating-point numbers. Dataset we are using is of diabetes patients and can be downloaded form here.
# Load data using pandas
filename = '/content/diabetes_data.csv' # Add the correct file path
df = pd.read_csv(filename)
mydata = df.values.tolist()
# Encode classes and convert attributes to float
mydata = encode_class(mydata)
for i in range(len(mydata)):
for j in range(len(mydata[i]) - 1):
mydata[i][j] = float(mydata[i][j])
10. Split Data into Training and Testing Sets
The code splits the data into training and testing sets using a specified ratio. It then trains the model by calculating the mean and standard deviation for each attribute in each class.
# Split the data into training and testing sets
ratio = 0.7
train_data, test_data = splitting(mydata, ratio)
print('Total number of examples:', len(mydata))
print('Training examples:', len(train_data))
print('Test examples:', len(test_data))
Output:
Total number of examples: 768
Training examples: 537
Test examples: 231
11. Train and Test the Model
Calculate mean and standard deviation for each attribute within each class for the training set. Finally, it tests the model on the test set and calculates the accuracy.
# Train the model
info = MeanAndStdDevForClass(train_data)
# Test the model
predictions = getPredictions(info, test_data)
accuracy = accuracy_rate(test_data, predictions)
print('Accuracy of the model:', accuracy)
Output:
Accuracy of the model: 100.0Naive Bayes proves to be an efficient and simple algorithm that works well for classification tasks. It is easy to understand since it is based on Bayes’ theorem and is simple to use and analyze. Naive Bayes Algorithm implementation from scratch in Python can be used to get insights and precise predictions for a variety of applications like spam filtering and text classification.
Frequently Asked Question(FAQs)
How to implement Naive Bayes from scratch with Python?
Implementing Naive Bayes from scratch in Python involves defining the necessary functions for calculating the probabilities required for Bayes’ theorem. This includes the prior probability of each class, the conditional probability of each feature given a class, and the likelihood of a given class. Once these probabilities are calculated, Bayes’ theorem can be used to classify new data points.
How does Naive Bayes Algorithm works?
Naive Bayes is a probabilistic classifier based on Bayes’ theorem, which states that the probability of an event (hypothesis) given evidence can be calculated as the product of the prior probability of the hypothesis and the likelihood of the evidence given the hypothesis, divided by the marginal probability of the evidence. In the context of Naive Bayes, the hypothesis represents the class label, the evidence represents the features of the data point, and the prior and likelihood probabilities are estimated from the training data.
What is Naive Bayes?
Naive Bayes is a classification algorithm based on Bayes’ theorem, which is a statistical method for calculating the probability of an event given a set of conditions. In Naive Bayes, the naive assumption is made that the features of the data are independent of each other, which simplifies the calculations.
Why is Naive Bayes a popular algorithm?
Naive Bayes is a popular algorithm due to its simplicity, efficiency, and effectiveness. It is often used as a baseline classifier for comparison with other more complex algorithms.
When should I use Naive Bayes?
Naive Bayes is a good choice for problems where the features of the data are relatively independent and where the training data is limited. It is also a good choice for problems where the computational cost is a concern.
