Data Science Course

Data Science
Powered by ChatGPT
Introduction
Welcome to the world of data science! This course is designed to give you a comprehensive
introduction to the tools, techniques, and concepts used in data science. Whether you are new to the
field or looking to expand your knowledge, this course will provide you with the foundational skills
and hands-on experience you need to start working with data in Python.
In this course, we will start by introducing you to the Python programming language and its popular
libraries for data science, including NumPy, pandas, matplotlib, and Scikit-learn. You will learn how to
load, manipulate, and visualize data using these libraries, as well as perform basic data analysis and
statistics.
We will then delve into the data science lifecycle, covering key concepts such as data mining, data
cleaning, and data exploration. You will learn how to use these concepts to prepare and understand
your data, and how to use the data to create models.
Next, we will cover the different types of data modeling, including supervised, unsupervised, and
reinforcement learning. You will learn how to use linear and logistic regression, evaluate
classification models, and use pipelines and hyperparameter tuning.
We will also cover Natural Language Processing (NLP) using python. This will give you an
understanding of how to process and analyze text data, which is an increasingly important area in
data science.
Throughout the course, you will have the opportunity to apply the concepts you have learned
through hands-on exercises and a capstone project, which will give you the chance to put your skills
to the test and create your own data science project.
By the end of the course, you will have a solid understanding of the key concepts and tools used in
data science, and you will be able to start working with data in Python with confidence. This course is
designed to be engaging and interactive, so you will have plenty of opportunities to ask questions,
get feedback, and work with other students to expand your knowledge. We look forward to helping
you get started on your data science journey!
1
Table of Contents
Introduction...........................................................................................................................................1
Data Science Lifecycle............................................................................................................................4
Programming languages in data science................................................................................................6
Numpy................................................................................................................................................8
Pandas................................................................................................................................................9
Sci-kit learn.......................................................................................................................................10
Data Collection.....................................................................................................................................12
Structured vs unstructured data......................................................................................................12
Human- versus computer-generated data.......................................................................................13
Quantitative versus qualitative data................................................................................................14
Data collection methods...................................................................................................................15
Data Cleaning.......................................................................................................................................18
Data Validation.................................................................................................................................18
Data Standardization........................................................................................................................19
Data Deduplication...........................................................................................................................20
Data Completion..............................................................................................................................21
Data Exploration...................................................................................................................................22
Data visualization.............................................................................................................................23
Statistical Analysis............................................................................................................................27
Feature engineering.............................................................................................................................35
Data Modelling.....................................................................................................................................37
General process................................................................................................................................37
Supervised learning..........................................................................................................................38
Model selection............................................................................................................................39
Model evaluation.........................................................................................................................46
Example of supervised learning....................................................................................................46
Ensembles....................................................................................................................................48
Pipelines.......................................................................................................................................50
Hyperparameter tuning................................................................................................................51
Complete example........................................................................................................................52
Overfitting versus underfitting......................................................................................................54
Unsupervised learning......................................................................................................................55
KMeans clustering........................................................................................................................56
Mean shift clustering....................................................................................................................57
2
Principal Component Analysis......................................................................................................58
Model evaluation.........................................................................................................................58
Reinforcement learning....................................................................................................................59
Techniques...................................................................................................................................59
Example........................................................................................................................................60
Natural Language Processing...............................................................................................................62
Bag of words.....................................................................................................................................63
Unigram BOW...............................................................................................................................63
N-gram BOW................................................................................................................................64
Term Frequency-Inverse Document Frequency BoW...................................................................64
Word embeddings........................................................................................................................66
3
Data Science Lifecycle
The data science lifecycle is the process of solving a business problem using data. It consists of
several stages, each with its own set of tools and techniques, that work together to turn raw data
into actionable insights. The stages of the data science lifecycle are:
- Problem definition: This is the starting point of the data science process, and it involves
posing a business question or identifying a problem that needs to be solved. This stage is
crucial as it sets the direction for the rest of the project and helps to ensure that the solution
addresses the right problem.
- Data collection: Once the problem has been defined, the next step is to collect the data
needed to solve it. This stage includes tasks such as identifying the data sources, extracting
the data, and cleaning it to make it usable. The quality and quantity of data can greatly
impact the outcome of the project, so it's important to ensure that the data collected is
accurate and complete.
- Data cleaning: A continuous process including tasks such as checking for missing values,
removing duplicate data, and handling outliers. This step is important as it ensures that the
data is accurate and complete, and that it can be used to answer the business question.
- Data exploration: After the data has been collected, the next step is to explore it. This stage
is used to gain a deeper understanding of the data and its characteristics. This can be done
by visualizing the data, looking for patterns, and identifying any outliers or missing values.
This stage is also used to identify any data preparation or cleaning that may be needed
before moving on to the next stage.
- Feature extraction: The most relevant features or variables in the data that can be used to
solve the business problem are identified. This step is important as it helps to reduce the
dimensionality of the data and make it more manageable for modeling. It also helps to
identify any correlated or redundant features that can be removed from the dataset, which
can improve the performance of the model. Note that this is a continuous process that must
be repeated with new insights or data collected.
- Data modeling: Once the data has been explored, the next step is to build models to solve
the problem. This stage includes tasks such as selecting the appropriate model, training it,
and evaluating its performance. The models built in this stage are used to generate insights
and predictions that can be used to make decisions.
- Deployment: After the model has been built and evaluated, it's time to put it into
production. This stage includes tasks such as deploying the model to a production
environment, creating an API to access it, and monitoring its performance. It's important to
ensure that the model is performing as expected in the production environment, and that it
can be easily updated or replaced as needed.
- Maintenance: Data science projects are not static, and data and business requirements
change over time. This stage is used to maintain the model, update it and retrain it as
necessary. This can include monitoring the performance of the model, retraining it with new
data, and updating the parameters or features used in the model.
The data science lifecycle is iterative, and it is important to go back and forth between different
stages to ensure that the solution addresses the problem and is accurate. Each stage of the data
science lifecycle requires different skills and tools, and it is important to have a diverse team with a
range of expertise to complete the project successfully. It's also important to remember that the data
4
science lifecycle is not a one-time process, but a continuous cycle that adapts to the changing
business needs.
It's important to note that the above stages are not always linear and that sometimes the stages may
be repeated or done in parallel. Additionally, the process may also include other stages such as data
governance, data quality, and data privacy. The process of data science also includes communication
and collaboration with other teams and stakeholders to ensure that the insights generated are useful
and can be implemented.
5
Programming languages in data science
The most common programming languages in the data science domain are:
1. Python: Python is a widely-used programming language in data science and it offers a vast
ecosystem of powerful libraries for data manipulation, visualization, and analysis. Some of
the most commonly used libraries in data science are NumPy, pandas, matplotlib, and scikit-
learn. Python's popularity in data science is due to its simplicity, expressiveness, and the
availability of a wide range of libraries and frameworks.
2. R: R is another popular programming language in data science and it has a strong focus on
data visualization and statistical analysis. It has a large community and a vast ecosystem of
packages and libraries for data manipulation, visualization, and modeling. R is often used in
academia and research due to its statistical capabilities.
3. SQL: SQL is a domain-specific language used for managing and querying relational databases.
It is widely used in data science for extracting and manipulating data from databases. Data
cleaning, data exploration and feature engineering are some of the tasks that SQL is used for.
4. Java: Java is a general-purpose programming language that is widely used in industry for
building enterprise-level applications. It is also used in data science for building distributed
and scalable systems.
5. Scala: Scala is a programming language that is often used for big data processing and
analytics. It is similar to Java and it runs on the JVM, but it has a more concise and expressive
syntax, and it's more suitable for functional programming.
In this course, we will use Python as our main programming language. There are many reasons for
choosing Python as a programming language for data science. Some of the most common reasons
include:
- Large community and ecosystem: Python has a large and active community of developers
and users, which means that there is a wealth of resources available for learning and
troubleshooting. Additionally, the Python ecosystem includes a wide range of powerful
libraries and frameworks for data manipulation, visualization, and analysis, such as NumPy,
pandas, matplotlib, and scikit-learn.
- Simplicity and expressiveness: Python has a simple and easy-to-learn syntax, which makes it
a great choice for beginners and non-programmers. It is also expressive, which means that it
is able to express complex ideas with fewer lines of code than other languages.
- Versatility: Python is a general-purpose programming language, which means that it can be
used for a wide range of tasks, including web development, scientific computing, and data
analysis. This makes it a good choice for data science projects that require the use of multiple
tools and techniques.
- Interoperability: Python can easily integrate with other languages and tools. It can be used
to call R functions and also has libraries to call C/C++ code. This makes it a good choice for
projects that require the use of multiple languages or tools.
- High-performance libraries and tools: Python has a wide range of high-performance libraries
and tools for data science, such as NumPy, pandas, Dask, and PyTorch, that allows for the
manipulation and analysis of large datasets.
6
- Good for Machine Learning: Python has a number of libraries, such as scikit-learn,
TensorFlow and PyTorch, which are very popular in the machine learning community. This
makes it a good choice for projects that involve machine learning and deep learning.
- Good for NLP: Python has a number of libraries for natural language processing, such as
NLTK, spaCy and gensim, which are widely used in the industry.
Python is an interpreted programming language, which means that it is executed by an interpreter

instead of being compiled to machine code like standard compiled languages like C, C++ and Java.
This means that Python code is executed line by line by an interpreter, rather than being compiled to
machine code and executed directly by the CPU. This makes it more convenient for development and
debugging, as changes to the code can be tested immediately without the need for a separate
compilation step.
Aside from being a interpreted programming language, another big difference with C# is that Python
is a dynamically typed language, which means that variable types are determined at runtime rather
than being explicitly defined by the programmer. This allows for more flexibility and less verbosity in
# Variables
x = 5
y = "Hello World"
# Functions
def add(a, b):
return a + b
result = add(x, 2)
print(result)
# Lists
my_list = [1, 2, 3, 4, 5]
# Lists are 0 indexed, so the first element is my_list[0]
print(my_list[0])
# Lists can be sliced, for example to get elements 1-3:
print(my_list[1:4])
# Dictionaries
my_dict = {"a": 1, "b": 2, "c": 3}
# Accessing elements in a dictionary is done using keys
print(my_dict["a"])
# You can add new elements to a dictionary
my_dict["d"] = 4
print(my_dict)
# And also modify existing ones
my_dict["a"] = 5
print(my_dict)
the code, but it can also make the code more prone to errors if the types of the variables are not
properly checked.
7
8
Numpy
NumPy is a powerful library for numerical computing in Python. It provides a wide range of functions
for working with arrays and matrices of numerical data, including mathematical and statistical
operations, linear algebra, and Fourier transforms. Some of the main features of NumPy include:
- N-dimensional arrays: NumPy provides a powerful and efficient n-dimensional array object,
called ndarray, that can be used to store and manipulate large amounts of numerical data.
- Mathematical functions: NumPy provides a wide range of mathematical functions that can
be applied to arrays and matrices, including basic arithmetic operations, trigonometric
functions, and linear algebra operations.
- Broadcasting: NumPy's broadcasting feature allows mathematical operations to be
performed on arrays of different shapes, without the need for explicit looping.
- C-API: NumPy provides a C-API that allows other libraries to access its functionality and
perform computations in C or C++.
Here is an example of some of the main functions of NumPy:
import numpy as np
# Creating an array
a = np.array([1, 2, 3, 4])
print(a)
# Mathematical operations
b = np.array([5, 6, 7, 8])
c = a + b
print(c)
# Broadcasting
d = a * 2
print(d)
# Linear Algebra
e = np.matrix([[1, 2], [3, 4]])
f = np.matrix([[5, 6], [7, 8]])
g = e + f
print(g)
h = np.dot(e, f)
print(h)
# Statistical functions
i = np.array([1, 2, 3, 4, 5])
j = np.mean(i)
print(j)
k = np.median(i)
print(k)
9
Pandas
Pandas is a powerful library for data manipulation and analysis in Python. It provides a wide range of
data structures and data analysis tools, including dataframes and series. The two main data
structures in pandas are:
- DataFrame: A DataFrame is a 2-dimensional table-like data structure that can hold data of
different types (e.g. integers, strings, floating-point numbers). It has rows and columns, and
can be thought of as a spreadsheet or a SQL table.
- Series: A Series is a 1-dimensional array-like object that can hold data of any type. It is similar
to a column in a DataFrame, and it has a label (index) associated with each element.
Some of the main features of Pandas include:
- Handling of missing data

- Data alignment and merging
- Data manipulation and cleaning
- Data visualization
Here is an example of some of the main functions of pandas:
import pandas as pd
# Creating a DataFrame
data = {'name': ['John', 'Jane', 'Bob'],
'age': [25, 22, 31],
'city': ['New York', 'Los Angeles', 'Chicago']}
df = pd.DataFrame(data)
print(df)
# Data manipulation
df['age'] += 1
df.rename(columns={'age':'Age'}, inplace=True)
# Data exploration
print(df.head())
print(df.describe())
# Data filtering
age_filter = df['Age'] > 25
print(df[age_filter])
# Data Grouping
grouped = df.groupby('city').mean()
print(grouped)
# Data Merging
df2 = pd.DataFrame({'city': ['Chicago', 'New York', 'Los
Angeles'],
'temperature': [21, 14, 35]})
merged_df = pd.merge(df, df2, on='city')
print(merged_df)
10
In this example, we start by creating a DataFrame df using a dictionary of data, then we perform
some data manipulation such as renaming a column and adding 1 to the age column. We also
perform some data exploration by displaying the first few rows of data and some statistics of the
dataframe, filtering the data based on a condition, grouping the data by city and finally merging the
data with another dataframe.
Sci-kit learn
Scikit-learn (also known as sklearn) is a powerful library for machine learning in Python. It provides a
wide range of tools for supervised and unsupervised learning, including regression, classification,
clustering, and dimensionality reduction. Some of the main features of scikit-learn include:
- Consistent interface to machine learning models: scikit-learn provides a consistent interface

to various machine learning models, making it easy to switch between different models and
apply them to different datasets.
- Algorithm selection: scikit-learn provides a wide range of algorithms to choose from,
including linear and non-linear models, as well as ensemble methods.
- Evaluation metrics: scikit-learn provides a wide range of evaluation metrics, such as
accuracy, precision, recall, and F1 score, to evaluate the performance of machine learning
models.
- Model selection: scikit-learn provides tools for model selection and hyperparameter tuning,
such as cross-validation and grid search, to help find the best model for a given dataset.
- Preprocessing: scikit-learn provides preprocessing functions for data cleaning and feature
extraction, such as imputation of missing values, normalization, and scaling.
Here is an example of some of the main functions of scikit-learn:
from sklearn.datasets import load_iris

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
# Loading the dataset

iris = load_iris()
X = iris.data
y = iris.target
# Splitting the data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
# Training a Logistic Regression model

clf = LogisticRegression()
clf.fit(X_train, y_train)
# Making predictions on the test set

y_pred = clf.predict(X_test)
# Evaluating the model

acc = accuracy_score(y_test, y_pred)
print(acc)
11
In this example, we start by loading the Iris dataset using the `load_iris()function from scikit-learn.
Then, we split the data into training and test sets using thetrain_test_split()function. Next, we train a
Logistic Regression model on the training data using thefit()method of the model class. Then, we use
the trained model to make predictions on the test set using thepredict()method. Finally, we evaluate
the performance of the model using theaccuracy_score()` function and print the accuracy.
This is just a simple example of what scikit-learn can do, but it provides many more functionalities.
Scikit-learn allows you to easily switch between different models, apply different preprocessing
techniques, and use advanced evaluation metrics. It also allows you to use ensemble methods, such
as Random Forest, Gradient Boosting and more. In addition, it allows for feature selection,
dimensionality reduction, and model selection and evaluation.
12
Data Collection
Data collection is the process of gathering and measuring information on variables of interest, in an
established systematic fashion that enables one to answer stated research questions, test
hypotheses, and evaluate outcomes. The data collection process can be divided into two main
categories: primary data collection and secondary data collection.
Primary data collection refers to the process of collecting data directly from its source. This can be
done through various methods such as surveys, experiments, observations, and interviews. The data
collected through these methods is original and is collected specifically for the research project at
hand. The advantages of primary data collection are that the data is current, specific to the research
project and can be tailored to the research questions.
Secondary data collection, on the other hand, refers to the process of collecting data that has already
been collected by someone else. This can be done through various sources such as books, journals,
newspapers, government reports, and online databases. The data collected through these sources is
not collected specifically for the research project at hand. The advantages of secondary data
collection are that it is less time-consuming and less expensive than primary data collection.
Primary Data Secondary Data

Collected specifically for the research project Already collected by someone else
Data is current and specific to the research Data may not be as current or specific to the
project research project
Collected through methods such as surveys, Collected through various sources such as
experiments, observations, and interviews books, journals, newspapers, government
reports, and online databases
More time-consuming and expensive to collect Less time-consuming and less expensive to
collect
Can be tailored to the research questions May not be tailored to the research questions
Structured vs unstructured data

Structured data refers to data that is organized in a specific format and can be easily stored,
searched, and analyzed using computer programs. The data is typically stored in a structured format,
such as a table, where each row represents a record, and each column represents a field or attribute.
Structured data is usually stored in a relational database or in a flat file format such as a CSV file.
However, working with structured data can also present some difficulties. One of the main difficulties
is ensuring the quality of the data. Structured data often contains errors, inconsistencies, and missing
values, which can make it difficult to clean and prepare the data for analysis. Additionally, structured
data can be difficult to integrate and merge with other data sources, especially when different data
sources use different data formats or schema.
Another difficulty is dealing with outliers, missing values, and errors in the data. Outliers, missing
values, and errors can greatly impact the results of the analysis and can lead to incorrect conclusions.
This can be addressed by applying appropriate data cleaning, imputation, and handling techniques.
13
Finally, working with structured data also requires a good understanding of the data and the analysis
techniques that are appropriate for that data, which can be a challenge for people who are new to
data science.
Examples of structured data include:
- Data from a customer relationship management (CRM) system

- Data from a financial transaction system
- Data from a survey with a fixed set of questions
On the other hand, unstructured data refers to data that does not have a specific format or structure.
It is typically unorganized and difficult to store, search, and analyze using computer programs.
Examples of unstructured data include:
- Social media posts and comments

- News articles
- Images and videos
- Audio recordings
Unstructured data can be in various forms, such as text, images, audio, or video. It can be in the form
of natural language text, such as emails, customer feedback, or social media posts. Unstructured data
is usually harder to manage and analyze than structured data, as it requires specialized tools and
techniques to extract insights from it.
Working with unstructured data can present several difficulties. One of the main challenges is that
unstructured data does not have a specific format or structure, making it difficult to store, search,
and analyze using computer programs. This requires specialized tools and techniques to extract
insights from it. Another difficulty is the complexity of unstructured data. Unstructured data can
come in various forms such as text, images, audio, or video and can be in the form of natural
language text, such as emails, customer feedback, or social media posts. This complexity makes it
harder to manage and analyze than structured data.
Another difficulty is the amount of data, unstructured data can be very large, making it difficult to
process and store. This can be challenging for organizations that need to analyze large volumes of
unstructured data in real-time, such as social media data, or large datasets like images, videos, and
audio files.
Additionally, unstructured data can be difficult to extract insights from, as it often requires the use of
natural language processing (NLP) techniques, which can be complex and computationally expensive.
This may require specialized knowledge and expertise in NLP, which can be difficult to find.
Moreover, unstructured data often contains a lot of noise and irrelevant information, which can
make it difficult to extract useful insights. This requires preprocessing techniques such as text
cleaning and feature extraction to remove noise and irrelevant information.
Finally, unstructured data is often not labeled, making it difficult to conduct supervised machine
learning tasks. This requires the use of unsupervised or semi-supervised techniques to extract
insights from unstructured data, which can be more complex and computationally expensive.
Human- versus computer-generated data

Human-generated data and computer-generated data are two different types of data that are
collected and used in different ways.
14
Human-generated data is data that is collected and generated by people, typically through surveys,
interviews, observations, and other forms of direct human input. This type of data is often subjective
and can be affected by human biases, emotions, and errors. Human-generated data can be in the
form of text, images, audio, or video and can be in the form of natural language text, such as emails,
customer feedback, or social media posts. Human-generated data is often used in social sciences,
market research, and psychology.
Computer-generated data, on the other hand, is data that is generated by computer programs or
machines. This type of data is often objective and can be generated in large volumes, at high speeds,
and with high accuracy. Computer-generated data can be in the form of sensor data, log files, and
other types of machine-generated data. Computer-generated data is often used in fields such as
finance, manufacturing, and healthcare.
Here is a comparison of human-generated data and computer-generated data:
Human-generated data Computer-generated data

Collected and generated by people Generated by computer programs or machines
Subjective and can be affected by human Objective and accurate
biases, emotions, and errors
Can be in the form of text, images, audio, or Can be in the form of sensor data, log files and
video other types of machine-generated data
Used in social sciences, market research, and Used in fields such as finance, manufacturing,
psychology and healthcare
Quantitative versus qualitative data

Quantitative data and qualitative data are two different types of data that are collected and used in
different ways.
Quantitative data is numerical data that can be measured and quantified. This type of data is often
objective and can be collected and analyzed using statistical methods. Examples of quantitative data
include numerical values such as age, income, and temperature, as well as binary data (yes/no,
true/false). Quantitative data is often used in fields such as economics, finance, and the natural
sciences.
Qualitative data, on the other hand, is non-numerical data that describes characteristics or qualities.
This type of data is often subjective and can be collected and analyzed using methods such as
observation, interviews, and document analysis. Examples of qualitative data include text, images,
and audio recordings. Qualitative data is often used in fields such as sociology, psychology, and
anthropology.
Here is a comparison of quantitative data and qualitative data:
Quantitative data Qualitative data

Numerical data that can be measured and Non-numerical data that describes
quantified characteristics or qualities
Often objective Often subjective
Collected and analyzed using statistical Collected and analyzed using methods such as
methods observation, interviews, and document analysis
Examples include numerical values such as age, Examples include text, images, and audio
15
income, and temperature recordings
Used in fields such as economics, finance, and Used in fields such as sociology, psychology,
natural sciences and anthropology
Data collection methods

Secondary data sources are a valuable resource for researchers and practitioners as they can provide
a wealth of information on a wide range of topics. The methods for collecting secondary data sources
include using existing online datasets, using API's offered by companies, countries or organizations
and scraping of websites. Each method has its own advantages and disadvantages and can be used to
gather information on a wide range of topics. However, it is important to consider the quality and
reliability of the data sources, and to have a clear understanding of the bias, errors, and limitations of
the data, as well as its relevance to the research question.
Using existing online datasets is a method of collecting secondary data by searching for and
downloading data that has been made available by other organizations or government agencies on
the internet. This can include datasets from open data portals, data repositories, and other publicly
available sources. These datasets can be in various formats such as CSV, Excel, JSON, among others.
This method can be an efficient way of collecting large amounts of information and can be used to
gather information on a wide range of topics. However, it is important to consider the quality and
reliability of the data sources, and to have a clear understanding of the bias, errors, and limitations of
the data, as well as its relevance to the research question. Examples of good sources for these type
of datasets are:
- Open Data portals: Many governments, organizations, and research institutions make their
datasets available for public use through open data portals. Examples of open data portals
include data.gov (U.S.), data.gov.uk (U.K.), and data.gc.ca (Canada).
- Data repositories: There are several data repositories that provide access to large collections
of datasets on a wide range of topics, such as the UCI Machine Learning Repository, the Data
Hub, and Kaggle.
- Academic journals: Many academic journals make datasets available on their websites, either
as supplementary material or through a data repository linked to the journal.
Using API's offered by companies, countries or organizations is a method of collecting secondary data
by accessing and extracting data from an Application Programming Interface (API) provided by an
organization. API's allow external parties to access and extract data from a website or application in a
structured format. This method is particularly useful when the data is not available for download but
is accessible through an API. Many companies, countries, and organizations provide API's that allow
access to their data, such as data on weather, finance, transportation and social media. However, it is
important to check the terms of use and data limits of the API before using it. For example, the
following companies offer an API for accessing their data:
- witter API: Allows developers to access and extract data from Twitter, such as tweets, user
profiles, and trending topics.
- Google Maps API: Allows developers to access and extract data from Google Maps, such as
location information, route information, and satellite imagery.
- OpenWeatherMap API: Allows developers to access and extract weather data from
OpenWeatherMap, such as current weather conditions, forecast, and historical weather
data.
16
- Quandl API: Allows developers to access and extract financial, economic and alternative
datasets from various sources such as governments, central banks, and private companies
- Facebook Graph API: Allows developers to access and extract data from Facebook, such as
user information, posts, and page insights.
- NASA API: Allows developers to access and extract data from NASA, such as satellite imagery,
climate data, and space exploration information.
- Spotify Web API: Allows developers to access and extract data from Spotify, such as track
information, album information, and artist information.
Scraping of websites is a method of collecting secondary data by extracting data from websites using
web scraping tools. This method involves automating the process of visiting a website and extracting
data from it, such as text, images, and links. Web scraping can be used to gather information on a
wide range of topics and can be an efficient way of collecting large amounts of information. Web
scraping can be performed in python using a number of different packages, for instance by using
scrapy. The following code is an example of this package:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):

for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('span small::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
next_page = response.css('li.next a::attr(href)').get()

if next_page is not None:
yield response.follow(next_page, self.parse)
This spider starts by visiting the first page of the quotes website
(http://quotes.toscrape.com/page/1/) and parses the HTML of the page looking for elements with
the CSS class "quote". It extracts the text, author, and tags of the quote and returns them as a
dictionary. Then it looks for the next page link by using the css selector li.next a::attr(href), it gets the
next link and sends a request to follow the next page.
Web scraping has several limitations, one of them is the legal and ethical limitations, since it can be
considered illegal or unethical if it is done without the permission of the website owner. Many
websites have terms of service that prohibit scraping and it is important to check these terms before
starting to scrape a website. Another limitation is the technical limitations, web scraping can be
challenging when websites use CAPTCHA, JavaScript, cookies, or other technologies to prevent
scraping. These technologies can make it difficult to extract the desired information, and may require
additional resources to overcome. Additionally, websites can block IP addresses that make too many
17
requests in a short period of time, or that make requests that look like they are coming from a
scraper, making it difficult to scrape large amounts of data without being detected. Web scraping can
also be limited by the quality of the data on the website, data may be unstructured, incomplete, or
inconsistent, making it difficult to extract useful information. Another limitation is that websites that
use AJAX, AngularJS and other dynamic technologies are harder to scrape than static ones, as the
content will be generated dynamically and may not be available in the page source. However, there
are smarter web scrapers that use techniques like headless browsers, browser automation, and
machine learning to overcome some of these limitations, but these scrapers may require additional
resources, such as more powerful hardware and specialized software, to run effectively.
18
Data Cleaning
Data cleaning is an essential step in the data science process that involves the process of identifying
and addressing errors, inconsistencies, and missing values in the data. The goal of this step is to
prepare the data for further analysis and modeling by ensuring that it is accurate, complete, and
consistent.
One of the main actions during data cleaning is data validation. This involves checking the data for
errors, such as missing values, duplicate records, or invalid data types. For example, a field that is
supposed to contain a date may have a string instead, or a field that is supposed to contain a number
may have letters.
Another important action is data standardization, which involves ensuring that the data conforms to
a specific format or structure. For example, standardizing the format of dates, phone numbers, or
addresses can make it easier to analyze and compare the data.
Data cleaning also includes data deduplication, which is the process of identifying and removing
duplicate records from the data. This can be done by comparing the values of specific fields, such as
the name or address of a person, to find exact or near-exact matches.
Data cleaning also includes data completion, which is the process of filling in missing values in the
data. This can be done by imputing the missing data using statistical techniques, such as mean
imputation or regression imputation or by inferring from other data points or external sources.
Data cleaning is important because it ensures that the data is accurate, complete and consistent,
which is essential for reliable analysis and modeling. Inaccurate or inconsistent data can lead to
incorrect conclusions and poor decision-making.
The amount of time spent on data cleaning can vary depending on the size and complexity of the
data, as well as the quality of the data. In general, the more data that needs to be cleaned, the more
time it will take. Additionally, if the data is of poor quality or has a high level of errors,
inconsistencies, or missing values, it will take longer to clean it. According to a survey by the Data
Warehouse Institute, data scientists spend 60% of their time on data cleaning and data preparation.
However, this is just an average and the proportion of time spent on data cleaning may vary from
one project to another.
Data Validation
Data validation is an important step in the data cleaning process that involves checking the data for
errors, inconsistencies, and invalid values. The goal of data validation is to ensure that the data is
accurate, complete, and conforms to the expected format or structure.
One common method of data validation is to check for missing values, this step consists of identifying
fields that are supposed to contain a value but are empty, or fields that contain a default value such
as "N/A" or "NULL" that doesn't make sense for the context.
Another method of data validation is to check for invalid data types, this step consists of identifying
fields that contain data that doesn't match the expected data type, for example, a field that is
supposed to contain a date may contain a string, or a field that is supposed to contain a number may
contain letters.
19
Another method is to check for outliers, this step consists of identifying values that are outside of the
expected range, for example, a temperature reading that is well below freezing, or a weight that is
above the maximum human weight.
Data validation can also include checking for duplicate records, this step involves identifying records
that have the same values in specific fields, such as name, address, or ID.
Finally, data validation can include checking the data against external sources, this step involves
comparing the data to other data sets or external sources, such as a list of valid postal codes or a list
of valid email domains to ensure that it is correct and consistent.
Data validation is important because it ensures that the data is accurate, complete, and conforms to
the expected format or structure, which is essential for reliable analysis and modeling. By identifying
and addressing errors, inconsistencies, and invalid values in the data, the data can be cleaned and
prepared for further analysis and modeling. An example of some commonly used functions in this
step can be seen below.
import pandas as pd
# read data from a CSV file

df = pd.read_csv("data.csv")
# check for missing values

print(df.isnull().sum())
# check for invalid data types

print(df.dtypes)
# check for duplicate records

print(df.duplicated().sum())
# check for outliers

print(df.describe())
# check for invalid values in specific

columns
print(df[df["age"] < 0])
Data Standardization
Data standardization is an important step in the data cleaning process that involves ensuring that the
data conforms to a specific format or structure. The goal of data standardization is to make it easier
to analyze and compare the data, by ensuring that data is in a consistent format.
One common method of data standardization is to standardize the format of fields that contain
dates, phone numbers, or addresses. For example, a date field may contain dates in different formats
such as "dd-mm-yyyy", "mm/dd/yyyy" or "yyyy-mm-dd". Data standardization will convert all the
dates to a single format like "yyyy-mm-dd"
20
Another method of data standardization is to convert data to a consistent unit of measurement. For
example, a field that contains weights may contain values in kilograms and pounds, data
standardization will convert all the values to a single unit of measurement.
Another method is to standardize categorical data, this step consists of ensuring that all the values of
a categorical field are in a consistent format, for example, a field that contains the color of a car may
contain "red", "Red" and "R" all referring to the same color, during standardization, all the values will
be converted to a consistent format, like "red"
Data standardization can also include handling missing data, this step involves replacing missing data
with a default value or a value that is inferred from other data points or external sources.
Data standardization is important because it ensures that data is in a consistent format, which is
essential for reliable analysis and modeling. By standardizing the data, it can be more easily
compared, analyzed, and modeled. Additionally, it makes it easier to join different data sets and to
combine data from different sources. Standardizing the data also increases the chances of finding
useful insights and patterns.
import pandas as pd

# standardize date format

df["date"] = pd.to_datetime(df["date"],
format='%Y-%m-%d')
# standardize phone number format

df["phone"] =
df["phone"].str.replace(r'[^0-9]+', '')
# standardize address format

df["address"] = df["address"].str.upper()
# standardize categorical data

df["color"] = df["color"].str.lower()
# standardize unit of measurement

df["weight"] = df["weight"] * 0.45359237
Data Deduplication
Data deduplication is an important step in the data cleaning process that involves identifying and
removing duplicate records from the data. The goal of data deduplication is to ensure that the data is
accurate, complete and consistent by removing duplicate records that can lead to confusion, and
errors in analysis and modeling.
21
One common method of data deduplication is to compare the values of specific fields, such as the
name or address of a person, to find exact or near-exact matches. For example, if the dataset
contains contact information, there may be multiple records with the same name, address, and
phone number, and these records can be considered as duplicates.
Another method of data deduplication is to use a unique identifier, such as a primary key or a unique
ID, to identify duplicate records. For example, in a database of customers, each customer may have a
unique ID that can be used to identify duplicate records.
Data deduplication also includes handling near-duplicate records, this step involves identifying
records that are similar but not exactly the same, for example, records that have slightly different
spellings or formatting of a name, address or phone number.
Data deduplication can also include handling the issue of data quality, for example, if the data
contains errors or inconsistencies, it may be difficult to identify duplicate records.
Data deduplication is important because it ensures that the data is accurate, complete and
consistent, which is essential for reliable analysis and modeling. By identifying and removing
duplicate records from the data, the data can be cleaned and prepared for further analysis and
modeling, also it helps to avoid confusion, and errors in analysis and modeling.
import pandas as pd

# remove duplicate records based on specific columns

df = df.drop_duplicates(subset=["name", "address", "phone"])
# remove duplicate records based on a unique identifier

df = df.drop_duplicates(subset=["id"])
#remove near-duplicate records using the Levenshtein distance

from Levenshtein import distance
df = df.drop_duplicates(subset=["name"], keep="first",
ignore_case=True,
threshold=lambda x,y:distance(x,y) <= 2)
Data Completion
Data completion is an important step in the data cleaning process that involves filling in missing or
incomplete data. The goal of data completion is to ensure that the data is accurate, complete, and
consistent, by filling in missing values that can affect the results of analysis and modeling.
One common method of data completion is to use a default value for missing data. For example, if a
record is missing a value for age, a default value of -1 can be used to indicate that the value is
missing. This is useful when missing values are not so important in the analysis, and it's better to
have a value than nothing
22
Another method of data completion is to use a value that is inferred from other data points or
external sources. For example, if a record is missing a value for income, a value can be inferred from
the occupation, education level, or location of the person.
Data completion also includes handling missing data that is important, this step involves identifying
missing values that will affect the analysis or modeling, and filling in the missing data with a value
that is inferred from other data points or external sources, or by using imputation techniques like
mean, median or mode.
Data completion is important because it ensures that the data is accurate, complete, and consistent,
which is essential for reliable analysis and modeling. By filling in missing values, the data can be
cleaned and prepared for further analysis and modeling.
import pandas as pd

# fill missing values with a default value

df["age"].fillna(-1, inplace=True)
# fill missing values with a value inferred from other data points
df["income"] = df.groupby("occupation")["income"].transform(
lambda x: x.fillna(x.mean()))
# fill missing values with imputation techniques

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='median')
df["weight"] = imputer.fit_transform(df[["weight"]]).ravel()
Data Exploration
The next phase in the data science lifecycle is the data exploration step. Data exploration involves
analyzing and understanding the data. The goal of data exploration is to uncover insights and
patterns in the data that can be used to inform the analysis and modeling stages.
One common method of data exploration is visualization, which involves creating visual
representations of the data, such as histograms, scatter plots, and bar charts, to uncover patterns
and trends. For example, a histogram can be used to visualize the distribution of a numerical
variable, while a scatter plot can be used to visualize the relationship between two numerical
variables.
Another method of data exploration is statistical analysis, which involves using statistical techniques
to summarize and understand the data. For example, calculating the mean and standard deviation of
a numerical variable can provide insight into the central tendency and spread of the data.
Additionally, calculating correlation and covariance between two variables can provide insight into
the relationship between the variables.
23
Data exploration is important because it helps to understand the structure, patterns, and trends in
the data. It helps to identify the variables that are important for the analysis and modeling and to
discover hidden patterns that can be used to inform the analysis and modeling stages. It also helps to
identify potential issues with the data, such as missing values or outliers, that need to be addressed
before proceeding to the next stages.
Data visualization
Visualization is a powerful method of data exploration that involves creating visual representations of
the data. It helps to uncover patterns, trends, and relationships in the data that are not immediately
obvious when looking at the raw data. There are several types of visualizations that can be used for
data exploration, such as:
- Histograms: which show the distribution of a numerical variable by dividing the range of
values into bins and counting the number of observations in each bin. Histograms can be
used to identify patterns such as skewness, outliers, and the presence of multiple modes.
- Scatter Plots: which show the relationship between two numerical variables by plotting each
observation as a point on a coordinate grid. Scatter plots can be used to identify patterns
such as linear or non-linear relationships, clusters, and outliers.
- Bar Charts: which show the distribution of a categorical variable by counting the number of
observations in each category and representing them as bars. Bar charts can be used to
identify patterns such as the relative frequencies of different categories and the presence of
outliers.
- Line Charts: which show the change in a numerical variable over time, by plotting the
observations as points on a line. Line charts can be used to identify patterns such as trends,
seasonality, and sudden changes.
- Heatmaps: which show the relationship between two categorical variables by plotting each
observation as a cell in a grid and coloring the cells according to a third variable. Heatmaps
can be used to identify patterns such as the relative frequencies of different combinations of
categories and the presence of outliers.
There are several Python packages that can be used for visualization, some of the most popular ones
are:
- Matplotlib: is a plotting library for the Python programming language and its numerical
mathematics extension NumPy. It provides an object-oriented API for embedding plots into
applications using general-purpose GUI toolkits like Tkinter, wxPython, Qt, or GTK.
- Seaborn: is a statistical data visualization library based on Matplotlib. It provides a high-level
interface for drawing attractive and informative statistical graphics. It is particularly well
suited for visualizing complex datasets with multiple variables.
- Plotly: is a Python library that allows you to create interactive, web-based visualizations. It
supports a wide variety of charts and maps, and it can be used to create visualizations that
can be embedded in web pages or exported as standalone HTML files.
Histogram
A histogram is a graph that shows the distribution of a numerical variable by dividing the range of
values into bins and counting the number of observations in each bin. The x-axis represents the range
of values for the variable and the y-axis represents the frequency of observations in each bin. The
height of each bar represents the number of observations that fall within the range of values
represented by the bin.
24
A histogram plot can be used to identify patterns such as skewness, outliers, and the presence of
multiple modes. Skewness refers to the asymmetry of the distribution, with a positive skew
indicating a long tail on the right side of the histogram and a negative skew indicating a long tail on
the left side of the histogram. Outliers are values that are far from the typical values in the
distribution, and multiple modes are the presence of more than one peak in the histogram.
Here's an example of how to create a histogram plot using Matplotlib:
import matplotlib.pyplot as plt

import numpy as np
# generate some data

data = np.random.normal(100, 15, 1000)
# create a histogram
plt.hist(data, bins=20)
# add labels
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Histogram of Data')
# show the plot

plt.show()
Scatter Plot
A scatter plot is a graph that shows the relationship between two numerical variables by plotting
each observation as a point on a coordinate grid. The x-axis represents the values of one variable and
the y-axis represents the values of the other variable. Each point on the plot represents an
observation in the data, with the x-coordinate of the point representing the value of one variable and
the y-coordinate of the point representing the value of the other variable.
A scatter plot can be used to identify patterns such as linear or non-linear relationships, clusters, and
outliers. A linear relationship between two variables is characterized by a straight line pattern in the
scatter plot, while a non-linear relationship is characterized by a curved pattern. Clusters are groups
of points that are close together on the plot, and outliers are points that are far away from the
typical pattern of points.
Here's an example of how to create a scatter plot using Matplotlib:
25
import numpy as np

x = np.random.normal(0, 1, 100)
y = np.random.normal(0, 1, 100)
# create a scatter plot

plt.scatter(x, y)
# add labels
plt.xlabel('x')
plt.ylabel('y')
plt.title('Scatter Plot of x and y')
# show the plot

plt.show()
Bar Chart
A bar chart, also known as a bar graph, is a graph that shows the distribution of a categorical variable
by counting the number of observations in each category and representing them as bars. The x-axis
represents the categories of the variable and the y-axis represents the frequency of observations in
each category. The height of each bar represents the number of observations that fall within the
category represented by the bar.
A bar chart can be used to identify patterns such as the relative frequencies of different categories,
and the presence of outliers. A bar chart can be used to compare the frequencies of different
categories, and to identify any categories that are over or underrepresented in the data. Outliers can
be identified as bars that are significantly higher or lower than the other bars in the chart.
Here's an example of how to create a bar chart using Matplotlib:

categories = ['A', 'B', 'C', 'D', 'E']
values = [10, 20, 30, 40, 50]
# create a bar chart

plt.bar(categories, values)
# add labels
plt.xlabel('Categories')
plt.ylabel('Values')
plt.title('Bar Chart of Categories and
Values')
# show the plot

plt.show()
26
Line Chart
A line chart is a graph that shows the relationship between two numerical variables over a period of
time or over a range of values. The x-axis represents the time or range of values of one variable, and
the y-axis represents the values of the other variable. The line connects the points of each
observation, and the slope of the line represents the rate of change of the values.
A line chart can be used to identify patterns such as trends, seasonality, and fluctuations. A trend is a
general direction of the values, a seasonality is the repeating patterns in the values, and a fluctuation
is the short-term variations in the values.
Here's an example of how to create a line chart using Matplotlib:

import numpy as np

x = np.linspace(0, 10, 100)
y = np.sin(x)
# create a line chart

plt.plot(x, y)
# add labels
plt.xlabel('x')
plt.ylabel('y')
plt.title('Line Chart of x and y')
# show the plot

plt.show()
Heatmap
A heat map is a graphical representation of data where individual values are represented as colors in
a 2-dimensional grid. The x-axis and y-axis represent the rows and columns of the grid, respectively,
and the color of each cell represents the value at that location in the grid. The colors are usually a
gradient, where the darkest color represents the highest value and the lightest color represents the
lowest value.
Heat maps can be used to identify patterns such as the distribution of values and the presence of
outliers. They can also be used to compare the values of different categories, and to identify any
categories that are over or underrepresented in the data.
Here's an example of how to create a heat map using Matplotlib:
27
import numpy as np

data = np.random.rand(10,10)
# create a heat map

plt.imshow(data, cmap="hot")
# add labels
plt.xlabel('Columns')
plt.ylabel('Rows')
plt.title('Heat map of data')
# show the plot

plt.show()
Statistical Analysis
There are many different statistical analyses that can be performed during the data exploration step,
depending on the type of data and the research question. Some common statistical analyses that can
be used during data exploration include:
- Descriptive statistics: Descriptive statistics provide a summary of the main characteristics of

the data, such as the mean, median, mode, standard deviation, and quartiles. These can be
used to understand the distribution and spread of the data.
- Frequency distributions: A frequency distribution is a table or graph that shows the number
of observations in each category of a categorical variable. Frequency distributions can be
used to understand the distribution of the data and to identify patterns and outliers.
- Correlation: Correlation is a statistical measure that quantifies the strength and direction of
the relationship between two numerical variables. Correlation can be used to understand the
relationships between different variables and to identify patterns and outliers.
- T-test: A t-test is a statistical method that is used to determine whether there is a significant
difference between the means of two groups. This can be used to understand the
relationships between different variables and to identify patterns and outliers.
- ANOVA: ANOVA (Analysis of Variance) is a statistical method that is used to determine
whether there is a significant difference between the means of two or more groups. This can
be used to understand the relationships between different variables and to identify patterns
and outliers.
- Chi-square test: A chi-square test is a statistical method that is used to determine whether
there is a significant association between two categorical variables. This can be used to
understand the relationships between different variables and to identify patterns and
outliers.
Descriptive statistics
Descriptive analysis is a method of summarizing and describing the main characteristics of the data. It
is one of the first steps in data exploration, and it is used to understand the distribution and spread
of the data. Descriptive statistics are used to summarize the data, and they include measures such as
the mean, median, mode, standard deviation, and quartiles.
28
Pandas is a powerful library in python that provides functions to perform descriptive statistics. You
can use the describe() function to get the basic statistics of a pandas DataFrame or Series. The
describe() function returns the count, mean, standard deviation, minimum, 25th percentile, median
(50th percentile), 75th percentile, and maximum of the data.
Here's an example of how to use the describe() function in pandas:
import pandas as pd
# create a sample dataframe

data = {'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
'age': [25, 30, 35, 40, 45],
'income': [50000, 60000, 70000, 80000, 90000]}
# get the basic statistics of the dataframe

df.describe()
In this example, the describe() function is used to get the basic statistics of the dataframe df which
contains the name, age and income of 5 people. The output will be a table containing the count,
mean, standard deviation, minimum, 25th percentile, median (50th percentile), 75th percentile, and
maximum of the data.
Pandas also provides other functions for descriptive statistics such as mean(), median(), mode(),
min(), max(), sum(), count(), var(), std(), skew(), kurt() and quantile() to get the statistics of a specific
column or series.
Frequency distributions
Frequency distribution is a table or graph that shows the number of observations in each category of
a categorical variable. It is a way to understand the distribution of the data and to identify patterns
and outliers. By counting the number of observations in each category, we can see which categories
are more or less common, and how the data is spread out.
Pandas provides a built-in function value_counts() that returns the frequency of each unique value in
a given column or series. Here's an example of how to use the value_counts() function in pandas:
29
import pandas as pd

data = {'name': ['Alice', 'Bob', 'Charlie', 'David', 'Charlie',
'Eve','Bob','Bob','Eve','David','David',
'David','Charlie','Charlie','Charlie','Alice',
'Bob','Bob','Eve','David','David','David',
'Charlie','Charlie','Charlie','Alice','Bob','Bob',
'Eve','David','David','David','Charlie','Charlie'],
'age': [25, 30, 35, 40, 45,40,40,40,40,40,40,40,40,40,40,
40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,
40,40,40,40],
'income': [50000, 60000, 70000, 80000, 90000,50000, 60000,
70000, 80000, 90000,50000, 60000, 70000, 80000,
90000,50000, 60000, 70000, 80000, 90000,50000,
60000, 70000, 80000, 90000,50000, 60000, 70000,
80000, 90000,50000, 60000, 70000, 80000, 90000]
}
# get the frequency of each unique value in the name column

df['name'].value_counts()
In this example, the value_counts() function is used to get the frequency of each unique value in the
name column of the dataframe df. The output will be a series that contains the unique values of the
column as the index and their frequency as the values.
The value_counts() function also accepts a parameter normalize which if set to True, the function
returns the relative frequency (proportion) of each unique value in the column or series.
df['name'].value_counts(normalize=True)
30
It's also possible to use the groupby() function of pandas to group the data by a specific column and
then apply the value_counts() function on the grouped data.
This will group the data on the 'age' column and for each group it will return the frequency of each
unique value in the 'name' column.
Correlation
Correlation analysis is a statistical method that quantifies the strength and direction of the
relationship between two numerical variables. Correlation can be used to understand the
relationships between different variables and to identify patterns and outliers. It is a measure of the
association between two variables, and it can range from -1 to 1. A value of -1 indicates a perfect
negative correlation, a value of 0 indicates no correlation, and a value of 1 indicates a perfect positive
correlation.
Pandas provides a built-in function corr() that calculates the correlation between the columns of a
DataFrame. Here's an example of how to use the corr() function in pandas:
In this example, the corr() function is used to get the correlation between the age and income
column of the dataframe df. The output will be a single value representing the correlation coefficient
between the two columns.
import pandas as pd

'age': [25, 30, 35, 40, 45],
'income': [50000, 60000, 70000, 80000, 90000]}
# get the correlation between the age and income column

df['age'].corr(df['income'])
31
import pandas as pd

'age': [25, 30, 35, 40, 45],
'income': [50000, 60000, 70000, 80000, 90000]}
# get the basic statistics of the dataframe

df.describe() import seaborn as sns
sns.heatmap(df.corr(),annot=True)
You can also use the corr() function on the entire dataframe to get the correlation matrix of all the
columns.
This will return a dataframe with the correlation coefficient between all the columns.
It's also possible to plot the correlation matrix using seaborn library, to visualize the correlation
between all the columns of the dataframe.
This will plot the correlation matrix in the form of a heatmap, where the darker the color, the
stronger the correlation between the columns.
It's important to note that correlation does not imply causality, it only measures the association
between two variables. It's also important to check for the outliers and the distribution of the
variables before performing correlation analysis.
T-test analysis
A t-test is a statistical test that is used to determine whether there is a significant difference between
the means of two groups. There are two main types of t-tests: the independent t-test and the
dependent t-test. The independent t-test is used to compare the means of two groups that are
independent of each other, while the dependent t-test is used to compare the means of two groups
that are related to each other.
The t-test can be used to test a null hypothesis that the means of two groups are equal against an
alternative hypothesis that they are not equal. It calculates the t-statistic, which is a measure of the
difference between the means of the two groups, and compares it to a t-distribution to determine
the probability of observing the difference by chance.
df.corr()
32
Pandas provides a built-in function ttest_ind() that performs an independent t-test. Here's an
example of how to use the ttest_ind() function in pandas:
In this example, the ttest_ind() function is used to perform an independent t-test on the group1 and
group2 columns of the dataframe df. The function returns a tuple containing the t-statistic and the p-
value. The p-value is the probability of observing the difference in means by chance.
A common threshold for the p-value is 0.05, which means that if the p-value is less than 0.05, we
reject the null hypothesis and conclude that there is a statistically significant difference in means
between the two groups.
It's important to note that t-test assumes that the data is normally distributed, it's also important to
check for the outliers and the distribution of the variables before performing t-test analysis.
Analysis of Variance
Analysis of Variance (ANOVA) is a statistical method that is used to determine whether there is a
significant difference between the means of two or more groups. It is an extension of the t-test and it
allows for the comparison of means for more than two groups.
There are three main types of ANOVA:
- One-way ANOVA: used to compare the means of two or more groups that are independent
of each other.
- Two-way ANOVA: used to compare the means of two or more groups that are independent
of each other and are related to two different factors.
- Repeated Measures ANOVA: used to compare the means of two or more groups that are
dependent on each other and are related to one factor.
ANOVA can be used to test a null hypothesis that the means of all the groups are equal against an
alternative hypothesis that they are not equal. It calculates an F-statistic, which is a ratio of the
variation between the groups to the variation within the groups, and compares it to an F-distribution
to determine the probability of observing the difference by chance.
import pandas as pd
import scipy.stats as stats

data = {'group1': [1, 2, 3, 4, 5],
'group2': [3, 4, 5, 6, 7]}
# perform independent t-test

stats.ttest_ind(df['group1'],
df['group2'])
33
Pandas provides a built-in function f_oneway() that performs a one-way ANOVA. Here's an example
of how to use the f_oneway() function in pandas:
In this example, the f_oneway() function is used to perform a one-way ANOVA on the group1,
group2, and group3 columns of the dataframe df. The function returns a tuple containing the F-value
and the p-value. The p-value is the probability of observing the difference in means by chance.
reject the null hypothesis and conclude that there is a statistically significant difference in means
between the groups.
It's important to note that ANOVA assumes that the data is normally distributed, it's also important
to check for the outliers and the distribution of the variables before performing ANOVA analysis.
In addition, scipy library provides a function AnovaRM for repeated measures ANOVA analysis, and
statsmodels library provides a ols function for two-way ANOVA analysis. It requires specifying the
different factors as well as the interaction between them.
Chi-square analysis
The chi-square test is a statistical test that is used to determine whether there is a significant
difference between observed and expected frequencies in a contingency table. It is used to test
hypotheses about the distribution of categorical variables.
The chi-square test is based on the chi-square statistic, which is calculated by comparing the
observed frequencies and the expected frequencies for each category and summing the differences.
The chi-square statistic follows a chi-square distribution, and the p-value is calculated by comparing
the calculated chi-square statistic to the chi-square distribution.
import pandas as pd
import scipy.stats as stats

data = {'group1': [1, 2, 3, 4, 5],
'group2': [3, 4, 5, 6, 7],
'group3': [5, 6, 7, 8, 9]}
# perform one-way ANOVA

stats.f_oneway(df['group1'],
df['group2'], df['group3'])
34
Pandas provides a built-in function crosstab() that can be used to create a contingency table and the
chi2_contingency() function in scipy.stats that can be used to perform chi-square test. Here's an
example of how to use the crosstab() and chi2_contingency() function in pandas:
In this example, the crosstab() function is used to create a contingency table of the dataframe df with
the gender and age columns. The chi2_contingency() function is used to perform a chi-square test on
the contingency table. It returns 4 values: the chi-square statistic, the p-value, the degrees of
freedom and the expected frequencies. The p-value is the probability of observing the difference in
frequencies by chance.
reject the null hypothesis and conclude that there is a statistically significant difference in
frequencies between the categories.
It's important to note that chi-square test assumes that the sample size is large enough and the
expected frequencies are greater than 5 for all cells. If this assumption is not met, other tests such as
Fisher's exact test should be used instead.
import pandas as pd
from scipy.stats import chi2_contingency

data = {'gender': ['male', 'female', 'male', 'male', 'female'],
'age': ['young', 'old', 'young', 'old', 'old']}
# create a contingency table

contingency_table = pd.crosstab(df['gender'], df['age'])
print(contingency_table)
# perform chi-square test

chi2, p, dof, expected = chi2_contingency(contingency_table)
print(p)
35
Feature engineering
The feature engineering phase in the data science lifecycle is the process of creating new features
from the raw data that can be used to improve the performance of machine learning models. The
goal of feature engineering is to extract the most relevant and informative features from the raw
data that can be used to make accurate predictions. During the feature engineering phase, data
scientists will typically perform a variety of tasks such as:
- Data transformation: transforming raw data into a format that can be used by machine
learning models, such as normalizing continuous variables and encoding categorical
variables.
- Feature extraction: extracting new features from the raw data that can be used to improve
the performance of machine learning models, such as calculating the average of a set of
values or creating a new feature that represents the interaction between two existing
features.
- Feature selection: selecting a subset of features from the raw data that are most informative
and relevant to the problem at hand, such as using statistical tests to identify the features
that are most strongly correlated with the target variable.
The feature engineering phase is crucial in the data science lifecycle as it can greatly impact the
performance of machine learning models. The features that are used as input for the model can
make a big difference in the accuracy of the predictions. Feature engineering can be a time-
consuming task that requires domain knowledge, creativity, and experimentation. There are different
techniques for feature engineering, such as:
- Polynomial feature engineering: map a set of features to a higher-dimensional space by

adding combinations of them
- Binning: group continuous variables into bins
- Encoding: map categorical features into numerical features by using ordinal encoding or one-
hot encoding.
- Counting the occurrences of certain values
- Grouping of similar values
- Deriving new features based on existing ones.
Python libraries like sklearn, numpy and pandas provide different functionalities that can be used for
feature engineering, such as sklearn.preprocessing for data transformation, sklearn.feature_selection
for feature selection and sklearn.feature_extraction for feature extraction. Here is a python code
example that demonstrates the different techniques for feature engineering:
36
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder
from sklearn.feature_extraction import PolynomialFeatures
from sklearn.feature_selection import SelectKBest,
mutual_info_classif
from sklearn.feature_extraction.text import CountVectorizer
# Data transformation
# Normalizing a continuous variable
data = {'age': [20, 25, 30, 35, 40]}
scaler = MinMaxScaler()
df['age_norm'] = scaler.fit_transform(df[['age']])
# Encoding a categorical variable

df['gender'] = ['male', 'female', 'male', 'female', 'male']
encoder = OneHotEncoder(sparse=False)
gender_encoded = encoder.fit_transform(df[['gender']])
# Feature extraction
# Creating a polynomial feature
poly = PolynomialFeatures(degree=2)
age_poly = poly.fit_transform(df[['age']])
# Counting occurrences of a certain value

df['text'] = ['I like apples', 'I like apples and bananas',
'I like apples and oranges',
'I like bananas', 'I like oranges']
count_vectorizer = CountVectorizer()
text_counts = count_vectorizer.fit_transform(df['text'])
# Feature selection
# Selecting the top 2 features based on mutual information
X = df[['age', 'age_norm']]
y = df['gender']
selector = SelectKBest(score_func=mutual_info_classif, k=2)
X_new = selector.fit_transform(X, y)
In this example, We first demonstrate data transformation by normalizing a continuous variable 'age'
and encoding a categorical variable 'gender'. Then we demonstrate feature extraction by creating a
polynomial feature of degree 2 from a continuous variable 'age' and counting occurrences of a
certain value in a text column 'text'. In addition, we demonstrate feature selection by selecting the
top 2 features based on mutual information between 'age' and 'age_norm' and the 'gender' using the
SelectKBest method.
It's important to note that the example is a simple example for demonstration purposes, in practice
it's important to test and compare different techniques and select the most suitable one for the
problem at hand.
37
38
Data Modelling
The data modeling phase in the data science lifecycle is the process of building and evaluating
machine learning models to make predictions or inferences from the data. The goal of data modeling
is to find the best model that can generalize well to new data and achieve a high level of accuracy.
During the data modeling phase, data scientists will typically perform a variety of tasks such as:
- Model selection: selecting the most appropriate machine learning algorithm for the problem
at hand based on the characteristics of the data and the problem.
- Model training: training the selected machine learning model on the training data using a set
of features and a target variable.
- Model evaluation: evaluating the performance of the trained model on the validation data
and comparing it to other models or to a baseline.
There are different types of machine learning models, based on the problem and the data
characteristics, such as:
- Supervised Learning: where the data is labeled and the model is trained to predict the labels
of new data. Examples of supervised learning models are linear regression, logistic
regression, decision trees, and support vector machines.
- Unsupervised Learning: where the data is not labeled and the model is trained to find
patterns or structure in the data. Examples of unsupervised learning models are k-means
clustering, hierarchical clustering, and principal component analysis.
- Reinforcement Learning: where the model learns from the feedback of the environment
through trial and error.
Python libraries like scikit-learn, TensorFlow and PyTorch provide a wide range of models and
functionalities that can be used for data modeling, such as sklearn.linear_model for linear regression,
sklearn.tree for decision trees, and sklearn.cluster for clustering.
In the data modeling phase, it's important to perform different model evaluation techniques, such as
cross-validation, to avoid overfitting and to get an idea of how well the model will generalize to new
data. It is also important to consider the interpretability of the model, as it's often important to
understand how the model is making its predictions and how different features are impacting the
model outcome.
General process
The general process of data modeling includes three main phases. In the first phase of data modeling
a model is chosen or built based on the type of problem to be solved, the characteristics of the data,
and the available resources. There are different types of models such as linear regression, logistic
regression, decision trees, and neural networks, each with their own advantages and disadvantages.
During this phase, the model's architecture, parameters, and hyperparameters are defined, and the
model is configured to receive the data.
Once the model is configured, it needs to be trained on a set of data. This phase is also known as
fitting the model. During training, the model's parameters are adjusted based on the data, so that the
model can learn the underlying patterns in the data. The training data is typically split into two sets: a
training set, which is used to train the model, and a validation set, which is used to evaluate the
model's performance during the training process.
39
After the model is trained, it needs to be evaluated to assess its performance. During this phase, the
model is tested on a separate set of data, called the test set, which is independent from the training
set. The test set is used to evaluate the model's ability to generalize to unseen data. The
performance of the model is measured using different metrics such as accuracy, precision, recall, F1-
score, and others, depending on the problem. Based on the results, the model can be further
improved by adjusting its parameters, hyperparameters, or architecture. It's also possible to try
different models and select the one that performs better.
Note that, this three phases could be iteratively carried out until a satisfactory accuracy is achieved.
It is also important to have a good understanding of the following terms:
Term Definition
Configuring a model Setting up the model and defining its
architecture, parameters, and hyperparameters
before training.
Parameters The values or variables of a model that are
learned from the data during the training
process. They are used to make predictions on
new data.
Hyperparameters The values or variables of a model that are not
learned from the data during the training
process, but are set before training. They
control the behavior or capacity of the model.
Model architecture The structure or design of the model, including
the number of layers, the number of neurons,
and the activation functions.
Training data The data used to train a model, typically a large
set of labeled examples.
Validation data The data used to evaluate the model's
performance during the training process,
typically a smaller set of labeled examples.
Test data The data used to evaluate the model's
performance after training, typically a separate
set of labeled examples.
Training of a model The process of adjusting the model's
parameters based on the training data, so that
the model can learn the underlying patterns in
the data.
Evaluating of a model The process of assessing the model's
performance on a separate set of data, called
the test data, which is independent from the
training data. The goal is to evaluate the
model's ability to generalize to unseen data.
Supervised learning
Supervised learning is a type of machine learning where the model is trained on labeled data to make
predictions or inferences about new data. In supervised learning, the goal is to learn a function that
maps inputs to outputs based on the labeled training data. The input data is also known as the
feature variables and the output data is known as the target variable. The function learned by the
40
model can be used to make predictions on new data by providing the model with new feature
variables and having it output a predicted target variable.
There are two main types of supervised learning: regression and classification.
- Regression is a type of supervised learning where the target variable is continuous. The goal
is to learn a function that can predict a continuous output for a given set of inputs. Examples
of regression problems include predicting the price of a stock, the temperature of a city, or
the length of a stay in a hospital.
- Classification is a type of supervised learning where the target variable is categorical. The
goal is to learn a function that can predict a categorical output for a given set of inputs.
Examples of classification problems include identifying spam emails, diagnosing a disease, or
identifying digits in an image.
Supervised learning can be applied to a wide range of problems and is a powerful tool for making
predictions and inferences from data. It's important to select the appropriate algorithm for the
problem and to evaluate the model performance by using different techniques such as cross-
validation and testing the model on unseen data.
Model selection
Supervised learning models can be implemented using various algorithms, such as:
- Linear Regression, which models the relationship between the data-points by fitting a best-fit
line through the data
- Logistic Regression, which is used to model the probability of a certain class or event existing
such as in a binary classification problem
- Decision Tree, which uses a tree-like model of decisions to classify and predict the outcome
of data
- Random Forest, which is an ensemble method of decision trees and aims to improve the
overall performance and reduce overfitting
- Neural Networks, which are a set of algorithms, modeled loosely after the human brain that
are designed to recognize patterns
- Support Vector Machines, which is a discriminative classifier formally defined by a separating
hyperplane.
- Naive Bayes: Naive Bayes is a probabilistic algorithm that makes classifications based on
Bayes' theorem with the assumption of independence among features. It's known for its
simplicity, fast training, and high accuracy. It's mainly used in text classification and spam
filtering.
- K-Nearest Neighbors (KNN): K-Nearest Neighbors is a non-parametric algorithm that makes
predictions based on the majority class of the k-nearest training examples to a given data
point. It's known for its simplicity, high interpretability, and good performance on small
datasets. It's mainly used for classification and regression problems.
The following table contains a comparison of the different algorithms using the following
characteristics:
- Problem Type: The type of problem that the algorithm is suitable for, either regression or
classification
- Model Type: The type of model that the algorithm builds, linear or non-linear
- Speed: The computational cost of the algorithm, fast or slow
- Scalability: The ability of the algorithm to handle large amounts of data, good or poor
41
- Interpratability: The ability to understand the model and how it makes predictions, high or
low
42
Algorithm Problem Type Model Type Speed Scalability Interpratability
Linear Regression Linear Fast Good High
Regression
Logistic Classification Linear Fast Good High
Regression
Decision Classification/Regression Non-linear Fast Good High
Tree
Random Classification/Regression Non-linear Fast Good Medium
Forest
Neural Classification/Regression Non-linear Slow Good Low
Networks
Support Classification Linear/Non- Slow Good Medium
Vector linear
Machines
Naive Classification Linear Fast Good High
Bayes
K-Nearest Classification/Regression Non-linear Slow Good High
Neighbors
(KNN)
Linear regression
Linear Regression is a supervised learning algorithm that models the relationship between a
dependent variable (target) and one or more independent variables (features) using a best-fit linear
equation. The goal of linear regression is to minimize the residual sum of squares (RSS) between the
predicted and actual values of the target variable.
The equation for a linear regression model is given by:
y=b0 +b1 x 1+ b2 x 2 +…+b n x n
where y is the target variable, x 1, x 2, ... x n are the independent variables, and b 0, b 1, b 2, ... b n are the
coefficients of the linear equation. The coefficients represent the slope and the y-intercept of the
line, and they are estimated using the training data. There are two main types of Linear Regression:
- Simple Linear Regression: It's used when there is a single independent variable. The
equation is represented by y=b0 +b1∗x1
- Multiple Linear Regression: It's used when there are multiple independent variables. The
equation is represented by y=b0 +b1 x 1+ b2 x 2 +…+b n x n
Linear regression can be solved using different techniques such as the ordinary least squares (OLS)
method, gradient descent, and others. The OLS method is the most common technique used to
estimate the coefficients of the linear equation. It minimizes the residual sum of squares (RSS)
between the predicted and actual values of the target variable.
Linear Regression is simple, easy to interpret, and computationally efficient. But it has several
assumptions that have to be met in order to produce accurate results such as linearity, independence
of errors, homoscedasticity and normality of errors. It's not suitable for non-linear problems.
There are several hyperparameters that can be used to configure a linear regression model. These
include:
43
- Regularization term: Linear regression models can be regularized to prevent overfitting by
adding a penalty term to the cost function. The most common regularization methods are L1
(Lasso) and L2 (Ridge) regularization, which add a penalty term to the cost function that is
proportional to the absolute or square value of the coefficients, respectively.
- Learning rate: Linear regression models are usually trained using a gradient descent
algorithm, which requires a learning rate hyperparameter that controls the step size of the
updates. A high learning rate can cause the model to converge quickly, but it may overshoot
the optimal solution, while a low learning rate may converge slowly but reach a better
solution.
- Iterations: Linear regression models are trained using a numerical optimization algorithm,
which requires a maximum number of iterations or epochs as hyperparameter. The number
of iterations will determine how many times the model will iterate over the data before
stopping.
- Solver: Linear regression models can be solved using different optimization techniques such
as Gradient Descent, Stochastic Gradient Descent, and others. Different solvers will have
different hyperparameters to configure.
- Normalization: Data normalization is important before training a linear regression model as
it helps in faster convergence and better prediction. There are different ways to normalize
data and it can be done using different hyperparameters as per the solver being used.
Logistic Regression
Logistic Regression is a supervised learning algorithm that is used for classification problems. It
models the probability of a certain class or event existing. Unlike Linear Regression, which is used to
predict a continuous outcome variable, logistic regression is used to predict a binary outcome
variable. The model uses a logistic function (also called sigmoid function) to predict a probability
value between 0 and 1, which can be mapped to binary classes. The equation for a logistic regression
model is given by:
1
P( y =1∨x)= −b0 −b1 x 1−b2 x 2−…−bn x n
1+e
Where P ( y=1|x ) is the probability of the target variable y being equal to 1 given the independent
variable x , x 1, x 2, ... x n are the independent variables, and b 0, b 1, b 2, ... b n are the coefficients of the
logistic equation. The coefficients represent the slope and the y-intercept of the logistic function, and
they are estimated using the training data.
The logistic function has an S-shaped curve, and it can be used to model the probability of a binary
outcome variable. The output of the logistic function is a probability value between 0 and 1, which
can be mapped to binary classes using a threshold value. For example, if the threshold value is 0.5,
then the predicted class is 1 if the probability is greater than or equal to 0.5, and it is 0 otherwise.
Logistic Regression is simple, easy to interpret, and computationally efficient, but it has several
assumptions that have to be met in order to produce accurate results such as independence of
errors, linearity in the logit and a large sample size. It's not suitable for non-linear problems or multi-
class classification.
Logistic regression is better suited for classification problems, while linear regression is better suited
for regression problems. In a classification problem, the goal is to predict a categorical variable, such
as a label or class, based on a set of features or independent variables. Logistic regression is a type of
generalized linear model that is commonly used for classification problems. The logistic function (also
44
known as the sigmoid function) is used to transform the linear combination of the independent
variables into a probability between 0 and 1. This probability is then used to make a binary or multi-
class prediction. Logistic regression can also be extended to handle more than two classes
(multinomial logistic regression).
On the other hand, linear regression is a type of statistical model that is used to predict a continuous
variable, such as a price, temperature, or weight, based on a set of independent variables. Linear
regression models the relationship between the independent variables and the dependent variable
as a linear equation. The goal is to find the best-fitting line or hyperplane that minimizes the
difference between the predicted and actual values. Linear regression can only be used for prediction
of continuous value.
There are several hyperparameters that can be used to configure a logistic regression model. These
include:
- Regularization term: Logistic regression models can be regularized to prevent overfitting by

adding a penalty term to the cost function. The most common regularization methods are L1
(Lasso) and L2 (Ridge) regularization, which add a penalty term to the cost function that is
proportional to the absolute or square value of the coefficients, respectively.
- Learning rate: Logistic regression models are usually trained using a gradient descent
algorithm, which requires a learning rate hyperparameter that controls the step size of the
updates. A high learning rate can cause the model to converge quickly, but it may overshoot
the optimal solution, while a low learning rate may converge slowly but reach a better
solution.
- Iterations: Logistic regression models are trained using a numerical optimization algorithm,
which requires a maximum number of iterations or epochs as hyperparameter. The number
of iterations will determine how many times the model will iterate over the data before
stopping.
- Solver: Logistic regression models can be solved using different optimization techniques such
as Gradient Descent, Stochastic Gradient Descent, and others. Different solvers will have
different hyperparameters to configure.
- Normalization: Data normalization is important before training a logistic regression model as
it helps in faster convergence and better prediction. There are different ways to normalize
data and it can be done using different hyperparameters as per the solver being used.
Decision Trees
Decision trees are a type of supervised learning algorithm that are commonly used for both
classification and regression problems. They are a powerful and interpretable method for building
predictive models by creating a flowchart-like structure that represents a series of decisions and their
possible consequences.
A decision tree is built using a recursive partitioning algorithm that splits the data into subsets based
on the values of the features or independent variables. At each node of the tree, a decision is made
based on the value of a certain feature. This decision leads to one of the branches or child nodes of
the node. The process continues recursively until a stopping criterion is met, such as reaching a
maximum depth or a minimum number of samples in a leaf node.
The final result of the decision tree is a set of if-then rules that can be used to make predictions on
new data. Each path from the root to a leaf node represents a rule, and the value of the target
variable (class label or output) associated with the leaf node is the prediction for that rule. The
45
decision tree algorithm can also provide an estimate of the probability of each class for the cases at
the leaf nodes.
Decision trees are known for their interpretability, as the rules are easy to understand and explain.
They are also versatile and can handle both categorical and numerical variables, missing values, and
outliers. However, decision trees can be prone to overfitting, especially when the tree is deep and
complex. This can be addressed using techniques such as pruning or by using ensemble methods like
random forests or gradient boosting.
Decision trees are a popular method for both classification and regression problems because of their
simplicity and interpretability. However, like any machine learning algorithm, they have their own set
of advantages and disadvantages. Advantages of decision trees:
- Easy to understand and interpret: Decision trees are flowchart-like structures that represent
a series of decisions and their possible consequences, making them easy to understand and
interpret.
- Handle both categorical and numerical variables: Decision trees can handle both categorical
and numerical variables, which makes them versatile and suitable for a wide range of
problems.
- Handle missing values: Decision trees can handle missing values, unlike some other machine
learning algorithms.
- Handle outliers: Decision trees can handle outliers, unlike some other machine learning
algorithms.
Disadvantages of decision trees:
- Prone to overfitting: Decision trees can be prone to overfitting, especially when the tree is
deep and complex. This can be addressed using techniques such as pruning or by using
ensemble methods like random forests or gradient boosting.
- Instability: Decision trees are sensitive to small variations in the data, which can cause them
to produce different trees for small changes in the training data.
- Bias: Decision trees can be biased towards features with many levels or a large number of
distinct values.
There are several hyperparameters that can be used to configure a decision tree model. These
include:
- Maximum depth: The maximum depth of the tree is one of the most important
hyperparameters. It determines the complexity of the tree and controls the number of splits
and the number of leaves. Increasing the maximum depth increases the complexity and the
chances of overfitting while decreasing it will make the tree simpler and reduce the chances
of overfitting.
- Minimum samples per leaf: This hyperparameter controls the minimum number of samples
that must be present in a leaf node. It is used to prevent creating overly complex trees by
ensuring that each leaf node has a minimum number of samples.
- Minimum samples for split: This hyperparameter controls the minimum number of samples
required to make a split. It helps prevent overfitting by ensuring that a split will only be made
if there are a sufficient number of samples to make a split.
- Maximum features: This hyperparameter controls the number of features that are
considered when making a split. By default, all features are considered but by setting this
parameter it can be reduced to a subset of the features.
46
- Criterion: Decision trees use different criteria to evaluate the quality of the split like Gini
impurity, information gain, and others. It's a hyperparameter that can be changed to use a
different criterion.
- Splitter: Decision trees have different splitters like best, random, etc. It can also be used as a
hyperparameter to change the way splits are made.
Random Forest
Random Forest is an ensemble method that combines multiple decision trees to improve the
predictive performance of the model. It is one of the most popular and widely used machine learning
algorithms for classification and regression problems.
The basic idea behind random forest is to train multiple decision trees on different subsets of the
training data and then combine their predictions to make a final decision. This is done by creating
multiple decision trees and training each tree on a different subset of the data, known as a bootstrap
sample. This allows each tree to learn from a different perspective and reduces the chances of
overfitting.
The final prediction is made by taking the majority vote of the predictions made by each tree. In the
case of regression problems, the average of the predictions made by each tree is taken as the final
prediction.
The main advantage of random forest is that it reduces overfitting by averaging predictions made by
multiple decision trees. Additionally, it provides a measure of feature importance and can handle
missing values and categorical variables.
In addition to the hyperparameters that are used in decision trees, random forest has several
additional hyperparameters that can be used to configure the model:
- Number of trees: This hyperparameter controls the number of decision trees that are used in
the ensemble. Increasing the number of trees generally improves the performance of the
model but can also increase the computational cost.
- Bootstrap: Bootstrapping is a method for sampling with replacement. It means that for each
tree, the samples are selected randomly with replacement. This is on by default, but it can be
turned off if desired.
- Random subspace: This hyperparameter controls the number of features that are considered
when making a split. Instead of considering all features, a random subset of features is
considered at each split.
- Out-of-bag samples: Random forest uses out-of-bag samples to estimate the generalization
performance of the model. Out-of-bag samples are samples that are not included in the
bootstrap sample used to train a particular tree.
Support Vector Machines

Support Vector Machines (SVMs) is a supervised learning algorithm that is mainly used for
classification problems. The main idea behind SVMs is to find a hyperplane that maximally separates
the different classes in the feature space. The hyperplane that maximizes the margin, which is the
distance between the hyperplane and the closest samples of the different classes, is chosen as the
decision boundary.
SVMs are particularly useful when the data is not linearly separable. In this case, a technique called
kernel trick is used to transform the data into a higher-dimensional space where it becomes linearly
separable. Common examples of kernel functions are linear, polynomial, and radial basis function
(RBF) kernels.
47
SVMs have several hyperparameters that can be used to configure the model, including the kernel
function, the regularization parameter (C), and the gamma parameter (for non-linear kernels). The
regularization parameter controls the trade-off between maximizing the margin and minimizing the
misclassification rate. The gamma parameter controls the width of the kernel function in non-linear
SVMs. SVMs also have a probabilistic interpretation, which allows for the estimation of the
probability of each class.
Naïve Bayes
Naive Bayes is a probabilistic algorithm that is based on Bayes' theorem, which states that the
probability of an event occurring is the product of the prior probability of the event and the
likelihood of the event given some observations. Naive Bayes algorithm is a simple and effective
method for classification problems, particularly when the number of features is large.
There are three main types of Naive Bayes algorithms: Gaussian Naive Bayes, Multinomial Naive
Bayes, and Bernoulli Naive Bayes. Gaussian Naive Bayes is used for continuous data, Multinomial
Naive Bayes is used for discrete data, and Bernoulli Naive Bayes is used for binary data.
The main assumption of Naive Bayes is that the features are independent, meaning that the presence
or absence of a feature does not depend on the presence or absence of any other feature. This
assumption is often not true in real-world data, but it is a good approximation for many problems.
The training process for Naive Bayes is straightforward and consists of computing the prior
probability of each class and the likelihood of each feature given each class. The prediction is made
by computing the posterior probability of each class given the features and choosing the class with
the highest probability.
Naive Bayes algorithm does not have any parameters that need to be tuned, the training process is
fast and the prediction process is also fast.
KNN
K-Nearest Neighbors (KNN) is a supervised learning algorithm that is used for both classification and
regression problems. The main idea behind KNN is that an instance is classified or predicted based on
the majority class or average value of its k-nearest neighbors. The number of nearest neighbors
considered is controlled by the hyperparameter k.
The training process for KNN is simple and consists of storing the feature vectors and labels of the
training instances. The prediction process is also simple and consists of computing the distance
between the test instance and all the training instances, finding the k-nearest neighbors, and
choosing the majority class or averaging the values of the k-nearest neighbors for regression.
The main hyperparameter of KNN is K, which controls the number of nearest neighbors that are
considered. A larger K will consider more neighbors and will have a smoother decision boundary,
while a smaller K will consider fewer neighbors and will have a more complex decision boundary.
KNN is a simple and effective algorithm, but it can be computationally expensive when the number of
training instances is large, because it requires computing the distance between the test instance and
all the training instances.
Model evaluation
Supervised regression techniques are typically evaluated using a combination of metrics that
measure the difference between the predicted values and the true values. Some of the most
commonly used metrics are:
48
- Mean Absolute Error (MAE) measures the average magnitude of the errors in a set of
predictions, without considering their direction. It is the sum of the absolute differences
between predictions and actual values, divided by the number of observations.
- Mean Squared Error (MSE) measures the average of the squared differences between
predictions and actual values. It is sensitive to outliers and penalizes large errors more than
MAE.
- Root Mean Squared Error (RMSE) is the square root of the mean squared error, which is
interpreted in the same units as the response variable.
- R-Squared (R2) is a metric that measures the proportion of variation in the response variable
that is explained by the predictor variables. It ranges between 0 and 1, with 1 indicating a
perfect fit.
Supervised classification techniques are typically evaluated using a combination of metrics that
measure the quality of the predictions in terms of true positives, true negatives, false positives, and
false negatives. Some of the most commonly used metrics are:
- Confusion matrix: A table that compares the predicted class labels with the true class labels,
and shows the number of correct and incorrect predictions for each class.
- Accuracy: The proportion of correct predictions out of all predictions. It is a simple and
intuitive metric, but it can be misleading in cases of imbalanced classes.
- Precision: The proportion of true positive predictions out of all positive predictions. It
measures how many of the instances that were predicted as positive are actually positive.
- Recall (sensitivity, hit rate, true positive rate): The proportion of true positive predictions
out of all actual positive instances. It measures how many of the actual positive instances
were predicted as positive.
- Specificity (true negative rate): The proportion of true negative predictions out of all actual
negative instances. It measures how many of the actual negative instances were predicted as
negative.
- F1-score: The harmonic mean of precision and recall. It balances precision and recall and is
considered a better measure than accuracy alone in cases of imbalanced classes.
- AUC-ROC (Area Under the Receiver Operating Characteristic curve): The ROC curve is a plot
of the true positive rate (recall) against the false positive rate. The AUC-ROC measures the
area under the ROC curve and ranges between 0 and 1, with 1 indicating a perfect
classification and 0.5 indicating a random classification.
- Log Loss (Cross-entropy loss): Log loss (or logistic loss) measures the performance of a
classifier where the predicted output is a probability value between 0 and 1.
Example of supervised learning

This example uses the LinearRegression class from the sklearn.linear_model module to create a
linear regression model. The model is trained on the X_train and y_train data using the fit() method,
and then used to make predictions on the X_test data using the predict() method. The mean squared
error and R-squared score are then calculated using the sklearn.metrics module to evaluate the
model's performance.
49
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
# Load the data

X_train, X_test, y_train, y_test
= train_test_split(X, y, test_size=0.2)
# Create a Linear Regression model

regressor = LinearRegression()
# Fit the model to the training data

regressor.fit(X_train, y_train)
# Make predictions on the test data

y_pred = regressor.predict(X_test)
# Calculate the evaluation metrics

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Mean Squared Error:", mse)

print("R-Squared:", r2)
This example uses the LogisticRegression class from the sklearn.linear_model module to create a
logistic regression model. The model is trained on the X_train and y_train data using the fit() method,
and then used to make predictions on the X_test data using the predict() method. The accuracy score
and confusion matrix are then calculated using the sklearn.metrics module to evaluate the model's
performance.
from sklearn.linear_model import LogisticRegression

from sklearn.metrics import accuracy_score, confusion_matrix
# Load the data

X_train, X_test, y_train, y_test
= train_test_split(X, y, test_size=0.2)
# Create a Logistic Regression model

classifier = LogisticRegression()
# Fit the model to the training data

classifier.fit(X_train, y_train)
# Make predictions on the test data

y_pred = classifier.predict(X_test)
# Calculate the evaluation metrics

accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
50
Ensembles
Ensemble techniques are a group of methods used to combine the predictions of multiple models in
order to improve the overall performance of a machine learning system. These techniques are based
on the idea that multiple models working together can often achieve better performance than a
single model working alone.There are several different ensemble techniques, including:
- Bagging: This method involves training multiple models independently on different subsets
of the training data, and then combining their predictions by averaging or voting.
- Boosting: This method involves training multiple models sequentially, where each model
tries to correct the mistakes of the previous model. The final predictions are made by
combining the predictions of all the models.
- Stacking: This method involves training multiple models independently on the same data,
and then using their predictions as input features for a higher-level model, which makes the
final predictions.
Ensemble techniques can be used for both regression and classification problems, and often provide
improved performance over single models. They can be a powerful way to improve the robustness
and accuracy of machine learning models, especially when the data is noisy or complex. In python,
scikit-learn library provides several ensemble methods such as RandomForestRegressor,
RandomForestClassifier, GradientBoostingRegressor and GradientBoostingClassifier.
Stacking is an ensemble technique that combines the predictions of multiple models in order to
produce a more accurate final prediction. In stacking, a new model, called the meta-model, is trained
on the predictions of the base models. The base models can be of any type, including other ensemble
models, and can be trained on different subsets of the data or using different parameters. The meta-
model is trained to learn the relationship between the base models' predictions and the true output.
The key idea behind stacking is that different models may perform well on different parts of the input
space, and by combining the predictions of the base models, the final prediction will be more
accurate. This is because the errors made by the base models are likely to be different and will
therefore cancel each other out.
from sklearn.ensemble import StackingRegressor

from sklearn.tree import DecisionTreeRegressor
# Using Linear Regression and Decision Tree as

base estimators
estimators = [('lr', LinearRegression()), ('dt',
DecisionTreeRegressor())]
# Creating Stacking Regressor

stack_reg =
StackingRegressor(estimators=estimators)
# Fit the model

stack_reg.fit(X_train, y_train)
51
Bagging (short for Bootstrap Aggregating) is an ensemble technique in which multiple models are
trained on different subsets of the training data and the final output is the combination of the
outputs from each of these models. The subsets of the training data are created by randomly
selecting samples from the original dataset with replacement, and each model is trained on a
different subset. Bagging can be used to improve the performance of a single model by reducing the
variance in the model's predictions. It is often used with decision trees and can be applied to both
regression and classification problems. One of the most popular ensemble techniques, Random
Forest, is based on bagging.
from sklearn.ensemble import BaggingRegressor

from sklearn.tree import DecisionTreeRegressor
# Using Decision Tree as base estimator

base_estimator = DecisionTreeRegressor()
# Creating Bagging Regressor

bag_reg = BaggingRegressor(base_estimator=base_estimator,
n_estimators=10)
# Fit the model

bag_reg.fit(X_train, y_train)
Boosting is an ensemble technique that attempts to improve the performance of a base model by
training a set of models to correct the errors made by the previous models. The models are trained
sequentially, with each model focusing on the mistakes made by the previous models. The final
predictions are typically made by combining the predictions of all the models, often using a weighted
majority vote. Common boosting algorithms include Adaboost, Gradient Boosting and XGBoost.
Boosting algorithms are often used for classification problems, but can also be used for regression
from sklearn.ensemble import AdaBoostClassifier

from sklearn.tree import DecisionTreeClassifier
# Using Decision Tree as base estimator

base_estimator = DecisionTreeClassifier()
# Creating AdaBoost Classifier

ada_clf =
AdaBoostClassifier(base_estimator=base_estimator,
n_estimators=10)
# Fit the model

ada_clf.fit(X_train, y_train)
problems.
52
Pipelines
Pipelines in data science are used to streamline the process of building, training, and evaluating
machine learning models. They allow you to encapsulate multiple steps of the modeling process into
a single, reusable object, which can make your code more organized and efficient.
One of the main advantages of using pipelines is that they can help prevent data leakage, which
occurs when information from the test set is used to fit the model. Pipelines ensure that the data is
properly transformed and scaled before it is used to train the model, and that the same
transformations are applied to the test set.
Pipelines also make it easier to try out different combinations of preprocessing steps and models,
without having to manually write code to combine them. This can save a lot of time and effort, and
make it easier to experiment with different approaches.
Another advantage of using pipelines is that they can help ensure that your code is consistent and
reproducible. Since all the steps of the pipeline are defined in one place, it is easy to see what was
done to the data and what models were used, which makes it easier to understand and replicate the
analysis.
Pipelines also make it simple to use cross-validation, which is a technique that helps to evaluate the
performance of a model by training it on multiple subsets of the data and evaluating it on the
remaining data. This can give you a better estimate of how well the model will perform on new data,
and can help you identify any overfitting or underfitting.
Overall, pipelines are an important tool in data science, as they help to automate and streamline the
modeling process, making it more efficient and less error-prone.
The following example is an example of a regression problem solved without using pipelines:
from sklearn.preprocessing import StandardScaler

from sklearn.metrics import mean_squared_error
# Preprocessing
X_train = StandardScaler().fit_transform(X_train)
X_test = StandardScaler().transform(X_test)
# Using fit_transform for X_test would result in data leakage
# Model building
model = LinearRegression()
model.fit(X_train, y_train)
# Evaluation
y_pred = model.predict(X_test)
print(f'Mean Squared Error: {mse}')
In this example, it would be easy to miss the fact that some test data would be used in a
preprocessing step. This can be solved by using pipelines as can be seen in the following code:
53
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
# Building the pipeline

pipe = make_pipeline(StandardScaler(), LinearRegression())
# Model building
pipe.fit(X_train, y_train)
# Evaluation
y_pred = pipe.predict(X_test)
print(f'Mean Squared Error: {mse}')
A pipeline in scikit-learn is not necessarily applied to all columns in a dataset. The

ColumnTransformer class can be used to specify which columns should be transformed by which
transformer in a pipeline. This allows for different preprocessing steps to be applied to different
subsets of the data, rather than applying the same steps to all columns. For example, one
transformer may be applied to all numerical columns while another transformer is applied to all
categorical columns. Additionally, the make_column_transformer function can be used to quickly
create a ColumnTransformer with specified transformers for different subsets of columns.
Hyperparameter tuning
Hyperparameter tuning is the process of selecting the best set of hyperparameters for a machine
learning model. Hyperparameters are parameters that are not learned from the data, but are set
before training the model. They include things like the learning rate, the number of trees in a random
forest, or the number of neighbors in a k-nearest-neighbors algorithm.
The process of hyperparameter tuning typically involves training a model with different combinations
of hyperparameters and evaluating their performance using a validation set. The goal is to find the
combination of hyperparameters that results in the best performance on the validation set.
There are several techniques that can be used for hyperparameter tuning, including grid search,
random search, and Bayesian optimization. Grid search is the simplest method, and involves
specifying a set of values for each hyperparameter and training a model for each combination of
values. Random search is similar to grid search, but instead of specifying a set of values for each
hyperparameter, a random value is chosen for each hyperparameter for each iteration. Bayesian
optimization is a more advanced method that uses Bayesian models to model the distribution of the
performance of the model with different hyperparameter settings.
Advantages of hyperparameter tuning are that it can lead to better model performance, it can be
used to prevent overfitting and underfitting and it can be used to select the best model among
several models that have been trained with different parameters.
In code using sklearn package, the GridSearchCV or RandomizedSearchCV classes can be used to
perform grid search or random search respectively.
54
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
param_grid = {'n_estimators': [50, 100, 200],

'max_depth': [None, 5, 10, 20]}
grid_search = GridSearchCV(RandomForestRegressor(), param_grid, cv=5)

grid_search.fit(X_train, y_train)
Complete example
In this example, we used the Iris dataset and the Support Vector Classifier (SVC) algorithm. The
pipeline starts by standardizing the data using the StandardScaler() function, then it one-hot encodes
the target variable using the OneHotEncoder() function, and finally, it fits the SVC model. In the
hyperparameter tuning step, we used the GridSearchCV() function to define a set of possible
hyperparameters and their values. The GridSearchCV() will train and evaluate the model using cross-
validation
55
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
# Load the Iris dataset

iris = load_iris()
X = iris.data
y = iris.target
# Split the data into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2, random_state=42)
# Create a pipeline for preprocessing and model fitting

pipe = Pipeline([
('scaler', StandardScaler()), # standardize the data
('encoder', OneHotEncoder()), # one-hot encode the target
variable
('classifier', SVC()) # fit the Support Vector Classifier
])
# Define the hyperparameters and their possible values

param_grid = {
'classifier__C': [0.1, 1, 10],
'classifier__kernel': ['linear', 'rbf'],
'classifier__degree': [1, 2, 3]
}
# Create the GridSearchCV object

grid = GridSearchCV(pipe, param_grid, cv=5)
# Fit the GridSearchCV object to the train data

grid.fit(X_train, y_train)
# Print the best hyperparameters

print("Best Hyperparameters: ", grid.best_params_)
# Predict the labels of the test set

y_pred = grid.predict(X_test)
# Print the accuracy score

print("Accuracy: ", grid.score(X_test, y_test))
56
Overfitting versus underfitting
Underfitting occurs when a model is not able to capture the underlying patterns in the data. This can
happen for a number of reasons, such as:
- The model is too simple and does not have enough capacity to fit the data.
- The model is over-regularized, meaning that it has been constrained too much and is unable
to fit the data.
- The data is too noisy and the model is not able to extract the relevant information.
Underfitting is characterized by a low training accuracy and a high bias. The model will perform
poorly on both the training and validation sets. This can be identified by comparing the training and
validation errors, where training error is low and validation error is high, this is an indication of
underfitting. To overcome underfitting, we can try several approaches such as:
- Collecting more data

- Using more complex models
- Using feature engineering
- Using fewer constraints or regularization
- Tuning hyperparameters
On the other hand, overfitting occurs when a model is trained too well on the training data, and as a
result, it performs poorly on unseen data or new data. This happens when the model is too complex
and has too many parameters, and it starts to fit the noise or random variations present in the
training data. In other words, the model has learned the training data too well, and it has not
generalized well to new data. The model is not able to generalize from the training data to the test
data, and this results in poor performance on unseen data.
Overfitting can be detected by comparing the performance of a model on the training data and test
data. If the performance on the training data is much better than on the test data, it is likely that the
model is overfitting. To combat overfitting, one can use techniques such as regularization, early
stopping, and ensemble methods.
Regularization is a technique used in machine learning to prevent overfitting by adding a penalty

term to the loss function. The penalty term is a function of the model's parameters, and it is designed
to shrink the parameters towards zero. This helps to reduce the complexity of the model and prevent
it from fitting the noise in the training data. There are several types of regularization techniques,
including L1, L2, and ElasticNet regularization. L1 regularization, also known as Lasso regularization,
adds a penalty term to the loss function that is proportional to the absolute value of the parameters.
L2 regularization, also known as Ridge regularization, adds a penalty term to the loss function that is
proportional to the square of the parameters. ElasticNet regularization is a combination of L1 and L2
regularization. The key hyperparameter to tune for regularization is the regularization strength, also
known as alpha. A smaller alpha value results in less regularization, and a larger alpha value results in
more regularization.
57
Unsupervised learning
Unsupervised learning is a type of machine learning in which the model is not provided with labeled
data. Instead, the model is given an unlabeled dataset and is expected to find patterns and structure
within the data on its own. The goal of unsupervised learning is to explore the underlying structure of
the data, identify patterns and relationships, and extract useful features for further analysis.
There are two main types of unsupervised learning algorithms: clustering and dimensionality
reduction. Clustering algorithms group similar data points together, while dimensionality reduction
algorithms reduce the number of features in a dataset by extracting the most important information.
Clustering algorithms include k-means, hierarchical clustering, and density-based clustering. These
algorithms are used to group similar data points together and identify patterns in the data.
Dimensionality reduction algorithms include Principal Component Analysis (PCA), Linear Discriminant
Analysis (LDA), and Multi-dimensional Scaling (MDS). These algorithms are used to reduce the
number of features in a dataset while preserving as much information as possible.
Unsupervised learning is useful in a variety of applications such as anomaly detection, market

segmentation, and image compression.
Unsupervised learning is used for anomaly detection by identifying patterns or structure in data that
do not conform to the expected behavior. This can be achieved by using techniques such as
clustering, which groups similar data points together, or dimensionality reduction, which simplifies
the data by removing less important features. Anomaly detection algorithms can then be applied to
the resulting data to identify data points that do not conform to the patterns or structure identified
by the unsupervised learning techniques. These data points can be considered as anomalies or
outliers.
For example, in a clustering algorithm, data points that are not assigned to any cluster or are far
away from the cluster center can be considered as anomalies. In dimensionality reduction, data
points that are far away from the main structure of the data can be considered as anomalies.
It's also possible to use unsupervised learning techniques to model the normal behavior of the data
and then use this model to identify data points that deviate significantly from the normal behavior.
Anomaly detection is useful in many applications such as intrusion detection, fraud detection, and
monitoring of manufacturing processes.
Unsupervised learning is used for market segmentation by grouping similar customers or products
together. This can be done using techniques such as clustering, which groups similar data points
based on certain features or characteristics. For example, a company may use clustering to segment
their customer base by demographics, purchase history, or other relevant information. The resulting
segments can then be used for targeted marketing or product development. Additionally,
unsupervised learning techniques such as dimensionality reduction can be used to identify patterns
and trends in the data that can be used for segmentation.
Unsupervised learning can be used for image compression by using techniques such as clustering or
dimensionality reduction. Clustering can be used to group similar pixels together, and then a
representative value can be chosen for each cluster to represent all the pixels in that cluster. This
reduces the number of unique values in the image, and thus compresses the image. Dimensionality
reduction can be used to identify the most important features in an image and then only keep those
features while discarding the less important ones. This also reduces the number of unique values in
58
the image and compresses it. These techniques can be used alone or in combination to compress an
image while still preserving its main features.
KMeans clustering
K-means is an unsupervised learning algorithm used for clustering. It groups similar data points
together into k clusters, where k is a user-specified number. The algorithm works by initializing k
centroids, which are points in the feature space representing the center of each cluster. The
algorithm then iteratively assigns each data point to the cluster with the nearest centroid and re-
computes the centroid of each cluster as the mean of all points in the cluster. This process is
repeated until the cluster assignments no longer change or a maximum number of iterations is
reached. The end result is k clusters with each cluster consisting of similar data points. The number of
clusters, k, is a hyperparameter that needs to be specified before running the algorithm. The choice
of k is often determined using techniques such as the elbow method or silhouette analysis. The
performance of the k-means algorithm is sensitive to the initialization of the centroids and the
algorithm can converge to a local optimum. To overcome this issue, the k-means algorithm can be
run multiple times with different initial centroids and the best solution can be selected. The k-means
algorithm is widely used in various domains such as image compression, market segmentation, and
customer segmentation.
K-means is a popular unsupervised machine learning technique for clustering. It is a centroid-based

algorithm, or a distance-based algorithm, where we calculate the distances to assign a point to a
cluster. The technique has the following advantages:
- Simple and easy to understand

- Fast and efficient in terms of computational cost, especially for large data sets
- Works well with a large number of variables
Of course, the technique also has some disadvantages, namely:
- Assumes that clusters are spherical in shape, which may not be the case in real-world data
sets
- Sensitive to initial conditions and may converge to a suboptimal solution
- Assumes prior knowledge of the number of clusters, which may be difficult to determine
- Not suitable for categorical data.
In the following example, we first generate some random data using the numpy library. Then, we
initialize the k-means model with 2 clusters, which is specified by the n_clusters parameter. Next, we
fit the model to the data using the fit method. Finally, we use the labels_ attribute to get the cluster
assignments for each data point and the cluster_centers_ attribute to get the coordinates of the
cluster centers.
59
from sklearn.cluster import KMeans
import numpy as np
# Generate random data

data = np.random.rand(100, 2)
# Initialize the k-means model with 2 clusters

kmeans = KMeans(n_clusters=2)
# Fit the model to the data

kmeans.fit(data)
# Get the cluster assignments for each data point

labels = kmeans.labels_
# Get the coordinates of the cluster centers

cluster_centers = kmeans.cluster_centers_
Mean shift clustering

Mean Shift is a density-based clustering algorithm that aims to discover "blobs" in a smooth density
of samples. It is a centroid-based algorithm, or a distance-based algorithm, where each dataset point
is represented by a center point called a centroid. The centroid is defined as the mean of all points
within the region of interest.
The main advantage of Mean Shift is that it does not require the user to specify the number of
clusters beforehand, as it automatically detects the number of clusters based on the density of the
data. Additionally, it can handle non-linearly shaped clusters and multi-modal distributions, which
can be a problem for other clustering algorithms like K-Means.
However, the main disadvantage of Mean Shift is that it can be sensitive to the choice of kernel and
bandwidth, which may require some experimentation to find the optimal values. Additionally, it can
be computationally expensive, especially for large datasets.
from sklearn.cluster import MeanShift
# Create an instance of the MeanShift class

ms = MeanShift()
# Fit the model to the data

ms.fit(X)
# Obtain the cluster labels

labels = ms.labels_
# Obtain the cluster centroids

cluster_centers = ms.cluster_centers_
60
Principal Component Analysis
Principal Component Analysis (PCA) is a technique used for dimensionality reduction and feature
extraction in unsupervised learning. It works by identifying the directions of maximum variance in the
data, and projecting the data onto a new, lower-dimensional space. The new axes, called principal
components, are the directions that maximize the variance of the data.
Advantages of PCA include that it can reveal patterns in the data that are not immediately apparent
and that it can be used to reduce the dimensionality of the data and make it more manageable.
Disadvantages of PCA include that it can be sensitive to the scale of the data, and that it can
sometimes produce poor results when the data has a non-linear structure.
In the following example we first generate a random data with numpy then we import PCA from
sklearn's decomposition library, and set number of components to 2. Then we fit and transform the
data using PCA. The transformed data is now stored in X_pca.
from sklearn.decomposition import PCA

import numpy as np
# Generating some random data

np.random.seed(0)
X = np.random.randn(5, 2)
# Initializing PCA
pca = PCA(n_components=2)
# Fitting and transforming the data

X_pca = pca.fit_transform(X)
print(X_pca)
Model evaluation
Unsupervised learning techniques are evaluated differently than supervised learning techniques
because there are no labeled output data for the model to predict. Instead, the goal is often to
identify patterns or structure in the input data. There are several ways to evaluate unsupervised
learning techniques:
- Visualization: One of the most common ways to evaluate unsupervised models is through
visualization. This can include creating scatter plots, heatmaps, or other visualizations to
explore the structure of the data and the clusters or components identified by the model.
- Internal evaluation metrics: There are several internal evaluation metrics that can be used to
evaluate unsupervised models, such as the silhouette score, which measures the similarity of
each point to its own cluster compared to other clusters.
- External evaluation metrics: These metrics compare the results of the unsupervised model
to some external measure or ground truth. For example, a clustering model can be evaluated
by comparing its clusters to known groups in the data.
- Domain knowledge: In some cases, the evaluation of unsupervised models is done by
experts in the field who have domain knowledge. They use their experience and knowledge
to interpret the results and assess the quality of the model.
61
It's important to note that unsupervised learning techniques are more subjective to evaluate than
supervised techniques, and the choice of evaluation metric depends on the problem and data at
hand.
Reinforcement learning
Reinforcement learning (RL) is a type of machine learning that is concerned with training agents to
make decisions in an environment. The agent is presented with a state, and it must take an action to
transition to the next state. The agent's goal is to learn a policy, which is a mapping from states to
actions, that maximizes a scalar reward signal.
In RL, an agent interacts with an environment over a sequence of time steps. At each time step, the
agent receives an observation of the environment's state, and it selects an action to perform. The
environment then transitions to a new state, and the agent receives a scalar reward signal. The
agent's goal is to learn a policy that maximizes the expected sum of rewards over the long term.
Reinforcement learning algorithms can be categorized into three main classes: value-based, policy-
based, and model-based. Value-based methods learn the value function, which estimates the
expected sum of future rewards for each state or state-action pair. Policy-based methods learn the
policy directly, without estimating the value function. Model-based methods learn a model of the
environment, which can be used to plan and make decisions.
There are several advantages of reinforcement learning, including the ability to learn from delayed or
sparse rewards, the ability to learn from trial-and-error, and the ability to learn in partially
observable environments. However, there are also several challenges and limitations, such as the
need for large amounts of data, the difficulty of balancing exploration and exploitation, and the
potential for poor performance in complex or high-dimensional environments.
Reinforcement learning has been successfully applied to a wide range of problems, such as game
playing, robotics, and control systems. It has also been used to optimize decision-making in areas
such as finance, healthcare, and energy management.
Techniques
The most common techniques for reinforcement learning include:
- Q-Learning: A model-free algorithm that uses a Q-table to estimate the optimal action-value
function.
- SARSA: A model-free algorithm that uses a Q-table to estimate the optimal action-value
function. It is similar to Q-learning but uses the expected value of the next action rather than
the max action.
- DDPG: A model-based algorithm that uses a neural network to approximate the optimal
action-value function. It is an off-policy algorithm.
- A3C: A model-free algorithm that uses a neural network to estimate the optimal action-value
function. It is an on-policy algorithm that uses multiple parallel agents to improve the
learning process.
- PPO: A model-free algorithm that uses a neural network to estimate the optimal action-value
function. It is an on-policy algorithm that uses a trust region optimization method to update
the policy.
- DQN: A model-free algorithm that uses a neural network to estimate the optimal action-
value function. It is an off-policy algorithm that uses experience replay and a target network
to stabilize the learning process.
62
In this section, we will only give a more detailed explanation of Q-learning since the others are too
advanced. Q-learning is a model-free reinforcement learning technique that uses a Q-table to store
the estimated value of taking a particular action in a given state. The Q-table is initially filled with
random values and is updated as the agent interacts with the environment. The Q-table is used to
determine the best action to take in a given state, based on the highest estimated value. The Q-
learning algorithm uses the Bellman equation to update the Q-table. The basic idea of Q-learning is
to learn a policy, which tells an agent what action to take under what circumstances. It is a type of
temporal difference learning, which is a combination of supervised and unsupervised learning. Q-
learning can be used for a variety of problems, including game playing, navigation, and control
systems.
The main advantage of Q-learning is that it can handle problems with large state spaces and
continuous state spaces, and it can learn directly from raw sensory inputs. Additionally, Q-learning
can be used in both deterministic and non-deterministic environments. However, Q-learning can be
very slow to converge in some problems, and it can require a lot of data to work well. Additionally, Q-
learning can be sensitive to the choice of initialization, and it can be prone to overfitting if the
function approximator is not chosen carefully.
Example
The following code is a simplified implementation of the Q-learning algorithm using an epsilon-
greedy policy. The epsilon-greedy policy is a strategy used in reinforcement learning to balance
exploration and exploitation. Exploration refers to the process of trying out new actions to gather
more information about the environment, while exploitation refers to using the current knowledge
of the environment to take actions that are likely to lead to the highest reward. The epsilon-greedy
policy sets a probability (epsilon) of taking a random action (exploration) and a probability of 1-
epsilon of taking the action that is currently estimated to be the best (exploitation). This allows the
agent to continue exploring the environment while also taking advantage of the knowledge it has
gained. The value of epsilon is typically decreased over time, as the agent becomes more confident in
its estimates of the best action.
63
import numpy as np
# Define the Q-table and some initial values

q_table = np.zeros((num_states, num_actions))
learning_rate = 0.8
discount_factor = 0.95
# Loop through the episodes

for episode in range(num_episodes):
# Initialize the state
current_state = initial_state
done = False
while not done:
# Choose an action using an epsilon-greedy policy
if np.random.uniform(0, 1) < epsilon:
action = np.random.choice(num_actions)
else:
action = np.argmax(q_table[current_state])
# Take the action and observe the next state and reward
next_state, reward, done, _ = env.step(action)
# Update the Q-value for the current state and action

change_action = np.max(q_table[next_state])
- q_table[current_state, action]
change_reward = reward + discount_factor * change_action
q_table[current_state, action] =
q_table[current_state, action]
+ learning_rate * change_reward
# Update the current state

current_state = next_state
64
Natural Language Processing
Natural Language Processing (NLP) is a field of Artificial Intelligence (AI) that focuses on the
interaction between computers and human languages. It involves the development of algorithms and
models that can understand, interpret, and generate human language. NLP is used in a wide range of
applications, including language translation, sentiment analysis, text summarization, question
answering, and chatbots.
There are several difficulties in working with NLP, including:
- Ambiguity: Human language is inherently ambiguous, and it can be difficult for machines to
understand the intended meaning of a statement.
- Structural complexity: Human language is complex and can have many different structures,
making it difficult for machines to understand.
- Vocabulary size: Human language has a very large vocabulary, and it can be difficult for
machines to learn and understand all the words and phrases.
- Context dependency: The meaning of a word or phrase can change depending on the
context in which it is used, making it difficult for machines to understand.
- Handling of idioms, colloquialism, sarcasm and other nuances in language.
- Lack of labeled data: NLP relies heavily on labeled data, which can be difficult and time-
consuming to acquire.
- Difficulty in evaluating the performance of NLP models as there isn't a clear metric for
comparison.
Some of the key techniques used in NLP include tokenization, stemming and lemmatization, part-of-
speech tagging, syntactic parsing, and semantic analysis. Tokenization is the process of breaking
down text into smaller units, such as words or phrases. Stemming and lemmatization are used to
reduce words to their base form, so that related words can be identified more easily. Part-of-speech
tagging is the process of identifying the grammatical role of words in a sentence. Syntactic parsing is
used to analyze the grammatical structure of a sentence. Semantic analysis is used to identify the
import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.tokenize import word_tokenize
# Create the stemmer and lemmatizer objects

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
# Define the text to be processed

text = "The quick brown fox jumps over the lazy dog."
# Tokenize the text

tokens = word_tokenize(text)
# Perform stemming and lemmatization on each token

stemmed_tokens = [stemmer.stem(token) for token in tokens]
lemmatized_tokens = [lemmatizer.lemmatize(token) for token in
tokens]
print("Original Tokens:", tokens)

print("Stemmed Tokens:", stemmed_tokens)
print("Lemmatized Tokens:", lemmatized_tokens)
65
meaning of words and phrases in context. Here is an example of Python code that performs
tokenization, stemming, and lemmatization using the NLTK library:
NLP also makes use of machine learning algorithms, such as supervised and unsupervised learning, to
train models to understand and generate human language. These models can be used for a variety of
tasks, such as sentiment analysis, language translation, and text summarization.
Overall, NLP is a complex field that requires a combination of techniques from computer science,
linguistics, and machine learning. It is an active area of research and development, with new
techniques and applications being developed all the time.
Bag of words
In the context of natural language processing (NLP), a bag of words is a representation of text data
that describes the occurrence of words within a document, without considering the order of the
words. It is called a bag of words because any information about the order or structure of words in
the document is discarded. The bag of words representation is typically used as a feature vector in
NLP tasks such as text classification, sentiment analysis, and language translation. The feature vector
can be a count of the words, a binary indicator of the presence or absence of a word, or the
frequency of the word within the document. The bag of words representation is simple to compute
and is widely used in NLP tasks.
There are several different forms of bag of words (BoW) that exist in the context of natural language
processing (NLP). Some of the most common forms include:
- Unigram BoW: This is the simplest form of BoW, where each word in a text is considered as a
separate feature.
- Bigram BoW: This form of BoW considers pairs of words (bigrams) as features instead of
individual words.
- N-gram BoW: This form of BoW considers groups of N words (N-grams) as features.
- Term Frequency-Inverse Document Frequency (TF-IDF) BoW: This form of BoW assigns a
weight to each word based on its frequency in a document and its rarity across a corpus of
documents.
- Word Embeddings: This form of BoW represent words or phrases as low-dimensional vectors
of real numbers, which are trained using a large corpus of text.
Each of these techniques transform a single continuous text into a certain feature vector. This
feature vector can be used by the different (un)supervised machine learning techniques for training a
model.
Unigram BOW
A unigram bag of words (BOW) is a representation of text data in which each word in the text is
considered as a separate feature. The text is first tokenized, i.e. divided into individual words, and
then a vocabulary is created from the set of unique words. Each word in the vocabulary corresponds
to a feature in the BOW representation. The value of the feature for a given text is the number of
occurrences of the corresponding word in the text. This representation is called unigram because it
only considers individual words, not combinations of words or phrases. It is a simple representation
that is easy to understand and implement, but it doesn't capture the meaning of words in context.
Here is an example of how to compute the unigram bag-of-words representation using the
CountVectorizer class from the sklearn.feature_extraction.text module in Python. Note that the
66
ngram_range parameter is set to (1, 1) to indicate that only unigrams (single words) should be
considered. The fit method is used to learn the vocabulary of the text data, and the transform
method is used to convert the text data into the unigram bag-of-words representation. The resulting
transformed data is stored in the variable unigram_bow.
N-gram BOW
In natural language processing, an n-gram bag of words (BOW) is a representation of text in which
words are grouped into "bags" based on the n-grams of the text. An n-gram is a contiguous sequence
of n items from a given sample of text or speech. For example, in the sentence "I love ice cream", a
unigram BOW would represent the text as a bag of individual words, where each word is a "token". A
bigram BOW would represent the text as a bag of word pairs, where each word pair is a "token" (e.g.
“I love” or “love ice”). A trigram BOW would represent the text as a bag of word triples, where each
word triple is a "token" (e.g. “I love ice” or “love ice cream”). The idea behind this representation is
that it captures the context of words in a text by taking into account the words that come before and
after each word. This can be useful for tasks such as text classification, language modeling, and
machine translation, where understanding the context of words is important.
Term Frequency-Inverse Document Frequency BoW

Term Frequency-Inverse Document Frequency (TF-IDF) is a method used to represent a text
document by a numerical vector. It is a statistical measure that reflects the importance of a word in a
document. The TF-IDF vector is composed of two elements: Term Frequency (TF) and Inverse
Document Frequency (IDF).
The Term Frequency (TF) is the number of times a word appears in a document divided by the total
number of words in the document. It represents the importance of a word in a single document. On
the other hand, the Inverse Document Frequency (IDF) is the logarithm of the ratio of the total
number of documents in a corpus to the number of documents containing the word. It represents
the rarity of a word in a corpus of documents. The final TF-IDF score for a word in a document is the
product of the TF and IDF scores for that word in the document. The resulting vector can be used as a
feature representation of the document.
TF-IDF has several advantages in working with text data:
- It can help to identify important words and phrases within a document.
67
- It can help to down-weight the importance of common words (such as "the" or "a") that
appear frequently across many documents.
- It can help to identify words that are unique or specific to a particular document, which can
be useful for document classification and information retrieval tasks.
- It can help to improve the performance of text-based machine learning models by providing
a more informative representation of the text data.
- It can also be used as a feature selection technique which helps to select important features.
For example, suppose we have a corpus of three documents:
- Document 1: "the dog jumped over the moon"

- Document 2: "the cat sat on the mat"
- Document 3: "the dog sat on the mat"
The unigram BoW for the first document will be:
{'the': 2, 'dog': 1, 'jumped': 1, 'over': 1, 'moon': 1}
The unigram BoW for the second document will be:
{'the': 2, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}
The unigram BoW for the third document will be:
{'the': 2, 'dog': 1, 'sat': 1, 'on': 1, 'mat': 1}
If we apply the TF-IDF to the unigram BoW, the resulting vector for each document will be different,
representing the importance of each word in that document with respect to the whole corpus.
The tfidf BoW for the first document will be:
3 3
{'the': 2∗log =0, 'dog': 1∗log =0 .176 , 'jumped': 0.477, 'over': 0.477, 'moon': 0.477}
3 2
The tfidf BoW for the second document will be:
{'the': 0, 'cat': 0.477, 'sat': 0.176, 'on': 0.176, 'mat': 0.176}
The tfidf BoW for the third document will be:
{'the': 0, 'dog': 0.176, 'sat': 0.176, 'on': 0.176, 'mat': 0.176}
From this example, we can deduce that words that do not occur frequently in the entire set of
documents, such as “moon”, receive a larger (relative) score than documents that occur very often,
such as “the”.
68
Here is an example of how to compute the TF-IDF BoW using the TfidfVectorizer class from the
sklearn.feature_extraction.text module:
from sklearn.feature_extraction.text import TfidfVectorizer
# Sample documents
documents = ["the dog jumped over the moon",
"the cat sat on the mat",
"the dog sat on the mat"]
# Create the TfidfVectorizer object

tfidf = TfidfVectorizer()
# Compute the TF-IDF BoW by fitting and transforming the documents

bow = tfidf.fit_transform(documents)
# View the resulting sparse matrix

print(bow.toarray())
This will output a sparse matrix representing the TF-IDF BoW of the input documents, where each
row is a document and each column is a unique word in the vocabulary. The values in the matrix
represent the TF-IDF weight of the word in the corresponding document.
Word embeddings
Word embeddings are a technique in natural language processing (NLP) for representing words in a
continuous vector space. The basic idea is to take a large corpus of text and use it to train a model
that can learn to map words to high-dimensional vectors, such that words that have similar meanings
or are used in similar contexts are close to each other in the vector space.
There are several methods for creating word embeddings, but the most popular ones include:
- Continuous Bag-of-Words (CBOW)

- Skip-Gram
- GloVe (Global Vectors for Word Representation)
The main advantage of using word embeddings is that they allow a machine learning model to
understand the meaning of words in a more robust way than traditional techniques such as one-hot
encoding or Bag-of-Words. Because the embeddings are learned from a large corpus of text, they are
able to capture the context in which words are used and the relationships between different words.
This means that a model trained on word embeddings is better able to understand natural language
text, leading to improved performance on tasks such as sentiment analysis, machine translation, and
text classification.
In addition to the advantages, there are also some limitations of word embeddings. For example,
they may not always perform well on rare or out-of-vocabulary words, and they are typically based
on a specific language and may not be easily transferable to other languages.
To compute word embeddings in python, one common library used is gensim. It provides a simple
API for training and using word embeddings, it also provides pre-trained embeddings for many
languages.
69
Word embeddings can be seen as a form of dimensionality reduction in the sense that they take a
high-dimensional one-hot encoded representation of a word and map it to a lower-dimensional
continuous vector. This allows for more efficient computation and better handling of sparsity in the
data. However, unlike traditional dimensionality reduction techniques such as PCA, the goal of word
embeddings is not to find a compact representation of the data, but rather to capture the meaning
and context of words in a way that can be used for downstream tasks.
70

Data Science Course

Uploaded by

Copyright:

Available Formats

Data Science Course

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Science Course

Uploaded by

Copyright:

Available Formats

What are the main programming languages and libraries covered in the course?

What are the main programming languages and libraries covered in the course?

What are the main steps in the data science lifecycle?

What are the main steps in the data science lifecycle?

Data Science

Python is an interpreted programming language, which means that it is executed by an interpreter

Here is an example of some of the main functions of NumPy:

Some of the main features of Pandas include:

- Handling of missing data

Here is an example of some of the main functions of pandas:

- Consistent interface to machine learning models: scikit-learn provides a consistent interface

Here is an example of some of the main functions of scikit-learn:

from sklearn.datasets import load_iris

# Loading the dataset

# Splitting the data into training and test sets

# Training a Logistic Regression model

# Making predictions on the test set

# Evaluating the model

Primary Data Secondary Data

Structured vs unstructured data

Examples of structured data include:

- Data from a customer relationship management (CRM) system

- Social media posts and comments

Human- versus computer-generated data

Here is a comparison of human-generated data and computer-generated data:

Human-generated data Computer-generated data

Quantitative versus qualitative data

Here is a comparison of quantitative data and qualitative data:

Quantitative data Qualitative data

Data collection methods

def parse(self, response):

next_page = response.css('li.next a::attr(href)').get()

# read data from a CSV file

# check for missing values

# check for invalid data types

# check for duplicate records

# check for outliers

# check for invalid values in specific

# read data from a CSV file

# standardize date format

# standardize phone number format

# standardize address format

# standardize categorical data

# standardize unit of measurement

# read data from a CSV file

# remove duplicate records based on specific columns

# remove duplicate records based on a unique identifier

#remove near-duplicate records using the Levenshtein distance

# read data from a CSV file

# fill missing values with a default value

# fill missing values with imputation techniques

Here's an example of how to create a histogram plot using Matplotlib:

import matplotlib.pyplot as plt

# generate some data

# show the plot

Here's an example of how to create a scatter plot using Matplotlib:

# generate some data

# create a scatter plot

# show the plot