Pandas Find Duplicate Rows
Most simple way to find duplicate rows in DataFrame is by using the duplicated() method. This method returns a boolean Series indicating whether each row is a duplicate of a previous row.
import pandas as pd
data = {'Name': ['John', 'Alice', 'Bob', 'Eve', 'John', 'Charlie'],
'Age': [25, 30, 22, 35, 25, 28],
'Gender': ['Male', 'Female', 'Male', 'Female', 'Male', 'Male'],
'Salary': [50000, 55000, 40000, 70000, 50000, 48000]}
df = pd.DataFrame(data)
# Find duplicate rows
duplicates = df[df.duplicated()]
print(duplicates)
Output
Name Age Gender Salary 4 John 25 Male 50000
In addition to the this method, there are several other approaches to find out the Duplicate rows in Pandas.
1. Identifying Duplicates Based on Specific Columns
Sometimes, you may only want to check for duplicates in specific columns, rather than the entire row. we can use duplicated() with the subset parameter to check for duplicates in specific columns.
The subset parameter allows you to specify which columns to consider when looking for duplicates. In this case, we're checking for duplicates based on the Name and Age columns.
# Find duplicates based on 'Name' and 'Age' columns
duplicates_by_columns = df[df.duplicated(subset=['Name', 'Age'])]
print(duplicates_by_columns)
Output
Name Age Gender Salary 4 John 25 Male 50000
2. Removing Duplicate Rows Using drop duplicates
Once duplicates are identified, you can remove them using the drop_duplicates() method. This method returns a new DataFrame with the duplicate rows removed.
The drop_duplicates() method removes all rows that are identical to a previous row. By default, it keeps the first occurrence of each row and drops subsequent duplicates.
# Remove duplicate rows
df_no_duplicates = df.drop_duplicates()
print(df_no_duplicates)
Output
Name Age Gender Salary 0 John 25 Male 50000 1 Alice 30 Female 55000 2 Bob 22 Male 40000 3 Eve 35 Female 70000 5 Charlie 28 Male 48000
Here are some key takeaways:
- Identify duplicates using duplicated()
- Find duplicates based on specific columns
- Remove duplicate rows with drop_duplicates()
