Python Pandas - Sparse Data

Sparse data structures in Pandas are used to store data in a compressed format. They are particularly useful when you have large datasets with many repeated values (such as NaN). The compression is achieved by not storing these repeated values, making the storage more efficient.

Pandas provides specialized data structures for efficiently storing sparse data. Unlike typical sparse structures that mostly store zeros, Pandas' sparse objects allow you to compress data by omitting any values matching a specific fill value (like NaN). This compression leads to significant memory savings, especially with large datasets.

In this tutorial we will learn about the Sparse objects in pandas.

Sparse Arrays and Dtypes

Pandas offers the SparseArray class for handling sparse data at the array level. You can access the dtype information, which includes both the data type of the stored elements and the fill value.

Example

Let's see an example of creating a series with sparse data structures and verifying it's datatype.

import pandas as pd
import numpy as np

# Generate random data
arr = np.random.randn(10)
arr[2:-2] = np.nan  # Introduce NaN values

# Convert to a sparse Series
sparse_series = pd.Series(pd.arrays.SparseArray(arr))

print("Output sparse Series:\n",sparse_series)
print("DataType of the Series:",sparse_series.dtype)

Its output is as follows −

Output sparse Series:
 0    0.763830
1    0.821392
2         NaN
3         NaN
4         NaN
5         NaN
6         NaN
7         NaN
8    0.532463
9    1.169153
dtype: Sparse[float64, nan]

DataType of the Series: Sparse[float64, nan]

Notice the dtype Sparse[float64, nan]. The nan indicates that NaN values are not actually stored, only the non-NaN elements are.

Memory Efficiency with Sparse DataFrames

Sparse objects are ideal for enhancing memory efficiency when working with large datasets containing many NaN values.

Example

Let us now assume you had a DataFrame with mostly NaN values and execute the following code −

import pandas as pd
import numpy as np

# Create a DataFrame and introduce NaN values 
df = pd.DataFrame(np.random.randn(10000, 4))
df.iloc[:9998] = np.nan

# Convert to a sparse DataFrame
sparse_df = df.astype(pd.SparseDtype("float", np.nan))

# Display the first few rows and data types
print("Sparse DataFrame: \n",sparse_df.head())
print("\nDataType:\n",sparse_df.dtypes)

# Compare memory usage
print("\nMemory Comparison:")
print('Dense: {:.2f} KB'.format(df.memory_usage().sum() / 1e3))
print('Sparse: {:.2f} KB'.format(sparse_df.memory_usage().sum() / 1e3))

Its output is as follows −

Sparse DataFrame: 
     0   1   2   3
0 NaN NaN NaN NaN
1 NaN NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN NaN NaN NaN
4 NaN NaN NaN NaN

DataType:
 0    Sparse[float64, nan]
1    Sparse[float64, nan]
2    Sparse[float64, nan]
3    Sparse[float64, nan]
dtype: object

Memory Comparison:
Dense: 320.13 KB
Sparse: 0.22 KB

By converting the DataFrame to a sparse format, memory usage is significantly reduced.

Converting Sparse Arrays to Dense

Any sparse object can be converted back to the standard dense form by calling sparse.to_dense() −

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(10, 2), columns=['A', 'B'])
df.iloc[:5] = np.nan

# Convert to a sparse DataFrame
sparse_df = df.astype(pd.SparseDtype("float", np.nan))

# Display input the sparse object
print("sparse object:\n",sparse_df.dtypes)

result = sparse_df.sparse.to_dense()

# Output Dense
print("Output Dense:\n", result.dtypes)

Its output is as follows −

sparse object:
 A    Sparse[float64, nan]
B    Sparse[float64, nan]
dtype: object

Output Dense:
 A    float64
B    float64
dtype: object

Working with Sparse Accessor

Pandas offers a .sparse accessor to work with sparse data structures, similar to .str for string data or .dt for datetime data.

Sparse data should have the same dtype as its dense representation. Currently, float64, int64 and booldtypes are supported. Depending on the original dtype, fill_value default changes −

• float64 − np.nan

• int64 − 0

• bool − False

Example

Let us execute the following code to understand the working of the sparse accessor −

import pandas as pd
import numpy as np

# Create a sparse object
sparse_series = pd.Series([0, 0, 1, 2], dtype="Sparse[int]")

# Display input of the sparse object
print("sparse object:\n",sparse_series)

# Output of working with the Sparse Accessor
print("Percent of non-fill_value points:",sparse_series.sparse.density)
print("Fill value:", sparse_series.sparse.fill_value)
print("The number of non- fill_value points:", sparse_series.sparse.npoints)
print("Non fill value:", sparse_series.sparse.sp_values)

Its output is as follows −

sparse object:
0    0
1    0
2    1
3    2
dtype: Sparse[int64, 0]

Percent of non-fill_value points: 0.5
Fill value: 0
The number of non- fill_value points: 2
Non fill value: [1 2]