Using Altair on Data Aggregated from Large Datasets

Altair is a powerful and easy-to-use Python library for creating interactive visualizations. It's based on a grammar of graphics, which means we can build complex plots from simple building blocks. When dealing with large datasets, Altair can be particularly handy for aggregating and visualizing data efficiently. Here we discuss how to use Altair to handle and visualize data aggregated from large datasets in a easy way.

Understanding Altair's Rendering Approach

Altair charts work by sending the entire dataset to the browser, where it is processed and rendered in the frontend. This approach can lead to performance issues when dealing with large datasets, as the browser may struggle to handle the volume of data. This limitation is not inherent to Altair itself but rather a consequence of its client-side rendering strategy.

Challenges with Large Datasets

When working with large datasets, Altair may encounter several challenges:

1. Browser Crashes: Attempting to render large datasets directly in the browser can cause it to crash, making it difficult to work with the data.
2. Performance Issues: Even if the browser does not crash, rendering large datasets can lead to slow performance, making it difficult to interact with the visualization.
3. Data Limitations: Altair has a default limit of 5000 rows for embedded datasets. Exceeding this limit raises a MaxRowsError, forcing the user to consider alternative approaches.

Efficient Techniques for Handling Large Datasets

To overcome the challenges associated with large datasets, several techniques can be employed:

• Pre-Aggregation and Filtering in Pandas: Performing data transformations such as aggregations and filters using pandas before passing the data to Altair can significantly reduce the dataset size. This approach ensures that only the necessary data is sent to the browser, improving performance and reducing the risk of browser crashes.
• Using VegaFusion: VegaFusion is a data transformer that pre-evaluates data transformations in Python, allowing Altair to handle larger datasets efficiently. Enabling VegaFusion raises the limit on embedded datasets, making it suitable for larger datasets.
• Local Data Server: Using the altair_data_server package, data can be served from a local threaded server, reducing the load on the browser. This approach is particularly useful for large datasets and improves interactivity performance.
• Passing Data by URL: Instead of embedding the data directly, it can be stored separately and passed to the chart by URL. This approach not only addresses the issue of large notebooks but also leads to better interactivity performance with large datasets.
• Disabling MaxRows Check: If the user is certain they want to embed their full untransformed dataset within the visualization specification, they can disable the MaxRows check. However, this approach should be used with caution, as it can lead to browser crashes or performance issues.

Understanding Data Aggregation

Data aggregation is the process of collecting and summarizing data to provide meaningful insights. It involves combining data from multiple sources and presenting it in a summarized format. Aggregation is essential for handling large datasets, as it simplifies data analysis and visualization.

Why Aggregate?

• Performance: Aggregated data significantly reduces the number of points plotted, improving rendering speeds and responsiveness.
• Clarity: Aggregations help uncover patterns, trends, and relationships that might be obscured in raw data.
• Customization: Altair excels at visualizing aggregated metrics (means, sums, counts) and allows for tailored insights.

Aggregating Data with Altair

Setting Up Altair:

Before diving into visualizations, you need to install Altair and the Vega datasets package. Use the following commands to install them:

pip install altair
pip install vega_datasets

Altair provides several methods for aggregating data within visualizations. These include using the aggregate property within encodings or the transform_aggregate() method for more explicit control.

1. Using the Aggregate Property

The aggregate property can be used within the encoding to compute summary statistics over groups of data. For example, to create a bar chart showing the mean acceleration grouped by the number of cylinders:

import altair as alt
from vega_datasets import data

cars = data.cars()

chart = alt.Chart(cars).mark_bar().encode(
    y='Cylinders:O',
    x='mean(Acceleration):Q'
)
chart

Output:

2. Using Transform Aggregate

The transform_aggregate() method provides more explicit control over the aggregation process. Here's the same bar chart using transform_aggregate():

chart = alt.Chart(cars).mark_bar().encode(
    y='Cylinders:O',
    x='mean_acc:Q'
).transform_aggregate(
    mean_acc='mean(Acceleration)',
    groupby=["Cylinders"]
)
chart

Output:

Data Aggregated from Large Datasets: Step-by-Step Implementation

Dataset Link - Weather History

Step 1: Loading and Aggregating Large Datasets

• Load the dataset and perform aggregation using Pandas.
• Imports the Pandas library for data manipulation.
• Reads the CSV file into a Pandas DataFrame.
• Groups the data by the 'Summary' column.
• Calculates the mean of the 'Temperature (C)' column for each group.
• Resets the index to turn the result into a DataFrame.

# Load the dataset
df = pd.read_csv("C:\\Users\\Tonmoy\\Downloads\\Dataset\\weatherHistory.csv")

# Aggregate the data (e.g., calculate the mean temperature grouped by 'Summary')
aggregated_df = df.groupby('Summary')['Temperature (C)'].mean().reset_index()

Step 2:Creating Visualizations with Altair

• Create a simple bar chart to visualize the aggregated data.
• Initializes a chart with the aggregated data.
• Specifies a bar mark for the chart.
• Encodes the x-axis with 'Summary' and the y-axis with 'Temperature (C)'.
• Saves the chart as an HTML file named 'chart_step3.html'.

# Create a bar chart
chart = alt.Chart(aggregated_df).mark_bar().encode(
    x='Summary',
    y='Temperature (C)'
)


# Save the chart as an HTML file
chart.save('chart_step3.html')

Output:

Step 3: Combining Multiple Aggregations

• Calculate mean and median values and visualize them together
• Groups the data by the 'Summary' column.
• Calculates the mean of the 'Temperature (C)' column for each group.
• Resets the index to turn the result into a DataFrame.
• Groups the data by the 'Summary' column.
• Calculates the median of the 'Temperature (C)' column for each group.
• Resets the index to turn the result into a DataFrame.
• Merges the mean and median DataFrames on the 'Summary' column.
• Adds suffixes to distinguish between mean and median columns.
• Initializes a chart with the merged data.
• Uses transform_fold to combine mean and median columns for plotting.
• Specifies a bar mark for the chart.
• Encodes the x-axis with 'Summary', the y-axis with 'value', and uses different colors for 'aggregation'.
• Saves the combined chart as an HTML file named 'chart_step4.html'.

# Calculate both mean and median
mean_df = df.groupby('Summary')['Temperature (C)'].mean().reset_index()

median_df = df.groupby('Summary')['Temperature (C)'].median().reset_index()

# Merge the two dataframes
merged_df = mean_df.merge(median_df, on='Summary', suffixes=('_mean', '_median'))

# Create a combined chart
chart = alt.Chart(merged_df).transform_fold(
    ['Temperature (C)_mean', 'Temperature (C)_median'],
    as_=['aggregation', 'value']
).mark_bar().encode(
    x='Summary',
    y='value:Q',
    color='aggregation:N'
)

# Save the combined chart as an HTML file
chart.save('chart_step4.html')

Output:

Step 4: Handling Very Large Datasets

• Samples 10,000 rows from the dataset with a fixed random state for reproducibility.
• Groups the sampled data by the 'Summary' column.
• Calculates the mean of the 'Temperature (C)' column for each group.
• Resets the index to turn the result into a DataFrame.
• Initializes a chart with the sampled and aggregated data.
• Specifies a bar mark for the chart.
• Encodes the x-axis with 'Summary' and the y-axis with 'Temperature (C)'.
• Saves the chart with the sampled data as an HTML file named 'chart_step5.html'.

# Create a chart with the sampled and aggregated data
chart = alt.Chart(aggregated_sampled_df).mark_bar().encode(
    x='Summary',
    y='Temperature (C)'
)

# Save the chart with the sampled data as an HTML file
chart.save('chart_step5.html')

Output:

Optimizing Performance

• Pre-Aggregate: Perform aggregations in your data pipeline before visualizing with Altair.
• Limit Data Points: For line charts or scatterplots with dense data, sample or reduce the number of points displayed.
• Simplify Visualizations: Avoid excessive chart elements or complex interactions that might slow down rendering.
• Hardware Acceleration: Consider using GPUs if available for faster plotting of very large datasets.

Conclusion

Using Altair for visualizing large datasets makes data analysis easy and effective. By combining Altair with Pandas, we can easily manipulate and visualize data. Altair's simple syntax and interactive features make it a great choice for creating clear and informative visualizations, even with large datasets.

M

mrmishraoofc

Follow

Improve

Article Tags :

• Blogathon
• Data Visualization
• AI-ML-DS
• AI-ML-DS With Python
• Data Science Blogathon 2024

+1 More