Driverless A I Booklet

Using H2O Driverless AI
Patrick Hall, Megan Kurka & Angela Bartz
http://docs.h2o.ai
October 2019: Version 1.8.0

Using H2O Driverless AI
by Patrick Hall, Megan Kurka & Angela Bartz
Published by H2O.ai, Inc.

2307 Leghorn St.
Mountain View, CA 94043
©2017-2019 H2O.ai, Inc. All rights reserved.
October 2019: Version 1.8.0
Photos by ©H2O.ai, Inc.
All copyrights belong to their respective owners.

While every precaution has been taken in the
preparation of this book, the publisher and
authors assume no responsibility for errors or
omissions, or for damages resulting from the
use of the information contained herein.
Printed in the United States of America.

Contents
1 Overview 6
1.1 Citation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Have Questions? . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Why Driverless AI? 7
3 Key Features 8
4 Supported Algorithms 10
5 Installing and Upgrading Driverless AI 11
6 Launching Driverless AI 12
6.1 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
7 The Datasets Page 13

7.1 Adding Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 14
7.2 Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2.1 Dataset Details Page . . . . . . . . . . . . . . . . . . 16
7.2.2 Dataset Rows Page . . . . . . . . . . . . . . . . . . . 17
7.2.3 Downloading Datasets . . . . . . . . . . . . . . . . . 18
7.3 Splitting Datasets . . . . . . . . . . . . . . . . . . . . . . . . 18
7.4 Visualizing Datasets . . . . . . . . . . . . . . . . . . . . . . . 19
7.4.1 The Visualization Page . . . . . . . . . . . . . . . . . 19
8 Running an Experiment 23
8.1 Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . 23
8.2 New Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 24
8.3 Completed Experiment . . . . . . . . . . . . . . . . . . . . . 27
8.3.1 Experiment Summary . . . . . . . . . . . . . . . . . . 28
8.4 Viewing Experiments . . . . . . . . . . . . . . . . . . . . . . 31
8.4.1 Checkpointing, Rerunning, and Retraining . . . . . . . 32
8.4.2 Deleting Experiments . . . . . . . . . . . . . . . . . . 34
9 Diagnosing a Model 34
10 Project Workspace 36
10.1 Linking Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 37
10.1.1 Selecting Datasets . . . . . . . . . . . . . . . . . . . . 37
10.2 Linking Experiments . . . . . . . . . . . . . . . . . . . . . . . 38
10.2.1 New Experiments . . . . . . . . . . . . . . . . . . . . 38
10.2.2 Checkpointing Experiments . . . . . . . . . . . . . . . 39
4 | CONTENTS
10.3 The Experiments Leaderboard . . . . . . . . . . . . . . . . . 39

10.3.1 Leaderboard Scoring . . . . . . . . . . . . . . . . . . . 40
10.3.2 Comparing Experiments . . . . . . . . . . . . . . . . . 41
10.4 Unlinking Data on a Projects Page . . . . . . . . . . . . . . . 43
10.5 Deleting Projects . . . . . . . . . . . . . . . . . . . . . . . . 43
11 Interpreting a Model 43
11.1 Interpret this Model button - Regular Experiments . . . . . . . 44
11.2 Interpret this Model button - Time-Series Experiments . . . . 45
11.2.1 Multi-Group Time Series MLI . . . . . . . . . . . . . . 45
11.2.2 Single Time Series MLI . . . . . . . . . . . . . . . . . 47
11.3 Model Interpretation - Driverless AI Models . . . . . . . . . . 49
11.4 Model Interpretation - External Models . . . . . . . . . . . . . 52
11.5 Understanding the Model Interpretation Page . . . . . . . . . 54
11.5.1 Summary Page . . . . . . . . . . . . . . . . . . . . . 56
11.5.2 DAI Model Dropdown . . . . . . . . . . . . . . . . . . 56
11.5.3 Surrogate Models Dropdown . . . . . . . . . . . . . . 61
11.5.4 Random Forest Dropdown . . . . . . . . . . . . . . . 66
11.5.5 Dashboard Page . . . . . . . . . . . . . . . . . . . . . 68
11.6 General Considerations . . . . . . . . . . . . . . . . . . . . . 69
11.6.1 Machine Learning and Approximate Explanations . . . 69
11.6.2 The Multiplicity of Good Models in Machine Learning . 70
11.6.3 Expectations for Consistency Between Explanatory Tech-
niques . . . . . . . . . . . . . . . . . . . . . . . . . . 70
12 Viewing Explanations 71
13 Score on Another Dataset 74
14 Transform Another Dataset 74
15 The Driverless AI Scoring Pipelines 76

15.1 Which Pipeline Should I Use? . . . . . . . . . . . . . . . . . . 76
15.2 Driverless AI Standalone Python Scoring Pipeline . . . . . . . 77
15.2.1 Python Scoring Pipeline Files . . . . . . . . . . . . . . 77
15.2.2 Quick Start - Recommended Method . . . . . . . . . . 78
15.2.3 Quick Start - Alternative Method . . . . . . . . . . . . 78
15.2.4 The Python Scoring Module . . . . . . . . . . . . . . 81
15.2.5 The Scoring Service . . . . . . . . . . . . . . . . . . . 82
15.2.6 Python Scoring Pipeline FAQ . . . . . . . . . . . . . . 85
15.2.7 Troubleshooting Python Environment Issues . . . . . . 85
15.3 Driverless AI MLI Standalone Scoring Package . . . . . . . . . 86
15.3.1 MLI Python Scoring Package Files . . . . . . . . . . . 87
15.3.2 Quick Start - Recommended Method . . . . . . . . . . 87
CONTENTS | 5
15.3.3 Quick Start - Alternative Method . . . . . . . . . . . . 88

15.3.4 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . 88
15.3.5 MLI Python Scoring Module . . . . . . . . . . . . . . 90
15.3.6 K-LIME vs Shapley Reason Codes . . . . . . . . . . . 90
15.3.7 MLI Scoring Service Overview . . . . . . . . . . . . . 91
15.4 Driverless AI MOJO Scoring Pipeline . . . . . . . . . . . . . . 93
15.4.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . 94
15.4.2 Enabling the MOJO Scoring Pipeline . . . . . . . . . . 95
15.4.3 MOJO Scoring Pipeline Files . . . . . . . . . . . . . . 95
15.4.4 Quickstart . . . . . . . . . . . . . . . . . . . . . . . . 95
15.5 MOJO Scoring Pipelines . . . . . . . . . . . . . . . . . . . . 96
15.5.1 MOJO Scoring Pipeline - Java Solution . . . . . . . . 96
15.5.2 MOJO Scoring Pipeline - C++ Solution . . . . . . . . 97
16 Deployment 101
16.1 Additional Resources . . . . . . . . . . . . . . . . . . . . . . 101
16.2 Deployments Overview Page . . . . . . . . . . . . . . . . . . 101
16.3 AWS Lambda Deployment . . . . . . . . . . . . . . . . . . . 102
16.3.1 Driverless AI Prerequisites . . . . . . . . . . . . . . . 102
16.3.2 AWS Access Permissions Prerequisites . . . . . . . . . 102
16.3.3 Deploying the Lambda . . . . . . . . . . . . . . . . . 103
16.3.4 Testing the Lambda Deployment . . . . . . . . . . . . 104
16.3.5 AWS Deployment Issues . . . . . . . . . . . . . . . . . 106
16.4 REST Server Deployment . . . . . . . . . . . . . . . . . . . . 106
16.4.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . 106
16.4.2 Deploying on REST Server . . . . . . . . . . . . . . . 107
16.4.3 Testing the REST Server Deployment . . . . . . . . . 108
16.4.4 REST Server Deployment Issues . . . . . . . . . . . . 110
17 About Driverless AI Transformations 110

17.1 Numeric Transformers . . . . . . . . . . . . . . . . . . . . . . 110
17.2 Time Series Experiments Transformers . . . . . . . . . . . . . 111
17.3 Categorical Transformers (String) . . . . . . . . . . . . . . . . 112
17.4 Text Transformers (String) . . . . . . . . . . . . . . . . . . . 113
17.5 Time Transformers (Date, Time) . . . . . . . . . . . . . . . . 114
18 Logs 114
18.1 Sending Logs to H2O . . . . . . . . . . . . . . . . . . . . . . 117
19 References 118
20 Authors 119
6 | Overview
1 Overview
H2O Driverless AI is an artificial intelligence (AI) platform for automatic
machine learning. Driverless AI automates some of the most difficult data
science and machine learning workflows such as feature engineering, model
validation, model tuning, model selection and model deployment. It aims to
achieve highest predictive accuracy, comparable to expert data scientists, but in
much shorter time thanks to end-to-end automation. Driverless AI also offers
automatic visualizations and machine learning interpretability (MLI). Especially
in regulated industries, model transparency and explanation are just as important
as predictive performance. Modeling pipelines (feature engineering and models)
are exported (in full fidelity, without approximations) both as Python modules
and as pure Java standalone scoring artifacts.
Driverless AI runs on commodity hardware. It was also specifically designed

to take advantage of graphical processing units (GPUs), including multi-GPU
workstations and servers such as IBM’s Power9-GPU AC922 server and the
NVIDIA DGX-1 for order-of-magnitude faster training.
This document describes how to use H2O Driverless AI UI and is updated

periodically. To view the latest Driverless AI User Guide, please go to http:
//docs.h2o.ai. To view an example for running the Driverless AI Python
Client, please refer to Appendix A: The Python Client in the Driverless AI User
Guide.
For more information about Driverless AI, please see https://www.h2o.

ai/driverless-ai/. For a third-party review, please see https://www.
infoworld.com/article/3236048/machine-learning/review-
h2oai-automates-machine-learning.html.
1.1 Citation
To cite this booklet, use the following: Hall, P., Kurka, M., and Bartz, A. (Sept
2018). Using H2O Driverless AI. http://docs.h2o.ai
1.2 Have Questions?

If you have questions about using Driverless AI, we recommend reviewing
the FAQ in the Driverless AI User Guide. If after reviewing the FAQ you
have additional questions, then you can post them on Stack Overflow us-
ing the driverless-ai tag at http://stackoverflow.com/questions/
tagged/driverless-ai.
Why Driverless AI? | 7
2 Why Driverless AI?

Over the last several years, machine learning has become an integral part of
many organizations’ decision-making processes at various levels. With not
enough data scientists to fill the increasing demand for data-driven business
processes, H2O.ai offers Driverless AI, which automates several time consuming
aspects of a typical data science workflow, including data visualization, feature
engineering, predictive modeling, and model explanation.
H2O Driverless AI is a high-performance, GPU-enabled computing platform

for automatic development and rapid deployment of state-of-the-art predictive
analytics models. It reads tabular data from plain text sources, Hadoop, or
S3 buckets and automates data visualization and building predictive models.
Driverless AI targets business applications such as loss-given-default, probability
of default, customer churn, campaign response, fraud detection, anti-money-
laundering, demand forecasting, and predictive asset maintenance models. (Or
in machine learning parlance: common regression, binomial classification, and
multinomial classification problems.)
How do you frame business problems in a data set for Driverless AI?
The data that is read into Driverless AI must contain one entity per row, like
a customer, patient, piece of equipment, or financial transaction. That row
must also contain information about what you will be trying to predict using
similar data in the future, like whether that customer in the row of data used a
promotion, whether that patient was readmitted to the hospital within thirty
days of being released, whether that piece of equipment required maintenance,
or whether that financial transaction was fraudulent. (In data science speak,
Driverless AI requires ”labeled” data.) Driverless AI runs through your data
many, many times looking for interactions, insights, and business drivers of the
phenomenon described by the provided dataset. Driverless AI can handle simple
data quality problems, but it currently requires all data for a single predictive
model to be in the same dataset, and that dataset must have already undergone
standard ETL, cleaning, and normalization routines before being loaded into
Driverless AI.
How do you use Driverless AI results to create commercial value?
Commercial value is generated by Driverless AI in a few ways.
• Driverless AI empowers data scientists or data analysts to work on projects

faster and more efficiently by using automation and state-of-the-art
computing power to accomplish tasks in just minutes or hours instead of
the weeks or months that it can take humans.
8 | Key Features
• Like in many other industries, automation leads to standardization of

business processes, enforces best practices, and eventually drives down
the cost of delivering the final product - in this case a predictive model.
• Driverless AI makes deploying predictive models easy - typically a difficult

step in the data science process. In large organizations, value from
predictive modeling is typically realized when a predictive model is moved
from a data analyst’s or data scientist’s development environment into a
production deployment setting. In this setting, the model is running on
live data and making quick and automatic decisions that make or save
money. Driverless AI provides both Java- and Python-based technologies
to make production deployment simpler.
Moreover, the system was designed with interpretability and transparency in

mind. Every prediction made by a Driverless AI model can be explained to
business users, so the system is viable even for regulated industries.
Visit https://www.h2o.ai/products/h2o-driverless-ai/ to download your free

21-day evaluation copy.
3 Key Features
Below are some of the key features available in Driverless AI.
Flexibility of Data and Deployment: Driverless AI works across a variety of

data sources including Hadoop HDFS, Amazon S3, and more. Driverless AI can
be deployed everywhere including all clouds (Microsoft Azure, AWS, Google
Cloud) and on premises on any system, but it is ideally suited for systems with
GPUs, including IBM Power 9 with GPUs built in.
NVIDIA GPU Acceleration: Driverless AI is optimized to take advantage of

GPU acceleration to achieve up to 40X speedups for automatic machine learning.
It includes multi-GPU algorithms for XGBoost, GLM, K-Means, and more. GPUs
allow for thousands of iterations of model features and optimizations.
Automatic Data Visualization (Autovis): For datasets, Driverless AI auto-

matically selects data plots based on the most relevant data statistics, generates
visualizations, and creates data plots that are most relevant from a statistical
perspective based on the most relevant data statistics. These visualizations
help users get a quick understanding of their data prior to starting the model
building process. They are also useful for understanding the composition of
very large datasets and for seeing trends or even possible issues, such as large
numbers of missing values or significant outliers that could impact modeling
results.
Key Features | 9
Automatic Feature Engineering: Feature engineering is the secret weapon

that advanced data scientists use to extract the most accurate results from
algorithms. H2O Driverless AI employs a library of algorithms and feature
transformations to automatically engineer new, high value features for a given
dataset. Included in the interface is an easy-to-read variable importance chart
that shows the significance of original and newly engineered features.
Automatic Model Documentation: To explain models to business users
and regulators, data scientists and data engineers must document the data,
algorithms, and processes used to create machine learning models. Driverless
AI provides an Autoreport (Autodoc) for each experiment, relieving the user
from the time-consuming task of documenting and summarizing their workflow
used when building machine learning models. The Autoreport includes details
about the data used, the validation schema selected, model and feature tuning,
and the final model created. With this capability in Driverless AI, practitioners
can focus more on drawing actionable insights from the models and save weeks
or even months in development, validation, and deployment process. Driverless
AI also provides a number of autodoc configuration options, giving users
even more control over output of the Autoreport.
Time Series Forecasting: Time series forecasting is one of the biggest chal-
lenges for data scientists. These models address key use cases, including demand
forecasting, infrastructure monitoring, and predictive maintenance. Driverless
AI delivers superior time series capabilities to optimize for almost any prediction
time window. Driverless AI incorporates data from numerous predictors, handles
structured character data and high-cardinality categorical variables, and handles
gaps in time series data and other missing values.
NLP with TensorFlow: Text data can contain critical information to inform
better predictions. Driverless AI automatically converts short text strings into
features using powerful techniques like TFIDF. With TensorFlow, Driverless AI
can also process larger text blocks and build models using all available data to
solve business problems like sentiment analysis, document classification, and
content tagging.
Automatic Scoring Pipelines: For completed experiments, Driverless AI au-
tomatically generates both Python scoring pipelines and new ultra-low latency
automatic scoring pipelines. The new automatic scoring pipeline is a unique
technology that deploys all feature engineering and the winning machine learning
model in a highly optimized, low-latency, production-ready Java code that can
be deployed anywhere.
Machine Learning Interpretability (MLI): Driverless AI provides robust in-
terpretability of machine learning models to explain modeling results in a
human-readable format. In the MLI view, Driverless AI employs a host of differ-
ent techniques and methodologies for interpreting and explaining the results of
10 | Supported Algorithms
its models. A number of charts are generated automatically, including K-LIME,

Shapley, Variable Importance, Decision Tree Surrogate, Partial Dependence,
Individual Conditional Expectation (ICE) and more. Additionally, you can
download a CSV of LIME and Shapley reasons codes from this view.
Automatic Reason Codes: In regulated industries, an explanation is often
required for significant decisions relating to customers (for example, credit
denial). Reason codes show the key positive and negative factors in a model’s
scoring decision in a simple language. Reasons codes are also useful in other
industries, such as healthcare, because they can provide insights into model
decisions that can drive additional testing or investigation.
Custom Recipe Support: Driverless AI allows you to import custom recipes for
MLI algorithms, feature engineering (transformers), scorers, and configuration.
You can use your custom recipes in combination with or instead of all built-
in recipes. This allows you to have greater influence over the Driverless AI
Automatic ML pipeline and gives you control over the optimization choices that
Driverless AI makes.
4 Supported Algorithms
XGBoost
XGBoost is a supervised learning algorithm that implements a process called
boosting to yield accurate models. Boosting refers to the ensemble learning
technique of building many models sequentially, with each new model attempting
to correct for the deficiencies in the previous model. In tree boosting, each
new model that is added to the ensemble is a decision tree. XGBoost provides
parallel tree boosting (also known as GBDT, GBM) that solves many data
science problems in a fast and accurate way. For many problems, XGBoost is
one of the best gradient boosting machine (GBM) frameworks today.
LightGBM
LightGBM is a gradient boosting framework developed by Microsoft that uses
tree based learning algorithms. It was specifically designed for lower memory
usage and faster training speed and higher efficiency. Similar to XGBoost, it is
one of the best gradient boosting implementations available. It is also used for
fitting Random Forest models inside of Driverless AI.
GLM
Generalized Linear Models (GLM) estimate regression models for outcomes
following exponential distributions. GLMs are an extension of traditional linear
models. They have gained popularity in statistical data analysis due to:
Installing and Upgrading Driverless AI | 11
• the flexibility of the model structure unifying the typical regression methods
(such as linear regression and logistic regression for binary classification)
• the recent availability of model-fitting software
• the ability to scale well with large datasets
TensorFlow
TensorFlow is an open source software library for performing high performance
numerical computation. Driverless AI includes a TensorFlow NLP recipe based
on CNN Deeplearning models.
RuleFit
The RuleFit ([3]) algorithm creates an optimal set of decision rules by first
fitting a tree model, and then fitting a Lasso (L1-regularized) GLM model to
create a linear model consisting of the most important tree leaves (rules).
FTRL
Follow the Regularized Leader (FTRL) is a DataTable implementation ([13]) of
the FTRL-Proximal online learning algorithm proposed in ”Ad click prediction:
a view from the trenches” ([10]). This implementation uses a hashing trick
and Hogwild approach ([11]) for parallelization. FTRL can do binomial and
multinomial classification, binomial and multinomial regressions, as well as
regression for continuous targets.
5 Installing and Upgrading Driverless AI

Installation and upgrade steps are provided in the Driverless AI User Guide. The
installation steps vary based on your platform and require a license key. Contact
[email protected] for information on how to purchase a Driverless AI license. Or visit
https://www.h2o.ai/download/ to obtain a free 21-day trial license.
For the best (and intended-as-designed) experience, install Driverless AI on
modern data center hardware with GPUs and CUDA support. Use Pascal or
Volta GPUs with maximum GPU memory for best results. (Note the older K80
and M60 GPUs available in EC2 are supported and very convenient, but not as
fast.)
To simplify cloud installation, Driverless AI is provided as an AMI. To simplify
local installation, Driverless AI is provided as a Docker image. For the best per-
formance, including GPU support, use nvidia-docker. For a lower-performance
experience without GPUs, use regular docker (with the same docker image).
RPM and Tar.sh options are available for native Driverless AI installations on
RedHat 7, CentOS 7, and SLES 12 operating systems. And finally, a DEB
12 | Launching Driverless AI
installation is available for Ubuntu 16.04 environments and on Windows 10

using Windows Subsystem for Linux (WSL).
For native installs (rpm, deb, tar.sh), Driverless AI requires a minimum of 5
GB of system memory in order to start experiments and a minimum of 5 GB of
disk space in order to run a small experiment. Note that these limits can be
changed in the config.toml file. We recommend that you have lots of system
CPU memory (64 GB or more) and 1 TB of free disk space available.
For Docker installs, we recommend 1TB of free disk space. Driverless AI uses
approximately 38 GB. In addition, the unpacking/temp files require space on
the same Linux mount /var during installation. Once DAI runs, the mounts
from the Docker container can point to other file system mount points.
If you are running Driverless AI with GPUs, be sure that your GPU has compute
capability of at least 3.5 and at least 4GB of RAM. If these requirements are
not met, then Driverless AI will switch to CPU-only mode.
Driverless AI supports unvalidated, none, local, LDAP, PAM, and OpenID
authentication. Authentication can be configured by setting environment
variables or via a config.toml file. Refer to the Setting Environment Variables
section in the User Guide.
Driverless AI also supports HDFS, S3, Azure Blob Store, BlueData DataTap,
Google Cloud Storage, Google Big Query, JDBC, KDB+, Minio, and Snowflake
access. Support for these data sources can be configured by setting environment
variables for the data connectors or via a config.toml file. Refer to the Data
Connectors section in the User Guide for more information.
6 Launching Driverless AI
1. After Driverless AI is installed and started, open a browser and navigate
to <driverless-ai-host-machine>:12345.
2. The first time you log in to Driverless AI, you will be prompted to read
and accept the Evaluation Agreement. You must accept the terms before
continuing. Review the agreement, then click I agree to these terms
to continue.
3. Log in by entering unique credentials. For example:
Username: h2oai
Password: h2oai
Note that these credentials do not restrict access to Driverless AI; they are
used to tie experiments to users. If you log in with different credentials,
for example, then you will not see any previously run experiments.
The Datasets Page | 13
4. As with accepting the Evaluation Agreement, the first time you log in,
you will be prompted to enter your License Key. Paste the License Key
into the License Key entry field, and then click Save to continue. This
license key will be saved in the host machine’s /license folder.
Note: Contact [email protected] for information on how to purchase a Driver-

less AI license.
Upon successful completion, you will be ready to add datasets and run experi-
ments.
6.1 Messages
A Messages menu option is available when you launch Driverless AI. Click this
to view news and upcoming events regarding Driverless AI.
7 The Datasets Page

The Datasets Overview page is the Driverless AI Home page. This shows all
datasets that have been imported. Note that the first time you log in, this list
will be empty.
14 | The Datasets Page
7.1 Adding Datasets

Driverless AI supports the following dataset file formats: csv, tsv, txt, dat, tgz,
gz, bz2, zip, xz, xls, xlsx, nff, jay, feather, bin, arff, and parquet. (See parquet
notes below.)
Parquet Notes
• For Parquet file formats, if you select to import multiple Parquet files,
those files will be imported as multiple datasets. If you select a folder
of Parquet files, the folder will be imported as a single dataset. Tools
like Spark/Hive export data as multiple Parquet files that are stored in a
directory with a user-defined name. For example, if you export with Spark
dataFrame.write.parquet("/data/big parquet dataset"),
Spark creates a folder /data/big parquet dataset, which will contain
multiple Parquet files (depending on the number of partitions in the input
dataset) + metadata.
• You may receive a ”Failed to ingest binary file with Parquet: lists with
structs are not supported” error when ingesting a Parquet file that has a
struct as an element of an array. This is because PyArrow cannot handle
a struct that’s an element of an array. In Sparkling Water, we provide
a workaround to flatten the Parquet file. Refer to our Sparkling Water
solution for more information.
You can add datasets using one of the following methods:
• Drag and drop files from your local machine directly onto this page. Note
that this method currently works for files that are less than 10 GB.
or
1. Click the Add Dataset or Drag and Drop button to upload or add a
dataset.
Notes:
• Upload File, File System, HDFS, and S3 are enabled by default.
These can be disabled by removing them from the enabled file systems
setting in the config.toml file.
• If File System is disabled, Driverless AI will open local filebrowser
by default.
• If Driverless AI was started with data connectors enabled for HDFS,
S3, Azure Blob Store, BlueData DataTap, Google Cloud Storage,
Google Big Query, JDBC, KDB+, Minio, and/or Snowflake, then
a dropdown will appear allowing you to specify where to begin
browsing for the dataset. Refer to the Data Connectors section in
the Driverless AI User Guide for more information.
Notes:
• When importing a folder, the entire folder and all of its contents are
read into Driverless AI as a single file.
• When importing a folder, all of the files in the folder must have the
same columns.
Upon completion, the datasets will appear in the Datasets Overview page. Click
on a dataset or click the [Click for Actions] button to open a submenu. From
this menu, you can specify to view Details, Split, Visualize, Predict, or Delete a
dataset. You can also delete an unused dataset by hovering over it, clicking the
X button, and then confirming the delete. Note: You cannot delete a dataset
that was used in an active experiment. You have to delete the experiment first.
7.2 Dataset Details
To view a summary of a dataset or to preview the dataset, click on the dataset

or select the [Click for Actions] button beside the dataset that you want to
view, and then click Details from the submenu that appears. This opens the
Dataset Details page.
7.2.1 Dataset Details Page
The Dataset Details page provides a summary of the dataset. This summary
lists each column that is included in the dataset along with the type, the count,
the mean, minimum, maximum, standard deviation, frequency, and the number
of unique values. Note: Driverless AI recognizes the following column types:
integer, string, real, and boolean. Date columns are given a str type.
Hover over the top of a column to view a summary of the first 20 rows of that
column.
To view information for a specific column, type the column name in the field
above the graph.
7.2.2 Dataset Rows Page
To switch the view and preview the dataset, click the Dataset Rows button in
the top right portion of the UI. Then click the Dataset Overview button to
return to the original view.
7.2.3 Downloading Datasets
In Driverless AI, you can download datasets from the Datasets Overview page.
To download a dataset, click on the dataset or select the [Click for Actions]
button beside the dataset that you want to download, and then select Download
from the submenu that appears.
7.3 Splitting Datasets

In Driverless AI, you can split a training dataset into test and validation datasets.
Perform the following steps to split a dataset.
1. On the Datasets page, select the [Click for Actions] button beside the
dataset that you want to split, and then select Split from the submenu
that appears.
2. The Dataset Splitter form displays. Specify an Output Name 1 and an

Output Name 2 for the first and second part of the split. (For example,
you can name one test and one valid.)
3. Optionally specify a Target column (for stratified sampling), a Fold column
(to keep rows belonging to the same group together), and/or a Time
column.
4. Use the slider to select a split ratio, or enter a value in the Train/Valid
Split Ratio field.
5. Click Save when you are done.
Upon completion, the split datasets will be available on the Datasets page.
7.4 Visualizing Datasets

Perform one of the following steps to visualize a dataset:
• On the Datasets page, select the [Click for Actions] button beside the
dataset that you want to view, and then click Visualize from the submenu
that appears.
• Click the Autoviz top menu link to go to the Visualizations list page,
click the New Visualization button, then select or import the dataset
that you want to visualize.
7.4.1 The Visualization Page
The Visualization page shows all available graphs for the selected dataset. Note
that the graphs on the Visualization page can vary based on the information in
your dataset. You can also view and download logs that were generated during
the visualization.
The following is a complete list of available graphs.

• Correlated Scatterplots: Correlated scatterplots are 2D plots with large
values of the squared Pearson correlation coefficient. All possible scatter-
plots based on pairs of features (variables) are examined for correlations.
The displayed plots are ranked according to the correlation. Some of
these plots may not look like textbook examples of correlation. The only
criterion is that they have a large value of squared Pearson’s r (greater
than .95). When modeling with these variables, you may want to leave
out variables that are perfectly correlated with others.
Note that points in the scatterplot can have different sizes. Because
Driverless AI aggregates the data and does not display all points, the
bigger the point is, the bigger number of exemplars (aggregated points)
the plot covers.
• Spikey Histograms: Spikey histograms are histograms with huge spikes.
This often indicates an inordinate number of single values (usually zeros)
or highly similar values. The measure of ”spikeyness” is a bin frequency
that is ten times the average frequency of all the bins. You should
be careful when modeling (particularly regression models) with spikey
variables.
• Skewed Histograms: Skewed histograms are ones with especially large
skewness (asymmetry). The robust measure of skewness is derived from
Groeneveld, R.A. and Meeden, G. (1984), ”Measuring Skewness and
Kurtosis.” The Statistician, 33, 391-399 ([6]). Highly skewed variables
are often candidates for a transformation (e.g., logging) before use in
modeling. The histograms in the output are sorted in descending order
of skewness.
• Varying Boxplots: Varying boxplots reveal unusual variability in a feature
across the categories of a categorical variable. The measure of variabil-
ity is computed from a robust one-way analysis of variance (ANOVA).
Sufficiently diverse variables are flagged in the ANOVA. A boxplot is
a graphical display of the fractiles of a distribution. The center of the

box denotes the median, the edges of a box denote the lower and upper
quartiles, and the ends of the ”whiskers” denote that range of values.
Sometimes outliers occur, in which case the adjacent whisker is shortened
to the next lower or upper value. For variables (features) having only a few
values, the boxes can be compressed, sometimes into a single horizontal
line at the median.
• Heteroscedastic Boxplots: Heteroscedastic boxplots reveal unusual
variability in a feature across the categories of a categorical variable.
Heteroscedasticity is calculated with a Brown-Forsythe test: Brown, M.
B. and Forsythe, A. B. (1974), ”Robust tests for equality of variances.”
Journal of the American Statistical Association, 69, 364-367. Plots
are ranked according to their heteroscedasticity values. A boxplot is a
graphical display of the fractiles of a distribution. The center of the
box denotes the median, the edges of a box denote the lower and upper
quartiles, and the ends of the ”whiskers” denote that range of values.
Sometimes outliers occur, in which case the adjacent whisker is shortened
to the next lower or upper value. For variables (features) having only a few
values, the boxes can be compressed, sometimes into a single horizontal
line at the median.
• Biplots: A Biplot is an enhanced scatterplot that uses both points and
vectors to represent structure simultaneously for rows and columns of a
data matrix. Rows are represented as points (scores), and columns are
represented as vectors (loadings). The plot is computed from the first two
principal components of the correlation matrix of the variables (features).
You should look for unusual (non-elliptical) shapes in the points that
might reveal outliers or non-normal distributions. And you should look for
purple vectors that are well-separated. Overlapping vectors can indicate
a high degree of correlation between variables.
• Outliers: Variables with anomalous or outlying values are displayed as
red points in a dot plot. Dot plots are constructed using an algorithm
in Wilkinson, L. (1999). ”Dot plots.” The American Statistician, 53,
276–281 ([14]). Not all anomalous points are outliers. Sometimes the
algorithm will flag points that lie in an empty region (i.e., they are not
near any other points). You should inspect outliers to see if they are
miscodings or if they are due to some other mistake. Outliers should
ordinarily be eliminated from models only when there is a reasonable
explanation for their occurrence.
• Correlation Graph: The correlation network graph is constructed from all
pairwise squared correlations between variables (features). For continuous-
continuous variable pairs, the statistic used is the squared Pearson corre-
lation. For continuous-categorical variable pairs, the statistic is based on

the squared intraclass correlation (ICC). This statistic is computed from
the mean squares from a one-way analysis of variance (ANOVA). The
formula is (MSbetween - MSwithin)/(MSbetween + (k - 1)MSwithin),
where k is the number of categories in the categorical variable. For
categorical-categorical pairs, the statistic is computed from Cramer’s V
squared. If the first variable has k1 categories and the second variable has
k2 categories, then a k1 x k2 table is created from the joint frequencies
of values. From this table, we compute a chi-square statistic. Cramer’s
V squared statistic is then (chi-square / n) / min(k1,k2), where n is the
total of the joint frequencies in the table. Variables with large values of
these respective statistics appear near each other in the network diagram.
The color scale used for the connecting edges runs from low (blue) to
high (red). Variables connected by short red edges tend to be highly
correlated.
• Parallel Coordinates Plot: A Parallel Coordinates Plot is a graph used

for comparing multiple variables. Each variable has its own vertical axis
in the plot. Each profile connects the values on the axes for a single
observation. If the data contain clusters, these profiles will be colored by
their cluster number.
• Radar Plot: A Radar Plot is a two-dimensional graph that is used for

comparing multiple variables. Each variable has its own axis that starts
from the center of the graph. The data are standardized on each variable
between 0 and 1 so that values can be compared across variables. Each
profile, which usually appears in the form of a star, connects the values
on the axes for a single observation. Multivariate outliers are represented
by red profiles. The Radar Plot is the polar version of the popular Parallel
Coordinates plot. The polar layout enables us to represent more variables
in a single plot.
• Data Heatmap: The heatmap graphic is constructed from the transposed

data matrix. Rows of the heatmap represent variables, and columns
represent cases (instances). The data are standardized before display
so that small values are yellow and large values are red. The rows and
columns are permuted via a singular value decomposition (SVD) of the
data matrix so that similar rows and similar columns are near each other.
• Missing Values Heatmap: The missing values heatmap graphic is

constructed from the transposed data matrix. Rows of the heatmap
represent variables and columns represent cases (instances). The data are
coded into the values 0 (missing) and 1 (nonmissing). Missing values are
colored red and nonmissing values are left blank (white). The rows and
Running an Experiment | 23
columns are permuted via a singular value decomposition (SVD) of the

data matrix so that similar rows and similar columns are near each other.
• Gaps Histogram: The gaps index is computed using an algorithm of

Wainer and Schacht based on work by John Tukey. (Wainer, H. and
Schacht, Psychometrika, 43, 2, 203-12.) Histograms with gaps can
indicate a mixture of two or more distributions based on possible subgroups
not necessarily characterized in the dataset.
The images on this page are thumbnails. You can click on any of the graphs to
view and download a full-scale image.
8 Running an Experiment
8.1 Before You Begin
This section describes how to run an experiment using the Driverless AI UI.
Before you begin, it is best that you understand the available options that you
can specify. Note that only a dataset and a target column are required to be
specified, but Driverless AI provides a variety of experiment and expert settings
that you can use to build your models. Hover over each option in the UI, or
review the Experiments section in the Driverless AI User Guide for information
about these options.
After you have a comfortable working knowledge of these options, you are ready
to start your own experiment.
24 | Running an Experiment
8.2 New Experiment

1. Run an experiment by selecting [Click for Actions] button beside the
dataset that you want to use. Click Predict to begin an experiment.
2. The Experiment Settings form displays and auto-fills with the selected
dataset. Optionally enter a custom name for this experiment. If you do
not add a name, Driverless AI will create one for you.
3. Optionally specify a validation dataset and/or a test dataset.
• The validation set is used to tune parameters (models, features, etc.).
If a validation dataset is not provided, the training data is used (with
holdout splits). If a validation dataset is provided, training data is not
used for parameter tuning - only for training. A validation dataset
can help to improve the generalization performance on shifting data
distributions.
• The test dataset is used for the final stage scoring and is the
dataset for which model metrics will be computed against. Test
set predictions will be available at the end of the experiment. This
dataset is not used during training of the modeling pipeline.
These datasets must have the same number of columns as the training
dataset. Also note that if provided, the validation set is not sampled
down, so it can lead to large memory usage, even if accuracy=1 (which
reduces the train size).
4. Specify the target (response) column. Note that not all explanatory
functionality will be available for multinomial classification scenarios
(scenarios with more than two outcomes). When the target column is
selected, Driverless AI automatically provides the target column type
and the number of rows. If this is a classification problem, then the UI
shows unique and frequency statistics for numerical columns. If this is a
regression problem, then the UI shows the dataset mean and standard
deviation values.
Notes Regarding Frequency:
• For data imported in versions <= 1.0.19, TARGET FREQ and
MOST FREQ both represent the count of the least frequent class
for numeric target columns and the count of the most frequent class
for categorical target columns.
• For data imported in versions 1.0.20-1.0.22, TARGET FREQ and
MOST FREQ both represent the frequency of the target class
(second class in lexicographic order) for binomial target columns;
the count of the most frequent class for categorical multinomial
target columns; and the count of the least frequent class for numeric
multinomial target columns.
• For data imported in version 1.0.23 (and later), TARGET FREQ
is the frequency of the target class for binomial target columns,
and MOST FREQ is the most frequent class for multinomial target
columns.
5. The next step is to set the parameters and settings for the experiment.
(Hover over each option and/or refer to the Experiment Settings section in
the Driverless AI User Guide for detailed information about these settings.)
You can set the parameters individually, or you can let Driverless AI infer
the parameters and then override any that you disagree with. Note that
Driverless AI will automatically infer the best settings for Accuracy, Time,
and Interpretability and provide you with an experiment preview based on
those suggestions. If you adjust these knobs, the experiment preview will
automatically update based on the new settings.
Expert settings (optional):
Optionally specify additional expert settings for the experiment. Refer to
the Expert Settings section in the Driverless AI User Guide for detailed
information about these settings. Note that the default values for these
options are derived from the environment variables in the config.toml file.
Additional settings (optional):
• Classification or Regression button. Driverless AI automatically
determines the problem type based on the response column. Though
not recommended, you can override this setting by clicking this
button.
• Reproducible: Click this button to build this with a random seed.
• Enable GPUs: Specify whether to enable GPUs. (Note that this
option is ignored on CPU-only systems.)
6. Click Launch Experiment to start the experiment.

The experiment launches with a randomly generated experiment name.
You can change this name at anytime during or after the experiment.
Mouse over the name of the experiment to view an edit icon, then type
in the desired name.
As the experiment runs, a running status displays in the upper middle por-
tion of the UI. First Driverless AI figures out the backend and determines
whether GPUs are running. Then it starts parameter tuning, followed by
feature engineering. Finally, Driverless AI builds the scoring pipeline.
In addition to the status, the UI also displays:
• Details about the dataset.
• The iteration data (internal validation) for each cross validation fold
along with the specified scorer value. Click on a specific iteration
or drag to view a range of iterations. Double click in the graph to
reset the view.
• The variable importance values. To view variable importance for
a specific iteration, just select that iteration in the Iteration Data
graph. The Variable Importance list will automatically update to
show variable importance information for that iteration. Hover over
an entry to view more info. Note: When hovering over an entry,
you may notice the term ”Internal[...] specification.” This label is
used for features that do not need to be translated/explained and
ensures that all features are uniquely identified.
• CPU/Memory information including Notifications, Logs, and Trace
info. (Note that Trace is used for development/debugging and to
show what the system is doing at that moment.)
For classification problems, the lower right section includes a toggle
between an ROC curve, Precision-Recall graph, Lift chart, Gains chart,
Kolmogorov-Smirnov chart, and GPU Usage information (if GPUs are
available). For regression problems, the lower right section includes a
toggle between a Residuals chart, an Actual vs. Predicted chart, and
GPU Usage information (if GPUs are available). Refer to the Experiment
Graphs section in the User Guide for more information about these graphs.
Upon completion, an Experiment Summary section will populate in the
lower right section.
You can stop experiments that are currently running. Click the Finish
button to stop the experiment. This jumps the experiment to the end
and completes the ensembling and the deployment package. You can
also click Abort to terminate the experiment. (You will be prompted to

confirm the abort.) Aborted experiments will display on the Experiments
page as Failed. You can restart aborted experiments by clicking the right
side of the experiment, then selecting Restart from Last Checkpoint.
This will start a new experiment based on the aborted one. Alternatively,
you can started a new experiment based on the aborted one by selecting
New Model with Same Params.
The final step that Driverless AI performs during an experiment is to
complete the experiment report. During this step, you can click Abort
to skip this report.
8.3 Completed Experiment

After an experiment status changes from RUNNING to COMPLETE, the UI
provides you with several options:
• Deploy. Refer to the Deployment section. (Only available after building
MOJO scoring pipelines.)
• Interpret this Model. Refer to the Interpreting a Model section. (Not sup-
ported for NLP experiments. Please contact H2O support for assistance
with interpreting NLP experiments.)
• Diagnose Model on New Dataset. Refer to the Diagnosing a Model
section.
• Score on Another Dataset. Refer to the Score on Another Dataset section.
• Transform Another Dataset. Refer to the Transform Another Dataset
section. (Not available for time series experiments.)
• Download Predictions dropdown:
– Training Holdout Predictions. In csv format, available if a validation

set was NOT used.
– Validation Set Predictions. In csv format, available if a validation
set was used.
– Test Set Predictions. In csv format, available if a test dataset is
used.
• Download Python Scoring Pipeline. A standalone Python scoring pipeline
for H2O Driverless AI.
• Build MOJO Scoring Pipeline. A standalone Model Object, Optimized
scoring pipeline. (Not available for TensorFlow, RuleFit, or FTRL models.)
• Download Experiment Summary. A zip file containing the following:
– A summary of the experiment
– The experiment features along with their relative importance
– Ensemble information
– An experiment preview
– Word version of an auto-generated report for the experiment
– A target transformations tuning leaderboard
– A tuning leaderboard
• Download Logs
8.3.1 Experiment Summary
An experiment summary is available for each completed experiment. Click the

Download Experiment Summary button to download the
h2oai experiment summary <experiment>.zip file.
The files within the experiment summary zip provide textual explanations of the
graphical representations that are shown on the Driverless AI UI. For example,
the preview.txt file provides the same information that was included on the
UI before starting the experiment; the summary.txt file provides the same
summary that appears in the lower-right portion of the UI for the experiment;
the features.txt file provides the relative importance values and descriptions
for the top features.
Experiment Report
A report file is included in the experiment summary. This report provides insight
into the training data and any detected shifts in distribution, the validation
schema selected, model parameter tuning, feature evolution and the final set of
features chosen during the experiment.
• report.docx: The report available in Word format
Experiment Overview Artifacts
The Experiment Summary contains artifacts that provide overviews of the
experiment.
• preview.txt: Provides a preview of the experiment. (This is the same
information that was included on the UI before starting the experiment.)
• summary.txt: Provides the same summary that appears in the lower-right
portion of the UI for the experiment.
Tuning Artifacts
During the Driverless AI experiment, model tuning is performed to determined
the optimal algorithm and parameter settings for the provided dataset. For
regression problems, target tuning is also performed to determine the best
way to represent the target column (i.e. does taking the log of the target
column improve results). The results from these tuning steps are available in
the Experiment Summary.
• tuning leaderboard: A table of the model tuning performed along with
the score generated from the model and training time. (Available in txt
or json.)
• target transform tuning leaderboard.txt: A table of the transforms
applied to the target column along with the score generated from the
model and training time. (This will be empty for binary and multiclass
use cases.)
Features Artifacts
Driverless AI performs feature engineering on the dataset to determine the
optimal representation of the data. The top features used in the final model
can be seen in the GUI. The complete list of features used in the final model is
available in the Experiment Summary artifacts.
The Experiment Summary also provides a list of the original features and their
estimated feature importance. For example, given the features in the final
Driverless AI model, we can estimate the feature importance of the original
features.
To calculate the feature importance of PAY 3, we can aggregate the feature

importance for all variables that used PAY 3:
• NumToCatWoE:PAY AMT2: 1 * 0 (PAY 3 not used.)
• PAY 3: 0.92 * 1 (PAY 3 is the only variable used.)
• ClusterDist9:BILL AMT1:LIMIT BAL:PAY 3: 0.90 * 1/3 (PAY 3 is one
of three variables used.)
Estimated Feature Importance = (1*0) + (0.92*1) + (0.9*(1/3)) = 1.22
Note: The feature importance is converted to relative feature importance.
• features: A complete list of all features used in the final model, a
description of the feature, and the feature importance. (Available in txt
or json.)
• features orig: A list of the original features provided and an estimate of
the relative feature importance of that original feature in the final model.
(Available in txt or json.)
Final Model Artifacts
The Experiment Summary includes artifacts that describe the final model. This
is the model that is used to score new datasets and create the MOJO scoring
pipeline. The final model may be an ensemble of models depending on the
Accuracy setting.
• ensemble.txt: A summary of the final model which includes a description
of the model(s), gains/lifts table, confusion matrix, and scores of the
final model for our list of scorers.
• ensemble description.txt: A sentence describing the final model. (For

example: Final TensorFlowModel pipeline with ensemble level=0 trans-
forming 21 original features -¿ 54 features in each of 1 models each fit
on full training data (i.e. no hold-out).)
• ensemble model description.json: A json file describing the model(s)

and for ensembles how the model predictions are weighted.
• ensemble model params.json: A json file decribing the parameters of

the model(s).
• ensemble scores.json: The scores of the final model for our list of
scorers.
• ensemble confusion matrix: The confusion matrix for the internal vali-
dation and test data if test data is provided.
• ensemble confusion matrix stats test.json: Confusion matrix statis-

tics on the test data. (Only available if test data provided)
• ensemble gains: The lift and gains table for the internal validation and
test data if test data is provided. (Visualization of lift and gains can be
seen in the UI.)
8.4 Viewing Experiments

The upper-right corner of the Driverless AI UI includes an Experiments link.
Click this link to open the Experiments page. From this page, you can rename
an experiment, view previous experiments, begin a new experiment, rerun an
experiment, and delete an experiment.
8.4.1 Checkpointing, Rerunning, and Retraining
In Driverless AI, you can retry an experiment from the last checkpoint, you
can run a new experiment using an existing experiment’s settings, and you can
retrain an experiment’s final pipeline.
Checkpointing Experiments
In real-world scenarios, data can change. For example, you may have a model
currently in production that was built using 1 million records. At a later date,
you may receive several hundred thousand more records. Rather than building
a new model from scratch, Driverless AI includes H2O.ai Brain, which enables
caching and smart re-use of prior models to generate features for new models.
You can configure one of the following Brain levels in the experiment’s Expert
Settings.
• Level -1: Dont use any brain cache
• Level 0: Dont use any brain cache but still write to cache
• Level 1: Smart checkpoint if an old experiment id is passed in (for example,

via running resume one like this in the GUI)
• Level 2: Smart checkpoint if the experiment matches all column names,

column types, classes, class labels, and time series options identically.
(default)
• Level 3: Smart checkpoint like level 1, but for the entire population. Tune
only if the brain population is of insufficient size.
• Level 4: Smart checkpoint like level 2, but for the entire population. Tune
only if the brain population is of insufficient size.
• Level 5: Smart checkpoint like level 4, but will scan over the entire brain
cache of populations (starting from resumed experiment if chosen) in
order to get the best scored individuals.
If you chooses Level 2 (default), then Level 1 is also done when appropriate.
To make use of smart checkpointing, be sure that the new data has:
• The same data column names as the old experiment
• The same data types for each column as the old experiment. (This won’t
match if, e.g,. a column was all int and then had one string row.)
• The same target as the old experiment
• The same target classes (if classification) as the old experiment
• For time series, all choices for intervals and gaps must be the same
When the above conditions are met, then you can:
• Start the same kind of experiment, just rerun for longer.
• Use a smaller or larger data set (i.e. fewer or more rows).
• Effectively do a final ensemble re-fit by varying the data rows and starting
an experiment with a new accuracy, time=1, and interpretability. Check
the experiment preview for what the ensemble will be.
• Restart/Resume a cancelled, aborted, or completed experiment
To run smart checkpointing on an existing experiment, click the right side of the
experiment that you want to retry, then select Restart from Last Checkpoint.
The experiment settings page opens. Specify the new dataset. If desired, you
can also change experiment settings, though the target column must be the
same. Click Launch Experiment to resume the experiment from the last
checkpoint and build a new experiment.
The smart checkpointing continues by adding a prior model as another model
used during tuning. If that prior model is better (which is likely if it was run for
more iterations), then that smart checkpoint model will be used during feature
evolution iterations and final ensemble.
Notes:
• Driverless AI does not guarantee exact continuation, only smart continua-
tion from any last point.
34 | Diagnosing a Model
• The directory where the H2O.ai Brain meta model files are stored is
tmp/H2O.ai brain. In addition, the default maximum brain size is
20GB. Both the directory and the maximum size can be changed in the
config.toml file.
Rerunning Experiments
To run a new experiment using an existing experiment’s settings, click the
right side of the experiment that you want to use as the basis for the new
experiment, then select New Model with Same Params. This opens the
experiment settings page. From this page, you can rerun the experiment using
the original settings, or you can specify to use new data and/or specify different
experiment settings. Click Launch Experiment to create a new experiment
with the same options.
Retrain Final Pipeline
To retrain an experiment’s final pipeline, click the right side of the experiment
that you want to use as the basis for the new experiment, then select Retrain
Final Pipeline. This opens the experiment settings page with the same settings
as the original experiment except that Time is set to 0. This retrain mode is
equivalent to setting feature brain level 3 with time 0 (no tuning or feature
evolution iterations).
8.4.2 Deleting Experiments
To delete an experiment, hover over the experiment that you want to delete.
An ”X” option displays. Click this to delete the experiment. A confirmation
message will display asking you to confirm the delete. Click OK to delete the
experiment or Cancel to return to the experiments page without deleting.
9 Diagnosing a Model
The Diagnosing Model on New Dataset option allows you to view model
performance for multiple scorers based on existing model and dataset.
On the completed experiment page, click the Diagnose Model on New
Dataset button.
Notes:
• You can also diagnose a model by selecting Diagnostic from the top
menu, then selecting an experiment and test dataset.
Diagnosing a Model | 35
• The Model Diagnostics page also automatically populates with any ex-
periments that were scored from the Project Leaderboard on the Projects
page.
Select a dataset to use when diagnosing this experiment. At this point, Driverless
AI will begin calculating all available scores for the experiment.
When the diagnosis is complete, it will be available on the Model Diagnostics
page. Click on the new diagnosis. From this page, you can download predictions.
You can also view scores and metric plots. The plots are interactive. Click a
graph to enlarge. In the enlarged view, you can hover over the graph to view
details for a specific point. You can also download the graph.
Classification metric plots include the following graphs:
• ROC Curve
• Precision-Recall Curve
• Cumulative Gains
• Lift Chart
• Kolmogorov-Smirnov Chart
• Confusion Matrix
36 | Project Workspace
Regression metric plots include the following graphs:

• Actual vs Predicted
• Residual Plot with LOESS curve
• Residual Histogram
10 Project Workspace
Driverless AI provides a Project Workspace for managing datasets and exper-
iments related to a specific business problem or use case. Whether you are
trying to detect fraud or predict user retention, datasets and experiments can
be stored and saved in the individual projects. A Leaderboard on the Projects
page allows you to easily compare performance and results and identify the best
solution for your problem.
To create a Project Workspace:
1. Click the Projects option on the top menu.
2. Click New Project.
3. Specify a name for the project and provide a description.
4. Click Create Project. This creates an empty Project page.
From the Projects page, you can link datasets and/or experiments, and you can
run new experiments. When you link an existing experiment to a Project, the
datasets used for the experiment will automatically be linked to this project (if
not already linked).
Project Workspace | 37
10.1 Linking Datasets

Any dataset that has been added to Driverless AI can be linked to a project. In
addition, datasets used for experiments are also automatically linked when an
experiment is linked to the project.
You can link a Training, Validation, or Test dataset by selecting the Train-
ing, Validation, or Test tab, clicking Link Dataset, and then selecting the
dataset(s) to include. The list available datasets include those that were added
on :ref:‘Datasets‘, or you can browse datasets in your file system. Be sure to
select the correct tab before linking a training, validation, or test dataset. This
is because, when you run a new experiment in the project, the training data,
validation data, and test data options for that experiment come from list of
datasets linked here. You will not be able to, for example, select any datasets
from within the Training tab when specifying a test dataset on the experiment.
When datasets are linked, the same menu options are available here as on the
Datasets page.
10.1.1 Selecting Datasets
In the Datasets section, click Select, then select a training and/or valida-
tion and/or testing dataset. The combination of selected datasets will show
experiments in the Project that use that combination of datasets.
10.2 Linking Experiments

Existing experiments can be selected and linked to a Project. Additionally, you
can run a new experiment or checkpointing an existing experiment from this
page, and those experiments will automatically be linked to this Project.
Link an existing experiment to the project by clicking Link Experiment and

then selecting the experiment(s) to include. When you link experiments, the
datasets used to create the experiments are also automatically linked.
10.2.1 New Experiments
When experiments are run from within a Project, only linked datasets can be
used.
1. Click the New Experiment link to begin a new experiment.
2. Select your training data and optionally your validation and/or testing
data.
3. Specify your desired experiment settings, and then click Launch Experi-
ment.
As the experiment is running, it will be listed at the top of the Experiments

Leaderboard until it is completed. It will also be available on the Experiments
page.
10.2.2 Checkpointing Experiments
When experiments are linked to a Project, the same checkpointing options for
experiments are available here as on the Experiments page.
10.3 The Experiments Leaderboard
When attempting to solve a business problem, a normal workflow will include

running multiple experiments, either with different/new data or with a variety
of settings, and the optimal solution can vary for different users and/or business
problems. For some users, the model with the highest accuracy for validation
and test data could be the most optimum one. Other users might be willing
to make an acceptable compromise on the accuracy of the model for a model
with greater performance (faster prediction). For some, it could also mean how
quickly the model could be trained with acceptable levels of accuracy. The
Experiments Leaderboard makes it easy for you to find the best solution for
your business problem.
The Leaderboard is organized by showing running experiments first, then com-

pleted experiments (sorted by validation score by default), then canceled experi-
ments. You can change the sorting of completed experiments by selecting the
sort dropdown menu.
Hover over experiments in the Leaderboard to view additional information about

the experiment, including the problem type, datasets used, and the target
column.
10.3.1 Leaderboard Scoring
The Leaderboard allows you to view scoring information for a variety of scorers.
You can change the scorer used by clicking the Scorer link and then selecting
a scorer.
Experiments linked to projects do not automatically include a test score. To

view Test Scores in the Leaderboard, you must first complete the scoring step for
a particular dataset and experiment combination. Without the scoring step, no
scoring data is available to populate in the Test Score and Score Time columns
of the Leaderboard. Experiments that do not include a test score or that have
an invalid scorer (for example, if the R2 scorer is selected for classification
experiments) show N/A in the Leaderboard. Also, if None is selected for the
scorer, then all experiments will show N/A.
To score the experiment:
1. Click the Select button at the top of the Leaderboard and select the
experiment or experiments that you want to score.
2. Select a Scorer Dataset.
3. Click Score n Items. This starts the Model Diagnostic process and scores
all experiments against the selected scorer dataset. Upon completion, the
experiment(s) on the Leaderboard will be populated with a test score. The

performance information will also be available on the Model Diagnostics
page.
Notes:
• If an experiment has already scored a dataset, it will not score it again.
The scoring step is deterministic, so for a particular scorer dataset and
experiment combination, the score will be same regardless of how many
times you repeat it.
• The scorer dataset absolutely needs to have all the columns that are
expected by the various experiments you are scoring it on. However, the
columns of the scorer dataset need not be exactly the same as input
features expected by the experiment. There can be additional columns
in the scorer dataset. If these columns were not used for training, they
will be ignored. This feature gives you the ability to train experiments
on different training datasets (i.e., having different features), and if you
have an ”uber test dataset” that includes all these feature columns, then
you can use the same dataset to score these experiments.
• You will notice a Score Time in the Experiments Leaderboard. This
values shows the total time (in seconds) that it took for calculating the
experiment scores for all applicable scorers for the experiment type. This
is valuable to users who need to estimate the runtime performance of an
experiment.
10.3.2 Comparing Experiments
You can compare two or three experiments and view side-by-side detailed
information about each.
1. Click the Select button at the top of the Leaderboard and select either
two or three experiments that you want to compare. You cannot compare
more than three experiments.
2. Click the Compare n Items button.
This opens the Compare Experiments page. This page includes the
experiment summary for each experiment as well as metric plots. The
metric plots vary depending on whether this is a classification or regression
experiment.
For classification experiments, this page includes:
• Variable Importance list
• Confusion Matrix
• ROC Curve
• Precision Recall Curve
• Lift Chart
• Gains Chart
• Kolmogorov-Smirnov Chart
For regression experiments, this page includes:
• Variable Importance list
• Actual vs. Predicted Graph
Interpreting a Model | 43
10.4 Unlinking Data on a Projects Page

Unlinking datasets and/or experiments does not delete that data from Driverless
AI. The datasets and experiments will still be available on the Datasets and
Experiments pages.
• Unlink a dataset by clicking on the dataset and selecting **Unlink**
from the menu. **Note**: You cannot unlink datasets that are tied to
experiments in the same project.
• Unlink an experiment by clicking on the experiment and selecting **Un-
link** from the menu. Note that this will not automatically unlink
datasets that were tied to the experiment.
10.5 Deleting Projects

To delete a project, click the Projects option on the top menu to open the
main Projects page. Hover over the project that you want to delete and click
the red X button.
Note that deleting projects does not delete datasets and experiments from
Driverless AI. Any datasets and experiments from deleted projects will still be
available on the Datasets and Experiments pages.
11 Interpreting a Model
Driverless AI provides robust interpretability of machine learning models to
explain modeling results in a human-readable format. In the Machine Learning
Interpetability (MLI) view, Driverless AI employs a host of different techniques
and methodologies for interpreting and explaining the results of its models.
A number of charts are generated automatically, including K-LIME, Shapley,
Variable Importance, Decision Tree Surrogate, Partial Dependence, Individual
Conditional Expectation, and more. Additionally, you can download a CSV of
LIME and Shapley reasons codes from this view.
44 | Interpreting a Model
This chapter describes Machine Learning Interpretability (MLI) in Driverless AI

for both regular and time-series experiments. There are two methods you can
use for interpreting models:
• Using the Interpret this Model button on a completed experiment page

to interpret a Driverless AI model on original and transformed features.
• Using the MLI link in the upper right corner of the UI to interpret either
a Driverless AI model or an external model.
Notes:
• This release deprecates experiments and MLI models from 1.7.0 and
earlier.
• MLI does not require an Internet connection to run on current models.
• MLI is not available for NLP experiments or for multiclass Time Series.
• For time series experiments, when the test set contains actuals, you will
see the time series metric plot and the group metrics table. If there are
no actuals, MLI will run, but you will see only the prediction value time
series and a Shapley table.
11.1 Interpret this Model button - Regular Ex-

periments
Clicking the Interpret this Model button on a completed experiment page

launches the Model Interpretation for that experiment. Python and Java logs
can be viewed while the interpretation is running.
For regular experiments, this page provides several visual explanations of the
trained Driverless AI model and its results. More information about this page is
available in the Understanding the Model Interpretation Page section later in
this chapter.
11.2 Interpret this Model button - Time-Series

Experiments
Driverless AI allows you to run MLI for multi-group and single-group time series
experiments.
11.2.1 Multi-Group Time Series MLI
This section describes how to run MLI on time series data for multiple groups.
1. Click the Interpret this Model button on a completed time series ex-
periment to launch Model Interpretation for that experiment. This page
includes the following:
• A Help panel describing how to read and use this page. Click the
Hide Help Button to hide this text.
• If a test set is provided and the test set includes actuals, then a panel
will display showing a time series plot and the top and bottom group
matrix tables based on the scorer that was used in the experiment.
The metric plot will show the metric of interest per time point
for holdout predictions and the test set. Likewise, the actual vs.
predicted plot will show actuals vs. predicted values per time point
for the holdout set and the test set. Note that this panel can be
resized if necessary.
• If a test set is not provided, then internal validation predictions will
be used. The metric plot will only show the metric of interest per
time point for holdout predictions. Likewise, the actual vs. predicted
plot will only show actuals vs. predicted values per time point for
the holdout set.
• A Download Logs button for retrieving logs that were generated

when this interpretation was built.
• A Download Group Metrics button for retrieving the averages of

each group’s scorer, as well as each group’s sample size.
• A Show Summary button that provide details about the experiment

settings that were used.
• A Group Search entry field for selecting the groups to view.
• Use the zoom feature to magnify any portion of a graph by clicking

the Enable Zoom icon near the top-right corner of a graph. While
this icon is selected, click and drag to draw a box over the portion
of the graph you want to magnify. Click the Disable Zoom icon to
return to the default view.
2. Scroll to the bottom of the panel and select a grouping in the Group
Search field to view a graph of Actual vs. Predicted values for the group.
The outputted graph can be downloaded to your local machine.
3. Click on a prediction point in the plot (white line) to view Shapley values
for that prediction point. The Shapley values plot can also be downloaded
to your local machine.
4. Click Add Panel to add a new MLI Time Series panel. This allows you to
compare different groups in the same model and also provides the flexibility to
do a ”side-by-side” comparison between different models.
11.2.2 Single Time Series MLI
Time Series MLI can also be run when only one group is available.
1. Click the Interpret this Model button on a completed time series ex-
periment to launch Model Interpretation for that experiment. This page
includes the following:
• A Help panel describing how to read and use this page. Click the
Hide Help Button to hide this text.
• If a test set is provided and the test set includes actuals, then a panel
will display showing a time series plot and the top and bottom group
matrix tables based on the scorer that was used in the experiment.
The metric plot will show the metric of interest per time point
for holdout predictions and the test set. Likewise, the actual vs.
predicted plot will show actuals vs. predicted values per time point
for the holdout set and the test set. Note that this panel can be
resized if necessary.
• If a test set is not provided, then internal validation predictions will
be used. The metric plot will only show the metric of interest per
time point for holdout predictions. Likewise, the actual vs. predicted
plot will only show actuals vs. predicted values per time point for
the holdout set.
• A Download Logs button for retrieving logs that were generated

when this interpretation was built.
• A Download Group Metrics button for retrieving the averages of

each group’s scorer, as well as each group’s sample size.
• A Show Summary button that provide details about the experiment

settings that were used.
• A Group Search entry field for selecting the groups to view.
• Use the zoom feature to magnify any portion of a graph by clicking

the Enable Zoom icon near the top-right corner of a graph. While
this icon is selected, click and drag to draw a box over the portion
of the graph you want to magnify. Click the Disable Zoom icon to
return to the default view.
2. Scroll to the bottom of the panel and select an option in the Group
Search field to view a graph of Actual vs. Predicted values for the group.
(Note that for Single Time Series MLI, there will only be one option in this
field.) The outputted graph can be downloaded to your local machine.
3. Click on a prediction point in the plot (white line) to view Shapley values
for that prediction point. The Shapley values plot can also be downloaded
to your local machine.
4. Click Add Panel to add a new MLI Time Series panel. This allows you
to do a ”side-by-side” comparison between different models.
11.3 Model Interpretation - Driverless AI Mod-

els
This method allows you to run model interpretation on a Driverless AI model.
This method is similar to clicking one of the Interpret This Model buttons
on an experiment summary page.
1. Click the MLI link in the upper-right corner of the UI to view a list of
interpreted models.
2. Click the New Interpretation button.

3. Select the dataset that was used to train the model that you will use for
interpretation.
4. Specify the Driverless AI model that you want to use for the interpretation.
Once selected, the target column used fro the model will be selected.
5. Select a LIME method of either K-LIME (default) or LIME-SUP.

• K-LIME creates one global surrogate GLM on the entire training data
and also creates numerous local surrogate GLMs on samples formed
from K-means clusters in the training data. The features used for
K-means are selected from the Random Forest surrogate model’s
variable importance. The number of features used for K-means is
the minimum of the top 25 percent of variables from the Random
Forest surrogate model’s variable importance and the max number
of variables that can be used for K-means, which is set by the user
in the config.toml setting for mli max number cluster vars.
(Note, if the number of features in the dataset are less than or
equal to 6, then all features are used for K-means clustering.) The
previous setting can be turned off to use all features for k-means
by setting use all columns klime kmeans in the config.toml
file to true. All penalized GLM surrogates are trained to model
the predictions of the Driverless AI model. The number of clusters
for local explanations is chosen by a grid search in which the R2
between the Driverless AI model predictions and all of the local
K-LIME model predictions is maximized. The global and local linear
model’s intercepts, coefficients, R2 values, accuracy, and predictions
can all be used to debug and develop explanations for the Driverless
AI model’s behavior.
• LIME-SUP explains local regions of the trained Driverless AI model
in terms of the original variables. Local regions are defined by
each leaf node path of the decision tree surrogate model instead of
simulated, perturbed observation samples - as in the original LIME.
For each local region, a local GLM model is trained on the original
inputs and the predictions of the Driverless AI model. Then the
parameters of this local GLM can be used to generate approximate,
local explanations of the Driverless AI model.
6. For K-LIME interpretations, specify the depth that you want for your
decision tree surrogate model. The tree depth value can be a value
from 2-5 and defaults to 3. For LIME-SUP interpretations, specify the
LIME-SUP tree depth. This can be a value from 2-5 and defaults to 3.
7. Optionally specify whether to use original features or transformed features
in the surrogate model for the new interpretation. Note: If this option is
disabled, then the K-LIME clustering column option will not be available,
and quantile binning will not be available.
8. Specify whether to perform the interpretation on a sample of the training
data. By default, MLI will sample the training dataset if it is greater
than 100k rows. (Note that this value can be modified in the config.toml
setting for mli sample size.) Turn this toggle off to run MLI on the
entire dataset.
9. Optionally specify weight and dropped columns.
10. For K-LIME interpretations, optionally specify a clustering column. Note
that this column should be categorical. Also note that this is only available
when K-LIME is used as the LIME method and when Use Original
Features is enabled. If the LIME method is changed to LIME-SUP, then
this option is no longer available.
11. Optionally specify the number of surrogate cross-validation folds to use
(from 0 to 10). When running experiments, Driverless AI automatically
splits the training data and uses the validation data to determine the
performance of the model parameter tuning and feature engineering steps.
For a new interpretation, Driverless AI uses 3 cross-validation folds by
default for the interpretation.
12. For K-LIME interpretations, optionally specify one or more columns to
generate decile bins (uniform distribution) to help with MLI accuracy.
Columns selected are added to top n columns for quantile binning selection.
If a column is not numeric or not in the dataset (transformed features),
then the column will be skipped. Note: This option is only available
when Use Original Features is enabled.
13. For K-LIME interpretations, optionally specify the number of top variable
importance numeric columns to run decile binning to help with MLI
accuracy. (Note that variable importances are generated from a Random
Forest model.) This defaults to 0, and the maximum value is 10. Note:
This option is only available when Use Original Features is enabled.
14. Click the Launch MLI button.
11.4 Model Interpretation - External Models

Model Interpretation does not need to be run on a Driverless AI experiment. You
can train an external model and run Model Interpretability on the predictions.
1. Click the MLI link in the upper-right corner of the UI to view a list of
interpreted models.
2. Click the New Interpretation button.

3. Select the dataset that you want to use for the model interpretation. This
must include a prediction column that was generated by the external
model. If the dataset does not have predictions, then you can join the
external predictions. An example showing how to do this using Python is
available in the Driverless AI User Guide.
Note: When running interpretations on an external model, leave the
Select Model option empty. That option is for selecting a Driverless AI
model.
4. Specify a Target Column (actuals) and the Prediction Column (scores
from the model).
5. Select a LIME method of either K-LIME (default) or LIME-SUP.
• K-LIME creates one global surrogate GLM on the entire training data
and also creates numerous local surrogate GLMs on samples formed
from K-means clusters in the training data. The features used for
K-means are selected from the Random Forest surrogate model’s
variable importance. The number of features used for K-means is
the minimum of the top 25 percent of variables from the Random
Forest surrogate model’s variable importance and the max number
of variables that can be used for K-means, which is set by the user
in the config.toml setting for mli max number cluster vars.
(Note, if the number of features in the dataset are less than or
equal to 6, then all features are used for K-means clustering.) The
previous setting can be turned off to use all features for k-means
by setting use all columns klime kmeans in the config.toml
file to true. All penalized GLM surrogates are trained to model
the predictions of the Driverless AI model. The number of clusters
for local explanations is chosen by a grid search in which the R2

between the Driverless AI model predictions and all of the local
K-LIME model predictions is maximized. The global and local linear
model’s intercepts, coefficients, R2 values, accuracy, and predictions
can all be used to debug and develop explanations for the Driverless
• LIME-SUP explains local regions of the trained Driverless AI model
in terms of the original variables. Local regions are defined by
each leaf node path of the decision tree surrogate model instead of
simulated, perturbed observation samples - as in the original LIME.
For each local region, a local GLM model is trained on the original
inputs and the predictions of the Driverless AI model. Then the
parameters of this local GLM can be used to generate approximate,
6. For K-LIME interpretations, specify the depth that you want for your
decision tree surrogate model. The tree depth value can be a value
from 2-5 and defaults to 3. For LIME-SUP interpretations, specify the
LIME-SUP tree depth. This can be a value from 2-5 and defaults to 3.
7. Specify whether to perform the interpretation on a sample of the training
data. By default, MLI will sample the training dataset if it is greater
than 100k rows. (Note that this value can be modified in the config.toml
setting for mli sample size.) Turn this toggle off to run MLI on the
entire dataset.
8. Optionally specify weight and dropped columns.
9. For K-LIME interpretations, optionally specify a clustering column. Also
note that this is only available when K-LIME is used as the LIME method.
If the LIME method is changed to LIME-SUP, then this option is no
longer available.
10. Optionally specify the number of surrogate cross-validation folds to use
(from 0 to 10). When running experiments, Driverless AI automatically
splits the training data and uses the validation data to determine the
performance of the model parameter tuning and feature engineering steps.
For a new interpretation, Driverless AI uses 3 cross-validation folds by
default for the interpretation.
11. For K-LIME interpretations, optionally specify one or more columns to
generate decile bins (uniform distribution) to help with MLI accuracy.
Columns selected are added to top n columns for quantile binning selection.
If a column is not numeric or not in the dataset (transformed features),
then the column will be skipped.
12. For K-LIME interpretations, optionally specify the number of top variable
importance numeric columns to run decile binning to help with MLI
accuracy. (Note that variable importances are generated from a Random
Forest model.) This value is combined with any specific columns selected
for quantile binning. This defaults to 0, and the maximum value is 10.
13. Click the Launch MLI button.
11.5 Understanding the Model Interpretation Page

The Model Interpretation page opens with a Summary of the interpretation.
This page also provides left-hand navigation for viewing additional plots. This
navigation includes:
• Summary: Provides a summary of the MLI experiment. See the Summary
Page section for more information.
• DAI Model: See DAI Model Dropdown for more information.
– For binary classification and regression experiments, the DAI Model
menu provides Feature Importance and Shapley plots for transformed
features as well as a Partial Dependence and ICE plot and a Disparate
Impact Analysis plot for Driverless AI models.
– For multiclass classification experiments, the DAI Model menu pro-
vides Feature Importance and Shapley plots for transformed features.
– Note: Shapley plots are not supported for RuleFit and TensorFlow
models.
• Surrogate Models: See Surrogate Models Dropdown for more informa-
tion.
– For binary classification and regression experiments, the Surrogate
Model menu provides KLIME and Decision Tree plots. This also
includes a Random Forest submenu, which includes Global and

Local Feature Importance plots for original features and a Partial
Dependence plot.
– For multiclass classification experiments, the Surrogate Model menu

provides a Random Forest submenu that includes a Global Feature
Importance plot for the Random Forest surrogate model.
• Dashboard: See the Dashboard Page section for more information.
– For binary classification and regression experiments, the Dashboard

page provides a single page with a Global Interpretable Model Expla-
nations plot, a Feature Importance plot, a Decision Tree plot, and a
Partial Dependence plot.
– The Dashboard page is not available for multinomial experiments.
• MLI Docs: A link to the Interpreting a Model section in the online help.
• Download MLI Logs: Downloads a zip file of the logs that were gener-
ated during this interpretation.
• Experiment: Provides a link back to the experiment that generated this

interpretation.
• Scoring Pipeline:
– For binomial experiments, Scoring Pipeline option downloads the

scoring pipeline for this interpretation.
– The Scoring Pipeline option is not available for multinomial experi-

ments.
• Download Reason Codes:
– For binomial experiments, download a CSV file of LIME and/or

Shapley reason codes.
– For multinomial experiments, download a CSV file of the Shapley

reason codes.
11.5.1 Summary Page
The Summary page is the first page that opens when you view an interpretation.
This page provides an overview of the interpretation, including the dataset and
Driverless AI experiment (if available) that were used for the interpretation
along with the feature space (original or transformed), target column, problem
type, and k-Lime information. If the interpretation was created from a Driverless
AI model, then a table with the Driverless AI model summary is also included
along with the top variables for the model.
11.5.2 DAI Model Dropdown
This menu provides a Feature Importance plot and a Shapley plot (not supported
for RuleFit and TensorFlow models) for transformed features as well as a Partial
Dependence plot and Disparate Impact Analysis (DIA) for Driverless AI models.
Feature Importance
This plot is available for all models for binary classification, multiclass classifica-
tion, and regression experiments.
This plot shows the Driverless AI feature importance. Driverless AI feature

importance is a measure of the contribution of an input variable to the overall
predictions of the Driverless AI model. Global feature importance is calculated
by aggregating the improvement in splitting criterion caused by a single variable
across all of the decision trees in the Driverless AI model.
Shapley Plot
This plot is not available for RuleFit or TensorFlow models. For all other
models, this plot is available for binary classification, multiclass classification,
and regression experiments.
Shapley explanations are a technique with credible theoretical support that
presents consistent global and local variable contributions. Local numeric
Shapley values are calculated by tracing single rows of data through a trained
tree ensemble and aggregating the contribution of each input variable as the
row of data moves through the trained ensemble. For regression tasks, Shapley
values sum to the prediction of the Driverless AI model. For classification
problems, Shapely values sum to the prediction of the Driverless AI model
before applying the link function. Global Shapley values are the average of the
absolute Shapley values over every row of a data set.
More information is available at https://arxiv.org/abs/1706.06060.
You can view a Shapley explanations plot by selecting the Interpret this Model
on Transformed Features button in an experiment.
Partial Dependence and Individual Conditional Expectation (ICE)

Partial dependence is a measure of the average model prediction with respect
to an input variable. Partial dependence plots display how machine-learned
response functions change based on the values of an input variable of interest,
while taking nonlinearity into consideration and averaging out the effects of all
other input variables. Partial dependence plots are well-known and described in
the Elements of Statistical Learning (Hastie et all, 2001 [4]). Partial dependence
plots enable increased transparency in Driverless AI models and the ability to
validate and debug Driverless AI models by comparing a variable’s average
predictions across its domain to known standards, domain knowledge, and
reasonable expectations. The partial dependence plot is available for binary
classification and regression models.
Individual conditional expectation (ICE) plots, a newer and less well-known
adaptation of partial dependence plots, can be used to create more localized
explanations for a single individual using the same basic ideas as partial de-
pendence plots. ICE Plots were described by Goldstein et al (2015 [5]). ICE
values are simply disaggregated partial dependence, but ICE is also a type of
nonlinear sensitivity analysis in which the model predictions for a single row
are measured while a variable of interest is varied over its domain. ICE plots
enable a user to determine whether the model’s treatment of an individual row
of data is outside one standard deviation from the average model behavior,
whether the treatment of a specific row is valid in comparison to average model
behavior, known standards, domain knowledge, and reasonable expectations,
and how a model will behave in hypothetical situations where one variable in a
selected row is varied across its domain. The ICE plot is available for binary
classification and regression models.
Given the row of input data with its corresponding Driverless AI and K -LIME
predictions:
Taking the Driverless AI model as F(X), assuming credit scores vary from 500
to 800 in the training data, and that increments of 30 are used to plot the ICE
curve, ICE is calculated as follows:
ICEcredit score,500 = F (30, 500, 1000)
...
The one-dimensional partial dependence plots displayed here do not take inter-
actions into account. Large differences in partial dependence and ICE are an
indication that strong variable interactions may be present. In this case partial
dependence plots may be misleading because average model behavior may not
accurately reflect local behavior.
Overlaying ICE plots onto partial dependence plots allow the comparison of
the Driverless AI model’s treatment of certain examples or individuals to the
model’s average predictions over the domain of an input variable of interest.
This plot shows the partial dependence when a variable is selected and the ICE
values when a specific row is selected. Users may select a point on the graph
to see the specific value at that point. Partial dependence (yellow) portrays the
average prediction behavior of the Driverless AI model across the domain of an
input variable along with +/- 1 standard deviation bands. ICE (grey) displays
the prediction behavior for an individual row of data when an input variable
is toggled across its domain. Currently, partial dependence and ICE is only
available for the top ten most important original input variables. Categorical
variables with 20 or more unique values are never included in these plots.
Disparate Impact Analysis

This plot is available for binary classification and regression models.
DIA is a technique that is used to evaluate fairness. Bias can be introduced
to models during the process of collecting, processing, and labeling dataas a
result, it is important to determine whether a model is harming certain users by
making a significant number of biased decisions.
DIA typically works by comparing aggregate measurements of unprivileged
groups to a privileged group. For instance, the proportion of the unprivileged
group that receives the potentially harmful outcome is divided by the proportion
of the privileged group that receives the same outcomethe resulting proportion
is then used to determine whether the model is biased. Refer to the Summary
section to determine if a categorical level (for example, Fairness Female) is
fair in comparison to the specified reference level and user-defined thresholds.
Fairness All is a true or false value that is only true if every category is fair in
comparison to the reference level.
Disparate impact testing is best suited for use with constrained models in Driver-
less AI, such as linear models, monotonic GBMs, or RuleFit. The average group
metrics reported in most cases by DIA may miss cases of local discrimination,
especially with complex, unconstrained models that can treat individuals very
differently based on small changes in their data attributes.
DIA allows you to specify a disparate impact variable (the group variable that
is analyzed), a reference level (the group level that other groups are compared
to), and user-defined thresholds for disparity. Several tables are provided as
part of the analysis:
• Group metrics: The aggregated metrics calculated per group. For
example, true positive rates per group.
• Group disparity: This is calculated by dividing the metric for group by
the reference group metric. Disparity is observed if this value falls outside
of the user-defined thresholds.
• Group parity: This builds on Group disparity by converting the above
calculation to a true or false value by applying the user-defined thresholds
to the disparity values.
In accordance with the established four-fifths rule, user-defined thresholds are set
to 0.8 and 1.25 by default. These thresholds will generally detect if the model is
(on average) treating the non-reference group 20% more or less favorably than
the reference group. Users are encouraged to set the user-defined thresholds to
align with their organization’s guidance on fairness thresholds.
Notes:
• Although the process of DIA is the same for both classification and re-
gression experiments, the returned information is dependent on the type
of experiment being interpreted. An analysis of a regression experiment
returns an actual vs. predicted plot, while an analysis of a binary clas-
sification experiment returns confusion matrices. The above tables are
provided for both types of experiments.
• Users are encouraged to consider the explanation dashboard to understand
and augment results from disparate impact analysis. In addition to its
established use as a fairness tool, users may want to consider disparate
impact for broader model debugging purposes. For example, users can
analyze the supplied confusion matrices and group metrics for important,
non-demographic features in the Driverless AI model.
Classification Experiment
Regression Experiment
11.5.3 Surrogate Models Dropdown
The Surrogate Models dropdown includes K-LIME/LIME-SUP and Decision

Tree plots as well as a Random Forest submenu, which includes Global and
Local Feature Importance plots for original features and a Partial Dependence
plot.
K-LIME and LIME-SUP
The MLI screen includes a K-LIME or LIME-SUP graph. A K-LIME graph

is available by default when you interpret a model from the experiment page.
When you create a new interpretation, you can instead choose to use LIME-SUP
as the LIME method. Note that these graphs are essentially the same, but the
K-LIME/LIME-SUP distinction provides insight into the LIME method that was
used during model interpretation.
The K-LIME Technique
K -LIME is a variant of the LIME technique proposed by Ribeiro at al [12].

K -LIME generates global and local explanations that increase the transparency
of the Driverless AI model, and allow model behavior to be validated and
debugged by analyzing the provided plots, and comparing global and local
explanations to one-another, to known standards, to domain knowledge, and to
reasonable expectations.
K-LIME creates one global surrogate GLM on the entire training data and
also creates numerous local surrogate GLMs on samples formed from K-means
clusters in the training data. The features used for K-means are selected from
the Random Forest surrogate model’s variable importance. The number of
features used for K-means is the minimum of the top 25 percent of variables
from the Random Forest surrogate model’s variable importance and the max
number of variables that can be used for K-means, which is set by the user in
the config.toml setting for mli max number cluster vars. (Note, if the
number of features in the dataset are less than or equal to 6, then all features
are used for K-means clustering.) The previous setting can be turned off to use
all features for k-means by setting use all columns klime kmeans in the
config.toml file to true. All penalized GLM surrogates are trained to model
the predictions of the Driverless AI model. The number of clusters for local
explanations is chosen by a grid search in which the R2 between the Driverless
AI model predictions and all of the local K-LIME model predictions is maximized.
The global and local linear model’s intercepts, coefficients, R2 values, accuracy,
and predictions can all be used to debug and develop explanations for the
Driverless AI model’s behavior.
LIME-SUP explains local regions of the trained Driverless AI model in terms

of the original variables. Local regions are defined by each leaf node path of
the decision tree surrogate model instead of simulated, perturbed observation
samples - as in the original LIME. For each local region, a local GLM model is
trained on the original inputs and the predictions of the Driverless AI model.
Then the parameters of this local GLM can be used to generate approximate,
The parameters of the global K -LIME model give an indication of overall linear
feature importance and the overall average direction in which an input variable
influences the Driverless AI model predictions. The global model is also used
to generate explanations for very small clusters (N < 20) where fitting a local
linear model is inappropriate.
The in-cluster linear model parameters can be used to profile the local region,
to give an average description of the important variables in the local region,
and to understand the average direction in which an input variable affects the
Driverless AI model predictions. For a point within a cluster, the sum of the local
linear model intercept and the products of each coefficient with their respective
input variable value are the K -LIME prediction. By disaggregating the K -LIME
predictions into individual coefficient and input variable value products, the
local linear impact of the variable can be determined. This product is sometimes
referred to as a reason code and is used to create explanations for the Driverless
In the following example, reason codes are created by evaluating and disaggre-
gating a local linear model.
predictions:
And the local linear model:

yK -LIME = 0.1 + 0.01 ∗ debt to income ratio + 0.0005 ∗ credit score + 0.0002 ∗ savings account balance
It can be seen that the local linear contributions for each variable are:
• debt to income ratio: 0.01 * 30 = 0.3
• credit score: 0.0005 * 600 = 0.3
• savings acct balance: 0.0002 * 1000 = 0.2
Each local contribution is positive and thus contributes positively to the Driver-
less AI model’s prediction of 0.85 for H2OAI predicted default. By taking into
consideration the value of each contribution, reason codes for the Driverless AI
decision can be derived. debt to income ratio and credit score would be the
two largest negative reason codes, followed by savings acct balance.
The local linear model intercept and the products of each coefficient and
corresponding value sum to the K -LIME prediction. Moreover it can be seen
that these linear explanations are reasonably representative of the nonlinear

model’s behavior for this individual because the K -LIME predictions are within
5.5% of the Driverless AI model prediction. This information is encoded into
English language rules which can be viewed by clicking the Explanations button.
Like all LIME explanations based on linear models, the local explanations are
linear in nature and are offsets from the baseline prediction, or intercept, which
represents the average of the penalized linear model residuals. Of course, linear
approximations to complex non-linear response functions will not always create
suitable explanations and users are urged to check the K -LIME plot, the local
model R2 , and the accuracy of the K -LIME prediction to understand the validity
of the K -LIME local explanations. When K -LIME accuracy for a given point
or set of points is quite low, this can be an indication of extremely nonlinear
behavior or the presence of strong or high-degree interactions in this local region
of the Driverless AI response function. In cases where K -LIME linear models
are not fitting the Driverless AI model well, nonlinear LOCO feature importance
values may be a better explanatory tool for local model behavior. As K -LIME
local explanations rely on the creation of k-means clusters, extremely wide input
data or strong correlation between input variables may also degrade the quality
of K -LIME local explanations.
The LIME-SUP Technique
LIME-SUP explains local regions of the trained Driverless AI model in terms

of the original variables. Local regions are defined by each leaf node path of
the decision tree surrogate model instead of simulated, perturbed observation
samples - as in the original LIME. For each local region, a local GLM model is
trained on the original inputs and the predictions of the Driverless AI model.
Then the parameters of this local GLM can be used to generate approximate,
The Global Interpretable Model Explanation Plot
This plot shows Driverless AI model predictions and LIME model predictions
in sorted order by the Driverless AI model predictions. This graph is interac-
tive. Hover over the Model Prediction, LIME Model Prediction, or Actual
Target radio buttons to magnify the selected predictions. Or click those radio
buttons to disable the view in the graph. You can also hover over any point in
the graph to view LIME reason codes for that value. By default, this plot shows
information for the global LIME model, but you can change the plot view to
show local results from a specific cluster. The LIME plot also provides a visual
indication of the linearity of the Driverless AI model and the trustworthiness
of the LIME explanations. The closer the local linear model approximates the
Driverless AI model predictions, the more linear the Driverless AI model and
the more accurate the explanation generated by the LIME local linear models.
Decision Tree
The decision tree surrogate model increases the transparency of the Driverless
AI model by displaying an approximate flowchart of the complex Driverless
AI model’s decision making process. The decision tree surrogate model also
displays the most important variables in the Driverless AI model and the most
important interactions in the Driverless AI model. The decision tree surrogate
model can be used for visualizing, validating, and debugging the Driverless AI
model by comparing the displayed decision-process, important variables, and
important interactions to known standards, domain knowledge, and reasonable
expectations.
A surrogate model is a data mining and engineering technique in which a

generally simpler model is used to explain another, usually more complex, model
or phenomenon. The decision tree surrogate is known to date back at least
to 1996 (Craven and Shavlik, [2]). The decision tree surrogate model here
is trained to predict the predictions of the more complex Driverless AI model
using the of original model inputs. The trained surrogate model enables a
heuristic understanding (i.e., not a mathematically precise understanding) of
the mechanisms of the highly complex and nonlinear Driverless AI model.
In the decision tree plot, the highlighted row shows the path to the highest
probability leaf node and indicates the globally important variables and inter-
actions that influence the Driverless AI model prediction for that row. The
decision tree plot is available for binary classification and regression models.
11.5.4 Random Forest Dropdown
The Random Forest dropdown provides a submenu that includes a Feature

Importance plot, a Partial Dependence plot, and a LOCO plot. These plots are
for original features rather than transformed features.
Global Feature Importance vs Local Feature Importance
Global feature importance (yellow) is a measure of the contribution of an input
variable to the overall predictions of the Driverless AI model. Global feature
importance is calculated by aggregating the improvement in splitting criterion
caused by a single variable across all of the decision trees in the Driverless AI
model. Local feature importance (grey) is a measure of the contribution of an
input variable to a single prediction of the Driverless AI model. Local feature
importance is calculated by removing the contribution of a variable from every
decision tree in the Driverless AI model and measuring the difference between
the prediction with and without the variable. Both global and local variable
importance are scaled so that the largest contributor has a value of 1.
Note: Engineered features are used for MLI when a time series experiment is
built. This is because munged time series features are more useful features for
MLI compared to raw time series features.
LOCO
Local feature importance describes how the combination of the learned model
rules or parameters and an individual row’s attributes affect a model’s prediction
for that row while taking nonlinearity and interactions into effect. Local feature
importance values reported here are based on a variant of the leave-one-covariate-
out (LOCO) method (Lei et al, 2017 [9]).
In the LOCO-variant method, each local feature importance is found by re-
scoring the trained Driverless AI model for each feature in the row of interest,
while removing the contribution to the model prediction of splitting rules that
contain that feature throughout the ensemble. The original prediction is then
subtracted from this modified prediction to find the raw, signed importance
for the feature. All local feature importance values for the row are then scaled
between 0 and 1 for direct comparison with global feature importance values.
predictions:
Taking the Driverless AI model as F(X), LOCO-variant feature importance

values are calculated as follows.
First, the modified predictions are calculated:
F debt to income ratio = F (N A, 600, 1000) = 0.99
F credit score = F (30, N A, 1000) = 0.73
F savings acct balance = F (30, 600, N A) = 0.82
Second, the original prediction is subtracted from each modified prediction to
generate the unscaled local feature importance values:
LOCOdebt to income ratio = F debt to income ratio − 0.85 = 0.99 −
0.85 = 0.14
LOCOcredit score = F credit score − 0.85 = 0.73 − 0.85 = −0.12
LOCOsavings acct balance = F savings acct balance − 0.85 = 0.82 −
0.85 = −0.03
Finally LOCO values are scaled between 0 and 1 by dividing each value for the
row by the maximum value for the row and taking the absolute magnitude of
this quotient.
Scaled(LOCOdebt to income ratio ) = Abs(LOCO debt to income ratio /0.14) =

1
Scaled(LOCOcredit score ) = Abs(LOCO credit score /0.14) = 0.86
Scaled(LOCOsavings acct balance ) = Abs(LOCO savings acct balance /0.14) =

0.21
One drawback to these LOCO-variant feature importance values is, unlike

K -LIME, it is difficult to generate a mathematical error rate to indicate when
LOCO values may be questionable.
Partial Dependence and Individual Conditional Expectation
A Partial Dependence and ICE plot is available for both Driverless AI and
surrogate models. Refer to the previous Partial Dependence and Individual
Conditional Expectation section for more information about this plot.
11.5.5 Dashboard Page
The Model Interpretation Dashboard includes the following information:
• Global interpretable model explanation plot
• Feature importance (Global for original features; LOCO for interpretations

with predictions and when interpreting on raw features)
• Decision tree surrogate model
• Partial dependence and individual conditional expectation plots

11.6 General Considerations
11.6.1 Machine Learning and Approximate Explanations
For years, common sense has deemed the complex, intricate formulas created by
training machine learning algorithms to be uninterpretable. While great advances
have been made in recent years to make these often nonlinear, non-monotonic,
and non-continuous machine-learned response functions more understandable
(Hall et al, 2017 [7]), it is likely that such functions will never be as directly or
universally interpretable as more traditional linear models.
Why consider machine learning approaches for inferential purposes? In general,

linear models focus on understanding and predicting average behavior, whereas
machine-learned response functions can often make accurate, but more difficult
to explain, predictions for subtler aspects of modeled phenomenon. In a sense,
linear models create very exact interpretations for approximate models. The
approach here seeks to make approximate explanations for very exact models.
It is quite possible that an approximate explanation of an exact model may
have as much, or more, value and meaning than the exact interpretations of
an approximate model. Moreover, the use of machine learning techniques for
inferential or predictive purposes does not preclude using linear models for
interpretation (Ribeiro et al, 2016 [12]).
11.6.2 The Multiplicity of Good Models in Machine Learn-

ing
It is well understood that for the same set of input variables and prediction
targets, complex machine learning algorithms can produce multiple accurate
models with very similar, but not exactly the same, internal architectures
(Brieman, 2001 [1]). This alone is an obstacle to interpretation, but when using
these types of algorithms as interpretation tools or with interpretation tools it is
important to remember that details of explanations will change across multiple
accurate models.
11.6.3 Expectations for Consistency Between Explanatory

Techniques
• The decision tree surrogate is a global, nonlinear description of the

Driverless AI model behavior. Variables that appear in the tree should
have a direct relationship with variables that appear in the global feature
importance plot. For certain, more linear Driverless AI models, variables
that appear in the decision tree surrogate model may also have large
coefficients in the global K -LIME model.
• K -LIME explanations are linear, do not consider interactions, and represent
offsets from the local linear model intercept. LOCO importance values
are nonlinear, do consider interactions, and do not explicitly consider
a linear intercept or offset. LIME explanations and LOCO importance
values are not expected to have a direct relationship but can align roughly
as both are measures of a variable’s local impact on a model’s predictions,
especially in more linear regions of the Driverless AI model’s learned
response function.
• ICE is a type of nonlinear sensitivity analysis which has a complex relation-
ship to LOCO feature importance values. Comparing ICE to LOCO can
only be done at the value of the selected variable that actually appears
in the selected row of the training data. When comparing ICE to LOCO
the total value of the prediction for the row, the value of the variable
in the selected row, and the distance of the ICE value from the average
prediction for the selected variable at the value in the selected row must
all be considered.
• ICE curves that are outside the standard deviation of partial dependence
would be expected to fall into less populated decision paths of the decision
tree surrogate; ICE curves that lie within the standard deviation of partial
dependence would be expected to belong to more common decision paths.
Viewing Explanations | 71
• Partial dependence takes into consideration nonlinear, but average, behav-

ior of the complex Driverless AI model without considering interactions.
Variables with consistently high partial dependence or partial dependence
that swings widely across an input variable’s domain will likely also have
high global importance values. Strong interactions between input variables
can cause ICE values to diverge from partial dependence values.
12 Viewing Explanations
Note: Not all explanatory functionality is available for multinomial classification
scenarios.
Driverless AI provides easy-to-read explanations for a completed model. You
can view these by clicking the Explanations button on the Model Interpretation
page. Note that this button is only available for completed experiments. Click
Close when you are done to return to the Model Interpretations page.
The UI allows you to view global, cluster-specific, and local reason codes. You
can also export the explanations to CSV.
• Global Reason Codes: To view global reason codes, select the Global
plot from the Cluster dropdown.
With Global selected, click the Explanations button beside the Cluster drop-
down.
• Cluster Reason Codes: To view reason codes for a specific cluster,

select a cluster from the Cluster dropdown.
72 | Viewing Explanations
With a cluster selected, click the Explanations button.
• Local Reason Codes by Row Number: To view local reason codes for
a specific row, select a point on the graph or type a value in the Value
field.
With a value selected, click the Explanations button in the upper-right

corner.
Viewing Explanations | 73
• Local Reason Codes by ID: To view local reason codes for a specific
row, change the dropdown to ID and then type a value in the ID field.
With a value selected, click the Explanations button in the upper-right

corner.
74 | Transform Another Dataset
13 Score on Another Dataset

After you generate a model, you can use that model to make predictions on
another dataset.
1. Click the Experiments link in the top menu and select the experiment
that you want to use.
2. On the Experiments page, click the Score on Another Dataset button.
3. Locate the new dataset that you want to score on. Note that this new
dataset must include the same columns as the dataset used in selected
experiment.
4. Click Select at the top of the screen. This immediately starts the scoring
process.
5. Click the Download Predictions button after scoring is complete.
14 Transform Another Dataset

When a training dataset is used in an experiment, Driverless AI transforms the
data into an improved, feature engineered dataset. (Refer to About Driverless
AI Transformations for more information about the transformations that are
provided in Driverless AI.) But what happens when new rows are added to your
dataset? In this case, you can specify to transform the new dataset after adding
it to Driverless AI, and the same transformations that Driverless AI applied to
the original dataset will be applied to these new rows.
Follow these steps to transform another dataset. Note that this assumes the
new dataset has been added to Driverless AI already.
Transform Another Dataset | 75
Note: Transform Another Dataset is not available for Time Series experi-
ments.
1. On the completed experiment page for the original dataset, click the
Transform Another Dataset button.
2. Select the new training dataset that you want to transform. Note that
this must have the same number columns as the original dataset.
3. In the Select drop down, specify a validation dataset to use with this
dataset, or specify to split the training data. If you specify to split the
data, then you also specify the split value (defaults to 25 percent) and
the seed (defaults to 1234). Note: To ensure the transformed dataset
respects the row order, choose a validation dataset instead of splitting
the training data. Splitting the training data will result in a shuffling of
the row order.
4. Optionally specify a test dataset. If specified, then the output also include
the final test dataset for final scoring.
5. Click Launch Transformation.
The following datasets will be available for download upon successful completion:
• Training dataset (not for cross validation)
• Validation dataset for parameter tuning
• Test dataset for final scoring. This option is available if a test dataset
was used.
76 | The Driverless AI Scoring Pipelines
15 The Driverless AI Scoring Pipelines

Driverless AI provides a Python Scoring Pipeline for experiments and interpreted
models and a MOJO (Java-based) Scoring Pipeline for experiments.
The Python Scoring Pipeline is implemented as a Python whl file. While
this allows for a single process scoring engine, the scoring service is generally
implemented as a client/server architecture and supports interfaces for TCP
and HTTP.
The MOJO Scoring Pipeline provides a standalone scoring pipeline that converts
experiments to MOJOs, which can be scored in real time.
Note: Scoring Pipelines are not available for Mac or Windows Docker image
installs.
15.1 Which Pipeline Should I Use?

Driverless AI provides a Python Scoring Pipeline, an MLI Standalone Scor-
ing Pipeline, and a MOJO Scoring Pipeline. Consider the following when
determining the scoring pipeline that you want to use.
• For all pipelines, the higher the accuracy, the slower the scoring.
• The Python Scoring Pipeline is slower but easier to use than the MOJO
scoring pipeline.
• When running the Python Scoring Pipeline:
– HTTP is easy and is supported by virtually any language. HTTP
supports RESTful calls via curl, wget, or supported packages in
various scripting languages.
– TCP is a bit more complex, though faster. TCP also requires Thrift,
which currently does not handle NAs.
• Use the MOJO Scoring Pipeline for a pure Java solution. This solution is
flexible and is faster than the Python Scoring Pipeline, but it requires a
bit more coding.
• The MLI Standalone Python Scoring Pipeline can be used to score
interpreted models but only supports k-LIME reason codes.
– For obtaining k-LIME reason codes from an MLI experiment, use
the MLI Standalone Python Scoring Pipeline. k-LIME reason codes
are available for all models.
– For obtaining Shapley reason codes from an MLI experiment, use the
DAI Standalone Python Scoring Pipeline. Shapley is only available
The Driverless AI Scoring Pipelines | 77
for XGBoost and LightGBM models. Note that obtaining Shapley

reason codes through the Python Scoring Pipeline can be time
consuming.
15.2 Driverless AI Standalone Python Scoring

Pipeline
As indicated earlier, a scoring pipeline is available after a successfully completed
experiment. This package contains an exported model and Python 3.6 source
code examples for productionizing models built using H2O Driverless AI.
The files in this package allow you to transform and score on new data in a
couple of different ways:
• From Python 3.6, you can import a scoring module, and then use the
module to transform and score on new data.
• From other languages and platforms, you can use the TCP/HTTP scoring
service bundled with this package to call into the scoring pipeline module
through remote procedure calls (RPC).
15.2.1 Python Scoring Pipeline Files
The scoring-pipeline folder includes the following notable files:

• example.py: An example Python script demonstrating how to import
and score new records.
• run example.sh: Runs example.py (also sets up a virtualenv with pre-
requisite libraries).
• tcp server.py: A standalone TCP server for hosting scoring services.
• http server.py: A standalone HTTP server for hosting scoring services.
• run tcp server.sh: Runs TCP scoring service (runs server.py).
• run http server.sh: Runs HTTP scoring service (runs server.py).
• example client.py: An example Python script demonstrating how to
communicate with the scoring server.
• run tcp client.sh: Demonstrates how to communicate with the scoring
service via TCP (runs example client.py).
• run http client.sh: Demonstrates how to communicate with the scoring
service via HTTP (using curl).
15.2.2 Quick Start - Recommended Method
This is the recommended method for running the Python Scoring Pipeline. Use
this method if:
• You have an air gapped environment with no access to the Internet.
• You are running Power.
• You want an easy quick start approach.
Prerequisites
• A valid Driverless AI license key
• A completed Driverless AI experiment
• Downloaded Python Scoring Pipeline
Running the Python Scoring Pipeline - Recommended
1. On https://www.h2o.ai/download/, download the TAR SH version of
Driverless AI (for either Linux or IBM Power).
2. Use bash to execute the download. This creates a new dai-nnn folder.
3. Change directories into the new Driverless AI folder.
cd dai-nnn directory.
4. Run the following to install the Python Scoring Pipeline for your completed
Driverless AI experiment:
./dai-env.sh pip install /path/to/your/scoring_experiment.whl
5. Run the following command to run the included scoring pipeline example:
DRIVERLESS_AI_LICENSE_KEY="pastekeyhere"
SCORING_PIPELINE_INSTALL_DEPENDENCIES=0 ./dai-env.sh
15.2.3 Quick Start - Alternative Method
This section describes an alternative method for running the Python Scoring
Pipeline. This version requires Internet access. It is also not supported on
Power machines.
Prerequisites
The following are required in order to run the downloaded scoring pipeline.
• The scoring module and scoring service are supported only on Linux
x86 64 with Python 3.6 and OpenBLAS.
• The scoring module and scoring service download additional packages
at install time and require Internet access. Depending on your network
environment, you might need to set up internet access via a proxy.
• Valid Driverless AI license. Driverless AI requires a license to be specified
in order to run the Python Scoring Pipeline.
• Apache Thrift (to run the TCP scoring service)
• Linux x86 64 environment
• Python 3.6
• libopenblas-dev (required for H2O4GPU)
• Internet access to download and install packages. Note that depending
on your environment, you may also need to set up proxy.
• OpenCL
Examples of how to install these prerequisites are below:
Installing Python 3.6 and OpenBlas Ubuntu 16.10+
$ sudo apt install python3.6 python3.6-dev python3-pip python3-dev \
python-virtualenv python3-virtualenv libopenblas-dev
Installing Python 3.6 and OpenBLAS on Ubuntu 16.4

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get update
$ sudo apt-get install python3.6 python3.6-dev python3-pip python3-dev \
python-virtualenv python3-virtualenv libopenblas-dev
Installing Conda 3.6

You can install Conda using either Anaconda or Miniconda. Refer to the links
below for more information:
• Anaconda: https://docs.anaconda.com/anaconda/install.
html
• Miniconda: https://conda.io/docs/user-guide/install/
index.html
License Specification
Driverless AI requires a license to be specified in order to run the Python Scoring
Pipeline. The license can be specified via an environment variable:
# Set DRIVERLESS_AI_LICENSE_FILE, the path to the Driverless AI license file

%env DRIVERLESS_AI_LICENSE_FILE="/home/ubuntu/license/license.sig"
# Set DRIVERLESS_AI_LICENSE_KEY, the Driverless AI license key (Base64

encoded string)
%env DRIVERLESS_AI_LICENSE_KEY="oLqLZXMI0y..."
Installing the Thrift Compiler

Thrift is required to run the scoring service in TCP mode, but it is not re-
quired to run the scoring module. The following steps are available on the
Thrift documentation site at: https://thrift.apache.org/docs/
BuildingFromSource.
$ sudo apt-get install automake bison flex g++ git libevent-dev \
libssl-dev libtool make pkg-config libboost-all-dev ant
$ wget https://github.com/apache/thrift/archive/0.10.0.tar.gz
$ tar -xvf 0.10.0.tar.gz
$ cd thrift-0.10.0
$ ./bootstrap.sh
$ ./configure
$ make
$ sudo make install
Run the following to refresh the runtime shared after installing Thrift:
$ sudo ldconfig /usr/local/lib
Running the Python Scoring Pipeline - Alternative Method

1. On the completed Experiment page, click on the Download Scoring
Pipeline button to download the scorer.zip file for this experiment onto
your local machine.
2. Unzip the scoring pipeline.
After the pipeline is downloaded and unzipped, you will be able to run the
scoring module and the scoring service.
Score from a Python Program
If you intend to score from a Python program, run the scoring module example.
(Requires Linux x86 64 and Python 3.6.)
$ export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"
$ bash run_example.sh
Score Using a Web Service

If you intend to score using a web service, run the HTTP scoring server example.
(Requires Linux x86 64 and Python 3.6.)
$ bash run_http_server.sh
$ bash run_http_client.sh
Score Using a Thrift Service

If you intend to score using a Thrift service, run the TCP scoring server example.
(Requires Linux x86 64, Python 3.6 and Thrift.)
$ bash run_tcp_server.sh
$ bash run_tcp_client.sh
Note: By default, the run *.sh scripts mentioned above create a virtual
environment using virtualenv and pip, within which the Python code is executed.
The scripts can also leverage Conda (Anaconda/Mininconda) to create Conda
virtual environment and install required package dependencies. The package
manager to use is provided as an argument to the script.
# to use conda package manager
$ bash run_example.sh --pm conda
# to use pip package manager

$ bash run_example.sh --pm pip
Note: If you experience errors while running any of the above scripts, please
check to make sure your system has a properly installed and configured Python
3.6 installation. Refer to the Troubleshooting Python Environment Issues section
at the end of this chapter to see how to set up and test the scoring module
using a cleanroom Ubuntu 16.04 virtual machine.
15.2.4 The Python Scoring Module
The scoring module is a Python module bundled into a standalone wheel file
(name scoring *.whl). All the prerequisites for the scoring module to work
correctly are listed in the requirements.txt file. To use the scoring module, all
you have to do is create a Python virtualenv, install the prerequisites, and then
import and use the scoring module as follows:
# See ’example.py’ for complete example.
from scoring_487931_20170921174120_b4066 import Scorer
scorer = Scorer() # Create instance.
score = scorer.score([ # Call score()
7.416, # sepal_len
3.562, # sepal_wid
1.049, # petal_len
2.388, # petal_wid
])
The scorer instance provides the following methods (and more):

• score(list): Score one row (list of values).
• score batch(df): Score a Pandas dataframe.
• fit transform batch(df): Transform a Pandas dataframe.
• get target labels(): Get target column labels (for classification problems).
The process of importing and using the scoring module is demonstrated by the
bash script run example.sh, which effectively performs the following steps:
# See ’run_example.sh’ for complete example.
$ virtualenv -p python3.6 env
$ source env/bin/activate
$ pip install -r requirements.txt
$ python example.py
15.2.5 The Scoring Service
The scoring service hosts the scoring module as an HTTP or TCP service.
Doing this exposes all the functions of the scoring module through remote
procedure calls (RPC). In effect, this mechanism allows you to invoke scoring
functions from languages other than Python on the same computer or from
another computer on a shared network or on the Internet.
The scoring service can be started in two ways:
• In TCP mode, the scoring service provides high-performance RPC calls
via Apache Thrift (https://thrift.apache.org/) using a binary
wire protocol.
• In HTTP mode, the scoring service provides JSON-RPC 2.0 calls served
by Tornado (http://www.tornadoweb.org).
Scoring operations can be performed on individual rows (row-by-row) or in
batch mode (multiple rows at a time).
Scoring Service - TCP Mode (Thrift)
The TCP mode allows you to use the scoring service from any language supported
by Thrift, including C, C++, C#, Cocoa, D, Dart, Delphi, Go, Haxe, Java,
Node.js, Lua, perl, PHP, Python, Ruby and Smalltalk.
To start the scoring service in TCP mode, you will need to generate the Thrift
bindings once, then run the server:
# See ’run_tcp_server.sh’ for complete example.

$ thrift --gen py scoring.thrift
$ python tcp_server.py --port=9090
Note that the Thrift compiler is only required at build-time. It is not a run time
dependency, i.e. once the scoring services are built and tested, you do not need
to repeat this installation process on the machines where the scoring services
are intended to be deployed.
To call the scoring service, simply generate the Thrift bindings for your language
of choice, then make RPC calls via TCP sockets using Thrift’s buffered transport
in conjunction with its binary protocol.
# See ’run_tcp_client.sh’ for complete example.
# See ’example_client.py’ for complete example.

socket = TSocket.TSocket(’localhost’, 9090)
transport = TTransport.TBufferedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ScoringService.Client(protocol)
transport.open()
row = Row()
row.sepalLen = 7.416 # sepal_len
row.sepalWid = 3.562 # sepal_wid
row.petalLen = 1.049 # petal_len
row.petalWid = 2.388 # petal_wid
scores = client.score(row)
transport.close()
You can reproduce the exact same result from other languages, e.g. Java:
$ thrift --gen java scoring.thrift
// Dependencies:
// commons-codec-1.9.jar
// commons-logging-1.2.jar
// httpclient-4.4.1.jar
// httpcore-4.4.1.jar
// libthrift-0.10.0.jar
// slf4j-api-1.7.12.jar
import ai.h2o.scoring.Row;
import ai.h2o.scoring.ScoringService;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;
import java.util.List;
public class Main {

public static void main(String[] args) {
try {
TTransport transport = new TSocket("localhost", 9090);
transport.open();
ScoringService.Client client = new ScoringService.Client(

new TBinaryProtocol(transport));
Row row = new Row(7.642, 3.436, 6.721, 1.020);

List<Double> scores = client.score(row);
System.out.println(scores);
transport.close();
} catch (TException ex) {
ex.printStackTrace();
}
}
}
Scoring Service - HTTP Mode (JSON-RPC 2.0)
The HTTP mode allows you to use the scoring service using plaintext JSON-
RPC calls. This is usually less performant compared to Thrift, but has the
advantage of being usable from any HTTP client library in your language of
choice, without any dependency on Thrift.
See http://www.jsonrpc.org/specification for JSON-RPC docu-

mentation.
To start the scoring service in HTTP mode:

# See ’run_http_server.sh’ for complete example.
$ python http_server.py --port=9090
To invoke scoring methods, compose a JSON-RPC message and make a HTTP

POST request to http://host:port/rpc as follows:
# See ’run_http_client.sh’ for complete example.
$ curl http://localhost:9090/rpc \
--header "Content-Type: application/json" \
--data @- <<EOF
{
"id": 1,
"method": "score",
"params": {
"row": [ 7.486, 3.277, 4.755, 2.354 ]
}
}
EOF
Similarly, you can use any HTTP client library to reproduce the above result.
For example, from Python, you can use the requests module as follows:
import requests
row = [7.486, 3.277, 4.755, 2.354]
req = dict(id=1, method=’score’, params=dict(row=row))
res = requests.post(’http://localhost:9090/rpc’, data=req)
print(res.json()[’result’])
15.2.6 Python Scoring Pipeline FAQ
Why am I getting a ”TensorFlow is disabled” message when I run the

Python Scoring Pipeline?
If you ran an experiment when TensorFlow was enabled and then attempt to
run the Python Scoring Pipeline, you may receive a message similar to the
following:
TensorFlow is disabled. To enable, export DRIVERLESS_AI_ENABLE_TENSORFLOW=1
or set enable_tensorflow=true in config.toml.
This can be fixed by enabling the DRIVERLESS AI ENABLE TENSORFLOW

flag when running the Python Scoring Pipeline. For example:
$ DRIVERLESS_AI_ENABLE_TENSORFLOW=1 bash run_example.sh
15.2.7 Troubleshooting Python Environment Issues
The following instructions describe how to set up a cleanroom Ubuntu 16.04

virtual machine to test that this scoring pipeline works correctly.
Prerequisites:
• Install Virtualbox: sudo apt-get install virtualbox
• Install Vagrant: https://www.vagrantup.com/downloads.html
1. Create configuration files for Vagrant.
• bootstrap.sh: contains commands to set up Python 3.6 and Open-
BLAS.
• Vagrantfile: contains virtual machine configuration instructions for
Vagrant and VirtualBox.
----- bootstrap.sh -----
#!/usr/bin/env bash
sudo apt-get -y update

sudo apt-get -y install apt-utils build-essential python-software-
properties software-properties-common zip libopenblas-dev
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt-get update -yqq
sudo apt-get install -y python3.6 python3.6-dev python3-pip python3-dev
python-virtualenv python3-virtualenv
# end of bootstrap.sh
----- Vagrantfile -----

# -*- mode: ruby -*-

# vi: set ft=ruby :
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision :shell, path: "bootstrap.sh", privileged: false
config.vm.hostname = "h2o"
config.vm.provider "virtualbox" do |vb|
vb.memory = "4096"
end
end
# end of Vagrantfile
2. Launch the VM and SSH into it. Note that we are also placing the scoring
pipeline in the same directory so that we can access it later inside the
VM.
cp /path/to/scorer.zip .
vagrant up
vagrant ssh
3. Test the scoring pipeline inside the virtual machine.

cp /vagrant/scorer.zip .
unzip scorer.zip
cd scoring-pipeline/
export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"
bash run_example.sh
At this point, you should see scores printed out on the terminal. If not, contact
us at [email protected].
15.3 Driverless AI MLI Standalone Scoring Pack-

age
This package contains an exported model and Python 3.6 source code examples
for productionizing models built using H2O Driverless AI Machine Learning
Interpretability (MLI) tool. This is only available for interpreted models.
The files in this package allow you to obtain reason codes for a given row of
data a couple of different ways:
• From Python 3.6, you can import a scoring module, and then use the
module to transform and score on new data.
• From other languages and platforms, you can use the TCP/HTTP scoring
service bundled with this package to call into the scoring pipeline module
through remote procedure calls (RPC).
15.3.1 MLI Python Scoring Package Files
The scoring-pipeline-mli folder includes the following notable files:

• example.py: An example Python script demonstrating how to import
and interpret new records.
• run example.sh: Runs example.py (This also sets up a virtualenv with
prerequisite libraries.)
• run example shapley.sh: Runs example shapley.py. This compares K-
LIME and Driverless AI Shapley reason codes.
• tcp server.py: A standalone TCP server for hosting MLI services.
• http server.py: A standalone HTTP server for hosting MLI services.
• run tcp server.sh: Runs the TCP scoring service (specifically, tcp server.py).
• run http server.sh: Runs HTTP scoring service (runs http server.py).
• example client.py: An example Python script demonstrating how to
communicate with the MLI server.
• example shapley.py: An example Python script demonstrating how to
compare K-LIME and Driverless AI Shapley reason codes.
• run tcp client.sh: Demonstrates how to communicate with the MLI
service via TCP (runs example client.py).
• run http client.sh: Demonstrates how to communicate with the MLI
service via HTTP (using curl).
15.3.2 Quick Start - Recommended Method
This is the recommended method for running the MLI Scoring Pipeline. Use
this method if:
• You have an air gapped environment with no access to the Internet.
• You are running Power.
• You want an easy quick start approach.
Prerequisites
• A valid Driverless AI license key.
• A completed Driverless AI experiment.
• Downloaded MLI Scoring Pipeline.
Running the MLI Scoring Pipeline - Recommended

1. Download the TAR SH version of Driverless AI from https://www.h2o.ai/download/
(for either Linux or IBM Power).
2. Use bash to execute the download. This creates a new dai-nnn folder.
3. Change directories into the new Driverless AI folder.
4. Run the following to install the Python Scoring Pipeline for your completed
Driverless AI experiment:
./dai-env.sh pip install /path/to/your/scoring_experiment.whl
5. Run the following command to run the included scoring pipeline example:
DRIVERLESS_AI_LICENSE_KEY="pastekeyhere"
SCORING_PIPELINE_INSTALL_DEPENDENCIES=0 ./dai-env.sh /path/to/
your/run_example.sh
15.3.3 Quick Start - Alternative Method
This section describes an alternative method for running the MLI Standalone
Scoring Pipeline. This version requires Internet access. It is also not supported
on Power machines.
15.3.4 Prerequisites
• Valid Driverless AI license.

• The scoring module and scoring service are supported only on Linux with
Python 3.6 and OpenBLAS.
• The scoring module and scoring service download additional packages
at install time and require internet access. Depending on your network
environment, you might need to set up internet access via a proxy.
• Apache Thrift (to run the scoring service in TCP mode)
Installing Python 3.6
# Installing Python 3.6 on Ubuntu 16.10+:
$ sudo apt install python3.6 python3.6-dev python3-pip python3-dev \
# Installing Python3.6 on Ubuntu 16.04

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get update
$ sudo apt-get install python3.6 python3.6-dev python3-pip python3-dev \
# Installing Conda 3.6

You can install Conda using either Anaconda or Miniconda. Refer to the links
below for more information:
- Anaconda - https://docs.anaconda.com/anaconda/install.html
- Miniconda - https://conda.io/docs/user-guide/install/index.html
Installing the Thrift Compiler

Refer to Thrift documentation at https://thrift.apache.org/docs/
BuildingFromSource for more information.
$ sudo apt-get install automake bison flex g++ git libevent-dev \
libssl-dev libtool make pkg-config libboost-all-dev ant
$ wget https://github.com/apache/thrift/archive/0.10.0.tar.gz
$ tar -xvf 0.10.0.tar.gz
$ cd thrift-0.10.0
$ ./bootstrap.sh
$ ./configure
$ make
$ sudo make install
Run the following to refresh the runtime shared after installing Thrift.
$ sudo ldconfig /usr/local/lib
Running the MLI Scoring Pipeline - Alternative Method

Before running the quickstart examples, be sure that the MLI Scoring Package
is already downloaded and unzipped.
1. On the MLI page, click the Scoring Pipeline button.
2. Unzip the scoring pipeline, and run the following examples in the scoring-
pipeline-mli folder.
Run the scoring module example. (This requires Linux x86 64 and Python 3.6.)
$ bash run_example.sh
Run the TCP scoring server example. Use two terminal windows. (This requires
Linux x86 64, Python 3.6 and Thrift.)
$ bash run_tcp_server.sh
$ bash run_tcp_client.sh
Run the HTTP scoring server example. Use two terminal windows. (This
requires Linux x86 64, Python 3.6 and Thrift.)
$ bash run_http_server.sh
$ bash run_http_client.sh
15.3.5 MLI Python Scoring Module
The MLI scoring module is a Python module bundled into a standalone wheel
file (name scoring *.whl). All the prerequisites for the scoring module to work
correctly are listed in the requirements.txt file. To use the scoring module, all
you have to do is create a Python virtualenv, install the prerequisites, and then
import and use the scoring module as follows:
----- See ’example.py’ for complete example. -----
from scoring_487931_20170921174120_b4066 import Scorer
scorer = KLimeScorer() # Create instance.
score = scorer.score_reason_codes([ # Call score_reason_codes()
7.416, # sepal_len
3.562, # sepal_wid
1.049, # petal_len
2.388, # petal_wid
])
The scorer instance provides the following methods:

• score reason codes(list): Get KLime reason codes for one row
(list of values).
• score reason codes batch(dataframe): Takes and outputs a
Pandas Dataframe
• get column names(): Get the input column names
• get reason code column names(): Get the output column names
The process of importing and using the scoring module is demonstrated by the
bash script run example.sh, which effectively performs the following steps:
----- See ’run_example.sh’ for complete example. -----
$ virtualenv -p python3.6 env
$ source env/bin/activate
$ pip install -r requirements.txt
$ python example.py
15.3.6 K-LIME vs Shapley Reason Codes
There are times when the K-LIME model score is not close to the Driverless
AI model score. In this case it may be better to use reason codes using the
Shapley method on the Driverless AI model. Note: The reason codes from
Shapley will be in the transformed feature space.
To see an example of using both K-LIME and Driverless AI Shapley reason
codes in the same Python session, run:
$ bash run_example_shapley.sh
For this batch script to succeed, MLI must be run on a Driverless AI model.
If you have run MLI in standalone (external model) mode, there will not be a
Driverless AI scoring pipeline.
If MLI was run with transformed features, the Shapley example scripts will not
be exported. You can generate exact reason codes directly from the Driverless
AI model scoring pipeline.
15.3.7 MLI Scoring Service Overview
The MLI scoring service hosts the scoring module as a HTTP or TCP service.
Doing this exposes all the functions of the scoring module through remote
procedure calls (RPC).
In effect, this mechanism allows you to invoke scoring functions from languages
other than Python on the same computer, or from another computer on a
shared network or the internet.
The scoring service can be started in two ways:
• In TCP mode, the scoring service provides high-performance RPC calls via
Apache Thrift (https://thrift.apache.org/) using a binary wire protocol.
• In HTTP mode, the scoring service provides JSON-RPC 2.0 calls served
by Tornado (http://www.tornadoweb.org).
Scoring operations can be performed on individual rows (row-by-row) using
score or in batch mode (multiple rows at a time) using score batch. Both
functions allow you to specify pred contribs=[True|False] to get MLI
predictions (KLime/Shapley) on a new dataset. See the example shapley.py
file for more information.
MLI Scoring Service - TCP Mode (Thrift)
The TCP mode allows you to use the scoring service from any language supported
by Thrift, including C, C++, C#, Cocoa, D, Dart, Delphi, Go, Haxe, Java,
Node.js, Lua, perl, PHP, Python, Ruby and Smalltalk.
To start the scoring service in TCP mode, you will need to generate the Thrift
bindings once, then run the server:
----- See ’run_tcp_server.sh’ for complete example. -----
$ python tcp_server.py --port=9090
Note that the Thrift compiler is only required at build-time. It is not a run time
dependency, i.e. once the scoring services are built and tested, you do not need
to repeat this installation process on the machines where the scoring services
are intended to be deployed.
To call the scoring service, simply generate the Thrift bindings for your language
of choice, then make RPC calls via TCP sockets using Thrift’s buffered transport
in conjunction with its binary protocol.
----- See ’run_tcp_client.sh’ for complete example. -----
----- See ’example_client.py’ for complete example. -----

socket = TSocket.TSocket(’localhost’, 9090)
transport = TTransport.TBufferedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ScoringService.Client(protocol)
transport.open()
row = Row()
row.sepalLen = 7.416 # sepal_len
row.sepalWid = 3.562 # sepal_wid
row.petalLen = 1.049 # petal_len
row.petalWid = 2.388 # petal_wid
scores = client.score_reason_codes(row)
transport.close()
You can reproduce the exact same result from other languages, e.g. Java:
$ thrift --gen java scoring.thrift
// Dependencies:
// commons-codec-1.9.jar
// commons-logging-1.2.jar
// httpclient-4.4.1.jar
// httpcore-4.4.1.jar
// libthrift-0.10.0.jar
// slf4j-api-1.7.12.jar
import ai.h2o.scoring.Row;
import ai.h2o.scoring.ScoringService;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;
import java.util.List;
public class Main {

public static void main(String[] args) {
try {
TTransport transport = new TSocket("localhost", 9090);
transport.open();
ScoringService.Client client = new ScoringService.Client(

new TBinaryProtocol(transport));
Row row = new Row(7.642, 3.436, 6.721, 1.020);

List<Double> scores = client.score_reason_codes(row);
System.out.println(scores);
transport.close();
} catch (TException ex) {
ex.printStackTrace();
}
}
}
Scoring Service - HTTP Mode (JSON-RPC 2.0)

The HTTP mode allows you to use the scoring service using plaintext JSON-
RPC calls. This is usually less performant compared to Thrift, but has the
advantage of being usable from any HTTP client library in your language of
choice, without any dependency on Thrift.
For JSON-RPC documentation, see http://www.jsonrpc.org/specification .
To start the scoring service in HTTP mode:
----- See ’run_http_server.sh’ for complete example. -----
$ python http_server.py --port=9090
To invoke scoring methods, compose a JSON-RPC message and make a HTTP

POST request to http://host:port/rpc as follows:
----- See ’run_http_client.sh’ for complete example. -----
$ curl http://localhost:9090/rpc \
--header "Content-Type: application/json" \
--data @- <<EOF
{
"id": 1,
"method": "score_reason_codes",
"params": {
"row": [ 7.486, 3.277, 4.755, 2.354 ]
}
}
EOF
Similarly, you can use any HTTP client library to reproduce the above result.
For example, from Python, you can use the requests module as follows:
import requests
row = [7.486, 3.277, 4.755, 2.354]
req = dict(id=1, method=’score_reason_codes’, params=dict(row=row))
res = requests.post(’http://localhost:9090/rpc’, data=req)
print(res.json()[’result’])
15.4 Driverless AI MOJO Scoring Pipeline

Note: MOJOs are currently not available for TensorFlow, RuleFit, or FTRL
models.
For completed experiments, Driverless AI converts models to MOJOs (Model
Objects, Optimized), which can be deployed for scoring in real time. A MOJO
is a scoring engine that can be deployed in any Java environment for scoring
in real time. The MOJO Scoring Pipeline is available as either a pure Java
solution or a C++-based solution.
Keep in mind that, similar to H2O-3, MOJOs are tied to experiments. Ex-
periments and MOJOs are not automatically upgraded when Driverless AI is
upgraded.
The following are required in order to run the MOJO scoring pipeline.
• Java 7 runtime (JDK 1.7) or newer.
• Valid Driverless AI license. You can download the license.sig file
from the machine hosting Driverless AI (usually in the license folder).
Copy the license file into the downloaded mojo-pipeline folder.
• mojo2-runtime.jar file. This is available from the top navigation menu in
the Driverless AI UI and in the downloaded mojo-pipeline.zip file for an
experiment.
License Specification
Driverless AI requires a license to be specified in order to run the MOJO Scoring
Pipeline. The license can be specified in one of the following ways:
• Via an environment variable:
– DRIVERLESS AI LICENSE FILE: Path to the Driverless AI li-
cense file, or
– DRIVERLESS AI LICENSE KEY: The Driverless AI license key
(Base64 encoded string)
• Via a system property of JVM (-D option):
– ai.h2o.mojos.runtime.license.file: Path to the Driver-
less AI license file, or
– ai.h2o.mojos.runtime.license.key: The Driverless AI
license key (Base64 encoded string)
• Via an application classpath:
– The license is loaded from a resource called /license.sig.
– The default resource name can be changed via the JVM system
property ai.h2o.mojos.runtime.license.filename.
For example:
$ java -Dai.h2o.mojos.runtime.license.file=/etc/dai/license.sig -cp mojo2-
runtime.jar ai.h2o.mojos.ExecuteMojo pipeline.mojo example.csv
15.4.2 Enabling the MOJO Scoring Pipeline
The MOJO Scoring Pipeline is disabled by default. As a result, a MOJO will

have to be built for each desired experiment by clicking on the Build MOJO
Scoring Pipeline button:
To enable MOJO Scoring Pipelines for each experiment, stop Driverless AI, then
restart using the DRIVERLESS AI MAKE MOJO SCORING PIPELINE=1 flag.
(Refer to the Config.toml File section in the User Guide. for more information.)
For example:
nvidia-docker run \
--add-host name.node:172.16.2.186 \
-e DRIVERLESS_AI_MAKE_MOJO_SCORING_PIPELINE=1 \
-p 12345:12345 \
--init -it --rm \
-v /tmp/dtmp/:/tmp \
-v /tmp/dlog/:/log \
-u $(id -u):$(id -g) \
opsh2oai/h2oai-runtime
Or you can change the value of make mojo scoring pipeline to true
in the config.toml file and specify that file when restarting Driverless AI.
15.4.3 MOJO Scoring Pipeline Files
The mojo-pipeline folder includes the following files:

• run example.sh: An bash script to score a sample test set.
• pipeline.mojo: Standalone scoring pipeline in MOJO format.
• mojo2-runtime.jar: MOJO Java runtime.
• example.csv: Sample test set (synthetic, of the correct format).
15.4.4 Quickstart
Before running the quickstart examples, be sure that the MOJO scoring pipeline
is already downloaded and unzipped:
1. On the completed Experiment page, click on the Download Scoring
Pipeline button to download the scorer.zip file for this experiment onto
your local machine.
Note: This button is Build MOJO Scoring Pipeline if the MOJO

Scoring Pipeline is disabled.
2. To score all rows in the sample test set (example.csv) with the
MOJO pipeline (pipeline.mojo) and license stored in the environment
variable DRIVERLESS AI LICENSE KEY:
$ bash run_example.sh license.sig
3. To score a specific test set (example.csv) with the MOJO pipeline

(pipeline.mojo) and the license file (license.sig):
$ bash run_example.sh pipeline.mojo example.csv license.sig
4. To run Java application for data transformation directly:

$ java -Dai.h2o.mojos.runtime.license.file=license.sig -cp mojo2-
runtime.jar ai.h2o.mojos.ExecuteMojo pipeline.mojo example.csv
15.5 MOJO Scoring Pipelines

As indicated previously, the MOJO Scoring Pipeline provides a standalone
scoring pipeline that converts experiments to MOJOs, which can be scored
in real time. The MOJO Scoring Pipeline is available as either a pure Java
solution or a C++-based solution.
Note: MOJOs are currently not available for TensorFlow, RuleFit, or FTRL
models.
15.5.1 MOJO Scoring Pipeline - Java Solution
1. Open a new terminal window and change directories to the experiment

folder:
$ cd experiment
2. Create your main program in the experiment folder by creating a new

file called Main.java (for example, using vim Main.java). Include
the following contents.
import java.io.IOException;
import ai.h2o.mojos.runtime.MojoPipeline;
import ai.h2o.mojos.runtime.frame.MojoFrame;
import ai.h2o.mojos.runtime.frame.MojoFrameBuilder;
import ai.h2o.mojos.runtime.frame.MojoRowBuilder;
import ai.h2o.mojos.runtime.utils.SimpleCSV;
public class Main {
public static void main(String[] args) throws IOException {

// Load model and csv
MojoPipeline model = MojoPipeline.loadFrom("pipeline.mojo");
// Get and fill the input columns

MojoFrameBuilder frameBuilder = model.getInputFrameBuilder();
MojoRowBuilder rowBuilder = frameBuilder.getMojoRowBuilder();
rowBuilder.setValue("AGE", "68");
rowBuilder.setValue("RACE", "2");
rowBuilder.setValue("DCAPS", "2");
rowBuilder.setValue("VOL", "0");
rowBuilder.setValue("GLEASON", "6");
frameBuilder.addRow(rowBuilder);
// Create a frame which can be transformed by MOJO pipeline

MojoFrame iframe = frameBuilder.toMojoFrame();
// Transform input frame by MOJO pipeline

MojoFrame oframe = model.transform(iframe);
// ‘MojoFrame.debug()‘ can be used to view the contents of a Frame
// oframe.debug();
// Output prediction as CSV

SimpleCSV outCsv = SimpleCSV.read(oframe);
outCsv.write(System.out);
}
}
3. Compile the source code:

$ javac -cp mojo2-runtime.jar -J-Xms2g -J-XX:MaxPermSize=128m Main.java
4. Run the MOJO example:

# Linux and OS X users
$ java -Dai.h2o.mojos.runtime.license.file=license.sig -cp .:mojo2-
runtime.jar Main
# Windows users
$ java -Dai.h2o.mojos.runtime.license.file=license.sig -cp .;mojo2-
runtime.jar Main
5. The following output is displayed:

CAPSULE.True
0.5442205910902282
15.5.2 MOJO Scoring Pipeline - C++ Solution
The C++ Scoring Pipeline is provided as R and Python packages for the
protobuf-based MOJO2 protocol. The packages are self contained, so no
additional software is required. Simply build the MOJO Scoring Pipeline and
begin using your preferred method. To download the MOJO Scoring Pipeline
onto your local machine, click the Download MOJO Scoring Pipeline button,
then click the same button again in the pop-up menu that appears. Refer to
the provided instructions for Java, Python, or R.
Notes:
• MOJOs are currently not available for TensorFlow, RuleFit, or FTRL
models.
• The Download MOJO Scoring Pipeline button appears as Build
MOJO Scoring Pipeline if the MOJO Scoring Pipeline is disabled.
Examples
The following examples show how to use the R and Python APIs of the C++
MOJO runtime.
R Example
Prerequisites
• methods
• Rcpp (≥1.0.0)
• data.table
# Load the MOJO
{.r}
library(daimojo)
m <- load.mojo("../data/dai/pipeline.mojo")
create.time(m)
## [1] "2018-12-17 22:00:24 UTC"
uuid(m)
## [1] "65875c15-943a-4bc0-a162-b8984fe8e50d"
# Load data and make predictions

{.r}
library(data.table)
col_class <- setNames(feature.types(m), feature.names(m))
d <- fread("../data/dai/example.csv", colClasses=col_class)
predict(m, d)
## label.B label.M
## 1 0.08287659 0.91712341
## 2 0.77655075 0.22344925
## 3 0.58438434 0.41561566
## 4 0.10570505 0.89429495
## 5 0.01685609 0.98314391
## 6 0.23656610 0.76343390
## 7 0.17410333 0.82589667
## 8 0.10157948 0.89842052
## 9 0.13546191 0.86453809
## 10 0.94778244 0.05221756
Python Example
Prerequisites
• Python 3.6
• datatable. Run the following to install:
pip install https://s3.amazonaws.com/h2o-release/datatable/stable/
datatable-0.8.0/datatable-0.8.0-cp36-cp36m-linux_x86_64.whl
• Python MOJO runtime. Run the following after downloading from the
GUI (Note: For PowerPC, replace x86 64 with ppc64le):
pip install daimojo-2.0.1+master.478-cp36-cp36m-linux\_x86\_64.whl
• Driverless AI License (either file or environment variable)

# import the daimojo model package
import daimojo.model
# specify the location of the MOJO

m = daimojo.model("../data/dai/pipeline.mojo")
# retrieve the full path of the MOJO

m.modelfile
’/home/<user>/Desktop/mojo2/data/dai/pipeline.mojo’
# retrieve the UUID

m.uuid
# verify the MOJO version

m.mojo_version
’2.19.3’
# view the creation time

m.created_time
’Mon May 6 14:00:24 2019’
# retrive a list of missing values

m.missing_values
[’’,
’?’,
’None’,
’nan’,
’NA’,
’N/A’,
’unknown’,
’inf’,
’-inf’,
’1.7976931348623157e+308’,
’-1.7976931348623157e+308’]
# retrieve the feature names

m.feature_names
[’clump_thickness’,
’uniformity_cell_size’,
’uniformity_cell_shape’,
’marginal_adhesion’,
’single_epithelial_cell_size’,
’bare_nuclei’,
’bland_chromatin’,
’normal_nucleoli’,
’mitoses’]
# retrieve the feature types

m.feature_types
[’float32’,
’float32’,
’float32’,
’float32’,
’float32’,
’float32’,
’float32’,
’float32’,
’float32’]
# retrieve the output names

m.output_names
[’label.B’, ’label.M’]
# retrieve the output types

[’float64’, ’float64’]
# import the datatable module

import datatable as dt
# parse the example.csv file

pydt = dt.fread("../data/dai/example.csv", na_strings=m.missing_values)
# retrieve the column types

pydt.stypes
(stype.float64,
stype.float64,
stype.float64,
stype.float64,
stype.float64,
stype.float64,
stype.float64,
stype.float64,
stype.float64)
# make predictions on the example.csv file

res = m.predict(pydt)
# retrieve the predictions

res
label.B. label.M
0 0.0828766 0.917123
1 0.776551 0.223449
2 0.584384 0.415616
3 0.105705 0.894295
4 0.0168561 0.983144
5 0.236566 0.763434
6 0.174103 0.825897
7 0.101579 0.898421
8 0.135462 0.864538
9 0.947782 0.0522176
[10 rows by 2 columns]
# retrieve the prediction column names

res.names
# (’label.B’, ’label.M’)
# retrieve the prediction column types

Deployment | 101
res.stypes
(stype.float64, stype.float64)
# convert datatable results to common data types

# res.to_pandas() # need pandas
# res.to_numpy() # need numpy
res.to_list()
16 Deployment
Driverless AI can deploy the MOJO scoring pipeline for you to test and/or to
integrate into a final product.
Notes:
• This section describes how to deploy a MOJO scoring pipeline and assumes
that a MOJO scoring pipeline exists.
• This is an early feature that will continue to support additional deploy-

ments.
16.1 Additional Resources

Refer to the https://github.com/h2oai/dai-deployment-templates repository
to see different deployment templates for Driverless AI scorers, including AWS
Lambda, KDB, and Local REST.
16.2 Deployments Overview Page

All of the MOJO scoring pipeline deployments are available in the Deployments
Overview page, which is available from the top menu. This page lists all active
deployments and the information needed to access the respective endpoints. In
addition, it allows you to stop any deployments that are no longer needed.
102 | Deployment
16.3 AWS Lambda Deployment

16.3.1 Driverless AI Prerequisites
To deploy a MOJO scoring pipeline as an AWS lambda function, the MOJO

pipeline archive has to be created first by choosing the Build MOJO Scoring
Pipeline option on the completed experiment page.
In addition, the Terraform tool has to be installed on the system running
Driverless AI. The tool is included in the Driverless AI Docker images but not in
native install packages. To install Terraform, please follow steps on Terraform
installation. Note: Terraform is not available on every platform. In particular,
there is no Power build, so AWS Lambda Deployment is currently not supported
on Power installations of Driverless AI.
16.3.2 AWS Access Permissions Prerequisites
The following AWS access permissions need to be provided to the role in order
for Driverless AI Lambda deployment to succeed.
• AWSLambdaFullAccess
• IAMFullAccess
• AmazonAPIGatewayAdministrator
The policy can be further stripped down to restrict Lambda and S3 rights using
the JSON policy definition as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:GetPolicyVersion",
"iam:DeletePolicy",
"iam:CreateRole",
"iam:AttachRolePolicy",
"iam:ListInstanceProfilesForRole",
Deployment | 103
"iam:PassRole",
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:GetRole",
"iam:GetPolicy",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:ListPolicyVersions"
],
"Resource": [
"arn:aws:iam::*:role/h2oai*",
"arn:aws:iam::*:policy/h2oai*"
]
},
{
"Effect": "Allow",
"Action": "apigateway:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"lambda:CreateFunction",
"lambda:ListFunctions",
"lambda:InvokeFunction",
"lambda:GetFunction",
"lambda:UpdateFunctionConfiguration",
"lambda:DeleteFunctionConcurrency",
"lambda:RemovePermission",
"lambda:UpdateFunctionCode",
"lambda:AddPermission",
"lambda:ListVersionsByFunction",
"lambda:GetFunctionConfiguration",
"lambda:DeleteFunction",
"lambda:PutFunctionConcurrency",
"lambda:GetPolicy"
],
"Resource": "arn:aws:lambda:*:*:function:h2oai*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::h2oai*/*",
"arn:aws:s3:::h2oai*"
]
}
]
}
16.3.3 Deploying the Lambda
Once the MOJO pipeline archive is ready, Driverless AI provides a Deploy

option on the completed experiment page.
Note: This button is not available on PPC64LE environments.
104 | Deployment
This option opens a new dialog for setting the AWS account credentials (or use
those supplied in the Driverless AI configuration file or environment variables),
AWS region, and the desired deployment name (which must be unique per
Driverless AI user and AWS account used).
Amazon Lambda deployment parameters:

• Deployment Name: A unique name of the deployment. By default,
Driverless AI offers a name based on the name of the experiment and the
deployment type. This has to be unique both for Driverless AI user and
the AWS account used.
• Region: The AWS region to deploy the MOJO scoring pipeline to. It
makes sense to choose a region geographically close to any client code
calling the endpoint in order to minimize request latency. (See also AWS
Regions and Availability Zones.)
• Use AWS environment variables: If enabled, the AWS credentials are
taken from the Driverless AI configuration file. This would usually be
entered by the Driverless AI installation administrator.
• AWS Access Key ID and AWS Secret Access Key: Credentials to
access the AWS account. This pair of secrets identifies the AWS user
and the account and can be obtained from the AWS account console.
16.3.4 Testing the Lambda Deployment
On a successful deployment, all the information needed to access the new

endpoint (URL and an API Key) is printed, and the same information is available
in the Deployments Overview Page after clicking on the deployment row.
Deployment | 105
Note that the actual scoring endpoint is located at the path /score. In addition,
to prevent DDoS and other malicious activities, the resulting AWS lambda is
protected by an API Key, i.e., a secret that has to be passed in as a part of the
request using the x-api-key HTTP header.
The request is a JSON object containing attributes:
• fields: A list of input column names that should correspond to the training
data columns.
• rows: A list of rows that are in turn lists of cell values to predict the
target values for.
• (optional) includeFieldsInOutput: A list of input columns that should
be included in the output.
An example request providing 2 columns on the input and asking to get one
column copied to the output looks as follows:
{
"fields": [
"age", "salary"
],
"includeFieldsInOutput": [
"salary"
],
"rows": [
[
"48.0", "15000.0"
],
[
"35.0", "35000.0"
],
[
"18.0", "22000.0"
]
]
}
Assuming the request is stored locally in a file named test.json, the request to
the endpoint can be sent, e.g., using the curl utility, as follows:
$ URL={place the endpoint URL here}
$ API_KEY={place the endpoint API key here}
$ curl \
-X POST \
-H "x-api-key: ${API_KEY}" \
-d @test.json ${URL}/score
The response is a JSON object with a single attribute “score“, which contains
the list of rows with the optional copied input values and the predictions.
For the example above with a two class target field, the result is likely to look
something like the following snippet. The particular values would of course
depend on the scoring pipeline:
106 | Deployment
{
"score": [
[
"48.0",
"0.6240277982943945",
"0.045458571508101536",
],
[
"35.0",
"0.7209441819603676",
"0.06299909138586585",
],
[
"18.0",
"0.7209441819603676",
"0.06299909138586585",
]
]
}
16.3.5 AWS Deployment Issues
We create a new S3 bucket per AWS Lambda deployment. The bucket names
have to be unique throughout AWS S3, and one user can create a maximum
of 100 buckets. Therefore, we recommend setting the bucket name used for
deployment with the deployment aws bucket name config option.
16.4 REST Server Deployment

This section describes how to deploy the trained MOJO scoring pipeline as a
local Representational State Transfer (REST) Server.
• Driverless AI MOJO Scoring Pipeline: To deploy a MOJO scoring pipeline

as a Local REST Scorer, the MOJO pipeline archive has to be created first
by choosing the Build MOJO Scoring Pipeline option on the completed
experiment page.
• When using a firewall or a virtual private cloud (VPC), the ports that are
used by the REST server must be exposed. Note that Docker users must
forward the REST server’s ports from the Docker container to the host
before proceeding.
• Ensure that you have enough memory and CPUs to run the REST scorer.
Typically, a good estimation for the amount of required memory is 12 times
the size of the pipeline.mojo file. For example, a 100MB pipeline.mojo
Deployment | 107
file will require approximately 1200MB of RAM. (Note: To conveniently

view in-depth information about your system in Driverless AI, click on
Resources at the top of the sceen, then click System Info.)
16.4.2 Deploying on REST Server
Once the MOJO pipeline archive is ready, Driverless AI provides a Deploy

(Local & Cloud) option on the completed experiment page.
Notes: This button is only available after the MOJO Scoring Pipeline has been
built.
This option opens a new dialog for setting the REST Server deployment name,
port number, and maximum heap size (optional).
108 | Deployment
1. Specify a name for the REST scorer in order to help track the deployed
REST scorers.
2. Provide a port number on which the REST scorer will run. For example,
if port number 8081 is selected, the scorer will be available at http://my-
ip-address:8081/models
3. Optionally specify the maximum heap size for the Java Virtual Machine
(JVM) running the REST scorer. This can help constrain the REST scorer
from overconsuming memory of the machine. Because the REST scorer
is running on the same machine as Driverless AI, it may be helpful to
limit the amount of memory that is allocated to the REST scorer. This
option will limit the amount of memory the REST scorer can use, but it
will also produce an error if the memory allocated is not enough to run
the scorer. (The amount of memory required is mostly dependent on the
size of MOJO. See Prerequisites for more information.)
16.4.3 Testing the REST Server Deployment
The request is a JSON object containing attributes:

• fields: A list of input column names that should correspond to the training
data columns.
• rows: A list of rows that are in turn lists of cell values to predict the
target values for.
• (optional) includeFieldsInOutput: A list of input columns that should
be included in the output.
Deployment | 109
An example request providing 2 columns on the input and asking to get one
column copied to the output looks as follows:
{
"fields": [
"age", "salary"
],
"includeFieldsInOutput": [
"salary"
],
"rows": [
[
"48.0", "15000.0"
],
[
"35.0", "35000.0"
],
[
"18.0", "22000.0"
]
]
}
Assuming the request is stored locally in a file named test.json, the request
to the endpoint can be sent, e.g., using the curl utility, as follows:
URL={place the endpoint URL here}
curl \
-X POST \
-d {"fields": [’age’, ’salary’, ’education’], "rows": [1, 2, 3], "
includeFieldsInOutput": ["education"]}\
-H "Content-Type: application/json" \
${URL}/score
The response is a JSON object with a single attribute score, which contains
the list of rows with the optional copied input values and the predictions.
For the example above with a two class target field, the result is likely to look
something like the following snippet (the exact values depend on the scoring
pipeline):
{
"score": [
[
"48.0",
"0.6240277982943945",
"0.045458571508101536",
],
[
"35.0",
"0.7209441819603676",
"0.06299909138586585",
],
[
"18.0",
"0.7209441819603676",
"0.06299909138586585",
]
]
}
110 | About Driverless AI Transformations
16.4.4 REST Server Deployment Issues
When using Docker, local REST scorers are deployed within the same container
as Driverless AI. As a result, all REST scorers will be turned off if the Driverless
AI container is closed. When using native installs (rpm/deb/tar.sh), the REST
scorers will continue to run even if Driverless AI is shut down.
17 About Driverless AI Transformations

Transformations in Driverless AI are applied to columns in the data. The
transformers create the engineered features in experiments.
Driverless AI provides a number of transformers. The downloaded experiment
logs include the transformations that were applied to your experiment. Note
that you can exclude transformations in the config.toml file, and that list of
excluded transformers will also be available in the experiment log.
The following transformers are available for classification (multiclass and binary)
and regression experiments.
17.1 Numeric Transformers

• ClusterDist Transformer: The Cluster Distance Transformer clusters
selected numeric columns and uses the distance to a specific cluster as a
new feature.
• ClusterTE Transformer: The Cluster Target Encoding Transformer
clusters selected numeric columns and calculates the mean of the response
column for each cluster. The mean of the response is used as a new
feature. Cross Validation is used to calculate mean response to prevent
overfitting.
• Interactions Transformer: The Interactions Transformer adds, divides,
multiplies, and subtracts two numeric columns in the data to create a
new feature.
• NumCatTE Transformer: The Numeric Categorical Target Encoding
Transformer calculates the mean of the response column for several
selected columns. If one of the selected columns is numeric, it is first
converted to categorical by binning. The mean of the response column
is used as a new feature. Cross Validation is used to calculate mean
response to prevent overfitting.
• NumToCatTE Transformer: The Numeric to Categorical Target En-
coding Transformer converts numeric columns to categoricals by binning
About Driverless AI Transformations | 111
and then calculates the mean of the response column for each group.
The mean of the response for the bin is used as a new feature. Cross
Validation is used to calculate mean response to prevent overfitting.
• NumToCatWoEMonotonic Transformer: The Numeric to Categorical
Weight of Evidence Transformer converts a numeric column to categorical
by binning and then calculates Weight of Evidence for each bin. The
Weight of Evidence is used as a new feature. Weight of Evidence measures
the strength of a grouping for separating good and bad risk and is
calculated by taking the log of the ratio of distributions for a binary
response column.
• Original Transformer: The Original Transformer applies an identity
transformation to a numeric column.
• TruncSVDNum Transformer: Truncated SVD Transformer trains a
Truncated SVD model on selected numeric columns and uses the compo-
nents of the truncated SVD matrix as new features.
17.2 Time Series Experiments Transformers

• DateTimeOriginal Transformer: The Date Original Transformer re-
trieves date values such as year, quarter, month, day, day of the year,
week, and weekday values.
• ClusterDist Transformer: The Date Time Original Transformer retrieves
date and time values such as year, quarter, month, day, day of the year,
week, weekday, hour, minute, and second values.
• EwmaLags Transformer: The Exponentially Weighted Moving Aver-
age (EWMA) Transformer calculates the exponentially weighted moving
average of target or feature lags.
• LagsAggregates Transformer: The Lags Aggregates Transformer cal-
culates aggregations of target/feature lags like mean(lag7, lag14, lag21)
with support for mean, min, max, median, sum, skew, kurtosis, std. The
aggregation is used as a new feature.
• LagsInteraction Transformer: The Lags Interaction Transformer creates
target/feature lags and calculates interactions between the lags (lag2 -
lag1, for instance). The interaction is used as a new feature.
• Lags Transformer: The Lags Transformer creates target/feature lags,
possibly over groups. Each lag is used as a new feature. Lag transformers
may apply to categorical (strings) features or binary/multiclass string
valued targets after they have been internally numerically encoded.
112 | About Driverless AI Transformations
17.3 Categorical Transformers (String)

• CatOriginal Transformer: The Categorical Original Transformer applies
an identity transformation that leaves categorical features as they are.
This transformer works with models that can handle non-numeric feature
values.
• CVCatNumEncode Transformer: The Cross Validation Categorical to

Numeric Encoding Transformer calculates an aggregation of a numeric
column for each value in a categorical column (ex: calculate the mean
Temperature for each City) and uses this aggregation as a new feature.
• CVTargetEncode Transformer: The Cross Validation Target Encoding

Transformer calculates the mean of the response column for each value
in a categorical column and uses this as a new feature. Cross Validation
is used to calculate mean response to prevent overfitting.
• Frequent Transformer: The Frequent Transformer calculates the fre-

quency for each value in categorical column(s) and uses this as a new
feature. This count can be either the raw count or the normalized count.
• LexiLabelEncoder Transformer: The Lexi Label Encoder sorts a cate-

gorical column in lexicographical order and uses the order index created
as a new feature.
• NumCatTE Transformer: The Numeric Categorical Target Encoding

Transformer calculates the mean of the response column for several
selected columns. If one of the selected columns is numeric, it is first
converted to categorical by binning. The mean of the response column
is used as a new feature. Cross Validation is used to calculate mean
response to prevent overfitting.
• OneHotEncoding Transformer: The One-hot Encoding transformer

converts a categorical column to a series of boolean features by performing
one-hot encoding. The boolean features are used as new features.
• SortedLE Transformer: The Sorted Label Encoding Transformer sorts

a categorical column by the response column and uses the order index
created as a new feature.
• WeightOfEvidence Transformer: The Weight of Evidence Transformer

calculates Weight of Evidence for each value in categorical column(s).
The Weight of Evidence is used as a new feature. Weight of Evidence
measures the strength of a grouping for separating good and bad risk and
is calculated by taking the log of the ratio of distributions for a binary
response column.
About Driverless AI Transformations | 113
This only works with a binary target variable. The likelihood needs to be
created within a stratified kfold if a fit transform method is used. More
information can be found here: http://ucanalytics.com/blogs/
information-value-and-weight-of-evidencebanking-case/.
17.4 Text Transformers (String)

• TextBiGRU Transformer: Trains a bi-directional GRU TensorFlow
model on word embeddings created from a text feature to predict the
response column. The GRU prediction is used as a new feature. Cross
Validation is used when training the GRU model to prevent overfitting.
• TextCharCNN Transformer: Trains a CNN TensorFlow model on char-

acter embeddings created from a text feature to predict the response
column. The CNN prediction is used as a new feature. Cross Validation
is used when training the CNN model to prevent overfitting.
• TextClustDist Transformer: The Text Cluster Distance Transformer

clusters a TF-IDF matrix created from a text feature and uses the distance
to a specific cluster as a new feature.
• TextClustTE Transformer: The Text Cluster Target Encoding Trans-

former clusters a TF-IDF matrix created from a text feature. The mean
of the response is calculated for each cluster and this is used as a new
feature. Cross Validation is used to calculate mean response to prevent
overfitting.
• TextCNN Transformer: The Text CNN Transformer trains a CNN

TensorFlow model on word embeddings created from a text feature to
predict the response column. The CNN prediction is used as a new a
feature. Cross Validation is used when training the CNN model to prevent
overfitting.
• TextLinModel Transformer: The Text Linear Model Transformer trains

a linear model on a TF-IDF matrix created from a text feature to predict
the response column. The linear model prediction is used as a new
feature. Cross Validation is used when training the linear model to
prevent overfitting.
114 | Logs
• Text Transformer: The Text Transformer tokenizes a text column and

creates a TFIDF matrix (term frequency-inverse document frequency)
or count (count of the word) matrix. This may be followed by dimen-
sionality reduction using truncated SVD. Selected components of the
TF-IDF/Count matrix are used as new features.
17.5 Time Transformers (Date, Time)
• Dates Transformer: The Dates Transformer retrieves any date values,

including year, quarter, month, day, day of the year, week, weekday, hour,
minute, and second values.
• IsHoliday Transformer: The Is Holiday Transformer determines if a date

column is a holiday. A boolean column indicating if the date is a holiday
is added as a new feature. Creates a separate feature for holidays in the
United States, United Kingdom, Germany, Mexico, and the European
Central Bank. Other countries available in the python Holiday package
can be added via the configuration file.
18 Logs
Driverless AI provides a number of logs that can be retrieved while visualizing
datasets, while an experiment is running, and after an experiment is completed.
While Visualizing Datasets
When running Autovisualization, you can access the Autoviz logs by clicking
the Display Logs button on the Visualize Datasets page.
Logs | 115
This page presents logs created while the dataset visualization was being
performed. You can download the vis-data-server.log file by clicking the
Download Logs button on this page. This file can be used to troubleshoot
any issues encountered during dataset visualization.
While an Experiment is Running
While the experiment is running, you can access the logs by clicking on the
Log button on the experiment screen. The Log button can be found in the
CPU/Memory section. Clicking on the Log button will present the experiment
logs in real time. You can download these logs by clicking on the Download
Logs button in the upper right corner.
Only the h2oai experiment.log will be downloaded while the experiment is

running (for example: h2oai experiment tobosoru.log). It will have the same
information as the logs being presented in real time on the screen.
For troubleshooting purposes, view the complete h2oai experiment.log (or
h2oai experiment anonymized.log). This will be available after the experi-
ment finishes, as described in the next section.
116 | Logs
After an Experiment has Finished

If the experiment has finished, you can download the logs by clicking on the
Download Logs button at the center of the experiment screen.
This will download a zip file which includes the following logs:
• h2oai experiment.log: This is the log corresponding to the experiment.
• h2oai experiment anonymized.log: This is the log corresponding to
the experiment where all data in the log is anonymized.
• h2oai server.log: Contains the logs for all experiments and all users.
• h2oai server anonymized.log: Contains the logs for all experiments
and all users where all data in the log is anonymized.
• h2o.log: This is the log corresponding to H2O-3. (H2O-3 is used
internally for parts of Driverless AI.)
For troubleshooting purposes, view the complete h2oai experiment.log or the
h2oai experiment anonymized.log.
The following additional information about your particular experiment will also
be included in the zip file:
• tuning leaderboard.txt: The results of the parameter tuning stage. This
contains the model parameters investigated and their performance.
• gene summary.txt: A summary of the feature transformations available
for each gene over the feature engineering iterations
• features.txt: The features used in the final Driverless AI model along
with feature importance and feature description
• details folder: Contains standard streams for each of the subprocesses
performed by Driverless A. This information is for debugging purposes
After Model Interpretation
You can view an MLI log for completed model interpretations by selecting the
Download MLI Logs link on the MLI page.
Logs | 117
This will download a zip file which includes the following logs:
• h2oai experiment mli key.log: This is the log corresponding to the model
interpretation.
• h2oai experiment mli key anonymized.log: This is the log corresponding
to the model interpretation where all data in the log is anonymized.
This file can be used to view logging information for successful interpretations.
If MLI fails, then those logs are in ./tmp/h2oai experiment mli key.log and
./tmp/h2oai experiment mli key anonymized.log.
18.1 Sending Logs to H2O

This section describes the logs to send in the event of failures when running
Driverless AI.
Dataset Failures
• Adding Datasets: If a dataset fails to import, a message on the screen
should provide the reason for the failure. The logs to send are available
in the Driverless AI ./tmp folder.
• Dataset Details: If a failure occurs when attempting to view Dataset
Details, the logs to send are available in the Driverless AI ./tmp folder.
• Autovisualization: If a failure occurs when attempting to Visualize
Datasets, a message on the screen should provide a reason for the failure.
The logs to send are available in the Driverless AI ./tmp folder.
Experiments
• While Running an Experiment: As indicated previously, a Log button
is available on the Experiment page. Clicking on the Log button will
present the experiment logs in real time. You can download these logs
by clicking on the Download Logs button in the upper right corner.
118 | References
You can also retrieve the h2oai experiment.log for the corresponding
experiment in the Driverless AI ./tmp folder.
MLI
• During Model Interpretation: If a failure occurs during model interpre-

tation, then the logs to send are ./tmp/h2oai experiment mli key.log
and ./tmp/h2oai experiment mli key anonymized.log.
19 References
1. L. Breiman. Statistical modeling: The two cultures (with comments
and a rejoinder by the author). Statistical Science, 16(3), 2001. URL
https://projecteuclid.org/euclid.ss/1009213726
2. M. W. Craven and J. W. Shavlik. Extracting tree-structured represen-

tations of trained networks. Advances in Neural Information Process-
ing Systems, 1996. URL http://papers.nips.cc/paper/1152-
extracting-tree-structured-representations-of-trained-
networks.pdf
3. J. Friedman. Predictive Learning via Rule Ensembles, October 2005.

URL http://statweb.stanford.edu/ ˜jhf/ftp/RuleFit.
pdf
4. J. Friedman, T. Hastie, and R. Tibshirani. The Elements of Statistical

Learning. Springer, New York, 2001. URL https://web.stanford.
edu/ ˜hastie/ElemStatLearn/printings/ESLII_print12.
pdf
5. A. Goldstein, A. Kapelner, J. Bleich, and E. Pitkin. Peeking inside the black

box: Visualizing statistical learning with plots of individual conditional
expectation. Journal of Computational and Graphical Statistics, 24(1),
2015
6. R. A. Groeneveld and G. Meeden. Measuring Skewness and Kurtosis.

Journal of the Royal Statistical Society. Series D (The Statistician), 33
(4):391–399, December 1984
7. P. Hall, W. Phan, and S. S. Ambati. Ideas on interpreting machine

learning. O’Reilly Ideas, 2017. URL https://www.oreilly.com/
ideas/ideas-on-interpreting-machine-learning
8. J. Hartigan and S. Mohanty. The RUNT test for Multimodality.

Journal of Classification, 9(1):63–70, January 1992
Authors | 119
9. J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman.

Distribution-free predictive inference for regression. Journal of the Amer-
ican Statistical Association just-accepted, 2017. URL http://www.
stat.cmu.edu/˜ryantibs/papers/conformal.pdf
10. H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie,
T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg,
A. M. Hrafnkelsson, T. Boulos, and J. Kubica. Ad Click Prediction: A
View from the Trenches. In Proceedings of the 19th ACM SIGKDD
international conference on Knowledge discovery and data mining, 2013.
URL https://research.google.com/pubs/archive/41159.
pdf
11. F. Niu, B. Recht, C. Re, and S. J. Wright. Hogwild!: A Lock-Free
Approach to Parallelizing Stochastic Gradient Descent. In NIPS,
2011. URL https://people.eecs.berkeley.edu/˜brecht/
papers/hogwildTR.pdf
12. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should I trust you?:
Explaining the predictions of any classifier. In Proceedings of the 22nd
ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, pages 1135–1144. ACM, 2016. URL http://www.kdd.
org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
13. L. Wilkinson. Dot Plots. The American Statistician, 53(3):276–281,
1999
14. L. Wilkinson, A. Anand, and R. Grossman. ”Graph-theoretic Scagnostics,”
in Proceedings of the IEEE Information Visualization. INFOVIS ’05. IEEE
Computer Society, Washington, DC, USA, 2005
15. H2O.ai Team. Datatable for python. URL https://github.com/
h2oai/datatable
20 Authors
Patrick Hall
Patrick Hall is senior director for data science products at H2O.ai where he
focuses mainly on model interpretability. Patrick is also currently an adjunct
professor in the Department of Decision Sciences at George Washington Univer-
sity, where he teaches graduate classes in data mining and machine learning.
Prior to joining H2O.ai, Patrick held global customer facing roles and research
and development roles at SAS Institute.
Follow him on Twitter: @jpatrickhall
120 | Authors
Megan Kurka
Megan is a customer data scientist at H2O.ai. Prior to working at H2O.ai,
she worked as a data scientist building products driven by machine learning for
B2B customers. Megan has experience working with customers across multiple
industries, identifying common problems, and designing robust and automated
solutions.
Angela Bartz
Angela is the doc whisperer at H2O.ai. With extensive experience in technical
communication, she brings our products to life by documenting the features and
functionality of the entire suite of H2O products. Having worked for companies
both large and small, she is an expert at understanding her audience and
translating complex ideas into consumable documents. Angela has a BA degree
in English from the University of Detroit Mercy.

Driverless A I Booklet

Uploaded by

Copyright:

Available Formats

Driverless A I Booklet

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Driverless A I Booklet

Uploaded by

Copyright:

Available Formats

Using H2O Driverless AI

Patrick Hall, Megan Kurka & Angela Bartz

October 2019: Version 1.8.0

Published by H2O.ai, Inc.

©2017-2019 H2O.ai, Inc. All rights reserved.

October 2019: Version 1.8.0

Photos by ©H2O.ai, Inc.

All copyrights belong to their respective owners.

Printed in the United States of America.

2 Why Driverless AI? 7

5 Installing and Upgrading Driverless AI 11

7 The Datasets Page 13

10.3 The Experiments Leaderboard . . . . . . . . . . . . . . . . . 39

13 Score on Another Dataset 74

14 Transform Another Dataset 74

15 The Driverless AI Scoring Pipelines 76

15.3.3 Quick Start - Alternative Method . . . . . . . . . . . . 88

17 About Driverless AI Transformations 110

Driverless AI runs on commodity hardware. It was also specifically designed

This document describes how to use H2O Driverless AI UI and is updated

For more information about Driverless AI, please see https://www.h2o.

1.2 Have Questions?

2 Why Driverless AI?

H2O Driverless AI is a high-performance, GPU-enabled computing platform

How do you use Driverless AI results to create commercial value?

Commercial value is generated by Driverless AI in a few ways.

• Driverless AI empowers data scientists or data analysts to work on projects

• Like in many other industries, automation leads to standardization of

• Driverless AI makes deploying predictive models easy - typically a difficult

Moreover, the system was designed with interpretability and transparency in

Visit https://www.h2o.ai/products/h2o-driverless-ai/ to download your free

Flexibility of Data and Deployment: Driverless AI works across a variety of

NVIDIA GPU Acceleration: Driverless AI is optimized to take advantage of

Automatic Data Visualization (Autovis): For datasets, Driverless AI auto-

Automatic Feature Engineering: Feature engineering is the secret weapon

its models. A number of charts are generated automatically, including K-LIME,

5 Installing and Upgrading Driverless AI

installation is available for Ubuntu 16.04 environments and on Windows 10

Note: Contact [email protected] for information on how to purchase a Driver-

7 The Datasets Page

7.1 Adding Datasets

7.2 Dataset Details

To view a summary of a dataset or to preview the dataset, click on the dataset

7.2.1 Dataset Details Page

7.2.2 Dataset Rows Page

7.2.3 Downloading Datasets

7.3 Splitting Datasets

2. The Dataset Splitter form displays. Specify an Output Name 1 and an

7.4 Visualizing Datasets

7.4.1 The Visualization Page

The following is a complete list of available graphs.

a graphical display of the fractiles of a distribution. The center of the

lation. For continuous-categorical variable pairs, the statistic is based on

• Parallel Coordinates Plot: A Parallel Coordinates Plot is a graph used

• Radar Plot: A Radar Plot is a two-dimensional graph that is used for

• Data Heatmap: The heatmap graphic is constructed from the transposed

• Missing Values Heatmap: The missing values heatmap graphic is

columns are permuted via a singular value decomposition (SVD) of the

• Gaps Histogram: The gaps index is computed using an algorithm of