Oracle Database 12c Data Masking and Subsetting Guide
Oracle Database 12c Data Masking and Subsetting Guide
Oracle Database 12c Data Masking and Subsetting Guide
April 2016
Contents
Preface ................................................................................................................................................................ xi
Audience ....................................................................................................................................................... xi
Related Documents...................................................................................................................................... xi
Conventions.................................................................................................................................................. xi
Changes in This Release for Oracle Data Masking and Subsetting Users Guide...... xiii
Versioning of Oracle Data Masking and Subsetting ............................................................................
xiii
xiii
Relationships) ...............................................................................................................................
xiii
1-1
Components...............................................................................................................................................
1-2
1-2
1-3
1-3
1-4
Application Templates.....................................................................................................................
1-4
Architecture ...............................................................................................................................................
1-5
1-5
In-Database........................................................................................................................................
1-5
In-Export ............................................................................................................................................
1-5
Heterogeneous ..................................................................................................................................
1-6
Methodology .............................................................................................................................................
1-6
Workflow ...................................................................................................................................................
1-7
iii
2-1
Access Control For Oracle Data Masking and Subsetting Objects ....................................................
2-2
2-3
2-3
2-3
Unsupported Objects................................................................................................................................
2-4
3-2
3-3
3-5
3-5
3-6
3-6
3-7
3-8
3-9
Data Masking
Overview of Oracle Data Masking.........................................................................................................
4-1
4-1
4-2
4-2
4-2
4-3
4-3
4-5
4-6
4-6
4-8
iv
Data Subsetting
Creating a Data Subset Definition ..........................................................................................................
5-1
5-6
5-8
5-9
5-9
Index
vi
List of Figures
1-1
1-2
1-3
3-1
3-2
4-1
1-5
1-7
1-8
3-1
3-2
4-4
vii
viii
List of Tables
4-1
4-2
4-3
4-4
4-5
4-32
4-32
4-32
4-33
4-33
ix
Preface
This preface contains the following topics:
Audience
Related Documents
Conventions
Audience
This document provides information about how to manage test data using the Oracle
Data Masking and Subsetting features of the Enterprise Manager for Oracle Database
Plug-in. This document is intended for database administrators, application designers,
and programmers who are responsible for performing real-world testing of Oracle
Database.
Related Documents
For more information about some of the topics discussed in this document, see the
following documents in the Oracle documentation set:
Oracle Database 2 Day DBA
Oracle Database 2 Day + Performance Tuning Guide
Oracle Database Administrator's Guide
Oracle Database Concepts
Oracle Database Performance Tuning Guide
Oracle Database SQL Tuning Guide
Oracle Database Installation Guide for Linux
Enterprise Manager Command Line Interface
Conventions
The following text conventions are used in this document:
xi
xii
Convention
Meaning
boldface
italic
monospace
xiii
Support for Application Data Modeling and Data Masking of Edition View Objects
Application Data Modeling and Data Masking components have been enhanced to
support Edition Views. Support for Edition Views is essential to discover, mask, and
subset applications such as Oracle E-Business Suite 12.2 or higher, which use editionbased redefinition for patching applications or upgrading databases, with minimal
downtime.
You can create an application data model with edition views, and mark the edition
view columns as sensitive.
Data Masking will mask the base column that the edition view column is referring to.
For more information on Edition Views, see the Oracle Database Advanced
Application Developer's Guide.
xiv
1
Introduction to Oracle Data Masking and
Subsetting
The Data Masking and Subsetting features of the Enterprise Manager for Oracle
Database Plug-in help you to securely manage test data.
When performing real-world testing, there is a risk of exposing sensitive data to nonproduction users in a test environment. The test data management features of Oracle
Database helps to minimize this risk by enabling you to perform data masking and
data subsetting on the test data.
When production data is copied into a testing environment, there is the risk of
breaching sensitive information to non-production users, such as application
developers or external consultants. In order to perform real-world testing, these nonproduction users need to access some of the original data, but not all the data,
especially when the information is deemed confidential.
Oracle Database offers test data management features that help reduce this risk by
enabling you to:
Store the list of applications, tables, and relationships between table columns using
Application Data Modeling, and sensitive columns. For more information, see
Application Data Modeling.
Replace sensitive data from your production system with fictitious data that can be
used during testing using data masking. For more information, see Data Masking.
Replicate information that pertains only to a particular site using data subsetting.
For more information, see Data Subsetting.
Note:
You must have the Oracle Data Masking and Subsetting Pack license to use
these security features.
Components
Components
This section discusses the components of Oracle Data Masking and Subsetting:
Application Data Modeling
Data Masking Format Library
Data Masking Transformations
Data Subsetting Techniques
Application Templates
Components
Oracle Data Masking and Subsetting also allows you to import and export Application
Data Models.
For more information, see the Application Data Modeling chapter.
Components
state, and postal codes. The values in these address fields will be consistent even
after the masking operation is completed.
Deterministic/Consistent Masking: generates consistent outputs for a given input
across various databases. This transformation is helpful to maintain data integrity
across multiple applications and also to preserve system integrity in a single signon environment. For example, consider three applications: a human capital
management application, a customer relationship management application, and a
sales data warehouse. These applications might have a common field such as
EMPLOYEE ID that must be masked consistently. The Substitute and Encrypt
masking formats provide deterministic masking transformation.
Shuffle: allows the fields within a column to be shuffled in a random manner. This
transformation is helpful in breaking the one-to-one mapping between sensitive
data elements. For example, columns containing personal health records can be
shuffled while masking health care information.
Key-based reversible masking (also known as Encrypt Format): encrypts and
decrypts the original data using a secure key string, while preserving the format of
the input data. This transformation uses the 3DES algorithm, and is helpful when
businesses need to mask and send their data to a third-party for analysis, reporting,
or any other business processing purpose. After the processed data is received from
the thirdparty, the original data can be recovered using the same key string that
was used to encrypt the data.
Format Preserving Randomization (also known as Auto Mask Format):
randomizes the data, preserving the input length, position of the characters and
numbers, case of the character (upper or lower), and special characters in the input.
Application Templates
Oracle Data Masking Application Templates deliver pre-identified sensitive columns,
their relationships, and industry-standard best practice masking techniques out-of-the
box for packaged applications such as Oracle E-Business Suite and Oracle Fusion
Applications. Use the Self Update feature to get the latest masking and subsetting
templates available from Oracle.
Architecture
Architecture
Oracle Data Masking and Subsetting architecture consists of a two-tier framework.
The Cloud Control UI provides a graphical user interface for administrators to handle
all management tasks. These client components can be installed locally or brought up
with a Web browser. The Oracle Enterprise Manager framework is comprised of
Oracle Management Server and a database repository.
The Management Server is the core of the Enterprise Manager framework. It provides
administrative user accounts, processes management functions such as jobs and
events, and manages the flow of information between the Cloud Control UI and the
nodes with Enterprise Management Agents. The Oracle Enterprise Manager Agent
communicates with the Oracle Management Server and performs tasks sent by Cloud
Control UI and other client applications.
The Enterprise Management Agent is installed on each monitored host and is
responsible for monitoring all of the targets running on those hosts, communicating
that information to the Oracle Management Server, and managing and maintaining
the hosts and its targets.
Figure 1-1
Execution Methods
Oracle Data Masking and Subsetting provides two modes for masking and subsetting
data:
In-Database
In-Export
In-Database
In-Database mode directly masks and subsets the data within a non-production
database with minimal or no impact on production environments. Since the InDatabase masking and subsetting mode permanently changes the data in a database, it
is strongly recommended to use this mode for non-production environments such as
staging, test, and development databases rather than using it on production databases.
In-Export
In-Export mode masks and subsets the data in near real-time while extracting the data
from a database. The extracted masked and subsetted data is written to data pump
Methodology
export files that are further imported into test, development, or QA databases. In
general, In-Export mode is used for production databases.
Heterogeneous
Oracle Data Masking and Subsetting also supports data masking of data from any
non-Oracle database, such as IBM DB2, Microsoft SQL Server, and Sybase.
Oracle Data Masking and Subsetting uses the DB links and the gateways to read data
from non-Oracle production databases, copy the data to an Oracle-based staging
environment, mask in the Oracle staging environment, and then write to the nonOracle test databases.
It uses the same masking techniques that are used in the Oracle databases for masking
the data.
Note:
Oracle Data Masking and Subsetting does not support automatic data
discovery for non-Oracle databases, and also does not perform referential
integrity for non-Oracle databases.
Methodology
Oracle Data Masking and Subsetting uses the following methodology to secure nonproduction database and replaces sensitive data with fictitious, but yet relevant data
that meets compliance requirements.
Creating an Application Data Model Discover sensitive data and data
relationships, and then create or assign an Application Data Model
Selecting Masking Formats and Criteria Create data masking definition and
masking format types and templates based on the sensitive data that is discovered.
Previewing and Validating Secure sensitive data by previewing the masking
algorithm results and the subset reduction results
Executing Masking Transformations Execute In-Database or In-Export masking
and subsetting transformations and validate the data that is masked.
The following figure describes the methodology used in Oracle Data Masking and
Subsetting.
Workflow
Figure 1-2
Methodology
Workflow
The following diagram explains the Oracle Data Masking and Subsetting workflow.
Workflow
Figure 1-3
The following steps describe the Oracle Data Masking and Subsetting Workflow:
1.
Create an Application Data Model To begin using Oracle Data Masking and
Subsetting, you must create an Application Data Model (ADM). ADMs capture
application metadata, referential relationships, and discover sensitive data from
the source database.
2.
Create a Data Masking Definition After an ADM is created, the next step is to
create a data masking definition. A masking definition includes information
regarding the table columns and the masking format for each of these columns.
The mask can be created by writing the masked data to the export file.
3.
2
Before You Begin
This chapter helps you prepare with the prerequisites and other important things that
you must consider before installing and using Oracle Data Masking and Subsetting.
ADMa user (other than Super Administrator) with ADM Operator privileges can
view an ADM but cannot edit and delete it, nor view its properties. To enforce this,
the Edit and Delete icons, and the Properties menu are disabled. Additionally, the
Sync option on the Create Verification Job page is disabled.
Data subset definitiona user (other than Super DSD Administrator) with Operator
privileges can view but not edit and delete a subset definition. To enforce this, the
Edit and Delete icons are disabled.
A user with Data Subset Definition Operator privileges can do any other operation
except edit and delete the data subset definition and has the following rights:
View the data subset definition.
Create a data subset to export files.
Create a data subset on a database.
Save the subset script.
Data masking definitiona user with Data Masking Definition Operator privileges
can do any other operation except edit and delete the data masking definition and
has the following rights:
View the data masking definition.
Generate a data masking script.
Schedule a data masking job.
Export a data masking definition.
Designer Privileges
Those granted Designer privileges can enhance, modify, and manage TDM objects.
These users can also grant and revoke Operator and Designer privileges to others.
Designer privileges imply the corresponding Operator privileges on a TDM object.
ADMa user with Designer privileges can perform all operations on an ADM
including delete.
Data subset definitiona user with Designer privileges can perform all operations
on a subset definition including delete.
Data masking definitiona user with Designer privileges can perform all operations
on a masking definition including delete.
Storage Requirements
Storage Requirements
Although Oracle Data Masking and Subsetting objects such as data models, masking
and subsetting definitions consume a negligible amount of storage space, depending
on the amount of data being stored over a period of time, you may need to allocate
additional storage space to Oracle Enterprise Manager's repository database.
This section details the storage recommendations for masking and subsetting.
In-Database Masking: 3X of additional space in the user tablespace (X being the
largest table in size) 2X of additional space in temporary tablespace
In-Export Masking: 2X additional space in the user tablespace (X being the largest
table in size) 2X of additional space in temporary tablespace Sufficient disk space to
store the generated export dump file
In-Database Subsetting: 2X additional space in the user tablespace (X being the
largest table in size) 2X additional space in temporary tablespace
In-Export Subsetting: X additional space in the user tablespace (X being the largest
table in size) Sufficient space to store the generated dump files
Note:
Unsupported Objects
The following Numeric Types can use Array List, Delete, Fixed Number, Null
Value, Post Processing Function, Preserve Original Data, Random Decimal
Numbers, Random Numbers, Shuffle, SQL Expression, Substitute, Table
Column, Truncate, Encrypt, and User Defined Function masking formats:
NUMBER
FLOAT
RAW
BINARY_FLOAT
BINARY_DOUBLE
String Types
The following String Types can use Array List, Delete, Fixed Number, Fixed
String, Null Value, Post Processing Function, Preserve Original Data, Random
Decimal Numbers, Random Digits, Random Numbers, Random Strings, Shuffle,
SQL Expression, Substitute, Substring, Table Column, Truncate, Encrypt, and
User Defined Function masking formats:
CHAR
NCHAR
VARCHAR2
NVARCHAR2
Date Types
The following Date Types can use Array List, Delete, Null Value, Post
Processing Function, Preserve Original Data, Random Dates, Shuffle, SQL
Expression, Substitute, Table Column, Truncate, Encrypt, and User Defined
Function masking formats:
DATE
TIMESTAMP
Grid Control 11g Release 1 (11.1) and Cloud Control 12c Release 1 (12.1.0.1) and
Release 2 (12.1.0.2)
Large Object (LOB) Data Types
The following Data Types can use Fixed Number, Fixed String, Null Value,
Regular Expression, and SQL Expression masking formats:
BLOB
CLOB
NCLOB
Unsupported Objects
Oracle Data Masking and Subsetting does not support:
clustered tables
Unsupported Objects
long columns
column of type XML; XML-type columns
virtual columns
Note:
Unsupported Objects
3
Application Data Modeling
Secure Test Data Management provides Enterprise Manager the capability to enable
operations such as sensitive data discovery, data subsetting, and data masking. These
capabilities enable scanning and tagging of sensitive data and modeling of data
relationships incorporated within an Application Data Model (ADM). You must have
the Oracle Data Masking and Subsetting Pack license to use test data management
features.
The following figure shows the workflow associated with an Application Data Model.
Figure 3-1
The ADM stores the list of applications, tables, and relationships between table
columns that are either declared in the data dictionary, imported from application
metadata, or user-specified. The ADM maintains sensitive data types and their
associated columns, and is used by test data operations, such as data subsetting and
data masking, to securely produce test data. Creating an ADM is a prerequisite for
data subsetting and data masking operations.
The following figure shows the Application Data Model's relationship to other test
data management components as well as the production and test environments.
Figure 3-2
You can perform several tasks related to Application Data Modeling, including the
following tasks discussed in this chapter:
Creating an Application Data Model
Creating and Managing Custom Sensitive Column Types
Associating a Database to an Existing ADM
Verifying or Synchronizing an ADM
Importing and Exporting an ADM
Assigning Privileges to an Existing ADM
Note:
See Also:
Job System
Named Credential
Oracle Data Masking and Subsetting resource privilege
Note:
When you create an Application Data Model, the PL/SQL metadata collection
packages are automatically deployed on the target database. The Database
user must have DBA privileges to auto-deploy the packages.
We recommend that you create an Application Data Model for the first time
with a highly privileged user so that all the necessary packages are deployed.
Subsequently, all other Application Data Models can be created with a low
privileged user.
Creating an ADM
To create an Application Data Model:
1. From the Application Data Modeling page, view the diagram that explains how
2. Click Create.
The Source Database is the source from which the metadata is extracted.
5. Select an Application Suite:
Oracle E-Business Suite Provide the database credentials for APPS user (or
equivalent) , and click Submit to create the ADM.
Oracle Fusion Applications Provide database credentials for FUSION user
(or equivalent), and click Submit to create the ADM.
Note the following points about metadata collections:
The metadata collection for the selected application suite populates the ADM
with the applications and tables in the suite.
The ADM can collect metadata for one or more schemas. An ADM application
typically represents a schema. Each schema you select becomes an ADM
application, and the ADM becomes populated with the tables in the schema,
particularly in the case of custom applications. However, please note that
multiple applications can also map to a single schema, as in the case of Fusion
Applications. The actual mapping depends on the application metadata
discovered by the metadata collection job.
6. Select the following:
+ icon, and specify the primary key and foreign key columns that must be matched.
You can chose to either run the metadata collection job immediately or schedule it
to start at a later time.
12. Click Submit to submit the metadata collection job.
The ADM you created appears in the Application Data Modeling page.
13. Click View Job Details to view the status and details of the metadata collection
process.
14. Review the Most Recent Job Status table column to monitor the status of the
The application data model is locked and cannot be edited when the metadata is
being collected. You must wait until the status indicates that the job is complete.
2. Select the Application Data Model you previously created, and then click Edit.
Add Application.
The Add Application pop-up window appears.
4. Specify a name for the application, a nick name/short name, description for the
The Applications and Objects subpage appears, displaying the applications and
objects found during metadata collection.
To see the tables for an application, click the expand ( > ) icon.
2. To add a table, click Add Application.
The Search and Select pop-up appears, displaying all of the tables from the selected
schema that are not assigned to an application.
4. Select an unassigned table, then click OK.
6. To remove a table, select the table from the Application and Objects view, and click
Remove.
Edit.
The Applications and Objects subpage appears, displaying the applications found
during metadata collection.
To view the tables for an application, click the expand ( > ) icon.
2. Click the Referential Relationships tab.
parent and dependent key relationships, or select Expand All from the View menu
to view all relationships.
Edit.
The Applications and Objects subpage appears, displaying the applications and
objects found during metadata collection.
To view the tables and objects for an application, click the expand ( > ) icon.
2. From the Referential Relationships tab, open the Actions menu, then select Add
Referential Relationship.
key column.
The new dependent column now appears in the referential relationships list.
From the Application Data Modeling page, select the model you created, then
click the Edit .
The Applications and Objects subpage appears, displaying the applications and
objects found during metadata collection. To view the tables for an application,
click the expand ( > ) icon.
2.
From the Sensitive Columns tab, open the Actions menu, then select Create
Sensitive Column Discovery Job.
The Parameters pop-up appears.
3.
4.
Click Continue.
The schedule pop-up window appears.
5.
Specify the required information, schedule the job, then click Submit when you
have finished.
The Sensitive Columns subpage reappears.
6.
Click Save and Return to return to the Application Data Modeling home page.
From the Application Data Modeling page, select the model you created, then
click the Edit.
The Applications and Objects subpage appears, displaying the applications and
objects found during metadata collection. To view the tables for an application,
click the expand ( > ) icon.
2.
3.
Select the sensitive column for which you want to change the type.
4.
Open the Actions menu, then select Set Sensitive Column Type.
Edit.
The Applications and Objects subpage appears, displaying the applications found
during metadata collection.
To view the tables for an application, click the expand ( > ) icon.
2. Select the Sensitive Columns tab, then click Discovery Results to view the
discovery results.
When the Most Recent Job Status column indicates that the job is Successful, select
the ADM, then click Edit.
2.
Select the Sensitive Columns tab, then click Discovery Results to view the job
results.
3.
To set the sensitive status of any column, select the row for the column you want
to define, open the Set Status menu, then select either Sensitive or Not Sensitive.
4.
5.
Click Save and Return to return to the Application Data Modelling page.
click OK.
The sensitive column now appears in the table for the Sensitive Columns tab.
From the Actions menu of the Application Data Models page, select Sensitive
Column Types.
The Sensitive Column Types page appears.
2.
Click Create.
The Create Sensitive Column Type pop-up appears.
3.
Specify a required name and regular expressions for the Column Name, Column
Comment, and Column Data search patterns.
The Or Search Type means that any of the patterns can match for a candidate
sensitive column.
The And Search Type means that all of the patterns must match for a candidate
sensitive column.
If you do not provide expressions for any of these parameters, the system does not
search for the entity.
4.
Click OK.
The sensitive column appears in the table in the Sensitive Column Types page.
From the Actions menu of the Application Data Models page, select Sensitive
Column Types.
The Sensitive Column Types page appears.
2.
Select either a sensitive column type you have already defined, or select one from
the out-of-box types that the product provides.
3.
4.
Specify a required name and alter the existing expressions for the Column Name,
Column Comment, and Column Data search patterns to suit your needs.
5.
Click OK.
The sensitive column appears in the table in the Sensitive Column Types page.
From the Application Data Models page, select an ADM, select Actions, then
Associated Databases.
This dialog lists all of the databases associated with this ADM and the schemas
assigned to each application per database. You can add more databases that give
you a choice of data sources when subsetting and databases to mask during
masking.
2.
3.
To change a schema, select the associated database on the left, select the
application on the right for which the schema is to be changed, then click Select
Schema.
4.
Select the missing schema from the list in the pop-up, then click Select.
2.
3.
Select the source database with the Invalid status, then click Create Verification
Job.
4.
Specify job parameters in the Create Verification Job pop-up, then click Submit.
5.
After the job completes successfully, click the source database and note the object
problems listed.
6.
Fix the object problems, rerun the Verification Job, then check that the Source
Database Status is now Valid.
if you are going to edit the exported file prior to import, it is clear which XML tags are
required and where they belong in the file.
Importing an ADM
Exporting an ADM
Note:
There are EMCLI verbs to export and import an ADM if you want to perform
these operations remotely or script them.
Importing an ADM
There are two methods of import:
Import an ADM XML file from the desktop
Import an ADM XML file from the Software Library
To import an ADM XML file from your desktop:
1.
From the Application Data Models page, select the ADM you want to import.
2.
From the Actions menu, select Import, then select File from Desktop.
3.
In the pop-up that appears, specify a name for the ADM, the source database you
want to assign to the ADM, and location on your desktop from which you want to
import the ADM.
4.
Click OK.
The ADM now appears on the Application Data Models page.
ADM XML file on the left, then specify a name and the source database you want
to assign to the ADM on the right.
4. Click Import.
Exporting an ADM
There are three methods of export:
Export a selected ADM to the desktop
From the Application Data Models page, select the ADM you want to export.
2.
From the Actions menu, select Export, then select Selected Application Data
Model.
3.
4.
In the Save As pop-up that appears, navigate to a file location and click Save.
The system converts the ADM into an XML file that now appears at the specified
location on your desktop.
From the Actions menu, select Export, then select File from Software Library.
2.
From the Actions menu, select Export, then select File from Software Library.
3.
In the Export File from Software Library pop-up that appears, select the desired
ADM and click Export.
4.
5.
In the Save As pop-up that appears, navigate to a file location and click Save.
The system converts the ADM into an XML file that now appears at the specified
location on your desktop.
4. In the Export Sensitive Data pop-up that appears, provide credentials for the
A message appears on the Application Data Models page confirming that the
sensitive data was copied to the database.
For detailed information on TSDP, see Oracle Database Security Guide.
Data Models.
granted.
4
Data Masking
This chapter provides conceptual information about the components that comprise
Oracle Data Masking, and procedural information about performing the task
sequence, such as creating masking formats and masking definitions. Data masking
presupposes that you have created an Application Data Model (ADM) with defined
sensitive columns.
The procedures in this chapter are applicable to Oracle Enterprise Manager 12.1 and
higher Cloud Control only. You must have the Oracle Data Masking and Subsetting
Pack license to use data masking features.
Note:
of the original data, but do not need to see every column of every table, especially
when the information is protected by government regulations.
Data masking enables organizations to generate realistic and fully functional data with
similar characteristics as the original data to replace sensitive or confidential
information. This contrasts with encryption or Virtual Private Database, which simply
hides data, and the original data can be retrieved with the appropriate access or key.
With data masking, the original sensitive data cannot be retrieved or accessed.
Names, addresses, phone numbers, and credit card details are examples of data that
require protection of the information content from inappropriate visibility. Live
production database environments contain valuable and confidential dataaccess to
this information is tightly controlled. However, each production system usually has
replicated development copies, and the controls on such test environments are less
stringent. This greatly increases the risks that the data might be used inappropriately.
Data masking can modify sensitive database records so that they remain usable, but
do not contain confidential or personally identifiable information. Yet, the masked test
data resembles the original in appearance to ensure the integrity of the application.
OMS_HOME/sysman/admin/scripts/db/reorg/reorganize.pl
The online help topic "Creating a Data Masking Definition" as well as the help
for each Data Masking page
Figure 4-1
For more information, see Creating New Masking Formats and Using Oraclesupplied Predefined Masking Formats.
3. Create a masking definition to associate table columns and edition view objects to
these mask formats. Data masking determines the database foreign key
relationships and adds foreign key columns to the mask.
For more information, see Masking with an Application Data Model and
Workloads .
4. Save the masking definition and generate the masking script.
5. Verify if the masked data meets the information security requirements. Otherwise,
refine the masking definition, restore the altered tables, and reapply the masking
definition until the optimal set of masking definitions has been identified.
6. Clone the production database to a staging area, selecting the masking definition to
be used after cloning. Note that you can clone using Oracle Enterprise Manager,
which enables you to add masking to the Oracle Enterprise Manager clone
workflow. However, if you clone outside of Oracle Enterprise Manager, you must
initiate masking from Oracle Enterprise Manager after cloning is complete. The
cloned database should be controlled with the same privileges as the production
system, because it still contains sensitive production data.
After cloning, make sure you change the passwords as well as update or disable
any database links, streams, or references to external data sources. Back up the
cloned database, or minimally the tables that contain masked data. This can help
you restore the original data if the masking definition needs to be refined further.
For more information, see Cloning the Production Database .
7. After masking, test all of your applications, reports, and business processes to
ensure they are functional. If everything is working, you can export the masking
definition to keep it as a back-up.
8. After masking the staging site, make sure to drop any tables named MGMT_DM_TT
before cloning to a test region. These temporary tables contain a mapping between
the original sensitive column value and the mask values, and are therefore
sensitive in nature.
During masking, Oracle Enterprise Manager automatically drops these temporary
tables for you with the default "Drop temporary tables created during masking"
option. However, you can preserve these temporary tables by deselecting this
option. In this case, you are responsible for deleting the temporary tables before
cloning to the test region.
9. After masking is complete, ensure that all tables loaded for use by the substitute
column format or table column format are going to be dropped. These tables
contain the mask values that table column or substitute formats will use. It is
recommended that you purge this information for security reasons.
For more information, see Deterministic Masking Using the Substitute Format .
10. Clone the database to a test region, or use it as the new test region. When cloning
the database to an external or unsecured site, you should use Export or Import.
Only supply the data in the database, rather than the database files themselves.
11. As part of cloning production for testing, provide the masking definition to the
From the Enterprise menu, select Quality Management, then Data Masking
Formats. Alternatively, if you are in the Database home page, select Data
Masking Format Library from the Schema menu.
The Format Library appears with predefined formats that Oracle Enterprise
Manager provides.
2.
Click Create.
The Create Format page appears, where you can define a masking format.
See Also:
3.
Provide a required name for the new format, select a format entry type from the
Add list, then click Go.
A page appears that enables you to provide input for the format entry you have
selected. For instance, if you select Array List, the subsequent page enables you to
enter a list of values, such as New York, New Jersey, and New Hampshire.
4.
5.
The online help for information on the Format Library and Create Format
pages
rowid is the min (rowid) of the rows that contain the value original_value
3rd argument.
column_name is the name of the column being masked.
original_value is the value being masked.
That is, it accepts the original value as an input string, and returns the mask value.
Both input and output values are varchar2. For instance, a user-defined function to
mask a number could receive 100 as input, the string representation of the number
100, and return 99, the string representation of the number 99. Values are cast
appropriately when inserting to the table. If the value is not castable, masking fails.
Post-processing functions
To provide a post-processing function, enter it in the Post Processing Function
field.
rowid is the min (rowid) of the rows that contain the value mask_value.
column_name is the name of the column being masked.
mask_value is the value being masked.
Category Definitions
The following sections discuss different categories of these definitions:
Credit Card Numbers
United States Social Security Numbers
ISBN Numbers
UPC Numbers
Canadian Social Insurance Numbers
North American Phone Numbers
UK National Insurance Numbers
Auto Mask
By default, these mask formats are also available in different format styles, such as a
hyphen ( - ) format. If needed, you can modify the format style.
Credit Card Numbers
Out of the box, the format library provides many different formats for credit cards.
The credit card numbers generated by these formats pass the standard credit card
validation tests by the applications, thereby making them appear like valid credit card
numbers.
Some of the credit card formats you can use include:
MasterCard numbers
Visa card numbers
American Express card numbers
Discover Card numbers
Any credit card number (credit card numbers belong to all types of cards)
You may want to use different styles for storing credit card numbers, such as:
Pure numbers
'Space' for every four digits
'Hyphen' ( - ) for every four digits, and so forth
To implement the masked values in a certain format style, you can set the
DM_CC_FORMAT variable of the DM_FMTLIB package. To install the package, see
Installing the DM_FMTLIB Package .
Copy these scripts to a directory in your target database installation and execute
them using SQL*Plus, connected as a user that can create packages in the
DBSNMP schema.
You can now use the predefined masking formats in your masking definitions.
3.
Select and import any predefined masking format into a masking definition by
clicking the Import Format button on the Define Column Mask page.
rowid is the min (rowid) of the rows that contains the value mask_value 3rd
argument.
column_name is the name of the column being masked.
mask_value is the value being masked.
Preserve Original Data
Retains the original values for rows that match the specified condition clause. This
is used in cases where some rows that match a condition do not need to be masked.
Random Dates
The uniqueness of the Date column is not maintained after masking. This format is
combinable.
Random Decimal Numbers
If used as part of a mixed random string, these have limited usage for generating
unique values. This masking format generates unique values within the specified
range. For example, a starting value of 5.5 and ending value of 9.99 generates a
decimal number ranging from 5.5 to 9.99, both inclusive. This masking format is
combinable.
Random Digits
This format generates unique values within the specified range. For example, for a
random digit with a length of [5,5], an integer between [0, 99999] is randomly
generated, left padded with '0's to satisfy the length and uniqueness requirement.
This is a complementary type of random number, which will not be padded. When
using random digits, the random digit pads to the appropriate length in a string. It
does not pad when used for a number column. This format is combinable.
Data masking ensures that the generated values are unique, but if you do not
specify enough digits, you could run out of unique values in that range.
Random Numbers
If used as part of a mixed random string, these have limited usage for generating
unique values. This format generates unique values within the specified range. For
example, a starting value of 100 and ending value of 200 generates an integer
number ranging from 100 to 200, both inclusive. Note that Oracle Enterprise
Manager release 10.2.0.4.0 does not support float numbers. This format is
combinable.
Random Strings
This format generates unique values within the specified range. For example, a
starting length of 2 and ending length of 6 generates a random string of 2 - 6
characters in length. This format is combinable.
Regular Expression
This format enables you to use regular expressions to search for sensitive data in
LOBs (BLOB, CLOB, NCLOB) and replace the data with a fixed string, a fixed
number, null, or SQL Expression. Use rules to search for multiple strings within a
LOB.
Examples:
Use the regular expression [0-9]{3}[.][0-9]{3}[.][0-9]{4} to match
strings of the format nnn.nnn.nnnn and replace with a masked value, for
example, ***.***.****
Use the regular expression <SALARY>[0-9]{2,6}</SALARY> to zero out
salary information by replacing with <SALARY>0</SALARY>
Use the regular expression [A-Z]+@[A-Z]+\.[A-Z]{2,4} to mask e-mail
addresses by replacing with [email protected]
You can also use this format with data type VARCHAR2 to mask part of a string.
Shuffle
This format randomly shuffles the original column data. It maintains data
distribution except when a column is conditionally masked and its values are not
unique.
For more information, see Using the Shuffle Format.
Substitute
This format uses a hash-based substitution for the original value and always yields
the same mask value for any given input value. Specify the substitution masking
table and column. This format has the following properties:
The masked data is not reversible. That is, this format is not vulnerable to
external security breaches because the original value is replaced, so it is not
possible to retrieve the original value from the mask value.
Masking multiple times with a hash substitute across different databases yields
the same mask value. This characteristic is valid across multiple databases or
multiple runs assuming that the same substitution values are used in the two
runs. That is, the actual rows and values in the substitution table do not change.
For example, suppose the two values Joe and Tom were masked to Henry and
Peter. When you repeat the same mask on another database using the same
substitution table, if there were Bob and Tom, they might be replaced with
Louise and Peter. Notice that even though the two runs have different data,
Tom is always replaced with Peter.
This format does not generate uniqueness.
SQL Expression
This masking format enables you to enter a SQL Expression for masking a column.
Data masking uses this expression to generate masked values to replace the
original values. You cannot combine a column using this masking format type with
other masking format types, such as Random Numbers or Random Strings.
The SQL Expression can consist of one or more values, operators, and SQL
functions that evaluates to a value. It can also contain substitution columns
(columns from the same table as the masked column). You should specify
substitution columns within percent signs (%). Use SQL Expressions with
dbms_lob and other user-defined functions for LOB (BLOB, CLOB, NCLOB)
masking.
Examples:
dbms_random.string('u', 8) || '@company.com'
Generates random e-mail addresses.
%first_name% || '.' || %last_name% || '@company.com'
Generates e-mail addresses using first_name and last_name column values.
In this example, first_name and last_name are the substitution columns.
CLOB Masking
dbms_lob.empty_clob()
Empties the CLOB.
custom_mask_clob(%CLOB_COL%)
Applies the custom mask function to the clob column CLOB_COL.
Conditional Mask
(case when %PARTY_TYPE%='PERSON' then %PERSON_FIRST_NAME%||
' ' ||%PERSON_LAST_NAME% else (select
dbms_random.string('U', 10) from dual) end)
Columns within %% are present in the same table. The expression masks
PERSON_FULL_NAME with the first and last name; otherwise, the mask is a
random string.
Substitution Mask
select MASK_ZIPCODE from data_mask.DATA_MASK_ADDR where
ADDR_SEQ = ora_hash( %ZIPCODE% , 1000, 1234)
Select 1000 rows in the substitution table data_mask.DATA_MASK_ADDR. Mask
%ZIPCODE% with the MASK_ZIPCODE column in the substitution table. The row
rowid is the min (rowid) of the rows that contain the value original_value
3rd argument.
column_name is the name of the column being masked.
original_value is the value being masked.
Note:
The Number, Date, and LOB columns do not support multiple formats.
As long as this table of values does not change, the mask is deterministic or consistent
across the three databases.
See Also:
The online help for Define Column Mask page for more information on the
Substitute format
If you plan to mask a test system intended for evaluating performance, the following
practices are recommended:
Try to preserve the production statistics and SQL profiles after masking by adding
a pre-masking script to export the SQL profiles and statistics to a temporary table,
then restoring after masking completes.
Run a SQL Performance Analyzer evaluation to understand the masking impact on
performance. Any performance changes other than what appears in the evaluation
report are usually related to application-specific changes on the masked database.
To create a masking definition:
1.
From the Enterprise menu, select Quality Management, then Data Masking
Definitions.
The Data Masking Definitions page appears, where you can create and schedule
new masking definitions and manage existing masking definitions.
2.
3.
Before proceeding to the next step, one or more sensitive columns must
already be defined in the Application Data Model. See Creating and
Managing Custom Sensitive Column Types for more information.
4.
Click Add to go to the Add Columns page, where you can choose which sensitive
columns in the ADM you want to mask. See Adding Columns for Masking for
information on adding columns.
The results appear in the Columns table. Primary key and foreign key columns
appear below the sensitive columns.
5.
Use filtering criteria to refine sensitive column results. For example, perhaps you
want to isolate all columns that have order in the name (column name=order%).
You first may have to expose the filter section (Show Filters).
6.
Use the disable feature to exclude certain columns from masking consideration.
All columns are enabled by default. You can disable selected or all columns. You
can also search for a subset of columns (column name=order%) to disable. The
Status column on the right changes to reflect a column's disabled state. Note that a
column's disabled state persists on export of a data masking definition.
7.
Optional. Click the edit icon in the Format column to review and edit the masking
format.
8.
Expand Show Advanced Options and decide whether the selected default data
masking options are satisfactory.
For more information, see Selecting Data Masking Advanced Options.
9.
Click OK to save your definition and return to the Data Masking Definitions page.
At this point, super administrators can see each other's masking definitions.
10. Select the definition and click Generate Script. The schedule job dialog opens.
You may have to log in to the database first. For information on script generation,
see Scheduling a Script Generation Job.
Complete the schedule job dialog by providing the required information, then
click Submit.
11. A message appears denoting that the job was submitted successfully and you
return to the Data Masking Definitions page, where the status is "Generating
Script." Click View Job Details to open the job summary page.
When the job completes, click Log Report to check whether sufficient disk space is
available for the operation, and to determine the impact on other destination
objects, such as users, after masking. If any tables included in the masking
definition have columns of data type LONG, a warning message may appear. For
more information, see Using Data Masking with LONG Columns.
12. When the status on the Data Masking Definitions page is "Script Generated,"
Clone Databaseto clone and mask the database using the Clone Database
wizard (this requires a Database Lifecycle Management Pack license). For more
information, see Cloning the Production Database.
Save Scriptto save the entire PL/SQL script to your desktop.
Save Mask Bundleto download a zip file containing the SQL files generated
as part of the At source masking script generation option. You can then extract
and execute the script to create a masked dump of the database.
View Scriptto view the PL/SQL script, which you can edit and save. You can
also view errors and warnings, if any, in the impact report.
Click Go to execute the selected action.
13. If you are already working with a test database and want to directly mask the data
in this database, click Schedule Job. For information on masking a database, see
Scheduling a Data Masking Job.
Provide the requisite information and desired options. You can specify the
database at execution time to any database. The system assumes that the
database you select is a clone of the source database. By default, the source
database from the ADM is selected.
Click Submit.
The Data Masking Definitions page appears. The job has been submitted to
Enterprise Manager and the masking process appears. The Status column on
this page indicates the current stage of the process.
Note that you can also perform data masking at the source as part of a data subsetting
definition. See Creating a Data Subset Definition for more information.
Editing a Data Masking Definition
Use the following procedure to edit or otherwise work with a masking definition:
1.
From the Data Masking Definitions page, click the radio button next to the
definition you want to edit, then click Edit.
2.
Add columns or edit format entries as you would when creating a new definition,
which is explained starting in Step 3 above.
3.
Click OK on the Edit Masking Definitions page when you have finished editing.
4.
Perform these other actions with an existing masking definition by choosing the
respective menu item from the Actions menu:
Create Liketo display the masking definition in the Create Masking
Definition page where you can customize the definition.
Grant Designerto grant Designer privileges on the masking definition to
selected roles or administrators, which means the grantees can view and edit
the definition. Privileges granted can also be revoked.
Grant Operatorto grant Operator privileges on the masking definition to
selected roles or administrators, which means the grantees can view and copy
but not edit the definition. Privileges granted can also be revoked.
Exportto export the definition as an XML file for use in other repositories.
Click Go to execute the selected action.
You need to add at least one column in the masking definition. Otherwise, you
cannot generate a script that creates an impact report that provides
information about the objects and resources examined and lists details of any
warnings or errors detected.
1. Enter search criteria, then click Search.
The sensitive columns you defined in the ADM appear in the table below.
2. Either select one or more columns for later formatting on the Create Masking
Definition page, or formatting now if the data types of the columns you have
selected are identical.
3. Optional: if you want to mask selected columns as a group, enable Mask selected
columns as a group. The columns that you want to mask as a group must all be
from the same table.
Enable this check box if you want to mask more than one column together, rather
than separately. When you select two or more columns and then later define the
format on the Define Group Mask page, the columns appear together, and any
choices you make for format type or masking table apply collectively to all of the
columns.
After you define the group and return to this page, the Column Group column in
the table shows an identical number for each entry row in the table for all members
of the group. For example, if you have defined your first group containing four
columns, each of the four entries in this page will show a number 1 in the Column
Group column. If you define another group, the entries in the page will show the
number 2, and so forth. This helps you to distinguish which columns belong to
which column groups.
4. Either click Add to add the column to the masking definition, return to the Create
Masking Definition page and define the format of the column later, or click Define
Format and Add to define the format for the column now.
The Define Format and Add feature can save you significant time. When you select
multiple columns to add that have the same data type, you do not need to define
the format for each column as you would when you click Add. For instance, if you
search for Social Security numbers (SSN) and the search yields 100 SSN columns,
you could select them all, then click Define Format and Add to import the SSN
format for all of them.
5. Do one of the following:
The Import Format page displays the formats that are marked with the same
sensitive type as the masked column.
Add another condition by clicking Add Condition to add a new condition
row, then provide one or more format entries as described in the previous
step.
When you have finished formatting the column, click OK to return to the
Create Masking Definition page.
If you clicked Define Format and Add in the previous step and checked Mask
selected columns as a group:
The Define Group Mask page appears, where you can add format entries for
group columns that appear in the Create Masking Definition page, as explained
below:
Select one of the available format types. For complete information on the
format types, see the online help for the Defining the Group Masking Format
topic.
Optionally add a column to the group.
When you have finished formatting the group, click OK to return to the
Create Masking Definition page.
The results appear in the Columns table. The sensitive columns you selected
earlier now appear on this page. Primary key and foreign key columns
appear below the sensitive columns.
Disabling this option compromises security. You must ensure this option is
enabled in the final mask performed on the copy of the production database.
Refresh statistics after masking
If you have already enabled statistics collection and would like to use special
options when collecting statistics, such as histograms or different sampling
percentages, it is beneficial to turn off this option to disable default statistics
collection and run your own statistics collection jobs.
Drop temporary tables created during masking
Masking creates temporary tables that map the original sensitive data values to
mask values. In some cases, you may want to preserve this information to track
how masking changed your data. Note that doing so compromises security. These
tables must be dropped before the database is available for unprivileged users.
Decrypt encrypted columns
This option decrypts columns that were previously masked using Encrypt format.
To decrypt a previously encrypted column, the seed value must be the same as the
value used to encrypt.
Decrypt only recovers the original value if the original format used for the
encryption matches the original value. If the originally encrypted value did not
conform to the specified regular expression, when decrypted, the encrypted value
cannot reproduce the original value.
Use parallel execution when possible
Oracle Database can make parallel various SQL operations that can significantly
improve their performance. Data Masking uses this feature when you select this
option. You can enable Oracle Database to automatically determine the degree of
parallelism, or you can specify a value. For more information about using parallel
execution and the degree of parallelism, see the Oracle Database Data Warehousing
Guide.
Recompile invalid dependent objects after masking
The masking process re-creates the table to be masked and as a consequence, all
existing dependent objects (packages, procedures, functions, MViews, Views,
Triggers) become invalid. You can specify that the masking process recompile these
invalid objects after creating the table, by selecting the check box. Otherwise,
invalid objects are not recompiled using utl_comp procedures at the end of
masking.
If you choose this option, indicate whether to use serial or parallel execution. You
can enable Oracle Database to automatically determine the degree, or you can
specify a value. For more information about using parallel execution and the
degree of parallelism, see the Oracle Database Data Warehousing Guide.
See Also:
The sub-options are available only if you select Mask In-Database in Step 4.
6. If you chose custom tablespace in Step 5, select a custom tablespace where the
Submit.
A message confirms that the job has been scheduled. Refresh the page to see the job
results.
Schedule Job.
2. Change the default job name if desired and enter an optional job description.
3. Select a database from the drop-down menu and indicate your preference:
Specify appropriate values if you want to override the defaults: enter a name
for the export file; specify a maximum file size in megabytes; specify the
maximum number of threads of active execution operating on behalf of the
export job. This enables you to consider trade-offs between resource
consumption and elapsed time.
Select whether to enable dump file compression and encryption. Enter and
confirm an encryption password, if appropriate. Log file generation is
selected by default.
5. Specify credentials to log in to the database host.
6. Specify credentials to log in to the reference database.
7. Specify to start the job immediately or at a later specified date and time, then click
Submit.
A message confirms that the job has been scheduled. Refresh the page to see the job
results.
Go to Data Discovery and Modeling and create a new Application Data Model
(ADM) using metadata collection for your packaged application suite.
When metadata collection is complete, edit the newly created ADM.
2.
From the Referential Relationships tab, open the Actions menu, then select
Add Referential Relationship.
The Add Referential Relationship pop-up window appears.
b.
c.
In the Columns Name list, select a dependent key column to associate with a
parent key column.
d.
3.
4.
5.
Select the newly created ADM and click Add, then Search to view this ADM's
sensitive columns.
6.
Select columns based on your search results, then import formats for the selected
columns.
Enterprise Manager displays formats that conform to the privacy attributes.
7.
8.
2.
Specify a name for the masking definition and select the ADM to associate with
the template. The Reference Database is automatically provided.
3.
Browse for the XML file, or specify the name of the XML file, then click Continue.
The Data Masking Definitions Page reappears and displays the imported
definition in the table list for subsequent viewing and masking.
On the Data Masking Definitions page, click Import from Software Library.
The Import Masking Definition page appears.
2.
3.
Specify a name for the masking definition and select the ADM to associate with
the template. The Reference Database is automatically provided.
4.
Click Continue.
The Data Masking Definitions Page reappears and displays the imported
definition in the table list for subsequent viewing and masking.
You can also export a data masking definition from the Software Library.
1. On the Data Masking Definitions page, click Export from Software Library.
2. Select a masking template in the Software Library list.
3. Click Export.
From the Data Masking Definitions page, select the masking definition you want
to clone, select Clone Database from the Actions list, then click Go.
The Clone Database: Source Type page appears.
The Clone Database wizard appears, where you can create a test system to run the
mask.
2.
Specify the type of source database backup to be used for the cloning operation,
then click Continue.
3.
Proceed through the wizard steps as you ordinarily would to clone a database.
For assistance, refer to the online help for each step.
4.
In the Database Configuration step of the wizard, add a masking definition, then
select the Run SQL Performance Analyzer option as well as other options as
desired or necessary.
5.
From the Data Masking Definitions page, select the masking definition to be
analyzed, then click Schedule Job.
The Schedule Data Masking Job page appears.
2.
3.
In the Encryption Seed section, provide a text string that you want to use for
encryption.
This section only appears for masking definitions that use the Substitute or
Encrypt formats. The seed is an encryption key used by the encryption/hashbased substitution APIs, and makes masking more deterministic instead of being
random.
4.
b.
Select the Capture Files option, if desired, then select a capture directory.
When you select this option, the contents of the directory is masked. The
capture file masking is executed consistently with the database.
5.
In the Detect SQL Plan Changes Due to Masking section, leave the Run SQL
Performance Analyzer option unchecked.
You do not need to enable this option because the pre- and post-masking scripts
you created, referenced in step 2, already execute the analyzer.
6.
7.
When the job completes successfully, click the link in the SQL Performance
Analyzer Task column to view the executed analysis tasks and Trial Comparison
Report, which shows any changes in plans, timing, and so forth.
2.
At step 4, the format of the Database Configuration step appears different from
the Schedule Data Masking Job page discussed in , but select options as you
would for the Schedule Data Masking Job page.
3.
Continue with the wizard steps to complete and submit the cloning and masking
job.
Upgrade Considerations
Upgrade Considerations
Upgrading data masking definitions from 10 or 11 Grid Control to 12c Cloud Control
assumes that you have completed the following tasks:
Upgraded Enterprise Manager to 12c
Downloaded the latest database plug-in using Self Update and deployed the plugin to OMS and Management Agent
Completing these tasks automatically upgrades the masking definitions and creates
for each a shell Application Data Model (ADM) that becomes populated with the
sensitive columns and their dependent column information from the legacy mask
definition. The ADM, and hence data masking, then remains in an unverified state,
because it is missing the dictionary relationships.
Proceed as follows to complete the masking definition upgrade:
1.
From the Enterprise menu, select Quality Management, then select Data
Discovery and Modeling.
2.
For each shell ADM (verification status is Needs Upgrade), do the following:
a.
b.
c.
3.
From the Enterprise menu, select Quality Management, then select Data
Masking Definitions.
4.
5.
a.
b.
c.
Next, schedule a script generation job for each upgraded masking definition.
You can now resume masking with the upgraded data masking definitions.
See Also:
An 11.1 Grid Control E-Business Suite (EBS) masking definition based on an EBS
masking template shipped from Oracle is treated as a custom application after the
upgrade. You can always use the approach discussed in the first bulleted item
above to move into a newly created EBS ADM with all of the metadata in place.
However, this is not required.
EmpName
Salary
10
90
10
10
90
20
If you mask the Salary column with this format, each of the original values is replaced
with one of the values from this set. Assume that the shuffle format replaces 10 with
20, 90 with 10, and 20 with 90 (Table 4-2).
Table 4-2
EmpName
Salary
10
20
90
10
20
90
The result is a shuffled Salary column as shown in the Masked Table (Table 4-3), but
the data distribution is changed. While the value 10 occurs three times in the Salary
column of the Original Table, it occurs only twice in the Masked Table.
Table 4-3
EmpName
Salary
20
10
20
20
Table 4-3
EmpName
Salary
10
90
If the salary values had been unique, the format would have maintained data
distribution.
Employee
Job Category
Salary
Shuffled Salary
Alice
90
88
Bill
88
90
Carol
72
70
Denise
57
45
Eddie
70
57
Frank
45
72
Employee
Job Category
Salary
Conditional Result
Alice
90
Bill
88
Table 4-5
Employee
Job Category
Salary
Conditional Result
Carol
72
01
Denise
57
01
Eddie
70
01
Frank
45
01
George
45
45
Conditional masking works when there are duplicate values provided there are no
dependent columns or foreign keys. If either of these is present, a "bleeding condition"
results in the first of two duplicate values becoming the value of the second. So, in the
example, George's salary is not preserved, but becomes 01.
5
Data Subsetting
This chapter covers the Integrated Subset and Mask capability, where you perform
data masking and subsetting in a single taskflow and outlines a number of scenarios to
demonstrate the process. You must have the Oracle Data Masking and Subsetting Pack
license to use data subsetting features.
Note:
Data subsetting is supported only in Oracle Database versions 10.1 and higher.
The procedures in this chapter are applicable only to Oracle Enterprise
Manager Cloud Control 12.1 and higher.
From the Enterprise menu, select Quality Management, then Data Subset
Definitions.
2.
Open the Actions menu in the Data Subset Definitions page, then select Create, or
just click the Create icon.
3.
Provide the requisite information in the General pop-up that appears, then
click Continue.
You can select any source database associated with the Application Data
Model.
If you are performing masking within the subset definition, you must select
the same ADM and target used in creating the masking definition.
b.
The space estimate collection job runs, and then displays the Data Subset
Definitions page. Your definition appears in the table, and the Most Recent Job
Status column should indicate Scheduled, Running, or Succeeded, depending on
the schedule option selected and time required to complete the job.
4.
Select the definition within the table, open the Actions menu, then select Edit.
The Database Login page appears.
5.
Select either Named Credentials or New Credentials if you have not already set
preferred credentials, then click Login.
6.
In the Applications subpage of the Edit page, move applications from the
Available list to the Selected list as follows:
If you intend only to mask the data (no subsetting), select all applications.
If you intend only to subset the data (no masking), select specific applications
as appropriate.
If you intend both to subset and mask the data, the applications selected must
include those that the masking definitions require.
The names of application suites, applications, or application modules are
maintained in the Application Data Model.
7.
Note:
If you are masking only, set the Default Table Rules option to include all rows
and skip to Step 13. The Column Mask Rules tab, Rule Parameters tab, and
additional features on the Object Rules tab pertain specifically to subsetting.
You can add rules here to define the data to include in the subset.
8.
Select Actions, then Create to display the Object Rule pop-up, or just click the
Create icon.
a.
b.
If you select All Objects, in the Rows to Include section, select the option that
best suits your needs for a representative sample of production data. If you
do not want to include all rows, you can include some rows by specifying a
percentage portion of the rows. For finer granularity, you could specify a
Where clause, such as where region_id=6.
For more information on specifying Where clauses, see Step 8f.
c.
If you select Specified as the object type, the tables from the selected
Application appear in the drop-down list.
If the selected table is partitioned, click Add/Remove Partitions to choose the
partitions and sub-partitions from which the objects must be included in the
subset.
d.
e.
If you want to specify a Where clause, go to the next step. Otherwise, skip to
Step 9.
f.
The colon ( : ) preceding emp_id is only present in the Where clause, and not
required when creating a new rule parameter.
Click OK to save the properties, which now appear in the Rule
Parameters tab.
Skip to Step 10.
9.
Click OK to save the rule and return to the Object Rules tab.
The new rule is displayed in the list. The related objects are displayed in the table
below. Related rows from the objects are included in the subset to provide
referential integrity in the subset database.
10. In the Related Objects section of the Object Rules tab, you can manage the size of
the subset by controlling the levels of ancestors and descendants within the
subset. Notice that each node in the table has a check box. By default, all nodes are
included in the subset, as indicated by the check mark. Deselect the check box to
exclude a node from the subset. The deselection option is disabled for parent rows
(the join columns to the right identify parent and child rows). In addition, you can
make these other refinements to subset content:
Click Allow Excluding Parent Objects. This enables the check marks that were
grayed out. You can now selectively exclude parent rows from the subset by
deselecting the check box.
Select a node within the table and click Add Descendants to include related
rows. In the dialog that opens, make appropriate selections and click OK.
As you make these refinements, columns on the right reflect the effect on space
estimates for the subset. The Related Objects section also denotes the processing
order of the ancestor and descendant tables, including the detailed impact of
including each object. When you are done with the refinements, go to the Space
Estimates tab to see a finer granularity of the impact on the overall size of the
subset.
11. In the Default Object Rows section of the Object Rules tab, choose whether you
want to include or exclude the objects not affected by the defined rules in the
subset.
When you select the Include All Rows option, all of the rows for the object are
selected as part of the subset.
This is a global rule and applies to the entire subset. You can only select the
Include All Rows option when all of the rules have a scope of None. A scope of
None is established when you uncheck the Include Related Rows option in the
Object Rule pop-up.
Note:
For a subset definition that has column rules (see Step 12), be sure to use object
rules to include the corresponding objects. You can use the Default Object
Rules option to include all objects not affected by object rules, if required.
12. Optional: Click the Column Mask Rules tab to perform inline masking as part of
Click Create and enter search criteria to filter on columns within the schema.
These would typically be vertical columns such as CLOB AND BLOB
columns.
Note:
If you are using column mask rules instead of masking definitions (see Step
13), you can select no more than 10 columns in a given table. This restriction
applies to the export method but not to the in-place delete method.
Click OK.
b.
Select a row or rows in the column search results and click Manage Masking
Formats.
c.
In the pop-up dialog, select a masking format and value to apply to the
columns. For multiselection, the same format must be appropriate for all
columns. If you select multiple columns, ensure that the column rule format
you choose is applicable to the selected columns. Use the columns (flags) not
null and unique to enforce compliance.
Click OK to apply the masking format to the columns.
13. Optional: Click the Data Masking Definitions tab to include masking definitions
as part of the subsetting operation or to perform at the source data masking only.
a.
Click Add.
b.
No single table within a masking definition can have more than 10 masked
columns if you are using the export method. The restriction does not apply to
the in-place delete method.
Click OK.
The search results appear in the data masking table.
14. Click the Space Estimates tab.
Note the value in the Estimated Subset Size MB column. The space estimates
depend on optimizer statistics, and the actual distribution of data can only be
calculated if histogram statistics are present.
Whenever you add new rules, recheck the space estimates for updated values.
Data in the Space Estimates subpage is sorted with the largest applications
appearing at the top.
Note:
You can specify a pre-subset script to run on the subset database before you
select subset data.
You can specify a post-subset script to run on the subset database after you
assemble the subset data.
Either script type runs on the source database.
16. Click Return.
The definition is complete and displayed in the Data Subsetting Definitions table.
You can now proceed with script generation. Alternatively, you may want to save the
script for future use. In either case, you must decide whether to export data to a dump
file or delete data from a target database.
Tip:
If you have a very large database of 4 terabytes, for instance, and you want to
export a small percentage of the rows, such as 10%, it is more advantageous to
use the export method. Using the in-place delete method would require 3.6
terabytes of data, which would not perform as quickly as the export method.
The in-place delete method is recommended when the amount of data being
deleted is a small percentage of the overall data size.
There is an EMCLI verb if you want to perform an in-place delete remotely or
script it.
Generating a Subset Script
Saving a Subset Script
2. Select a target database that is either the same target database you used to create
the subset model, or similar to this database regarding the table schema and
objects.
3. Decide if you want to create a subset by writing subset data to export files, or by
depend on whether you chose the export or delete option in the previous step.
For Writing Subset Data to Export Files, provide the requisite information, then
click Continue to schedule the job.
Specify a subset directory where to save the export dump. The drop-down list
consists of directory objects for which you have access. Alternatively, you can
select a custom directory path. Click the check box if you want to speed the
process by using an external directory. Recommended default:
DATA_PUMP_DIR.
Specify appropriate values if you want to override the defaults: enter a name for
the export file; specify a maximum file size in megabytes; specify the maximum
number of threads of active execution operating on behalf of the export job. This
enables you to consider trade-offs between resource consumption and elapsed
time.
Select whether to enable dump file compression and encryption. Enter and
confirm an encryption password, if appropriate. Log file generation is selected
by default.
For Deleting Data From a Target Database, provide the requisite information, then
click Continue to schedule the job.
Specify a subset directory where to save the subset scripts. The drop-down list
consists of directory objects for which you have access. Alternatively, you can
select a custom directory path. Recommended default: DATA_FILE_DIR.
You must enable the check box indicating that the selected target is not a
production database in order to proceed.
5. Click Continue to schedule the job from the Generate Subset Schedule pop-up,
then click Submit. For the delete option, you must specify and confirm an
encryption seed.
The Data Subset Definitions page reappears, and the Most Recent Job Status
column shows that the subset job is running, and subsequently that it has
succeeded.
After performing this procedure, you can now create a subset database with the
generated export files at any time.
Select the definition within the table, open the Actions menu, then select Save
Subset Script. The Subset Mode pop-up appears.
2.
Select a target database that is either the same target database you used to create
the subset model, or similar to this database regarding the table schema and
objects.
3.
Decide if you want to create a subset by writing subset data to export files, or by
deleting data from a target database.
Choosing to delete data creates an in-place subset by removing/deleting
unwanted data from a cloned copy of the production database, rather than a
production database. Only data satisfying the rules are retained. You should
never use this option on a production database.
Select either Named Credentials or New Credentials if you have not already set
preferred credentials.
If you have defined any parameters from the Rule Parameters tab, they appear in
the table at the bottom. You can change a parameter value by clicking on the
associated field in the Value column.
4.
Click Continue to access the Parameters pop-up. The contents of the pop-up
depend on whether you chose the export or delete option in the previous step.
For Writing Subset Data to Export Files, provide the requisite information, then
click Continue to schedule the job.
Specify a subset directory where to save the export dump. The drop-down list
consists of directory objects for which you have access. Alternatively, you can
select a custom directory path. Click the check box if you want to speed the
process by using an external directory. Recommended default:
DATA_PUMP_DIR.
Specify appropriate values if you want to override the defaults: enter a name
for the export file; specify a maximum file size in megabytes; specify the
maximum number of threads of active execution operating on behalf of the
export job. This enables you to consider trade-offs between resource
consumption and elapsed time.
Select whether to enable dump file compression and encryption. Enter and
confirm an encryption password, if appropriate. Log file generation is selected
by default.
For Deleting Data From a Target Database, provide the requisite information,
then click Continue to schedule the job.
Specify a subset directory where to save the subset scripts. The drop-down list
consists of directory objects for which you have access. Alternatively, you can
select a custom directory path. Recommended default: DATA_FILE_DIR.
You must enable the check box indicating that the selected target is not a
production database in order to proceed.
5.
Click Continue. A progress indicator tracks script generation. When complete, the
Files table lists the results of script generation.
6.
Click Download. In the File Download pop-up that appears, click Save.
7.
In the Save As pop-up that appears, navigate to a file location and click Save.
The file containing the scripts (SubsetBundle.zip) now appears at the specified
location on your desktop.
Note that if you want to change generated parameter settings, you can do so by
editing the following file in a text editor prior to executing the script:
subset_exec_params.lst
There are EMCLI verbs to export and import subset definitions and subset
dumps if you want to perform these operations remotely or script them.
From the Actions menu, select Import, then select File from Desktop.
2.
3.
From the Actions menu, select Import, then select Subset Dump.
2.
3.
4.
Click Submit
The job reads the dump files and loads the data into the selected target database.
To import a subset definition XML file from the Software Library:
1. From the Actions menu, select Import, then select File from Software Library.
2. In the Import Data Subset Definition from Software Library pop-up that appears:
From the Data Subset Definitions page, select the subset definition you want to
export.
2.
From the Actions menu, select Export, then select Selected Subset Definition.
3.
4.
In the Save As pop-up that appears, navigate to a file location and click Save.
The system converts the subset definition into an XML file that now appears at the
specified location on your desktop.
The system converts the subset definition into an XML file that now appears at the
specified location on your desktop.
After the job runs successfully, the subset template appears in the list of subsets in the
table on the Data Subset Definitions page.
2.
3.
4.
When the job completes, the subset definition is unlocked and available for use.
Definitions.
4. In the dialog that opens, select the type (administrator or role, or both). Search by
See Oracle Data Masking and Subsetting Access Rights for more information on
privileges within the test data management area.
Inline masking is available only with Oracle Database 11g and higher releases.
The benefits of integrating data masking with subsetting include the following:
Prepare the test system in a single flow
Avoid the necessity of maintaining large-size masked databases for test purposes
Exported data in the form of a dump file can be imported into multiple databases
without exposing sensitive data
Subsetting is enhanced by ability to discard columns containing chunks of large
data
You can select one or more data masking definitions during subset creation. The
masking definitions must be based on the same ADM as the current subset definition.
At the same time, you can significantly reduce the subset size by defining column
rules to set CLOB and BLOB columns to null (or another supported format such as
Fixed String, Fixed Number).
You generate a subset in two ways:
Export Dumpif masking definitions are part of the subset model, mapping tables
are created during generation, and the resulting dump contains masked values
In-Place Deletesubsetting is performed on a cloned copy of the production
database; if data masking is part of the subset model, pregenerated masking scripts
are executed on the target sequentially
Advantages of inline masking include the following:
Sensitive data never leaves the production environment and thus is not exposed
(Export Dump option).
There is no need to temporarily store data in a staging area.
Exported data can subsequently be imported into multiple environments.
You can define table rules to export only a subset of data, and can further trim the
volume by using column rules to eliminate large vertical columns.
You can mask the same data in different ways and import into different test
databases.
You can use the provisioning framework to create multiple copies of trimmed
down, referentially intact databases containing no sensitive data (in-place delete),
or import a dump file into multiple databases (export dump).
The section Creating a Data Subset Definition includes instructions for combining
data subsetting and data masking within the process of creating a subset definition.
See Data Masking,for information on data masking and creating a data masking
definition.
2.
3.
b.
c.
d.
b.
4.
2.
3.
b.
c.
d.
Save.
b.
4.
2.
b.
c.
d.
Save.
3.
4.
a.
b.
b.
On the Data Masking tab, search and select masking definitions. System
validation checks for overlapping columns that use multiple masking
definitions.
2.
3.
4.
b.
c.
d.
Save.
b.
b.
On the Data Masking tab, search and select masking definitions. System
validation checks for overlapping columns that use multiple masking
definitions.
2.
3.
b.
Click the Table Rules tab and select from existing options, if desired.
b.
c.
Specify filtering criteria to search for large-sized columns and select the
desired columns in the results table.
d.
Click Manage Masking Formats and select a format from the drop-down list.
Enter a value if appropriate to the selection.
e.
Generate the subset, using either the Export or In-Place Delete option.
2.
b.
b.
3.
On the Data Masking tab, search and select masking definitions. System
validation checks for overlapping columns that use multiple masking
definitions.
Select the subset definition on the Subset home page and export it.
The subset definition is saved on the client machine as an XML file that potentially
contains the following:
Information on selected applications
Rules and rule parameters
Selected masking definitions
Columns to set to null
Pre- and post-scripts
Had column rules been used, they would replace the masking definitions in the list.
Import a Subset Definition That Uses Inline Masking
As the Security Administrator, you want to import a subset definition XML file to
create replicas of the subset definition previously exported.
1.
2.
3.
On the subset home page, select Import Subset Dump from the Actions menu.
2.
3.
Provide the import type, tablespace options, and log file location details.
Lifecycle Management
4.
The job reads the dump files and loads the data into the selected target database.
Save Subset Script Bundle
As the Security Administrator, you want to save a subset script bundle so that it can be
executed on a target database external to Enterprise Manager.
This example presupposes the existence of a subset model that has required table rules
and masking definitions.
1.
On the subset home page, from the Actions menu, select Generate, then select
Subset.
2.
3.
a.
b.
c.
d.
Click Continue.
b.
If the subset is to be stored externally, click the check box and select the
location.
c.
Specify an export file name. Note that you can use the % wildcard.
d.
e.
Indicate whether to generate a log file and specify a log file name.
f.
Click Continue.
4.
Note the progress of the script file generation. When complete, click Download.
5.
Lifecycle Management
This section discusses the lifecycle management of Application Data Models, Data
Masking, and Data Subsetting definitions.
In the event of an Enterprise Manager user being dropped or modified, the user can
reassign the Application Data Model, data masking and data subsetting definitions to
another user in the system. However, if the user doesnt reassign the Application Data
Model, data masking and data subsetting definitions to another user, these definitions
are automatically reassigned to the SYSMAN user.
When you try to reassign the Application Data Model, data masking and data
subsetting definitions to another user in the system, and if the reassigned user already
has a definition with the same name, the original definitions are renamed.
For example: User A has an Application Data Model, ADM1, and user B also has an
Application Data Model, named ADM1. If user B is being dropped, and you choose to
Lifecycle Management
assign its definitions to user A, the original definition ADM1 is renamed to ADM1_B,
The original definitions with the same name are renamed by suffixing "_" and adding
the user name that is being dropped. After the reassignment, user A will now have
both definitions ADM1 and ADM1_B.
Lifecycle Management
Index
A
Application Data Model (ADM)
application tables, viewing and editing, 3-5
associating a database to, 3-8, 3-9
creating
prerequisites for, 3-2
definition of, 3-1
discovering sensitive columns, 3-6, 4-27
editing application tables, 3-5
manually adding referential relationships, 3-6,
4-27
C
changing the type for sensitive columns, 3-7
cloning production database in data masking, 4-5
creating
data masking definition, 4-16
data subset definition, 5-1
masking definition, 4-3
masking formats, 4-6
sensitive column type, 3-8
D
data masking
adding columns to a masking definition, 4-6
advanced options, 4-21
auto mask, 4-10
Canadian social insurance numbers, 4-10
cloning and masking, 4-30
cloning the production database, 4-5, 4-29
Create Masking Definition page, 4-17
Index-1
E
editing application tables, 3-5
exporting
subset templates, data subsetting, 5-11
H
hiding data using data masking, 4-1
I
importing
data masking templates, 4-28
subset templates, data subsetting, 5-9
M
manually adding sensitive columns, 3-8
mask format libraries, 4-3
masking definitions
credit card numbers, 4-9
masking formats
entry options, 4-11
predefined, 4-8
R
referential relationships
manually adding, 3-6, 4-27
viewing, 3-6
regulatory compliance using masked data, 2-3
Right to Financial Privacy Act of 1978, 2-3
S
Sarbanes-Oxley regulatory requirements, 2-3
Secure Test Data Management, 3-1
security
compliance with masked data, 2-3
data masking, 4-1
list of Oracle products, 4-2
mask format libraries, 4-3
masking definitions, 4-3
security administrator, data masking and, 4-4
sensitive columns
changing the type for, 3-7
creating the type for, 3-8
discovering, 3-6, 4-27
discovering automatically, 3-7
manually adding, 3-8
performing discovery of, 4-27
staging region, data masking and, 4-3
statuses for Source Database Status column, ADM,
3-10
U
upgrading
data masking, 4-31
V
O
Oracle Data Masking and Subsetting Pack, 3-1
P
post-processing functions, data masking, 4-7
prerequisites for creating an ADM, 3-2
Index-2
viewing
application tables, 3-5
referential relationships, 3-6
W
Where clause, specifying for data subsetting, 5-3