Datastage Parallel Job Advanced Developers Guide

Ascential DataStage
for Ascential DataStage™ Enterprise Edition
Parallel Job Advanced Developer’s

Guide
Version 7.5.1
Part No. 00D-030DS751

December 2004
his document, and the software described or referenced in it, are confidential and proprietary to Ascential Software
Corporation ("Ascential"). They are provided under, and are subject to, the terms and conditions of a license
agreement between Ascential and the licensee, and may not be transferred, disclosed, or otherwise provided to
third parties, unless otherwise permitted by that agreement. No portion of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying,
recording, or otherwise, without the prior written permission of Ascential. The specifications and other
information contained in this document for some purposes may not be complete, current, or correct, and are
subject to change without notice. NO REPRESENTATION OR OTHER AFFIRMATION OF FACT CONTAINED IN THIS
DOCUMENT, INCLUDING WITHOUT LIMITATION STATEMENTS REGARDING CAPACITY, PERFORMANCE, OR
SUITABILITY FOR USE OF PRODUCTS OR SOFTWARE DESCRIBED HEREIN, SHALL BE DEEMED TO BE A
WARRANTY BY ASCENTIAL FOR ANY PURPOSE OR GIVE RISE TO ANY LIABILITY OF ASCENTIAL WHATSOEVER.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL ASCENTIAL BE LIABLE FOR ANY CLAIM, OR
ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. If you
are acquiring this software on behalf of the U.S. government, the Government shall have only "Restricted Rights" in
the software and related documentation as defined in the Federal Acquisition Regulations (FARs) in Clause
52.227.19 (c) (2). If you are acquiring the software on behalf of the Department of Defense, the software shall be
classified as "Commercial Computer Software" and the Government shall have only "Restricted Rights" as defined
in Clause 252.227-7013 (c) (1) of DFARs.
This product or the use thereof may be covered by or is licensed under one or more of the following issued
patents: US6604110, US5727158, US5909681, US5995980, US6272449, US6289474, US6311265, US6330008,
US6347310, US6415286; Australian Patent No. 704678; Canadian Patent No. 2205660; European Patent No. 799450;
Japanese Patent No. 11500247.
© 2005 Ascential Software Corporation. All rights reserved. DataStage®, EasyLogic®, EasyPath®, Enterprise Data
Quality Management®, Iterations®, Matchware®, Mercator®, MetaBroker®, Application Integration, Simplified®,
Ascential™, Ascential AuditStage™, Ascential DataStage™, Ascential ProfileStage™, Ascential QualityStage™,
Ascential Enterprise Integration Suite™, Ascential Real-time Integration Services™, Ascential MetaStage™, and
Ascential RTI™ are trademarks of Ascential Software Corporation or its affiliates and may be registered in the
United States or other jurisdictions.
The software delivered to Licensee may contain third-party software code. See Legal Notices (legalnotices.pdf) for
more information.
How to Use this Guide
Ascential DataStage™ is a powerful software suite that is used to

develop and run DataStage jobs. A DataStage job can extract from
different sources, and then cleanse, integrate, and transform the data
according to your requirements. The clean data is ready to be
imported into a data warehouse for analysis and processing by
business information software.
This manual gives information that might be required by advanced
users of parallel jobs. For basic information about using DataStage,
see DataStage Designer Guide and DataStage Manager Guide. For
basic information about designing parallel jobs, see Parallel Job
Developer’s Guide.
To find particular topics you can:
Use the Guide’s contents list (at the beginning of the Guide).
Use the Guide’s index (at the end of the Guide).
Use the Adobe Acrobat Reader bookmarks.
Use the Adobe Acrobat Reader search facility (select Edit ➤
Search).
The guide contains links both to other topics within the guide, and to
other guides in the DataStage manual set. The links are shown in blue.
Note that, if you follow a link to another manual, you will jump to that
manual and lose your place in this manual. Such links are shown in
italics.
Organization of This Manual

This manual contains the following:
Chapter 1 contains an inroduction to the manual, including some
of the terminology used.
Chapter 2 contains job design tips and gives a guide to good
design paractice for parallel jobs.
Chapter 3 gives some tips for improving theperformance of a
parallel job.
Parallel Job Advanced Developer’s Guide iii

Documentation Conventions How to Use this Guide
Chapter 4 explains the link buffering used by parallel jobs in detail.

Chapter 5 gives a guide to specifying your own, customized
parallel job stages.
Chapter 6 describes the environment variables available in the
parallel job environment.
Chapter 7 describes the job control interfaces that you can use to
run DataStage jobs from other programs or from the command
line.
Documentation Conventions
This manual uses the following conventions:
Convention Usage
Bold In syntax, bold indicates commands, function names, keywords,
and options that must be input exactly as shown. In text, bold
indicates keys to press, function names, and menu selections.
UPPERCASE In syntax, uppercase indicates BASIC statements and functions

and SQL statements and keywords.
Italic In syntax, italic indicates information that you supply. In text,

italic also indicates UNIX commands and options, file names,
and pathnames.
Plain In text, plain indicates Windows commands and options, file

names, and path names.
Lucida The Lucida Typewriter font indicates examples of source code
Typewriter and system output.
Lucida In examples, Lucida Typewriter bold indicates characters that
Typewriter the user types or keys the user presses (for example,
<Return>).
[] Brackets enclose optional items. Do not type the brackets unless

indicated.
{} Braces enclose nonoptional items from which you must select

at least one. Do not type the braces.
itemA | itemB A vertical bar separating items indicates that you can choose
only one item. Do not type the vertical bar.
... Three periods indicate that more of the same type of item can
optionally follow.
➤ A right arrow between menu commands indicates you should

choose each command in sequence. For example, “Choose File
➤ Exit” means you should choose File from the menu bar,
then choose Exit from the File pull-down menu.
iv Parallel Job Advanced Developer’s Guide

How to Use this Guide Documentation Conventions
Convention Usage
This line The continuation character is used in source code examples to
➥ continues indicate a line that is too long to fit on the page, but must be
entered as a single line on screen.
DataStage Documentation
DataStage documentation includes the following:
DataStage Director Guide: This guide describes the DataStage
Director and how to validate, schedule, run, and monitor
DataStage parallel jobs.
DataStage Manager Guide: This guide describes the DataStage
Manager and describes how to use and maintain the DataStage
Repository.
DataStage Designer Guide: This guide describes the DataStage
Designer, and gives a general description of how to create, design,
and develop a DataStage application.
DataStage Server: Server Job Developer’s Guide: This guide
describes the tools that are used in building a server job, and it
supplies programmer’s reference information..
DataStage Enterprise Edition: Parallel Job Developer’s
Guide: This guide describes the tools that are used in building a
parallel job, and it supplies programmer’s reference information.
DataStage Enterprise Edition: Parallel Job Advanced
Developer’s Guide: This guide gives more specialized
information about parallel job design.
DataStage Enterprise MVS Edition: Mainframe Job
Developer’s Guide: This guide describes the tools that are used
in building a mainframe job, and it supplies programmer’s
reference information..
DataStage Administrator Guide: This guide describes
DataStage setup, routine housekeeping, and administration.
DataStage Install and Upgrade Guide. This guide contains
instructions for installing DataStage on Windows and UNIX
platforms, and for upgrading existing installations of DataStage.
DataStage NLS Guide. This Guide contains information about
using the NLS features that are available in DataStage when NLS
is installed.
These guides are also available online in PDF format. You can read
them using the Adobe Acrobat Reader supplied with DataStage. See
Install and Upgrade Guide for details on installing the manuals and
the Adobe Acrobat Reader.
Parallel Job Advanced Developer’s Guide v

Documentation Conventions How to Use this Guide
You can use the Acrobat search facilities to search the whole
DataStage document set. To use this feature, select Edit ➤ Search
then choose the All PDF documents in option and specify the
DataStage docs directory (by default this is C:\Program
Files\Ascential\DataStage\Docs).
Extensive online help is also supplied. This is particularly useful when
you have become familiar with DataStage, and need to look up
specific information.
vi Parallel Job Advanced Developer’s Guide

Contents
How to Use this Guide

Organization of This Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Chapter 1
Introduction
Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Chapter 2
Job Design Tips
DataStage Designer Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Processing Large Volumes of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Modular Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Designing for Good Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
Combining Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Sorting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
Default and Explicit Type Conversions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
Using Transformer Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-7
Using Sequential File Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
Using Database Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
Database Sparse Lookup vs. Join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
DB2 Database Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
Oracle Database Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10
Teradata Database Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
Parallel Job Advanced Developer’s Guide vii

Contents
Chapter 3
Improving Performance
Understanding a Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Score Dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Example Score Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Tips for Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Performance Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
JOB MONITOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
Iostat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Load Average . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Runtime Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
OS/RDBMS Specific Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6
Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6
Selectively Rewriting the flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6
Identifying Superfluous Repartitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7
Identifying Buffering Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7
Resolving Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Choosing the Most Efficient Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Partitioner Insertion, Sort Insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Combinable Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
Disk I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
Ensuring Data is Evenly Partitioned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
Platform Specific Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-11
Tru64 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-11
HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
Disk Space Requirements of Post-Release 7.0.1 Datasets . . . . . . . . . . . . . . . 3-12
Chapter 4
Link Buffering
Buffering Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Controlling Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Buffering Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Overriding Default Buffering Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Operators with Special Buffering Requirements. . . . . . . . . . . . . . . . . . . . . 4-5
viii Parallel Job Advanced Developer’s Guide

Contents
Chapter 5
Specifying Your Own Parallel Stages
Defining Custom Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Defining Build Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-10
Build Stage Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-21
How Your Code is Executed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-23
Inputs and Outputs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-24
Example Build Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25
Defining Wrapped Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-31
Example Wrapped Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-40
Chapter 6
Environment Variables
Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-6
APT_BUFFER_FREE_RUN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-6
APT_BUFFER_MAXIMUM_MEMORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7
APT_BUFFER_MAXIMUM_TIMEOUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7
APT_BUFFER_DISK_WRITE_INCREMENT . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7
APT_BUFFERING_POLICY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7
APT_SHARED_MEMORY_BUFFERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7
Building Custom Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
DS_OPERATOR_BUILDOP_DIR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
OSH_BUILDOP_CODE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
OSH_BUILDOP_HEADER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
OSH_BUILDOP_OBJECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
OSH_BUILDOP_XLC_BIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-8
OSH_CBUILDOP_XLC_BIN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
APT_COMPILER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
APT_COMPILEOPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
APT_LINKER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
APT_LINKOPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
DB2 Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
APT_DB2INSTANCE_HOME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
APT_DB2READ_LOCK_TABLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
APT_DBNAME. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
APT_RDBMS_COMMIT_ROWS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
DB2DBDFT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
Parallel Job Advanced Developer’s Guide ix

Contents
Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-10
APT_DEBUG_OPERATOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-11
APT_DEBUG_MODULE_NAMES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-11
APT_DEBUG_PARTITION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-11
APT_DEBUG_SIGNALS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-11
APT_DEBUG_STEP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-12
APT_DEBUG_SUBPROC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-12
APT_EXECUTION_MODE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-12
APT_PM_DBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_PM_GDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_PM_LADEBUG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_PM_SHOW_PIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_PM_XLDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_PM_XTERM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
APT_SHOW_LIBLOAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
Decimal Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_DECIMAL_INTERM_PRECISION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_DECIMAL_INTERM_SCALE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_DECIMAL_INTERM_ROUND_MODE . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
Disk I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_BUFFER_DISK_WRITE_INCREMENT . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_CONSISTENT_BUFFERIO_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-14
APT_EXPORT_FLUSH_COUNT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
APT_IO_MAP/APT_IO_NOMAP and
APT_BUFFERIO_MAP/APT_BUFFERIO_NOMAP . . . . . . . . . . . . . . . . . 6-15
APT_PHYSICAL_DATASET_BLOCK_SIZE. . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
General Job Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
APT_CHECKPOINT_DIR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
APT_CLOBBER_OUTPUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
APT_CONFIG_FILE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
APT_DISABLE_COMBINATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
APT_EXECUTION_MODE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-16
APT_ORCHHOME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-17
APT_STARTUP_SCRIPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-17
APT_NO_STARTUP_SCRIPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-17
APT_STARTUP_STATUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
APT_THIN_SCORE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
x Parallel Job Advanced Developer’s Guide

Contents
Job Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18

APT_MONITOR_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
APT_MONITOR_TIME. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
APT_NO_JOBMON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
APT_PERFORMANCE_DATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19
APT_COPY_TRANSFORM_OPERATOR. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19
APT_DATE_CENTURY_BREAK_YEAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19
APT_IMPEXP_ALLOW_ZERO_LENGTH_FIXED_NULL . . . . . . . . . . . . . . . . 6-19
APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS . . . . . . . . . . . . . . . . 6-19
APT_INSERT_COPY_BEFORE_MODIFY . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19
APT_OLD_BOUNDED_LENGTH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-19
APT_OPERATOR_REGISTRY_PATH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-20
APT_PM_NO_SHARED_MEMORY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-20
APT_PM_NO_NAMED_PIPES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-20
APT_PM_SOFT_KILL_WAIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-20
APT_PM_STARTUP_CONCURRENCY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-20
APT_RECORD_COUNTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21
APT_SAVE_SCORE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21
APT_SHOW_COMPONENT_CALLS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21
APT_STACK_TRACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21
APT_WRITE_DS_VERSION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-21
OSH_PRELOAD_LIBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-22
Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-22
APT_IO_MAXIMUM_OUTSTANDING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-22
APT_IOMGR_CONNECT_ATTEMPTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-22
APT_PM_CONDUCTOR_HOSTNAME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-22
APT_PM_NO_TCPIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
APT_PM_NODE_TIMEOUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
APT_PM_SHOWRSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
APT_PM_USE_RSH_LOCALLY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
Parallel Job Advanced Developer’s Guide xi

Contents
NLS Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23

APT_COLLATION_SEQUENCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
APT_COLLATION_STRENGTH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23
APT_ENGLISH_MESSAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_IMPEXP_CHARSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_INPUT_CHARSET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_OS_CHARSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_OUTPUT_CHARSET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_STRING_CHARSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
Oracle Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-24
APT_ORACLE_LOAD_DELIMITED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-25
APT_ORACLE_LOAD_OPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-25
APT_ORACLE_NO_OPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-25
APT_ORACLE_PRESERVE_BLANKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-25
APT_ORA_IGNORE_CONFIG_FILE_PARALLELISM . . . . . . . . . . . . . . . . . . 6-26
APT_ORA_WRITE_FILES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-26
APT_ORAUPSERT_COMMIT_ROW_INTERVAL
APT_ORAUPSERT_COMMIT_TIME_INTERVAL . . . . . . . . . . . . . . . . . . 6-26
Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-26
APT_NO_PART_INSERTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-26
APT_PARTITION_COUNT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-26
APT_PARTITION_NUMBER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-27
Reading and Writing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-27
APT_DELIMITED_READ_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-27
APT_FILE_IMPORT_BUFFER_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-27
APT_FILE_EXPORT_BUFFER_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-28
APT_IMPORT_PATTERN_USES_FILESET. . . . . . . . . . . . . . . . . . . . . . . . . . 6-28
APT_MAX_DELIMITED_READ_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-28
APT_PREVIOUS_FINAL_DELIMITER_COMPATIBLE . . . . . . . . . . . . . . . . . 6-28
APT_STRING_PADCHAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-28
xii Parallel Job Advanced Developer’s Guide

Contents
Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-28
APT_DUMP_SCORE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-29
APT_ERROR_CONFIGURATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-29
APT_MSG_FILELINE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-30
APT_PM_PLAYER_MEMORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-30
APT_PM_PLAYER_TIMING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
APT_RECORD_COUNTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
OSH_DUMP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
OSH_ECHO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
OSH_EXPLAIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
OSH_PRINT_SCHEMAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31
SAS Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_HASH_TO_SASHASH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_NO_SASOUT_INSERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_NO_SAS_TRANSFORMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_SAS_ACCEPT_ERROR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_SAS_CHARSET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-32
APT_SAS_CHARSET_ABORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_COMMAND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SASINT_COMMAND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_DEBUG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_DEBUG_IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_DEBUG_LEVEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_DEBUG_VERBOSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-33
APT_SAS_NO_PSDS_USTRING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
APT_SAS_S_ARGUMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
APT_SAS_SCHEMASOURCE_DUMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
APT_SAS_SHOW_INFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
APT_SAS_TRUNCATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-34
APT_NO_SORT_INSERTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35
APT_SORT_INSERTION_CHECK_ONLY . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35
Parallel Job Advanced Developer’s Guide xiii

Contents
Teradata Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35

APT_TERA_64K_BUFFERS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35
APT_TERA_NO_ERR_CLEANUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35
APT_TERA_NO_SQL_CONVERSION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35
APT_TERA_NO_PERM_CHECKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
APT_TERA_SYNC_DATABASE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
APT_TERA_SYNC_PASSWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
APT_TERA_SYNC_USER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
Transport Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
APT_AUTO_TRANSPORT_BLOCK_SIZE. . . . . . . . . . . . . . . . . . . . . . . . . . . 6-36
APT_LATENCY_COEFFICIENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-37
APT_DEFAULT_TRANSPORT_BLOCK_SIZE. . . . . . . . . . . . . . . . . . . . . . . . 6-37
APT_MAX_TRANSPORT_BLOCK_SIZE/
APT_MIN_TRANSPORT_BLOCK_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . 6-37
Guide to Setting Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-37
Environment Variable Settings for all Jobs . . . . . . . . . . . . . . . . . . . . . . . . 6-37
Optional Environment Variable Settings . . . . . . . . . . . . . . . . . . . . . . . . . . 6-38
Chapter 7
DataStage Development Kit (Job Control Interfaces)
DataStage Development Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-2
The dsapi.h Header File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-2
Data Structures, Result Data, and Threads . . . . . . . . . . . . . . . . . . . . . . . . . 7-2
Writing DataStage API Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-3
Building a DataStage API Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4
Redistributing Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4
API Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4
DSAddEnvVar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-7
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-7
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-7
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-7
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-8
DSAddProject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9
xiv Parallel Job Advanced Developer’s Guide

Contents
DSCloseJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10
DSCloseProject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11
Return Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-11
DSDeleteEnvVar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12
DSDeleteProject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
Return Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
DSFindFirstLogEntry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-14
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-14
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-14
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-15
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-15
DSFindNextLogEntry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16
DSGetProjectList . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-17
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-17
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-17
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-17
DSGetJobInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-19
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-19
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-19
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-20
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-20
Parallel Job Advanced Developer’s Guide xv

Contents
DSGetLastError . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-21
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-21
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-21
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-21
DSGetLastErrorMsg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-22
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-22
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-22
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-22
Rermarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-22
DSGetLinkInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-23
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-23
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-23
Return Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-24
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-24
DSGetLogEntry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-25
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-25
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-25
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-25
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-25
DSGetNewestLogId . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-26
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-26
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-26
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-26
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-27
DSGetParamInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-28
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-28
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-28
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-28
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-28
DSGetProjectInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-30
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-30
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-30
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-30
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-30
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-31
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-31
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-31
xvi Parallel Job Advanced Developer’s Guide

Contents
DSGetReposInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-32
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-32
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-32
Return Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-33
DSGetReposUsage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-34
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-34
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-34
Return Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-35
DSGetStageInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-36
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-36
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-36
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-37
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-37
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-38
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-38
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-38
DSListEnvVars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-39
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-39
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-39
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-39
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-39
DSListProjectProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-40
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-40
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-40
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-40
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-41
DSLockJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-42
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-42
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-42
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-42
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-42
DSLogEvent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Parallel Job Advanced Developer’s Guide xvii

Contents
DSMakeJobReport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-44
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-44
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-44
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-44
DSOpenJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-45
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-45
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-45
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-45
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-45
DSOpenProject. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
DSRunJob. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-47
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-47
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-47
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-47
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-47
DSSetEnvVar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-49
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-49
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-49
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-49
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-50
DSSetGenerateOpMetaData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
DSSetJobLimit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
DSSetParam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-54
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-54
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-54
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-54
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-55
xviii Parallel Job Advanced Developer’s Guide

Contents
DSSetProjectProperty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-56
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-56
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-56
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-57
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-57
DSSetServerParams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-58
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-58
Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-58
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-58
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-58
DSStopJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-59
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-59
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-59
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-59
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-59
DSUnlockJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-60
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-60
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-60
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-60
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-60
DSWaitForJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-61
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-61
Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-61
Return Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-61
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-61
Data Structures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-62
DSCUSTINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-63
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-63
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-63
DSJOBINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-64
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-64
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-64
DSLINKINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-67
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-67
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-67
DSLOGDETAIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-68
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-68
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-68
Parallel Job Advanced Developer’s Guide xix

Contents
DSLOGEVENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-69
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-69
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-69
DSPARAM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-70
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-70
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-70
DSPARAMINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-72
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-72
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-72
DSPROJECTINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-74
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-74
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-74
DSREPOSINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-75
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-75
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-75
DSREPOSUSAGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-76
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-76
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-76
DSSTAGEINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-77
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-77
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-77
DSLINKINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-79
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-79
Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-79
Error Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-80
DataStage BASIC Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-87
DSAttachJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-90
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-90
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-90
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-90
DSCheckRoutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-91
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-91
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-91
DSDetachJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-92
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-92
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-92
xx Parallel Job Advanced Developer’s Guide

Contents
DSExecute. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-93
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-93
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-93
DSGetCustInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-94
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-94
DSIPCPageProps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-95
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-95
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-95
DSGetJobInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-96
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-96
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-99
Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-99
DSGetJobMetaBag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-100
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-100
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-100
DSGetLinkInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-101
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-101
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-102
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-102
DSGetLinkMetaData. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-103
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-103
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-103
DSGetLogEntry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-104
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-104
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-104
DSGetLogSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-105
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-105
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-106
DSGetNewestLogId . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-107
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-107
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-107
DSGetParamInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-108
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-108
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-109
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-109
DSGetProjectInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-111
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-111
Parallel Job Advanced Developer’s Guide xxi

Contents
DSGetStageInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-112
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-112
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-114
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-114
DSGetStageLinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-115
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-115
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-115
DSGetStagesOfType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-116
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-116
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-116
DSGetStagesTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-117
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-117
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-117
DSGetProjectInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-118
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-118
DSLogEvent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-119
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-119
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-119
DSLogFatal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-120
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-120
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-120
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-120
DSLogInfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-121
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-121
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-121
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-121
DSLogToController . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-122
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-122
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-122
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-122
DSLogWarn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-123
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-123
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-123
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-123
DSMakeJobReport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-124
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-124
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-124
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-124
xxii Parallel Job Advanced Developer’s Guide

Contents
DSMakeMsg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-125
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-125
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-125
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-125
DSPrepareJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-126
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-126
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-126
DSRunJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-127
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-127
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-127
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-127
DSSendMail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-128
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-128
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-129
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-129
DSSetGenerateOpMetaData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-130
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-130
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-130
DSSetJobLimit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-131
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-131
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-131
DSSetParam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-132
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-132
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-132
DSSetUserStatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-133
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-133
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-133
DSStopJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-134
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-134
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-134
DSTransformError . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-135
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-135
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-135
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-135
DSTranslateCode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-136
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-136
Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-136
Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-136
Parallel Job Advanced Developer’s Guide xxiii

Contents
DSWaitForFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-137
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-137
Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-137
DSWaitForJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-138
Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-138
Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-138
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-138
Job Status Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-139
Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-140
Commands for Controlling DataStage Jobs . . . . . . . . . . . . . . . . . . . . . . 7-140
Commands for Administering DataStage . . . . . . . . . . . . . . . . . . . . . . . . 7-151
Commands for Searching Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-157
XML Schemas and Sample Stylesheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-162
Appendix A
Header Files
C++ Classes – Sorted By Header File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
C++ Macros – Sorted By Header File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-6
xxiv Parallel Job Advanced Developer’s Guide

1
Introduction
This manual is intended for the DataStage Enterprise Edition user who
has mastered the basics of parallel job design and now wants to
progress further.
The manual covers the following topics:
Job Design Tips. This chapter contains miscellaneous tips about
designing parallel jobs, from use of the DataStage Designer
interface to handling large volumes of data.
Improving Performance. This chapter describes methods by which
you can evaluate the performance of your parallel job designs and
come up with strategies for improving them.
Link Buffering. This chapter contains an in-depth description of
when and how DataStage buffers data within a job, and how you
can change the automatic settings if required.
Specifying Your Own Parallel Stages. This chapter describe the
interface DataStage provides for defining your own parallel job
stage types.
Environment Variables. This chapter list all the environment
variables that are available for affecting the set up and operation
of parallel jobs.
DataStage Development Kit (Job Control Interfaces). This chapter
lists the various interfaces that enable you to run and control
DataStage jobs without using the DataStage Director client.
Parallel Job Advanced Developer’s Guide 1-1

Terminology Introduction
Terminology
Because of the technical nature of some of the descriptions in this
manual, we sometimes talks about details of the engine that drives
parallel jobs. This involves the use of terms that may be unfamiliar to
ordinary parallel job users.
Operators. These underlie the stages in a DataStage job. A single
stage may correspond to a single operator, or a number of
operators, depending on the properties you have set, and whether
you have chosen to partition or collect or sort data on the input
link to a stage. At compilation, DataStage evaluates your job
design and will sometimes optimize operators out if they are
judged to be superfluous, or insert other operators if they are
needed for the logic of the job.
OSH. This is the scripting language used internally by the
DataStage Enterprise Edition engine.
Players. Players are the workhorse processes in a parallel job.
There is generally a player for each operator on each node. Players
are the children of section leaders; there is one section leader per
processing node. Section leaders are started by the conductor
process running on the conductor node (the conductor node is
defined in the configuration file).
1-2 Parallel Job Advanced Developer’s Guide

2
Job Design Tips
This chapter gives some hints and tips for the good design of parallel
jobs.
DataStage Designer Interface

The following are some tips for smooth use of the DataStage Designer
when actually laying out your job on the canvas.
To re-arrange an existing job design, or insert new stage types into
an existing job flow, first disconnect the links from the stage to be
changed, then the links will retain any meta data associated with
them.
A Lookup stage can only have one input stream, one output
stream, and, optionally, one reject stream. Depending on the type
of lookup, it can have several reference links. To change the use of
particular Lookup links in an existing job flow, disconnect the links
from the Lookup stage and then right-click to change the link type,
for example, Stream to Reference.
The Copy stage is a good placeholder between stages if you
anticipate that new stages or logic will be needed in the future
without damaging existing properties and derivations. When
inserting a new stage, simply drag the input and output links from
the Copy placeholder to the new stage. Unless the Force property
is set in the Copy stage, DataStage optimizes the actual copy out
at runtime.

Processing Large Volumes of Data Job Design Tips
Processing Large Volumes of Data

The ability to process large volumes of data in a short period of time
depends on all aspects of the flow and the environment being
optimized for maximum throughput and performance. Performance
tuning and optimization are iterative processes that begin with job
design and unit tests, proceed through integration and volume
testing, and continue throughout the production life cycle of the
application. Here are some performance pointers:
When writing intermediate results that will only be shared
between parallel jobs, always write to persistent data sets (using
Data Set stages). You should ensure that the data is partitioned,
and that the partitions, and sort order, are retained at every stage.
Avoid format conversion or serial I/O.
Data Set stages should be used to create restart points in the
event that a job or sequence needs to be rerun. But, because data
sets are platform and configuration specific, they should not be
used for long-term backup and recovery of source data.
Depending on available system resources, it may be possible to
optimize overall processing time at run time by allowing smaller
jobs to run concurrently. However, care must be taken to plan for
scenarios when source files arrive later than expected, or need to
be reprocessed in the event of a failure.
Parallel configuration files allow the degree of parallelism and
resources used by parallel jobs to be set dynamically at runtime.
Multiple configuration files should be used to optimize overall
throughput and to match job characteristics to available hardware
resources in development, test, and production modes.
The proper configuration of scratch and resource disks and the
underlying filesystem and physical hardware architecture can
significantly affect overall job performance.
Within clustered ETL and database environments, resource-pool
naming can be used to limit processing to specific nodes,
including database nodes when appropriate.
Modular Development
You should aim to use modular development techniques in your job
designs in order to maximize the reuse of parallel jobs and
components and save yourself time.

Job Design Tips Designing for Good Performance
Use job parameters in your design and supply values at run time.
This allows a single job design to process different data in
different circumstances, rather than producing multiple copies of
the same job with slightly different arguments.
Using job parameters allows you to exploit the DataStage
Director’s multiple invocation capability. You can run several
invocations of a job at the same time with different runtime
arguments.
Use shared containers to share common logic across a number of
jobs. Remember that shared containers are inserted when a job is
compiled. If the shared container is changed, the jobs using it will
need recompiling (you can use the Usage Analysis tool in the
DataStage Manager to help you identify the jobs, and the multiple
job compile tool to recompile them).
Designing for Good Performance

Here are some tips for designing good performance into your job
from the outset.
Avoid unnecessary type conversions.

Be careful to use proper source data types, especially from Oracle.
You can set the OSH_PRINT_SCHEMAS environment variable to verify
that runtime schemas match the job design column definitions.
If you are using stage variables on a Transformer stage, ensure that
their data types match the expected result types.
Use Transformer stages sparingly and wisely

Transformer stages can slow down your job. Do not have multiple
stages where the functionality could be incorporated into a single
stage, and use other stage types to perform simple transformation
operations (see "Using Transformer Stages" on page 2-7 for more
guidance).
Increase Sort performance where possible

Careful job design can improve the performance of sort operations,
both in standalone Sort stages and in on-link sorts specified in the
Inputs page Partitioning tab of other stage types. See "Sorting Data"
on page 2-5 for guidance.

Combining Data Job Design Tips
Remove Unneeded Columns

Remove unneeded columns as early as possible within the job flow.
Every additional unused column requires additional buffer memory,
which can impact performance and make each row transfer from one
stage to the next more expensive. If possible, when reading from
databases, use a select list to read just the columns required, rather
than the entire table.
Avoid reading from sequential files using the Same

partitioning method.
Unless you have specified more than one source file, this will result in
the entire file being read into a single partition, making the entire
downstream flow run sequentially unless you explicitly repartition
(see "Using Sequential File Stages" on page 2-8 for more tips on using
Sequential file stages).
Combining Data
The two major ways of combining data in a DataStage job are via a
Lookup stage or a Join stage. How do you decide which one to use?
Lookup and Join stages perform equivalent operations: combining
two or more input datasets based on one or more specified keys.
When one unsorted input is very large or sorting is not feasible,
Lookup is preferred. When all inputs are of manageable size or are
pre-sorted, Join is the preferred solution.
The Lookup stage is most appropriate when the reference data for all
Lookup stages in a job is small enough to fit into available physical
memory. Each lookup reference requires a contiguous block of
physical memory. The Lookup stage requires all but the first input (the
primary input) to fit into physical memory.
If the reference to a lookup is directly from a DB2 or Oracle table and
the number of input rows is significantly smaller than the reference
rows, 1:100 or more, a Sparse Lookup may be appropriate.
If performance issues arise while using Lookup, consider using the
Join stage. The Join stage must be used if the datasets are larger than
available memory resources.

Job Design Tips Sorting Data
Sorting Data
Look at job designs and try to reorder the job flow to combine
operations around the same sort keys if possible, and coordinate your
sorting strategy with your hashing strategy. It is sometimes possible
to rearrange the order of business logic within a job flow to leverage
the same sort order, partitioning, and groupings.
If data has already been partitioned and sorted on a set of key
columns, specify the “don’t sort, previously sorted” option for the key
columns in the Sort stage. This reduces the cost of sorting and takes
greater advantage of pipeline parallelism.
When writing to parallel data sets, sort order and partitioning are
preserved. When reading from these data sets, try to maintain this
sorting if possible by using Same partitioning method.
The stable sort option is much more expensive than non-stable sorts,
and should only be used if there is a need to maintain row order other
than as needed to perform the sort.
The performance of individual sorts can be improved by increasing
the memory usage per partition using the Restrict Memory Usage
(MB) option of the Sort stage. The default setting is 20 MB per
partition. Note that sort memory usage can only be specified for
standalone Sort stages, it cannot be changed for inline (on a link)
sorts.
Default and Explicit Type Conversions

When you are mapping data from source to target you may need to
perform data type conversions. Some conversions happen
automatically, and these can take place across the output mapping of
any parallel job stage that has an input and an output link. Other
conversions need a function to explicitly perform the conversion.
These functions can be called from a Modify stage or a Transformer
stage, and are listed in Appendix B of DataStage Parallel Job
Developer’s Guide. (Modify is the preferred stage for such conversions
– see "Using Transformer Stages" on page 2-7.)
The following table shows which conversions are performed
automatically and which need to be explicitly performed. “d”
indicates automatic (default) conversion, “m” indicates that manual

Default and Explicit Type Conversions Job Design Tips
conversion is required, a blank square indicates that conversion is not

possible:
Destination Field
Source
timestamp
Field
decimal
uint32
uint64
uint16
string
dfloat
sfloat
uint8
int32
int64
int16
time
date
raw
int8 int8
d, d d d d d d d d d, d d, m m m
m m m
uint8 d d d d d d d d d d d
int16 d, d d d d d d d d d d,
m m
int32 d, d d d d d d d d d d, m m
m m
uint32 d d d d d d d d d d m m
int64 d, d d d d d d d d d d
m
sfloat d, d d d d d d d d d d
m
dfloat d, d d d d d d d d d, d, d, m m
m m m m
decimal d, d d d d, d d, d, d d, d, d,
m m m m m m m
string d, d d, d d d, d d d d, d, d, m m m
m m m m m m
raw m m d
date m m m m m m m
time m m m m d d,
m
timestamp m m m m m m d
You should also note the following points about type conversion:
When converting from variable-length to fixed-length strings
using default conversions, parallel jobs pad the remaining length
with NULL (ASCII zero) characters.

Job Design Tips Using Transformer Stages
The environment variable APT_STRING_PADCHAR can be used to

change the default pad character from an ASCII NULL (0x0) to
another character; for example, an ASCII space (Ox20) or a
unicode space (U+0020).
As an alternate solution, the PadString function can be used to
pad a variable-length (Varchar) string to a specified length using a
specified pad character. Note that PadString does not work with
fixed-length (Char) string types. You must first convert Char to
Varchar before using PadString.
Using Transformer Stages

In general, it is good practice not to use more Transformer stages than
you have to. You should especially avoid using multiple Transformer
stages where the logic can be combined into a single stage.
It is often better to use other stage types for certain types of operation:
Use a Copy stage rather than a Transformer for simple operations
such as:
– Providing a job design placeholder on the canvas. (Provided
you do not set the Force property to True on the Copy stage,
the copy will be optimized out of the job at run time.)
– Renaming columns.
– Dropping columns.
– Implicit type conversions (see "Default and Explicit Type
Conversions" on page 2-5).
Note that, if runtime column propagation is disabled, you can also
use output mapping on a stage to rename, drop, or convert
columns on a stage that has both inputs and outputs.
Use the Filter stage or the Switch stage to separate rows into
multiple output links based on SQL-like constraint expressions.
Use the Modify stage for explicit type conversion (see "Default
and Explicit Type Conversions" on page 2-5) and null handling.
Where complex, reusable logic is required, or where existing
Transformer-stage based job flows do not meet performance
requirements, consider building your own custom stage (see
Chapter 5, "Specifying Your Own Parallel Stages.")
Use a BASIC Transformer stage for large-volume job flows, or
where you want to take advantage of user-defined functions and
routines.

Using Sequential File Stages Job Design Tips
Using Sequential File Stages

Certain considerations apply when reading and writing fixed-length
fields using the Sequential File stage.
If reading columns that have an inherently variable-width type (for
example, integer, decimal, or varchar) then you should set the
Field Width property to specify the actual fixed-width of the input
column. Do this by selecting Edit Row… from the shortcut menu
for a particular column in the Columns tab, and specify the width
in the Edit Column Meta Data dialog box.
If writing fixed-width columns with types that are inherently
variable-width, then set the Field Width property and the Pad char
property in the Edit Column Meta Data dialog box to match the
width of the output column.
Other considerations are as follows:
If a column is nullable, you must define the null field value and
length in the Edit Column Meta Data dialog box.
Be careful when reading delimited, bounded-length varchar
columns (i.e., varchars with the length option set). If the source file
has fields which are longer than the maximum varchar length,
these extra characters are silently discarded.
Avoid reading from sequential files using the Same partitioning
method. Unless you have specified more than one source file, this
will result in the entire file being read into a single partition,
making the entire downstream flow run sequentially unless you
explicitly repartition.
Using Database Stages

In general you are better using ‘native’ database stages to access
certain databases rather than plug-in stages as the former give
maximum parallel performance and features. The natives stages are:
DB2/UDB Enterprise
Informix Enterprise
Oracle Enterprise
Teradata Enterprise
You should avoid generating target tables in the database from your
DataStage job (i.e., using the Create write mode on the database
stage) unless they are intended for temporary storage only. This is
because this method does not allow you to, for example, specify
target table space, and you may inadvertently violate data-

Job Design Tips Using Database Stages
management policies on the database. If you want to create a table on

a target database from within a job, use the Open command property
on the database stage to explicitly create the table and allocate
tablespace, or any other options required.
The Open command property allows you to specify a command (for
example some SQL) that will be executed by the database before it
processes any data from the stage. There is also a Close property that
allows you to specify a command to execute after the data from the
stage has been processed. (Note that, when using user-defined Open
and Close commands, you may need to explicitly specify locks where
appropriate.)
Database Sparse Lookup vs. Join

Data read by any database stage can serve as the reference input to a
Lookup stage. By default, this reference data is loaded into memory
like any other reference link.
When directly connected as the reference link to a Lookup stage, both
DB2/UDB Enterprise and Oracle Enterprise stages allow the lookup
type to be changed to Sparse and send individual SQL statements to
the reference database for each incoming Lookup row. Sparse Lookup
is only available when the database stage is directly connected to the
reference link, with no intermediate stages.
It is important to note that the individual SQL statements required by
a Sparse Lookup are an expensive operation from a performance
perspective. In most cases, it is faster to use a DataStage Join stage
between the input and DB2 reference data than it is to perform a
Sparse Lookup.
For scenarios where the number of input rows is significantly smaller
(1:100 or more) than the number of reference rows in a DB2 or Oracle
table, a Sparse Lookup may be appropriate.
DB2 Database Tips

Always use the DB2/UDB Enterprise stage in preference to the DB2/
API plugin stage for reading from, writing to, and performing lookups
against a DB2 Enterprise Server Edition with the Database Partitioning
Feature (DBF). The DB2/UDB Enterprise stage is designed for
maximum performance and scaleability against very large partitioned
DB2 UNIX databases.
The DB2/API plugin should only be used to read from and write to DB2
on other, non-UNIX platforms. You might, for example, use it to
access mainframe editions through DB2 Connect.

Using Database Stages Job Design Tips
Write vs. Load

The DB2/UDB Enterprise stage offers the choice between SQL
methods (insert, update, upsert, delete) or fast loader methods when
writing to a DB2 database. The choice between these methods
depends on the required performance, database log usage, and
recoverability considerations as follows:
The write method (using insert, update, upsert, or delete)
communicates directly with DB2 database nodes to execute
instructions in parallel. All operations are logged to the DB2
database log, and the target table(s) can be accessed by other
users. Time and row-based commit intervals determine the
transaction size and availability of new rows to other applications.
The load method requires that the user running the job has
DBADM privilege on the target database. During a load operation
an exclusive lock is placed on the entire DB2 tablespace into which
the data is being loaded, and so this tablespace cannot be
accessed by anyone else while the load is taking place. The load is
also non-recoverable: if the load operation is terminated before it
is completed, the contents of the table are unusable and the
tablespace is left in the load pending state. If this happens, the
DataStage job must be re-run with the stage set to truncate mode
to clear the load pending state.
Oracle Database Tips

When designing jobs that use Oracle sources or targets, note that the
parallel engine will use its interpretation of the Oracle meta data (e.g,
exact data types) based on interrogation of Oracle, overriding what
you may have specified in the Columns tab. For this reason it is best
to import your Oracle table definitions using the Import ➤
Orchestrate Schema Definitions command from the Table
Definitions category of the Repository view in the DataStage Designer
(also available in the Manager under Import ➤ Table Definitions ➤
Orchestrate Schema Definitions). Choose the Database table
option and follow the instructions from the wizard.
Loading and Indexes

When you use the Load write method in an Oracle Enterprise stage,
you are using the Parallel Direct Path load method. If you want to use
this method to write tables that have indexes on them (including
indexes automatically generated by primary key constraints), you
must specify the Index Mode property (you can set it to Maintenance
or Rebuild). An alternative is to set the environment variable
APT_ORACLE_LOAD_OPTIONS to “OPTIONS (DIRECT=TRUE,

Job Design Tips Using Database Stages
PARALLEL=FALSE). This allows the loading of indexed tables without

index maintenance, but the load is performed sequentially.
You can use the upsert write method to insert rows into an Oracle
table without bypassing indexes or constraints. In order to
automatically generate the SQL needed, set the Upsert Mode property
to Auto-generated and identify the key column(s) on the Columns tab
by selecting the Key check boxes.
Teradata Database Tips

You can use the Additional Connections Options property in the
Teradata Enterprise stage (which is a dependent of DB Options Mode)
to specify details about the number of connections to Teradata. The
possible values of this are:
sessionsperplayer. This determines the number of connections
each player in the job has to Teradata. The number should be
selected such that:
(sessions per player * number of nodes * players per node) = total
requested sessions
The default value is 2. Setting this too low on a large system can
result in so many players that the job fails due to insufficient
resources.
requestedsessions. This is a number between 1 and the number of
vprocs in the database. The default is the maximum number of
available sessions.

Using Database Stages Job Design Tips

3
Improving Performance
This chapter is intended to help resolve any performance problems. It

assumes that basic steps to assure performance have been taken: a
suitable configuration file has been set up (see "The Parallel Engine
Configuration File" in Parallel Job Developer’s Guide), reasonable
swap space configured etc. (see "Configuring for Enterprise Edition" in
DataStage Install and Upgrade Guide), and that you have followed the
design guidelines laid down in Chapter 1.
Understanding a Flow
In order to resolve any performance issues it is essential to have an
understanding of the flow of DataStage jobs.
Score Dumps
To help understand a job flow we suggest you take a score dump. Do
this by setting the APT_DUMP_SCORE environment variable true and
running the job (APT _DUMP_SCORE can be set in the Administrator
client, under the Parallel ➤ Reporting branch). This causes a report to
be produced which shows the operators, processes and data sets in
the job. The report includes information about:
Where and how data is repartitioned.
Whether DataStage had inserted extra operators in the flow.
The degree of parallelism each operator runs with, and on which
nodes.
Information about where data is buffered.

Tips for Debugging Improving Performance
The dump score information is included in the job log when you run a
job.
The score dump is particularly useful in showing you where
DataStage is inserting additional components in the job flow. In
particular DataStage will add partition and sort operators where the
logic of the job demands it. Sorts in particular can be detrimental to
performance and a score dump can help you to detect superfluous
operators and amend the job design to remove them.
Example Score Dump

The following score dump shows a flow with a single data set, which
has a hash partitioner, partitioning on key “a”. It shows three
operators: generator, tsort, and peek. Tsort and peek are “combined”,
indicating that they have been optimized into the same process. All
the operators in this flow are running on one node.
##I TFSC 004000 14:51:50(000) <main_program>
This step has 1 dataset:
ds0: {op0[1p] (sequential generator)
eOther(APT_HashPartitioner { key={ value=a }
})->eCollectAny
op1[2p] (parallel APT_CombinedOperatorController:tsort)}
It has 2 operators:
op0[1p] {(sequential generator)
on nodes (
lemond.torrent.com[op0,p0]
)}
op1[2p] {(parallel APT_CombinedOperatorController:
(tsort)
(peek)
) on nodes (
)}
It runs 3 processes on 2 nodes.
Tips for Debugging

Use the Data Set Management utility, which is available in the
Tools menu of the DataStage Designer or the DataStage Manager,
to examine the schema, look at row counts, and delete a Parallel
Data Set. You can also view the data itself.
Check the DataStage job log for warnings. These may indicate an
underlying logic problem or unexpected data type conversion.

Improving Performance Performance Monitoring
Enable the APT_DUMP_SCORE and APT_RECORD_COUNTS

environment variables. Also enable OSH_PRINT_SCHEMAS to
ensure that a runtime schema of a job matches the design-time
schema that was expected.
The UNIX command od –xc displays the actual data contents of
any file, including any embedded ASCII NULL characters.
The UNIX command, wc –lc filename, displays the number of lines
and characters in the specified ASCII text file. Dividing the total
number of characters by the number of lines provides an audit to
ensure that all rows are the same length. It is important to know
that the wc utility works by counting UNIX line delimiters, so if the
file has any binary columns, this count may be incorrect.
Performance Monitoring
There are various tools you can you use to aid performance
monitoring, some provided with DataStage and some general UNIX
tools.
JOB MONITOR
You access the DataStage job monitor through the DataStage Director
(see "Monitoring Jobs" in DataStage Director Guide). You can also use
certain dsjob commands from the command line to access monitoring
functions (see "Retrieving Information" on page 7-145 for details).
The Job Monitor provides a useful snapshot of a job’s performance at
a moment of execution, but does not provide thorough performance
metrics. That is, a Job Monitor snapshot should not be used in place
of a full run of the job, or a run with a sample set of data. Due to
buffering and to some job semantics, a snapshot image of the flow
may not be a representative sample of the performance over the
course of the entire job.
The CPU summary information provided by the Job Monitor is useful
as a first approximation of where time is being spent in the flow.
However, it does not include any sorts or similar that may be inserted
automatically in a parallel job. For these components, the score dump
can be of assistance. See "Score Dumps" on page 3-1.
A worst-case scenario occurs when a job flow reads from a data set,
and passes immediately to a sort on a link. The job will appear to
hang, when, in fact, rows are being read from the data set and passed
to the sort.

Performance Monitoring Improving Performance
The operation of the job monitor is controlled by two environment

variables: APT_MONITOR_TIME and APT_MONITOR_SIZE. By default
the job monitor takes a snapshot every five seconds. You can alter the
time interval by changing the value of APT_MONITOR_TIME, or you
can have the monitor generate a new snapshot every so-many rows
by following this procedure:
1 Select APT_MONITOR_TIME on the DataStage Administrator
environment variable dialog box, and press the set to default
button.
2 Select APT_MONITOR_SIZE and set the required number of rows
as the value for this variable.
Iostat
The UNIX tool Iostat is useful for examining the throughput of various
disk resources. If one or more disks have high throughput,
understanding where that throughput is coming from is vital. If there
are spare CPU cycles, IO is often the culprit.
The specifics of Iostat output vary slightly from system to system.
Here is an example from a Linux machine which slows a relatively
light load:
(The first set of output is cumulative data since the machine was
booted)
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 13.50 144.09 122.33 346233038 293951288
...
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 4.00 0.00 96.00 0 96
Load Average
Ideally, a performant job flow should be consuming as much CPU as is
available. The load average on the machine should be two to three
times the value as the number of processors on the machine (for
example, an 8-way SMP should have a load average of roughly 16-24).
Some operating systems, such as HPUX, show per-processor load
average. In this case, load average should be 2-3, regardless of
number of CPUs on the machine.
If the machine is not CPU-saturated, it indicates a bottleneck may exist
elsewhere in the flow. A useful strategy in this case is to over-partition
your data, as more partitions cause extra processes to be started,
utilizing more of the available CPU power.
If the flow cause the machine to be fully loaded (all CPUs at 100%),
then the flow is likely to be CPU limited, and some determination

Improving Performance Performance Monitoring
needs to be made as to where the CPU time is being spent (setting the
APT_PM_PLAYER _TIMING environment variable can be helpful here -
see the following section).
The commands top or uptime can provide the load average.
Runtime Information
When you set the APT_PM_PLAYER_TIMING environment variable,
information is provided for each operator in a job flow. This
information is written to the job log when the job is run.
An example output is:
##I TFPM 000324 08:59:32(004) <generator,0> Calling runLocally: step=1,
node=rh73dev04, op=0, ptn=0
##I TFPM 000325 08:59:32(005) <generator,0> Operator completed. status:
APT_StatusOk elapsed: 0.04 user: 0.00 sys: 0.00 suser: 0.09 ssys:
0.02 (total CPU: 0.11)
##I TFPM 000324 08:59:32(006) <peek,0> Calling runLocally: step=1,
node=rh73dev04, op=1, ptn=0
##I TFPM 000325 08:59:32(012) <peek,0> Operator completed. status:
0.02 (total CPU: 0.11)
##I TFPM 000324 08:59:32(013) <peek,1> Calling runLocally: step=1,
node=rh73dev04a, op=1, ptn=1
##I TFPM 000325 08:59:32(019) <peek,1> Operator completed. status:
0.02 (total CPU: 0.11)¨
This output shows us that each partition of each operator has

consumed about one tenth of a second of CPU time during its runtime
portion. In a real world flow, we’d see many operators, and many
partitions.
It is often useful to see how much CPU each operator (and each
partition of each component) is using. If one partition of an operator is
using significantly more CPU than others, it may mean the data is
partitioned in an unbalanced way, and that repartitioning, or choosing
different partitioning keys might be a useful strategy.
If one operator is using a much larger portion of the CPU than others,
it may be an indication that you’ve discovered a problem in your flow.
Common sense is generally required here; a sort is going to use
dramatically more CPU time than a copy. This will, however, give you
a sense of which operators are the CPU hogs, and when combined
with other metrics presented in this document can be very
enlightening.
Setting the environment variable APT_DISABLE_COMBINATION may
be useful in some situations to get finer-grained information as to

Performance Analysis Improving Performance
which operators are using up CPU cycles. Be aware, however, that

setting this flag will change the performance behavior of your flow, so
this should be done with care.
Unlike the job monitor cpu percentages, setting
APT_PM_PLAYER_TIMING will provide timings on every operator
within the flow.
OS/RDBMS Specific Tools

Each OS and RDBMS has its own set of tools which may be useful in
performance monitoring. Talking to the sysadmin or DBA may provide
some useful monitoring strategies.
Performance Analysis
Once you have carried out some performance monitoring, you can
analyze your results.
Bear in mind that, in a parallel job flow, certain operators may
complete before the entire flow has finished, but the job isn’t
successful until the slowest operator has finished all its processing.
Selectively Rewriting the flow

One of the most useful mechanisms in detecting the cause of
bottlenecks in your flow is to rewrite portions of it to exclude stages
from the set of possible causes. The goal of modifying the flow is to
see the new, modified, flow run noticeably faster than the original
flow. If the flow is running at roughly an identical speed, change the
flow further.
While editing a flow for testing, it is important to keep in mind that
removing one stage may have unexpected affects in the flow.
Comparing the score dump between runs is useful before concluding
what has made the performance difference.
When modifying the flow, be aware of introducing any new
performance problems. For example, adding a Data Set stage to a
flow might introduce disk contention with any other data sets being
read. This is rarely a problem, but might be significant in some cases.
Moving data into and out of parallel operation are two very obvious
areas of concern. Changing a job to write into a Copy stage (with no
outputs) will throw the data away. Keep the degree of parallelism the
same, with a nodemap if necessary. Similarly, landing any read data

Improving Performance Performance Analysis
to a data set can be helpful if the data’s point of origin is a flat file or
RDBMS.
This pattern should be followed, removing any potentially suspicious
operators while trying to keep the rest of the flow intact. Removing
any custom stages should be at the top of the list.
Identifying Superfluous Repartitions

Superfluous repartitioning should be identified. Due to operator or
license limitations (import, export, RDBMS ops, SAS, etc.) some
stages will run with a degree of parallelism that is different than the
default degree of parallelism. Some of these can’t be eliminated, but
understanding the where, when and why these repartitions occur is
important for flow analysis. Repartitions are especially expensive
when the data is being repartitioned on an MPP system, where
significant network traffic will result.
Sometimes you may be able to move a repartition upstream in order
to eliminate a previous, implicit repartition. Imagine an Oracle stage
performing a read (using the oraread operator). Some processing is
done on the data and it is then hashed and joined with another data
set. There might be a repartition after the oraread operator, and then
the hash, when only one repartitioning is really necessary.
Similarly, specifying a nodemap for an operator may prove useful to
eliminate repartitions. In this case, a transform stage sandwiched
between a DB2 stage reading (db2read) and another one writing
(db2write) might benefit from a nodemap placed on it to force it to run
with the same degree of parallelism as the two db2 operators to avoid
two repartitions.
Identifying Buffering Issues

Buffering is one of the more complex aspects to parallel job
performance tuning. Buffering is described in detail in Chapter 4, "Link
Buffering."
The goal of buffering on a specific link is to make the producing
operator’s output rate match the consumption rate of the downstream
operator. In any flow where this is incorrect behavior for the flow (for
example, the downstream operator has two inputs, and waits until it
had exhausted one of those inputs before reading from the next)
performance is degraded. Identifying these spots in the flow requires
an understanding of how each operator involved reads its record, and
is often only found by empirical observation.
You can diagnose a buffering tuning issue when a flow runs slowly
when it is one massive flow, but each component runs quickly when

Resolving Bottlenecks Improving Performance
broken up. For example, replacing an Oracle write stage with a copy
stage vastly improves performance, and writing that same data to a
data set, then loading via an Oracle stage, also goes quickly. When the
two are put together, performance is poor.
"Buffering" on page 3-10 details specific, common buffering
configurations aimed at resolving various bottlenecks.
Resolving Bottlenecks
Choosing the Most Efficient Operators

Because DataStage Enterprise Edition offers a wide range of different
stage types, with different operators underlying them, there can be
several different ways of achieving the same effects within a job. This
section contains some hint as to preferred practice when designing
for performance is concerned. When analyzing your flow you should
try substituting preferred operators in particular circumstances.
Modify and Transform

Modify, due to internal implementation details, is a particularly
efficient operator. Any transformation which can be implemented in
the Modify stage will be more efficient than implementing the same
operation in a Transformer stage. Transformations that touch a single
column (for example, keep/drop, type conversions, some string
manipulations, null handling) should be implemented in a Modify
stage rather than a Transformer.
Lookup and Join

Lookup and join perform equivalent operations: combining two or
more input datasets based on one or more specified keys.
Lookup requires all but one (the first or primary) input to fit into
physical memory. Join requires all inputs to be sorted.
When one unsorted input is very large or sorting isn’t feasible, lookup
is the preferred solution. When all inputs are of manageable size or
are pre-sorted, join is the preferred solution.
Partitioner Insertion, Sort Insertion

Partitioner insertion and sort insertion each make writing a flow easier
by alleviating the need for a user to think about either partitioning or

Improving Performance Resolving Bottlenecks
sorting data. By examining the requirements of operators in the flow,

the parallel engine can insert partitioners, collectors and sorts as
necessary within a dataflow.
However, there are some situations where these features can be a
hindrance.
If data is pre-partitioned and pre-sorted, and the DataStage job is
unaware of this, you could disable automatic partitioning and sorting
for the whole job by setting the following environment variables while
the job runs:
APT_NO_PART_INSERTION
APT_NO_SORT_INSERTION
You can also disable partitioning on a per-link basis within your job
design by explicitly setting a partitioning method of Same on the
Input page Partitioning tab of the stage the link is input to.
To disable sorting on a per-link basis, insert a Sort stage on the link,
and set the Sort Key Mode option to Don’t Sort (Previously Sorted).
We advise that average users leave both partitioner insertion and sort
insertion alone, and that power users perform careful analysis before
changing these options.
Combinable Operators
Combined operators generally improve performance at least slightly
(in some cases the difference is dramatic). There may also be
situations where combining operators actually hurts performance,
however. Identifying such operators can be difficult without trial and
error.
The most common situation arises when multiple operators are
performing disk I/O (for example, the various file stages and sort). In
these sorts of situations, turning off combination for those specific
stages may result in a performance increase if the flow is I/O bound.
Combinable operators often provide a dramatic performance increase
when a large number of variable length fields are used in a flow.
Disk I/O
Total disk throughput is often a fixed quantity that DataStage has no
control over. It can, however, be beneficial to follow some rules.
If data is going to be read back in, in parallel, it should never be
written as a sequential file. A data set or file set stage is a much
more appropriate format.

Resolving Bottlenecks Improving Performance
When importing fixed-length data, the Number of Readers per

Node property on the Sequential File stage can often provide a
noticeable performance boost as compared with a single process
reading the data.
Some disk arrays have read ahead caches that are only effective
when data is read repeatedly in like-sized chunks. Setting the
environment variable APT_CONSISTENT_BUFFERIO_SIZE=N will
force stages to read data in chunks which are size N or a multiple
of N.
Memory mapped I/O, in many cases, contributes to improved
performance. In certain situations, however, such as a remote disk
mounted via NFS, memory mapped I/O may cause significant
performance problems. Setting the environment variables
APT_IO_NOMAP and APT_BUFFERIO_NOMAP true will turn off
this feature and sometimes affect performance. (AIX and HP-UX
default to NOMAP. Setting APT_IO_MAP and APT_BUFFERIO_MAP
true can be used to turn memory mapped I/O on for these
platforms.)
Ensuring Data is Evenly Partitioned

Because of the nature of parallel jobs, the entire flow runs only as fast
as its slowest component. If data is not evenly partitioned, the slowest
component is often slow due to data skew. If one partition has ten
records, and another has ten million, then a parallel job cannot make
ideal use of the resources.
Setting the environment variable APT_RECORD_COUNTS displays the
number of records per partition for each component. Ideally, counts
across all partititions should be roughly equal. Differences in data
volumes between keys often skew data slightly, but any significant
(e.g., more than 5-10%) differences in volume should be a warning
sign that alternate keys, or an alternate partitioning strategy, may be
required.
Buffering
Buffering is intended to slow down input to match the consumption
rate of the output. When the downstream operator reads very slowly,
or not at all, for a length of time, upstream operators begin to slow
down. This can cause a noticeable performance loss if the buffer’s
optimal behavior is something other than rate matching.
By default, each link has a 3 MB in-memory buffer. Once that buffer
reaches half full, the operator begins to push back on the upstream
operator’s rate. Once the 3 MB buffer is filled, data is written to disk in
1 MB chunks.

Improving Performance Platform Specific Tuning
In most cases, the easiest way to tune buffering is to eliminate the

pushback and allow it to buffer the data to disk as necessary. Setting
APT_BUFFER_FREE_RUN=N or setting Buffer Free Run in the
Output page Advanced tab on a particular stage will do this. A
buffer will read N * max_memory (3 MB by default) bytes before
beginning to push back on the upstream. If there is enough disk space
to buffer large amounts of data, this will usually fix any egregious
slowdown issues cause by the buffer operator.
If there is a significant amount of memory available on the machine,
increasing the maximum in-memory buffer size is likely to be very
useful if buffering is causing any disk I/O. Setting the
APT_BUFFER_MAXIMUM_MEMORY environment variable or
Maximum memory buffer size on the Output page Advanced tab on
a particular stage will do this. It defaults to 3145728 (3 MB).
For systems where small to medium bursts of I/O are not desirable,
the 1 MB write to disk size chunk size may be too small. The
environment variable APT_BUFFER_DISK_WRITE_INCREMENT or
Disk write increment on the Output page Advanced tab on a
particular stage controls this and defaults to 1048576 (1 MB). This
setting may not exceed max_memory * 2/3.
Finally, in a situation where a large, fixed buffer is needed within the
flow, setting Queue upper bound on the Output page Advanced
tab (no environment variable exists) can be set equal to max_memory
to force a buffer of exactly max_memory bytes. Such a buffer will
block an upstream operator (until data is read by the downstream
operator) once its buffer has been filled, so this setting should be used
with extreme caution. This setting is rarely, if ever, necessary to
achieve good performance, but may be useful in an attempt to
squeeze every last byte of performance out of the system where it is
desirable to eliminate buffering to disk entirely. No environment
variable is available for this flag, and therefore this can only be set at
the individual stage level.
Platform Specific Tuning
Tru64
In some cases improved performance can been achieved by setting
the virtual memory “eager” setting (vm_aggressive_swap kernel
parameter). This will aggressively swap processes out of memory to
free up physical memory for the running processes.

Disk Space Requirements of Post-Release 7.0.1 Datasets Improving Performance
Some environments have experienced better memory management

when the vm_swap_eager kernel is set. This swaps out idle processes
more quickly, allowing more physical memory for parallel jobs. A
higher degree of parallelism may be available as a result of this
setting, but system interactivity may suffer as a result.
We recommend that you set the environment variable
APT_PM_NO_SHARED memory for Tru64 version 51A (only).
HP-UX
HP-UX has a limitation when running in 32-bit mode, which limits
memory mapped I/O to 2 GB per machine. This can be an issue when
dealing with large lookups. The Memory Windows options can
provide a work around for this memory limitation. Ascential Product
Support can provide this document on request.
AIX
If you are running DataStage Enterprise Edition on an RS/6000 SP or a
network of workstations, verify your setting of the network parameter
thewall (see "Configuring for Enterprise Edition" in DataStage Install
and Upgrade Guide for details).
Disk Space Requirements of Post-Release

7.0.1 Datasets
Some parallel datasets generated with DataStage 7.0.1 and later
releases require more disk space when the columns are of type
VarChar when compared to 7.0. This is due to changes added for
performance improvements for bounded length VarChars in 7.0.1.
The preferred solution is to use unbounded length VarChars (don't set
any length) for columns where the maximum length is rarely used.
Alternatively, you can set the environment variable,
APT_OLD_BOUNDED_LENGTH (see page 6-19), but this is not
recommended, as it leads to performance degradation.

4
Link Buffering
DataStage automatically performs buffering on the links of certain

stages. This is primarily intended to prevent deadlock situations
arising (where one stage is unable to read its input because a previous
stage in the job is blocked from writing to its output).
Deadlock situations can occur where you have a fork-join in your job.
This is where a stage has two output links whose data paths are
joined together later in the job. The situation can arise where all the
stages in the flow are waiting for each other to read or write, so none
of them can proceed. No error or warning message is output for
deadlock; your job will be in a state where it will wait forever for an
input.
DataStage automatically inserts buffering into job flows containing
fork-joins where deadlock situations might arise. In most
circumstances you should not need to alter the default buffering
implemented by DataStage. However you may want to insert buffers
in other places in your flow (to smooth and improve performance) or
you may want to explicitly control the buffers inserted to avoid
deadlocks. DataStage allows you to do this, but we advise caution
when altering the default buffer settings.
Buffering Assumptions
This section describes buffering in more detail, and in particular the
design assumptions underlying its default behavior.
Buffering in DataStage is designed around the following assumptions:
Buffering is primarily intended to remove the potential for
deadlock in flows with fork-join structure.

Controlling Buffering Link Buffering
Throughput is preferable to overhead. The goal of the DataStage

buffering mechanism is to keep the flow moving with as little
memory and disk usage as possible. Ideally, data should simply
stream through the data flow and rarely land to disk. Upstream
operators should tend to wait for downstream operators to
consume their input before producing new data records.
Stages in general are designed so that on each link between
stages data is being read and written whenever possible. While
buffering is designed to tolerate occasional backlog on specific
links due to one operator getting ahead of another, it is assumed
that operators are at least occasionally attempting to read and
write data on each link.
Buffering is implemented by the automatic insertion of a hidden
buffer operator on links between stages. The buffer operator attempts
to match the rates of its input and output. When no data is being read
from the buffer operator by the downstream stage, the buffer operator
tries to throttle back incoming data from the upstream stage to avoid
letting the buffer grow so large that it must be written out to disk.
The goal is to avoid situations where data will be have to be moved to
and from disk needlessly, especially in situations where the consumer
cannot process data at the same rate as the producer (for example,
due to a more complex calculation).
Because the buffer operator wants to keep the flow moving with low
overhead, it is assumed in general that it is better to cause the
producing stage to wait before writing new records, rather than allow
the buffer operator to consume resources.
Controlling Buffering
DataStage offers two ways of controlling the operation of buffering:
you can use environment variables to control buffering on all links of
all stages in all jobs, or you can make individual settings on the links
of particular stages via the stage editors.
Buffering Policy
You can set this via the APT_BUFFERING_POLICY environment
variable, or via the Buffering mode field on the Inputs or Outputs
Page Advanced tab for individual stage editors.
The environment variable has the following possible values:

Link Buffering Controlling Buffering
AUTOMATIC_BUFFERING. Buffer a data set only if necessary to

prevent a dataflow deadlock. This setting is the default if you do
not define the environment variable.
FORCE_BUFFERING. Unconditionally buffer all links.
NO_BUFFERING. Do not buffer links. This setting can cause
deadlock if used inappropriately.
The possible settings for the Buffering mode field are:
(Default). This will take whatever the default settings are as
specified by the environment variables (this will be Auto buffer
unless you have explicitly changed the value of the
APT_BUFFERING _POLICY environment variable).
Auto buffer. Buffer data only if necessary to prevent a dataflow
deadlock situation.
Buffer. This will unconditionally buffer all data output from/input
to this stage.
No buffer. Do not buffer data under any circumstances. This
could potentially lead to deadlock situations if not used carefully.
Overriding Default Buffering Behavior

Since the default value of APT_BUFFERING_POLICY is
AUTOMATIC_BUFFERING, the default action of DataStage is to
buffer a link only if required to avoid deadlock. You can, however,
override the default buffering operation in your job.
For example, some operators read an entire input data set before
outputting a single record. The Sort stage is an example of this. Before
a sort operator can output a single record, it must read all input to
determine the first output record. Therefore, these operators
internally buffer the entire output data set, eliminating the need of the
default buffering mechanism. For this reason, DataStage never inserts
a buffer on the output of a sort.
You may also develop a customized stage that does not require its
output to be buffered, or you may want to change the size parameters
of the DataStage buffering mechanism. In this case, you can set the
various buffering parameters. These can be set via environment
variables or via the Advanced tab on the Inputs or Outputs page for
individual stage editors. What you set in the Outputs page
Advanced tab will automatically appear in the Inputs page
Advanced tab of the stage at the other end of the link (and vice versa)
The available environment variables are as follows:

APT_BUFFER_MAXIMUM_MEMORY. Specifies the maximum

amount of virtual memory, in bytes, used per buffer. The default
size is 3145728 (3 MB). If your step requires 10 buffers, each
processing node would use a maximum of 30 MB of virtual
memory for buffering. If DataStage has to buffer more data than
Maximum memory buffer size, the data is written to disk.
APT_BUFFER_DISK_WRITE_INCREMENT. Sets the size, in
bytes, of blocks of data being moved to/from disk by the buffering
operator. The default is 1048576 (1 MByte.) Adjusting this value
trades amount of disk access against throughput for small
amounts of data. Increasing the block size reduces disk access, but
may decrease performance when data is being read/written in
smaller units. Decreasing the block size increases throughput, but
may increase the amount of disk access.
APT_BUFFER_FREE_RUN. Specifies how much of the available
in-memory buffer to consume before the buffer offers resistance
to any new data being written to it, as a percentage of Maximum
memory buffer size. When the amount of buffered data is less
than the Buffer free run percentage, input data is accepted
immediately by the buffer. After that point, the buffer does not
immediately accept incoming data; it offers resistance to the
incoming data by first trying to output data already in the buffer
before accepting any new input. In this way, the buffering
mechanism avoids buffering excessive amounts of data and can
also avoid unnecessary disk I/O. The default percentage is 0.5
(50% of Maximum memory buffer size or by default 1.5 MB). You
must set Buffer free run greater than 0.0. Typical values are
between 0.0 and 1.0. You can set Buffer free run to a value greater
than 1.0. In this case, the buffer continues to store data up to the
indicated multiple of Maximum memory buffer size before writing
data to disk.
The available settings in the Input or Outputs page Advanced tab
of stage editors are:
Maximum memory buffer size (bytes). Specifies the maximum
amount of virtual memory, in bytes, used per buffer. The default
size is 3145728 (3 MB).
Buffer free run (percent). Specifies how much of the available
in-memory buffer to consume before the buffer resists. This is
expressed as a percentage of Maximum memory buffer size.
When the amount of data in the buffer is less than this value, new
data is accepted automatically. When the data exceeds it, the
buffer first tries to write some of the data it contains before
accepting more.
The default value is 50% of the Maximum memory buffer size. You
can set it to greater than 100%, in which case the buffer continues

Link Buffering Controlling Buffering
to store data up to the indicated multiple of Maximum memory

buffer size before writing to disk.
Queue upper bound size (bytes). Specifies the maximum
amount of data buffered at any time using both memory and disk.
The default value is zero, meaning that the buffer size is limited
only by the available disk space as specified in the configuration
file (resource scratchdisk). If you set Queue upper bound size
(bytes) to a non-zero value, the amount of data stored in the buffer
will not exceed this value (in bytes) plus one block (where the data
stored in a block cannot exceed 32 KB).
If you set Queue upper bound size to a value equal to or slightly
less than Maximum memory buffer size, and set Buffer free run to
1.0, you will create a finite capacity buffer that will not write to
disk. However, the size of the buffer is limited by the virtual
memory of your system and you can create deadlock if the buffer
becomes full.
(Note that there is no environment variable for Queue upper
bound size).
Disk write increment (bytes). Sets the size, in bytes, of blocks
of data being moved to/from disk by the buffering operator. The
default is 1048576 (1 MB). Adjusting this value trades amount of
disk access against throughput for small amounts of data.
Increasing the block size reduces disk access, but may decrease
performance when data is being read/written in smaller units.
Decreasing the block size increases throughput, but may increase
the amount of disk access.
Operators with Special Buffering Requirements

If you have built a custom stage that is designed to not consume one
of its inputs, for example to buffer all records before proceeding, the
default behavior of the buffer operator can end up being a
performance bottleneck, slowing down the job. This section describes
how to fix this problem.
Although the buffer operator is not designed for buffering an entire
data set as output by a stage, it is capable of doing so assuming
sufficient memory and/or disk space is available to buffer the data. To
achieve this you need to adjust the settings described above
appropriately, based on your job. You may be able to solve your
problem by modifying one buffering property, the Buffer free run
setting. This controls the amount of memory/disk space that the buffer
operator is allowed to consume before it begins to push back on the
upstream operator.

The default setting for Buffer free run is 0.5 for the environment
variable, (50% for Buffer free run on the Advanced tab), which
means that half of the internal memory buffer can be consumed
before pushback occurs. This biases the buffer operator to avoid
allowing buffered data to be written to disk.
If your stage needs to buffer large data sets, we recommend that you
initially set Buffer free run to a very large value such as 1000, and then
adjust according to the needs of your application. This will allow the
buffer operator to freely use both memory and disk space in order to
accept incoming data without pushback.
Ascential Software recommends that you set the Buffer free run
property only for those links between stages that require a non-
default value; this means altering the setting on the Inputs page or
Outputs page Advanced tab of the stage editors, not the
environment variable.

5
Specifying Your Own Parallel Stages
In addition to the wide range of parallel stage types available,

DataStage allows you to define your own stage types, which you can
then use in parallel jobs.There are three different types of stage that
you can define:
Custom. This allows knowledgeable Orchestrate users to specify
an Orchestrate operator as a DataStage stage. This is then
available to use in DataStage Parallel jobs.
Build. This allows you to design and build your own operator as a
stage to be included in DataStage Parallel Jobs.
Wrapped. This allows you to specify a UNIX command to be
executed by a DataStage stage. You define a wrapper file that in
turn defines arguments for the UNIX command and inputs and
outputs.
The DataStage Manager provides an interface that allows you to
define a new DataStage Parallel job stage of any of these types. This
interface is also available from the Repository window of the
DataStage Designer. This chapter describes how to use this interface.

Defining Custom Stages Specifying Your Own Parallel Stages
Defining Custom Stages

You can define a custom stage in order to include an Orchestrate
operator in a DataStage stage which you can then include in a
DataStage job. The stage will be available to all jobs in the project in
which the stage was defined. You can make it available to other
projects using the DataStage Manager Export/Import facilities. The
stage is automatically added to the job palette.
To define a custom stage type from the DataStage Manager:
1 Select the Stage Types category in the Repository tree.
2 Choose File ➤ New Parallel Stage ➤ Custom from the main
menu or New Parallel Stage ➤ Custom from the shortcut
menu. The Stage Type dialog box appears.
3 Fill in the fields on the General page as follows:

– Stage type name. This is the name that the stage will be
known by to DataStage. Avoid using the same name as
existing stages.
– Category. The category that the new stage will be stored in
under the stage types branch of the Repository tree view. Type
in or browse for an existing category or type in the name of a
new one. The category also determines what group in the
palette the stage will be added to. Choose an existing category
to add to an existing group, or specify a new category to create
a new palette group.

Specifying Your Own Parallel Stages Defining Custom Stages
– Parallel Stage type. This indicates the type of new Parallel

job stage you are defining (Custom, Build, or Wrapped). You
cannot change this setting.
– Execution Mode. Choose the execution mode. This is the mode
that will appear in the Advanced tab on the stage editor. You
can override this mode for individual instances of the stage as
required, unless you select Parallel only or Sequential only. See
"Advanced Tab" in Parallel Job Developer’s Guide for a
description of the execution mode.
– Mapping. Choose whether the stage has a Mapping tab or
not. A Mapping tab enables the user of the stage to specify
how output columns are derived from the data produced by
the stage. Choose None to specify that output mapping is not
performed, choose Default to accept the default setting that
DataStage uses.
– Preserve Partitioning. Choose the default setting of the
Preserve Partitioning flag. This is the setting that will appear in
the Advanced tab on the stage editor. You can override this
setting for individual instances of the stage as required. See
description of the preserve partitioning flag.
– Partitioning. Choose the default partitioning method for the
stage. This is the method that will appear in the Inputs page
Partitioning tab of the stage editor. You can override this
method for individual instances of the stage as required. See
"Partitioning Tab" in Parallel Job Developer’s Guide for a
description of the partitioning methods.
– Collecting. Choose the default collection method for the
stage. This is the method that will appear in the Inputs page
Partitioning tab of the stage editor. You can override this
method for individual instances of the stage as required. See
"Partitioning Tab" in Parallel Job Developer’s Guide for a
description of the collection methods.
– Operator. Enter the name of the Orchestrate operator that you
want the stage to invoke.
– Short Description. Optionally enter a short description of the
stage.
– Long Description. Optionally enter a long description of the
stage.

Go to the Links page and specify information about the links

allowed to and from the stage you are defining.
Use this to specify the minimum and maximum number of input

and output links that your custom stage can have, and to enable
the ViewData feature for target data (you cannot enable target
ViewData if your stage has any output links). When the stage is
used in a job design, a ViewData button appears on the Input
page, which allows you to view the data on the actual data target
(provided some has been written there).
In order to use the target ViewData feature, you have to specify an
Orchestrate operator to read the data back from the target. This
will usually be different to the operator that the stage has used to
write the data (i.e., the operator defined in the Operator field of
the General page). Specify the reading operator and associated
arguments in the Operator and Options fields.
If you enable target ViewData, a further field appears in the
Properties grid, called ViewData (see page 5-7).
4 Go to the Creator page and optionally specify information about
the stage you are creating. We recommend that you assign a
version number to the stage so you can keep track of any
subsequent changes.
You can specify that the actual stage will use a custom GUI by
entering the ProgID for a custom GUI in the Custom GUI Prog ID
field.
You can also specify that the stage has its own icon. You need to
supply a 16 x 16 bibitmap and a 32 x 32 bit bitmap to be displayed

in vatious place in the DataStage user interface. Click the 16 x 16

Bitmap button and browse for the smaller bitmap file. Click the
32 x 32 Bitmap button and browse for the large bitmap file. Click
the Reset Bitmap Info button to revert to using the default
DataStage icon for this stage..
5 Go to the Properties page. This allows you to specify the options

that the Orchestrate operator requires as properties that appear in
the Stage Properties tab. For custom stages the Properties tab
always appears under the Stage page.
6 Fill in the fields as follows:

– Property name. The name of the property.

– Data type. The data type of the property. Choose from:
Boolean
Float
Integer
String
Pathname
List
Input Column
Output Column
If you choose Input Column or Output Column, when the stage is
included in a job a drop-down list will offer a choice of the defined
input or output columns.
If you choose list you should open the Extended Properties
dialog box from the grid shortcut menu to specify what appears in
the list.
– Prompt. The name of the property that will be displayed on
the Properties tab of the stage editor.
– Default Value. The value the option will take if no other is
specified.
– Required. Set this to True if the property is mandatory.
– Repeats. Set this true if the property repeats (i.e., you can
have multiple instances of it).
– Conversion. Specifies the type of property as follows:
-Name. The name of the property will be passed to the
operator as the option value. This will normally be a hidden
property, i.e., not visible in the stage editor.
-Name Value. The name of the property will be passed to the
operator as the option name, and any value specified in the
stage editor is passed as the value.
-Value. The value for the property specified in the stage editor
is passed to the operator as the option name. Typically used to
group operator options that are mutually exclusive.
Value only. The value for the property specified in the stage
editor is passed as it is.

Input Schema. Specifies that the property will contain a

schema string whose contents are populated from the Input
page Columns tab.
Output Schema. Specifies that the property will contain a
schema string whose contents are populated from the Output
page Columns tab.
None. This allows the creation of properties that do not
generate any osh, but can be used for conditions on other
properties (for example, for use in a situation where you have
mutually exclusive properties, but at least one of them must be
specified).
– Schema properties require format options. Select this
check box to specify that the stage being specified will have a
Format tab.
If you have enabled target ViewData on the Links page, the
following property is also displayed:
– ViewData. Select Yes to indicate that the value of this
property should be used when viewing data. For example, if
this property specifies a file to write to when the stage is used
in a job design, the value of this property will be used to read
the data back if ViewData is used in the stage.
If you select a conversion type of Input Schema or Output
Schema, you should note the following:
– Data Type is set to String.
– Required is set to Yes.
– The property is marked as hidden and will not appear on the
Properties page when the custom stage is used in a job
design.
If your stage can have multiple input or output links there would
be a Input Schema property or Output Schema property per-link.
When the stage is used in a job design, the property will contain
the following OSH for each input and/or output link:
–property_name record {format_props} ( column_definition
{format_props}; …)
Where:
– property_name is the name of the property (usually ‘schema’)
– format_properties are formatting information supplied on the
Format page (if the stage has one).

– there is one column_definition for each column defined in the

Columns tab for that link. The format_props in this case refers
to per-column format information specified in the Edit
Column Meta Data dialog box.
Schema properties are mutually exclusive with schema file
properties. If your custom stage supports both, you should use the
Extended Properties dialog box to specify a condition of
“schemafile= “ for the schema property. The schema property is
then only valid provided the schema file property is blank (or does
not exist).
7 If you want to specify a list property, or otherwise control how
properties are handled by your stage, choose Extended
Properties from the Properties grid shortcut menu to open the
Extended Properties dialog box.
The settings you use depend on the type of property you are
specifying:
– Specify a category to have the property appear under this
category in the stage editor. By default all properties appear in
the Options category.
– Specify that the property will be hidden and not appear in the
stage editor. This is primarily intended to support the case
where the underlying operator needs to know the JobName.
This can be passed using a mandatory String property with a
default value that uses a DS Macro. However, to prevent the
user from changing the value, the property needs to be hidden.
– If you are specifying a List category, specify the possible values
for list members in the List Value field.

– If the property is to be a dependent of another property, select

the parent property in the Parents field.
– Specify an expression in the Template field to have the actual
value of the property generated at compile time. It is usually
based on values in other properties and columns.
– Specify an expression in the Conditions field to indicate that
the property is only valid if the conditions are met. The
specification of this property is a bar '|' separated list of
conditions that are AND'ed together. For example, if the
specification was a=b|c!=d, then this property would only be
valid (and therefore only available in the GUI) when property a
is equal to b, and property c is not equal to d.
Click OK when you are happy with the extended properties.
8 If your custom stage will create columns, go to the Mapping
Additions page. It contains a grid that allows for the specification
of columns created by the stage. You can also specify that column
details are filled in from properties supplied when the stage is
used in a job design, allowing for dynamic specification of
columns.
The grid contains the following fields:

– Column name. The name of the column created by the stage.
You can specify the name of a property you specified on the
Property page of the dialog box to dynamically allocate the
column name. Specify this in the form #property_name#, the
created column will then take the value of this property, as
specified at design time, as the name of the created column.

Defining Build Stages Specifying Your Own Parallel Stages
– Parallel type. The type of the column (this is the underlying

data type, not the SQL data type). Again you can specify the
name of a property you specified on the Property page of the
dialog box to dynamically allocate the column type. Specify
this in the form #property_name#, the created column will then
take the value of this property, as specified at design time, as
the type of the created column. (Note that you cannot use a
repeatable property to dynamically allocate a column type in
this way.)
– Nullable. Choose Yes or No to indicate whether the created
column can contain a null.
– Conditions. Allows you to enter an expression specifying the
conditions under which the column will be created. This could,
for example, depend on the setting of one of the properties
specified in the Property page.
You can propagate the values of the Conditions fields to other
columns if required. Do this by selecting the columns you want to
propagate to, then right-clicking in the source Conditions field
and choosing Propagate from the shortcut menu. A dialog box
asks you to confirm that you want to propagate the conditions to
all columns.
Defining Build Stages

You define a Build stage to enable you to provide a custom operator
that can be executed from a DataStage Parallel job stage. The stage
will be available to all jobs in the project in which the stage was
defined. You can make it available to other projects using the
DataStage Manager Export facilities. The stage is automatically added
to the job palette.
When defining a Build stage you provide the following information:
Description of the data that will be input to the stage.
Whether records are transferred from input to output. A transfer
copies the input record to the output buffer. If you specify auto
transfer, the operator transfers the input record to the output
record immediately after execution of the per record code. The
code can still access data in the output buffer until it is actually
written.
Any definitions and header file information that needs to be
included.
Code that is executed at the beginning of the stage (before any
records are processed).

Specifying Your Own Parallel Stages Defining Build Stages
Code that is executed at the end of the stage (after all records have
been processed).
Code that is executed every time the stage processes a record.
Compilation and build details for actually building the stage.
The Code for the Build stage is specified in C++. There are a number of
macros available to make the job of coding simpler (see "Build Stage
Macros" on page 5-21). There are also a number of header files
available containing many useful functions, see Appendix A.
When you have specified the information, and request that the stage
is generated, DataStage generates a number of files and then
compiles these to build an operator which the stage executes. The
generated files include:
Header files (ending in .h)
Source files (ending in .c)
Object files (ending in .so)
The following shows a build stage in diagrammatic form:
Pre-loop code - Post-loop code -

executed before any executed after all
records are processed records are processed
Input Per-record code. Output

buffer Used to process buffer
each record
Input port - records from Output port - records to

input link output link
Transfer directly copies records from input buffer

to output buffer. Records can still be accessed by
code while in the buffer.
Build Stage
To define a Build stage from the DataStage Manager:


2 Choose File ➤ New Parallel Stage ➤ Build from the main
menu or New Parallel Stage ➤ Build from the shortcut menu.
The Stage Type dialog box appears:

existing stages.
under the stage types branch. Type in or browse for an existing
category or type in the name of a new one. The category also
determines what group in the palette the stage will be added
to. Choose an existing category to add to an existing group, or
specify a new category to create a new palette group.
– Class Name. The name of the C++ class. By default this takes
the name of the stage type.
– Parallel Stage type. This indicates the type of new parallel
– Execution mode. Choose the default execution mode. This is
the mode that will appear in the Advanced tab on the stage
editor. You can override this mode for individual instances of
the stage as required, unless you select Parallel only or
Sequential only. See "Advanced Tab" in Parallel Job
Developer’s Guide for a description of the execution mode.

– Preserve Partitioning. This shows the default setting of the

Preserve Partitioning flag, which you cannot change in a Build
stage. This is the setting that will appear in the Advanced tab
on the stage editor. You can override this setting for individual
instances of the stage as required. See "Advanced Tab" in
Parallel Job Developer’s Guide for a description of the preserve
partitioning flag..
– Partitioning. This shows the default partitioning method,
which you cannot change in a Build stage. This is the method
that will appear in the Inputs Page Partitioning tab of the
stage editor. You can override this method for individual
instances of the stage as required. See "Partitioning Tab" in
Parallel Job Developer’s Guide for a description of the
partitioning methods.
– Collecting. This shows the default collection method, which
you cannot change in a Build stage. This is the method that will
appear in the Inputs Page Partitioning tab of the stage editor.
You can override this method for individual instances of the
stage as required. See "Partitioning Tab" in Parallel Job
Developer’s Guide for a description of the collection methods.
– Operator. The name of the operator that your code is defining
and which will be executed by the DataStage stage. By default
this takes the name of the stage type.
stage.
stage.
release number to the stage so you can keep track of any
subsequent changes.
field.


DataStage icon for this stage.
5 Go to the Properties page. This allows you to specify the options

that the Build stage requires as properties that appear in the Stage
Properties tab. For custom stages the Properties tab always
appears under the Stage page.
Fill in the fields as follows:

– Property name. The name of the property. This will be passed

to the operator you are defining as an option, prefixed with ‘-’
and followed by the value selected in the Properties tab of the
stage editor.
Boolean
Float
Integer
String
Pathname
List
Input Column
Output Column
If you choose Input Column or Output Column, when the stage is
included in a job a drop-down list will offer a choice of the defined
input or output columns.
dialog box from the grid shortcut menu to specify what appears in
the list.
specified.
operator as the option value. This will normally be a hidden
property, i.e., not visible in the stage editor.
operator as the option name, and any value specified in the
stage editor is passed as the value.
is passed to the operator as the option name. Typically used to
group operator options that are mutually exclusive.


specifying:

7 Click on the Build page. The tabs here allow you to define the
actual operation that the stage will perform.
The Interfaces tab enable you to specify details about inputs to

and outputs from the stage, and about automatic transfer of
records from input to output. You specify port details, a port being
where a link connects to the stage. You need a port for each
possible input link to the stage, and a port for each possible output
link from the stage.
You provide the following information on the Input sub-tab:
– Port Name. Optional name for the port. The default names for
the ports are in0, in1, in2 … . You can refer to them in the code
using either the default name or the name you have specified.
– Alias. Where the port name contains non-ascii characters, you
can give it an alias in this column.
– AutoRead. This defaults to True which means the stage will
automatically read records from the port. Otherwise you
explicitly control read operations in the code.
– Table Name. Specify a table definition in the DataStage
Repository which describes the meta data for the port. You can
browse for a table definition by choosing Select Table from
the menu that appears when you click the browse button. You
can also view the schema corresponding to this table definition
by choosing View Schema from the same menu. You do not
have to supply a Table Name. If any of the columns in your
table definition have names that contain non-ascii characters,

you should choose Column Aliases from the menu. The

Build Column Aliases dialog box appears. This lists the
columns that require an alias and let you specify one.
– RCP. Choose True if runtime column propagation is allowed for
inputs to this port. Defaults to False. You do not need to set this
if you are using the automatic transfer facility.
You provide the following information on the Output sub-tab:
– Port Name. Optional name for the port. The default names for
the links are out0, out1, out2 … . You can refer to them in the
code using either the default name or the name you have
specified.
– Alias. Where the port name contains non-ascii characters, you
can give it an alias in this column.
– AutoWrite. This defaults to True which means the stage will
automatically write records to the port. Otherwise you
explicitly control write operations in the code. Once records are
written, the code can no longer access them.
– Table Name. Specify a table definition in the DataStage
Repository which describes the meta data for the port. You can
browse for a table definition. You do not have to supply a Table
Name. A shortcut menu accessed from the browse button
offers a choice of Clear Table Name, Select Table, Create
Table,View Schema, and Column Aliases. The use of these
is as described for the Input sub-tab.
– RCP. Choose True if runtime column propagation is allowed for
outputs from this port. Defaults to False. You do not need to set
this if you are using the automatic transfer facility.
The Transfer sub-tab allows you to connect an input buffer to an
output buffer such that records will be automatically transferred
from input to output. You can also disable automatic transfer, in
which case you have to explicitly transfer data in the code.

Transferred data sits in an output buffer and can still be accessed

and altered by the code until it is actually written to the port.
You provide the following information on the Transfer tab:

– Input. Select the input port to connect to the buffer from the
drop-down list. If you have specified an alias, this will be
displayed here.
– Output. Select the output port to transfer input records from
the output buffer to from the drop-down list. If you have
specified an alias, this will be displayed here.
– Auto Transfer. This defaults to False, which means that you
have to include code which manages the transfer. Set to True to
have the transfer carried out automatically.
– Separate. This is False by default, which means this transfer
will be combined with other transfers to the same port. Set to
True to specify that the transfer should be separate from other
transfers.
The Logic tab is where you specify the actual code that the stage
executes.
The Definitions sub-tab allows you to specify variables, include
header files, and otherwise initialize the stage before processing
any records.

The Pre-Loop sub-tab allows you to specify code which is

executed at the beginning of the stage, before any records are
processed.
The Per-Record sub-tab allows you to specify the code which is
executed once for every record processed.
The Post-Loop sub-tab allows you to specify code that is
executed after all the records have been processed.
You can type straight into these pages or cut and paste from
another editor. The shortcut menu on the Pre-Loop, Per-Record,
and Post-Loop pages gives access to the macros that are
available for use in the code.
The Advanced tab allows you to specify details about how the
stage is compiled and built. Fill in the page as follows:
– Compile and Link Flags. Allows you to specify flags that are
passed to the C++ compiler.
– Verbose. Select this check box to specify that the compile and
build is done in verbose mode.
– Debug. Select this check box to specify that the compile and
build is done in debug mode. Otherwise, it is done in optimize
mode.
– Suppress Compile. Select this check box to generate files
without compiling, and without deleting the generated files.
This option is useful for fault finding.
– Base File Name. The base filename for generated files. All
generated files will have this name followed by the appropriate
suffix. This defaults to the name specified under Operator on
the General page.
– Source Directory. The directory where generated .c files are
placed. This defaults to the buildop folder in the current project
directory. You can also set it using the
DS_OPERATOR_BUILDOP_DIR environment variable in the
DataStage Administrator (see DataStage Administrator Guide).
– Header Directory. The directory where generated .h files are
– Object Directory. The directory where generated .so files are

Specifying Your Own Parallel Stages Build Stage Macros
– Wrapper directory. The directory where generated .op files

are placed. This defaults to the buildop folder in the current
project directory. You can also set it using the
8 When you have filled in the details in all the pages, click Generate
to generate the stage. A window appears showing you the result
of the build.
Build Stage Macros

There are a number of macros you can use when specifying Pre-Loop,
Per-Record, and Post-Loop code. Insert a macro by selecting it from
the short cut menu. They are grouped into the following categories:
Informational
Flow-control
Input and output
Transfer
Informational Macros
Use these macros in your code to determine the number of inputs,
outputs, and transfers as follows:
inputs(). Returns the number of inputs to the stage.
outputs(). Returns the number of outputs from the stage.
transfers(). Returns the number of transfers in the stage.
Flow-Control Macros
Use these macros to override the default behavior of the Per-Record
loop in your stage definition:
endLoop(). Causes the operator to stop looping, following
completion of the current loop and after writing any auto outputs
for this loop.
nextLoop() Causes the operator to immediately skip to the start of
next loop, without writing any outputs.
failStep() Causes the operator to return a failed status and
terminate the job.

Build Stage Macros Specifying Your Own Parallel Stages
Input and Output Macros

These macros allow you to explicitly control the read and write and
transfer of individual records.
Each of the macros takes an argument as follows:
input is the index of the input (0 to n). If you have defined a name
for the input port you can use this in place of the index in the form
portname.portid_.
output is the index of the output (0 to n). If you have defined a
name for the output port you can use this in place of the index in
the form portname.portid_.
index is the index of the transfer (0 to n).
The following macros are available:
readRecord(input). Immediately reads the next record from input,
if there is one. If there is no record, the next call to inputDone() will
return true.
writeRecord(output). Immediately writes a record to output.
inputDone(input). Returns true if the last call to readRecord() for
the specified input failed to read a new record, because the input
has no more records.
holdRecord(input). Causes auto input to be suspended for the
current record, so that the operator does not automatically read a
new record at the start of the next loop. If auto is not set for the
input, holdRecord() has no effect.
discardRecord(output). Causes auto output to be suspended for
the current record, so that the operator does not output the record
at the end of the current loop. If auto is not set for the output,
discardRecord() has no effect.
discardTransfer(index). Causes auto transfer to be suspended, so
that the operator does not perform the transfer at the end of the
current loop. If auto is not set for the transfer, discardTransfer()
has no effect.
Transfer Macros
These macros allow you to explicitly control the transfer of individual
records.
Each of the macros takes an argument as follows:
input is the index of the input (0 to n). If you have defined a name
for the input port you can use this in place of the index in the form
portname.portid_.

output is the index of the output (0 to n). If you have defined a

name for the output port you can use this in place of the index in
the form portname.portid_.
index is the index of the transfer (0 to n).
The following macros are available:
doTransfer(index). Performs the transfer specified by index.
doTransfersFrom(input). Performs all transfers from input.
doTransfersTo(output). Performs all transfers to output.
transferAndWriteRecord(output). Performs all transfers and writes
a record for the specified output. Calling this macro is equivalent
to calling the macros doTransfersTo() and writeRecord().
How Your Code is Executed

This section describes how the code that you define when specifying
a Build stage executes when the stage is run in a DataStage job.
The sequence is as follows:
1 Handles any definitions that you specified in the Definitions sub-
tab when you entered the stage details.
2 Executes any code that was entered in the Pre-Loop sub-tab.
3 Loops repeatedly until either all inputs have run out of records, or
the Per-Record code has explicitly invoked endLoop(). In the
loop, performs the following steps:
a Reads one record for each input, except where any of the
following is true:
– The input has no more records left.
– The input has Auto Read set to false.
– The holdRecord() macro was called for the input last time
around the loop.
b Executes the Per-Record code, which can explicitly read and
write records, perform transfers, and invoke loop-control
macros such as endLoop().
c Performs each specified transfer, except where any of the
following is true:
– The input of the transfer has no more records.
– The transfer has Auto Transfer set to False.
– The discardTransfer() macro was called for the transfer during
the current loop iteration.

d Writes one record for each output, except where any of the
following is true:
– The output has Auto Write set to false.
– The discardRecord() macro was called for the output during
the current loop iteration.
4 If you have specified code in the Post-loop sub-tab, executes it.
5 Returns a status, which is written to the DataStage Job Log.
Inputs and Outputs

The input and output ports that you defined for your Build stage are
where input and output links attach to the stage. By default, links are
connected to ports in the order they are connected to the stage, but
where your stage allows multiple input or output links you can change
the link order using the Link Order tab on the stage editor.
When you specify details about the input and output ports for your
Build stage, you need to define the meta data for the ports. You do this
by loading a table definition from the DataStage Repository.
When you actually use your stage in a job, you have to specify meta
data for the links that attach to these ports. For the job to run
successfully the meta data specified for the port and that specified for
the link should match. An exception to this is where you have runtime
column propagation enabled for the job. In this case the input link
meta data can be a super-set of the port meta data and the extra
columns will be automatically propagated.
Using Multiple Inputs

Where you require your stage to handle multiple inputs, there are
some special considerations. Your code needs to ensure the
following:
The stage only tries to access a column when there are records
available. It should not try to access a column after all records
have been read (use the inputDone() macro to check), and should
not attempt to access a column unless either Auto Read is enabled
on the link or an explicit read record has been performed.
The reading of records is terminated immediately after all the
required records have been read from it. In the case of a port with
Auto Read disabled, the code must determine when all required
records have been read and call the endLoop() macro.
In most cases we recommend that you keep Auto Read enabled when
you are using multiple inputs, this minimizes the need for explicit
control in your code. But there are circumstances when this is not

appropriate. The following paragraphs describes some common

scenarios:
Using Auto Read for all Inputs

All ports have Auto Read enabled and so all record reads are handled
automatically. You need to code for Per-record loop such that each
time it accesses a column on any input it first uses the inputDone()
macro to determine if there are any more records.
This method is fine if you want your stage to read a record from every
link, every time round the loop.
Using Inputs with Auto Read Enabled for Some and Disabled
for Others
You define one (or possibly more) inputs as Auto Read, and the rest
with Auto Read disabled. You code the stage in such a way as the
processing of records from the Auto Read input drives the processing
of the other inputs. Each time round the loop, your code should call
inputDone() on the Auto Read input and call exitLoop() to complete
the actions of the stage.
This method is fine where you process a record from the Auto Read
input every time around the loop, and then process records from one
or more of the other inputs depending on the results of processing the
Auto Read record.
Using Inputs with Auto Read Disabled.

Your code must explicitly perform all record reads. You should define
Per-Loop code which calls readRecord() once for each input to start
processing. Your Per-record code should call inputDone() for every
input each time round the loop to determine if a record was read on
the most recent readRecord(), and if it did, call readRecord() again for
that input. When all inputs run out of records, the Per-Loop code
should exit.
This method is intended where you want explicit control over how
each input is treated.
Example Build Stage

This section shows you how to define a Build stage called Divide,
which basically divides one number by another and writes the result
and any remainder to an output link. The stage also checks whether
you are trying to divide by zero and, if you are, sends the input record
down a reject link.

To demonstrate the use of properties, the stage also lets you define a
minimum divisor. If the number you are dividing by is smaller than
the minimum divisor you specify when adding the stage to a job, then
the record is also rejected.
The input to the stage is defined as auto read, while the two outputs
have auto write disabled. The code has to explicitly write the data to
one or other of the output links. In the case of a successful division the
data written is the original record plus the result of the division and
any remainder. In the case of a rejected record, only the original
record is written.
The input record has two columns: dividend and divisor. Output 0 has
four columns: dividend, divisor, result, and remainder. Output 1 (the
reject link) has two columns: dividend and divisor.
If the divisor column of an input record contains zero or is less than
the specified minimum divisor, the record is rejected, and the code
uses the macro transferAndWriteRecord(1) to transfer the data to port
1 and write it. If the divisor is not zero, the code uses doTransfersTo(0)
to transfer the input record to Output 0, assigns the division results to
result and remainder and finally calls writeRecord(0) to write the
record to output 0.
The following screen shots show how this stage is defined in
DataStage using the Stage Type dialog box:

1 First general details are supplied in the General tab:
2 Details about the stage’s creation are supplied on the Creator

page:

3 The optional property of the stage is defined in the Properties

tab:
4 Details of the inputs and outputs is defined on the interfaces tab of

the Build page.
Details about the single input to Divide are given on the Input
sub-tab of the Interfaces tab. A table definition for the inputs link
is available to be loaded from the DataStage Repository.

Details about the outputs are given on the Output sub-tab of the
Interfaces tab.
Note When you use the stage in a job, make sure that you use
table definitions compatible with the tables defined in
the input and output sub-tabs.
Details about the transfers carried out by the stage are defined on
the Transfer sub-tab of the Interfaces tab.

5 The code itself is defined on the Logic tab. In this case all the
processing is done in the Per-Record loop and so is entered on the
Per-Record sub-tab.
6 As this example uses all the compile and build defaults, all that
remains is to click Generate to build the stage.

Specifying Your Own Parallel Stages Defining Wrapped Stages
Defining Wrapped Stages

You define a Wrapped stage to enable you to specify a UNIX
command to be executed by a DataStage stage. You define a wrapper
file that handles arguments for the UNIX command and inputs and
outputs. The DataStage Manager provides an interface that helps you
define the wrapper. The stage will be available to all jobs in the project
in which the stage was defined. You can make it available to other
projects using the DataStage Manager Export facilities. You can add
the stage to your job palette using palette customization features in
the DataStage Designer.
When defining a Wrapped stage you provide the following
information:
Details of the UNIX command that the stage will execute.
Description of the data that will be input to the stage.
Description of the data that will be output from the stage.
Definition of the environment in which the command will execute.
The UNIX command that you wrap can be a built-in command, such
as grep, a utility, such as SyncSort, or your own UNIX application. The
only limitation is that the command must be ‘pipe-safe’ (to be pipe-
safe a UNIX command reads its input sequentially, from beginning to
end).
You need to define meta data for the data being input to and output
from the stage. You also need to define the way in which the data will
be input or output. UNIX commands can take their inputs from
standard in, or another stream, a file, or from the output of another
command via a pipe. Similarly data is output to standard out, or
another stream, to a file, or to a pipe to be input to another command.
You specify what the command expects.
DataStage handles data being input to the Wrapped stage and will
present it in the specified form. If you specify a command that expects
input on standard in, or another stream, DataStage will present the
input data from the jobs data flow as if it was on standard in. Similarly
it will intercept data output on standard out, or another stream, and
integrate it into the job’s data flow.
You also specify the environment in which the UNIX command will be
executed when you define the wrapped stage.
To define a Wrapped stage from the DataStage Manager:

Defining Wrapped Stages Specifying Your Own Parallel Stages

2 Choose File ➤ New Parallel Stage ➤ Wrapped from the main
menu or New Parallel Stage ➤ Wrapped from the shortcut
menu. The Stage Type dialog box appears:

existing stages or the name of the actual UNIX command you
are wrapping.
under the stage types branch. Type in or browse for an existing
category or type in the name of a new one. The category also
determines what group in the palette the stage will be added
to. Choose an existing category to add to an existing group, or
specify a new category to create a new palette group.
– Parallel Stage type. This indicates the type of new Parallel
– Wrapper Name. The name of the wrapper file DataStage will
generate to call the command. By default this will take the
same name as the Stage type name.
– Execution mode. Choose the default execution mode. This is
the mode that will appear in the Advanced tab on the stage
editor. You can override this mode for individual instances of

the stage as required, unless you select Parallel only or

Sequential only. See "Advanced Tab" in Parallel Job
Developer’s Guide for a description of the execution mode.
– Preserve Partitioning. This shows the default setting of the
Preserve Partitioning flag, which you cannot change in a
Wrapped stage. This is the setting that will appear in the
Advanced tab on the stage editor. You can override this
setting for individual instances of the stage as required. See
description of the preserve partitioning flag.
– Partitioning. This shows the default partitioning method,
which you cannot change in a Wrapped stage. This is the
method that will appear in the Inputs Page Partitioning tab
of the stage editor. You can override this method for individual
instances of the stage as required. See "Partitioning Tab" in
Parallel Job Developer’s Guide for a description of the
partitioning methods.
– Collecting. This shows the default collection method, which
you cannot change in a Wrapped stage. This is the method that
will appear in the Inputs Page Partitioning tab of the stage
editor. You can override this method for individual instances of
the stage as required. See "Partitioning Tab" in Parallel Job
Developer’s Guide for a description of the collection methods.
– Command. The name of the UNIX command to be wrapped,
plus any required arguments. The arguments that you enter
here are ones that do not change with different invocations of
the command. Arguments that need to be specified when the
Wrapped stage is included in a job are defined as properties for
the stage.
stage.
stage.
release number to the stage so you can keep track of any
subsequent changes.
field.

DataStage icon for this stage.
5 Go to the Properties page. This allows you to specify the

arguments that the UNIX command requires as properties that
appear in the stage Properties tab. For wrapped stages the
Properties tab always appears under the Stage page.
Fill in the fields as follows:

– Property name. The name of the property that will be

displayed on the Properties tab of the stage editor.
Boolean
Float
Integer
String
Pathname
List
Input Column
Output Column
If you choose Input Column or Output Column, when the stage
is included in a job a drop-down list will offer a choice of the
defined input or output columns.
dialog box from the grid shortcut menu to specify what
appears in the list.
specified.
– Repeats. Set this true if the property repeats (i.e. you can have
multiple instances of it).
command as the argument value. This will normally be a
hidden property, i.e., not visible in the stage editor.
command as the argument name, and any value specified in
the stage editor is passed as the value.
is passed to the command as the argument name. Typically
used to group operator options that are mutually exclusive.


specifying:
7 Go to the Wrapped page. This allows you to specify information
about the command to be executed by the stage and how it will be
handled.

The Interfaces tab is used to describe the inputs to and outputs

from the stage, specifying the interfaces that the stage will need to
function.
Details about inputs to the stage are defined on the Inputs sub-
tab:
– Link. The link number, this is assigned for you and is read-
only. When you actually use your stage, links will be assigned
in the order in which you add them. In our example, the first
link will be taken as link 0, the second as link 1 and so on. You
can reassign the links using the stage editor’s Link Ordering
tab on the General page.
– Table Name. The meta data for the link. You define this by
loading a table definition from the Repository. Type in the
name, or browse for a table definition. Alternatively, you can
specify an argument to the UNIX command which specifies a
table definition. In this case, when the wrapped stage is used in
a job design, the designer will be prompted for an actual table
definition to use.

– Stream. Here you can specify whether the UNIX command

expects its input on standard in, or another stream, or whether
it expects it in a file. Click on the browse button to open the
Wrapped Stream dialog box.
In the case of a file, you should also specify whether the file to be
read is given in a command line argument, or by an environment
variable.
Details about outputs from the stage are defined on the Outputs
sub-tab:
– Link. The link number, this is assigned for you and is read-
only. When you actually use your stage, links will be assigned
in the order in which you add them. In our example, the first
link will be taken as link 0, the second as link 1 and so on. You
can reassign the links using the stage editor’s Link Ordering
tab on the General page.
– Table Name. The meta data for the link. You define this by
loading a table definition from the Repository. Type in the
name, or browse for a table definition.
– Stream. Here you can specify whether the UNIX command
will write its output to standard out, or another stream, or
whether it outputs to a file. Click on the browse button to open
the Wrapped Stream dialog box.
In the case of a file, you should also specify whether the file to
be written is specified in a command line argument, or by an

The Environment tab gives information about the environment

in which the command will execute.
Set the following on the Environment tab:

– All Exit Codes Successful. By default DataStage treats an
exit code of 0 as successful and all others as errors. Select this
check box to specify that all exit codes should be treated as
successful other than those specified in the Failure codes
grid.
– Exit Codes. The use of this depends on the setting of the All
Exits Codes Successful check box.
If All Exits Codes Successful is not selected, enter the codes in
the Success Codes grid which will be taken as indicating
successful completion. All others will be taken as indicating
failure.
If All Exits Codes Successful is selected, enter the exit codes in
the Failure Code grid which will be taken as indicating failure. All
others will be taken as indicating success.
– Environment. Specify environment variables and settings that
the UNIX command requires in order to run.
8 When you have filled in the details in all the pages, click Generate
to generate the stage.

Example Wrapped Stage

This section shows you how to define a Wrapped stage called exhort
which runs the UNIX sort command in parallel. The stage sorts data in
two files and outputs the results to a file. The incoming data has two
columns, order number and code. The sort command sorts the data
on the second field, code. You can optionally specify that the sort is
run in reverse order.
Wrapping the sort command in this way would be useful if you had a
situation where you had a fixed sort operation that was likely to be
needed in several jobs. Having it as an easily reusable stage would
save having to configure a built-in sort stage every time you needed it.
When included in a job and run, the stage will effectively call the Sort
command as follows:
sort -r -o outfile -k 2 infile1 infile2
The following screen shots show how this stage is defined in

DataStage using the Stage Type dialog box:

1 First general details are supplied in the General tab. The

argument defining the second column as the key is included in the
command because this does not vary:
2 The reverse order argument (-r) are included as properties

because it is optional and may or may not be included when the
stage is incorporated into a job.
3 The fact that the sort command expects two files as input is
defined on the Input sub-tab on the Interfaces tab of the
Wrapper page.

4 The fact that the sort command outputs to a file is defined on the
Output sub-tab on the Interfaces tab of the Wrapper page.
Note When you use the stage in a job, make sure that you use
table definitions compatible with the tables defined in
the input and output sub-tabs.
5 Because all exit codes other than 0 are treated as errors, and
because there are no special environment requirements for this
command, you do not need to alter anything on the Environment
tab of the Wrapped page. All that remains is to click Generate to
build the stage.

6
There are many environment variables that affect the design and
running of parallel jobs in DataStage. Commonly used ones are
exposed in the DataStage Administrator client, and can be set or
unset using the Administrator (see"Setting Environment Variables" in
DataStage Administrator Guide). There are additional environment
variables, however. This chapter describes all the environment
variables that apply to parallel jobs. They can be set or unset as you
would any other UNIX system variables, or you can add them to the
User Defined section in the DataStage Administrator environment
variable tree.
The available environment variables are grouped according to
function. They are summarized in the following table.
The final section in this chapter gives some guidance to setting the
environment variables..
Category Environment Variable

Buffering APT_BUFFER_FREE_RUN
APT_BUFFER_MAXIMUM_MEMORY
APT_BUFFER_MAXIMUM_TIMEOUT
APT_BUFFER_DISK_WRITE_INCREMENT
APT_BUFFERING_POLICY
APT_SHARED_MEMORY_BUFFERS
Building Custom Stages DS_OPERATOR_BUILDOP_DIR
OSH_BUILDOP_CODE
OSH_BUILDOP_HEADER


OSH_BUILDOP_OBJECT
OSH_BUILDOP_XLC_BIN
OSH_CBUILDOP_XLC_BIN
Compiler APT_COMPILER
APT_COMPILEOPT
APT_LINKER
APT_LINKOPT
DB2 Support APT_DB2INSTANCE_HOME
APT_DB2READ_LOCK_TABLE
APT_DBNAME
APT_RDBMS_COMMIT_ROWS
DB2DBDFT
Debugging APT_DEBUG_OPERATOR
APT_DEBUG_MODULE_NAMES
APT_DEBUG_PARTITION
APT_DEBUG_SIGNALS
APT_DEBUG_STEP
APT_DEBUG_SUBPROC
APT_EXECUTION_MODE
APT_PM_DBX
APT_PM_GDB
APT_PM_LADEBUG
APT_PM_SHOW_PIDS
APT_PM_XLDB
APT_PM_XTERM
APT_SHOW_LIBLOAD
Decimal Support APT_DECIMAL_INTERM_PRECISION
APT_DECIMAL_INTERM_SCALE
APT_DECIMAL_INTERM_ROUND_MODE
Disk I/O APT_BUFFER_DISK_WRITE_INCREMENT
APT_CONSISTENT_BUFFERIO_SIZE


APT_EXPORT_FLUSH_COUNT
APT_IO_MAP/APT_IO_NOMAP and
APT_BUFFERIO_MAP/APT_BUFFERIO_NOMAP
APT_PHYSICAL_DATASET_BLOCK_SIZE
General Job APT_CHECKPOINT_DIR

Administration
APT_CLOBBER_OUTPUT
APT_CONFIG_FILE
APT_DISABLE_COMBINATION
APT_EXECUTION_MODE
APT_ORCHHOME
APT_STARTUP_SCRIPT
APT_NO_STARTUP_SCRIPT
APT_STARTUP_STATUS
APT_THIN_SCORE
Job Monitoring APT_MONITOR_SIZE
APT_MONITOR_TIME
APT_NO_JOBMON
APT_PERFORMANCE_DATA
Miscellaneous APT_COPY_TRANSFORM_OPERATOR
APT_DATE_CENTURY_BREAK_YEAR
APT_IMPEXP_ALLOW_ZERO_LENGTH_FIXED_NULL
APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS
APT_INSERT_COPY_BEFORE_MODIFY
APT_OLD_BOUNDED_LENGTH
APT_OPERATOR_REGISTRY_PATH
APT_PM_NO_SHARED_MEMORY
APT_PM_NO_NAMED_PIPES
APT_PM_SOFT_KILL_WAIT
APT_PM_STARTUP_CONCURRENCY
APT_RECORD_COUNTS
APT_SAVE_SCORE


APT_SHOW_COMPONENT_CALLS
APT_STACK_TRACE
APT_WRITE_DS_VERSION
OSH_PRELOAD_LIBS
Network APT_IO_MAXIMUM_OUTSTANDING
APT_IOMGR_CONNECT_ATTEMPTS
APT_PM_CONDUCTOR_HOSTNAME
APT_PM_NO_TCPIP
APT_PM_NODE_TIMEOUT
APT_PM_SHOWRSH
APT_PM_USE_RSH_LOCALLY
NLS APT_COLLATION_SEQUENCE
APT_COLLATION_STRENGTH
APT_ENGLISH_MESSAGES
APT_IMPEXP_CHARSET
APT_INPUT_CHARSET
APT_OS_CHARSET
APT_OUTPUT_CHARSET
APT_STRING_CHARSET
Oracle Support APT_ORACLE_LOAD_DELIMITED
APT_ORACLE_LOAD_OPTIONS
APT_ORACLE_NO_OPS
APT_ORACLE_PRESERVE_BLANKS
APT_ORA_IGNORE_CONFIG_FILE_PARALLELISM
APT_ORA_WRITE_FILES
APT_ORAUPSERT_COMMIT_TIME_INTERVAL
Partitioning APT_NO_PART_INSERTION
APT_PARTITION_COUNT
APT_PARTITION_NUMBER
Reading and Writing APT_DELIMITED_READ_SIZE

Files


APT_FILE_IMPORT_BUFFER_SIZE
APT_FILE_EXPORT_BUFFER_SIZE
APT_IMPORT_PATTERN_USES_FILESET
APT_MAX_DELIMITED_READ_SIZE
APT_PREVIOUS_FINAL_DELIMITER_COMPATIBLE
APT_STRING_PADCHAR
Reporting APT_DUMP_SCORE
APT_ERROR_CONFIGURATION
APT_MSG_FILELINE
APT_PM_PLAYER_MEMORY
APT_PM_PLAYER_TIMING
APT_RECORD_COUNTS
OSH_DUMP
OSH_ECHO
OSH_EXPLAIN
OSH_PRINT_SCHEMAS
SAS Support APT_HASH_TO_SASHASH
APT_NO_SASOUT_INSERT
APT_NO_SAS_TRANSFORMS
APT_SAS_ACCEPT_ERROR
APT_SAS_CHARSET
APT_SAS_CHARSET_ABORT
APT_SAS_COMMAND
APT_SASINT_COMMAND
APT_SAS_DEBUG
APT_SAS_DEBUG_IO
APT_SAS_DEBUG_LEVEL
APT_SAS_DEBUG_VERBOSE
APT_SAS_NO_PSDS_USTRING
APT_SAS_S_ARGUMENT
APT_SAS_SCHEMASOURCE_DUMP

Buffering Environment Variables

APT_SAS_SHOW_INFO
APT_SAS_TRUNCATION
Sorting APT_NO_SORT_INSERTION
APT_SORT_INSERTION_CHECK_ONLY
Teradata Support APT_TERA_64K_BUFFERS
APT_TERA_NO_ERR_CLEANUP
APT_TERA_NO_PERM_CHECKS
APT_TERA_NO_SQL_CONVERSION
APT_TERA_SYNC_DATABASE
APT_TERA_SYNC_USER
Transport Blocks APT_AUTO_TRANSPORT_BLOCK_SIZE
APT_LATENCY_COEFFICIENT
APT_DEFAULT_TRANSPORT_BLOCK_SIZE
APT_MIN_TRANSPORT_BLOCK_SIZE
Buffering
These environment variable are all concerned with the buffering
DataStage performs on stage links to avoid deadlock situations. These
settings can also be made on the Inputs page or Outputs page
Advanced tab of the parallel stage editors.
APT_BUFFER_FREE_RUN
This environment variable is available in the DataStage Administrator,
under the Parallel category. It specifies how much of the available in-
memory buffer to consume before the buffer resists. This is expressed
as a decimal representing the percentage of Maximum memory buffer
size (for example, 0.5 is 50%). When the amount of data in the buffer is
less than this value, new data is accepted automatically. When the
data exceeds it, the buffer first tries to write some of the data it
contains before accepting more.
The default value is 50% of the Maximum memory buffer size. You can
set it to greater than 100%, in which case the buffer continues to store

Environment Variables Buffering
data up to the indicated multiple of Maximum memory buffer size

before writing to disk.
APT_BUFFER_MAXIMUM_MEMORY
Sets the default value of Maximum memory buffer size. The default
value is 3145728 (3 MB). Specifies the maximum amount of virtual
memory, in bytes, used per buffer.
APT_BUFFER_MAXIMUM_TIMEOUT
DataStage buffering is self tuning, which can theoretically lead to long
delays between retries. This environment variable specified the
maximum wait before a retry in seconds, and is by default set to 1.
Sets the size, in bytes, of blocks of data being moved to/from disk by
the buffering operator. The default is 1048576 (1 MB). Adjusting this
value trades amount of disk access against throughput for small
amounts of data. Increasing the block size reduces disk access, but
may decrease performance when data is being read/written in smaller
units. Decreasing the block size increases throughput, but may
increase the amount of disk access.
APT_BUFFERING_POLICY
under the Parallel category. Controls the buffering policy for all virtual
data sets in all steps. The variable has the following settings:
AUTOMATIC_BUFFERING (default). Buffer a data set only if
necessary to prevent a data flow deadlock.
FORCE_BUFFERING. Unconditionally buffer all virtual data sets.
Note that this can slow down processing considerably.
NO_BUFFERING. Do not buffer data sets. This setting can cause
data flow deadlock if used inappropriately.
APT_SHARED_MEMORY_BUFFERS
Typically the number of shared memory buffers between two
processes is fixed at 2. Setting this will increase the number used. The
likely result of this is POSSIBLY both increased latency and increased
performance. This setting can significantly increase memory use.

Building Custom Stages Environment Variables
Building Custom Stages

These environment variables are concerned with the building of
custom operators that form the basis of customized stages (as
described in Chapter 5, "Specifying Your Own Parallel Stages.")
DS_OPERATOR_BUILDOP_DIR
Identifies the directory in which generated buildops are created. By
default this identifies a directory called buildop under the current
project directory. If the directory is changed, the corresponding entry
in APT_OPERATOR_REGISTRY_PATH needs to change to match the
buildop folder.
OSH_BUILDOP_CODE
Identifies the directory into which buildop writes the generated .C file
and build script. It defaults to the current working directory. The -C
option of buildop overrides this setting.
OSH_BUILDOP_HEADER
Identifies the directory into which buildop writes the generated .h file.
It defaults to the current working directory. The -H option of buildop
overrides this setting.
OSH_BUILDOP_OBJECT
Identifies the directory into which buildop writes the dynamically
loadable object file, whose extension is .so on Solaris, .sl on HP-UX, or
.o on AIX. Defaults to the current working directory.
The -O option of buildop overrides this setting.
OSH_BUILDOP_XLC_BIN
AIX only. Identifies the directory specifying the location of the shared
library creation command used by buildop.
On older AIX systems this defaults to /usr/lpp/xlC/bin/
makeC++SharedLib_r for thread-safe compilation. On newer AIX
systems it defaults to /usr/ibmcxx/bin/makeC++SharedLib_r. For non-
thread-safe compilation, the default path is the same, but the name of
the file is makeC++SharedLib.

Environment Variables Compiler
OSH_CBUILDOP_XLC_BIN
AIX only. Identifies the directory specifying the location of the shared
library creation command used by cbuildop. If this environment
variable is not set, cbuildop checks the setting of
OSH_BUILDOP_XLC_BIN for the path. On older AIX systems
OSH_CBUILDOP_XLC_BIN defaults to /usr/lpp/xlC/bin/
makeC++SharedLib_r for thread-safe compilation. On newer AIX
systems it defaults to /usr/ibmcxx/bin/makeC++SharedLib_r. For non-
threadsafe compilation, the default path is the same, but the name of
the command is makeC++SharedLib.
Compiler
These environment variables specify details about the C++ compiler
used by DataStage in connection with parallel jobs.
APT_COMPILER
This environment variable is available in the DataStage Administrator
under the Parallel ➤ Compiler branch. Specifies the full path to the
C++ compiler.
APT_COMPILEOPT
under the Parallel ➤ Compiler branch. Specifies extra options to be
passed to the C++ compiler when it is invoked.
APT_LINKER
under the Parallel ➤ Compiler branch. Specifies the full path to the
C++ linker.
APT_LINKOPT
under the Parallel ➤ Compiler branch. Specifies extra options to be
passed to the C++ linker when it is invoked.

DB2 Support Environment Variables
DB2 Support
These environment variables are concerned with setting up access to
DB2 databases from DataStage.
APT_DB2INSTANCE_HOME
Specifies the DB2 installation directory. This variable is set by
DataStage to values obtained from the DB2Server table, representing
the currently selected DB2 server.
APT_DB2READ_LOCK_TABLE
If this variable is defined and the open option is not specified for the
DB2 stage, DataStage performs the following open command to lock
the table:
lock table 'table_name' in share mode
APT_DBNAME
Specifies the name of the database if you choose to leave out the
Database option for DB2 stages. If APT_DBNAME is not defined as
well, DB2DBDFT is used to find the database name. These variables
are set by DataStage to values obtained from the DB2Server table,
representing the currently selected DB2 server.
APT_RDBMS_COMMIT_ROWS
Specifies the number of records to insert into a data set between
commits. The default value is 2048.
DB2DBDFT
For DB2 operators, you can set the name of the database by using the
-dbname option or by setting APT_DBNAME. If you do not use either
method, DB2DBDFT is used to find the database name. These
variables are set by DataStage to values obtained from the DB2Server
table, representing the currently selected DB2 server.
Debugging
These environment variables are concerned with the debugging of
DataStage parallel jobs.

Environment Variables Debugging
APT_DEBUG_OPERATOR
Specifies the operators on which to start debuggers. If not set, no
debuggers are started. If set to an operator number (as determined
from the output of APT_DUMP_SCORE), debuggers are started for
that single operator. If set to -1, debuggers are started for all
operators.
APT_DEBUG_MODULE_NAMES
This comprises a list of module names separated by white space that
are the modules to debug, i.e., where internal IF_DEBUG statements
will be run. The subproc operator module (module name is
"subproc") is one example of a module that uses this facility.
APT_DEBUG_PARTITION
Specifies the partitions on which to start debuggers. One instance, or
partition, of an operator is run on each node running the operator. If
set to a single number, debuggers are started on that partition; if not
set or set to -1, debuggers are started on all partitions.
See the description of APT_DEBUG_OPERATOR for more information
on using this environment variable.
For example, setting APT_DEBUG_STEP to 0,
APT_DEBUG_OPERATOR to 1, and APT_DEBUG_PARTITION to -1
starts debuggers for every partition of the second operator in the first
step.
APT_DEBUG_ APT_DEBUG_ Effect

OPERATOR PARTITION
not set any value no debugging
-1 not set or -1 debug all partitions of all operators
-1 >= 0 debug a specific partition of all

operators
>= 0 -1 debug all partitions of a specific

operator
>= 0 >= 0 debug a specific partition of a specific

operator
APT_DEBUG_SIGNALS
You can use the APT_DEBUG_SIGNALS environment variable to
specify that signals such as SIGSEGV, SIGBUS, etc., should cause a

Debugging Environment Variables
debugger to start. If any of these signals occurs within an

APT_Operator::runLocally() function, a debugger such as dbx is
invoked.
Note that the UNIX and DataStage variables DEBUGGER, DISPLAY,
and APT_PM_XTERM, specifying a debugger and how the output
should be displayed, must be set correctly.
APT_DEBUG_STEP
Specifies the steps on which to start debuggers. If not set or if set to -
1, debuggers are started on the processes specified by
APT_DEBUG_OPERATOR and APT_DEBUG_PARTITION in all steps. If
set to a step number, debuggers are started for processes in the
specified step.
APT_DEBUG_SUBPROC
Display debug information about each subprocess operator.
APT_EXECUTION_MODE
under the Parallel branch. By default, the execution mode is parallel,
with multiple processes. Set this variable to one of the following
values to run an application in sequential execution mode:
ONE_PROCESS one-process mode
MANY_PROCESS many-process mode
NO_SERIALIZE many-process mode, without serialization
In ONE_PROCESS mode:
The application executes in a single UNIX process. You need only
run a single debugger session and can set breakpoints anywhere
in your code.
Data is partitioned according to the number of nodes defined in
the configuration file.
Each operator is run as a subroutine and is called the number of
times appropriate for the number of partitions on which it must
operate.
In MANY_PROCESS mode the framework forks a new process for
each instance of each operator and waits for it to complete rather than
calling operators as subroutines.
In both cases, the step is run entirely on the Conductor node rather
than spread across the configuration.

Environment Variables Debugging
NO_SERIALIZE mode is similar to MANY_PROCESS mode, but the

DataStage persistence mechanism is not used to load and save
objects. Turning off persistence may be useful for tracking errors in
derived C++ classes.
APT_PM_DBX
Set this environment variable to the path of your dbx debugger, if a
debugger is not already included in your path. This variable sets the
location; it does not run the debugger.
APT_PM_GDB
Linux only. Set this environment variable to the path of your xldb
debugger, if a debugger is not already included in your path. This
variable sets the location; it does not run the debugger.
APT_PM_LADEBUG
Tru64 only. Set this environment variable to the path of your dbx
debugger, if a debugger is not already included in your path. This
variable sets the location; it does not run the debugger.
APT_PM_SHOW_PIDS
If this variable is set, players will output an informational message
upon startup, displaying their process id.
APT_PM_XLDB
Set this environment variable to the path of your xldb debugger, if a
debugger is not already included in your path. This variable sets the
location; it does not run the debugger.
APT_PM_XTERM
If DataStage invokes dbx, the debugger starts in an xterm window;
this means DataStage must know where to find the xterm program.
The default location is /usr/bin/X11/xterm. You can override this
default by setting the APT_PM_XTERM environment variable to the
appropriate path. APT_PM_XTERM is ignored if you are using xldb.

Decimal Support Environment Variables
APT_SHOW_LIBLOAD
If set, dumps a message to stdout every time a library is loaded. This
can be useful for testing/verifying the right library is being loaded.
Note that the message is output to stdout, NOT to the error log.
Decimal Support
APT_DECIMAL_INTERM_PRECISION
Specifies the default maximum precision value for any decimal
intermediate variables required in calculations. Default value is 38.
APT_DECIMAL_INTERM_SCALE
Specifies the default scale value for any decimal intermediate
variables required in calculations. Default value is 10.
APT_DECIMAL_INTERM_ROUND_MODE
Specifies the default rounding mode for any decimal intermediate
variables required in calculations. The default is round_inf.
Disk I/O
These environment variables are all concerned with when and how
DataStage parallel jobs write information to disk.
For systems where small to medium bursts of I/O are not desirable,
the default 1MB write to disk size chunk size may be too small.
APT_BUFFER_DISK_WRITE_INCREMENT controls this and can be set
larger than 1048576 (1 MB). The setting may not exceed
max_memory * 2/3.
APT_CONSISTENT_BUFFERIO_SIZE
Some disk arrays have read ahead caches that are only effective when
data is read repeatedly in like-sized chunks. Setting

Environment Variables General Job Administration
APT_CONSISTENT_BUFFERIO_SIZE=N will force stages to read data

in chunks which are size N or a multiple of N.
APT_EXPORT_FLUSH_COUNT
Allows the export operator to flush data to disk more often than it
typically does (data is explicitly flushed at the end of a job, although
the OS may choose to do so more frequently). Set this variable to an
integer which, in number of records, controls how often flushes
should occur. Setting this value to a low number (such as 1) is useful
for real time applications, but there is a small performance penalty
associated with setting this to a low value.
APT_IO_MAP/APT_IO_NOMAP and APT_BUFFERIO_MAP/

APT_BUFFERIO_NOMAP
In many cases memory mapped I/O contributes to improved
performance. In certain situations, however, such as a remote disk
mounted via NFS, memory mapped I/O may cause significant
performance problems. Setting the environment variables
APT_IO_NOMAP and APT_BUFFERIO_NOMAP true will turn off this
feature and sometimes affect performance. (AIX and HP-UX default to
NOMAP. Setting APT_IO_MAP and APT_BUFFERIO_MAP true can be
used to turn memory mapped I/O on for these platforms.)
APT_PHYSICAL_DATASET_BLOCK_SIZE
Specify the block size to use for reading and writing to a data set
stage. The default is 128 KB.
General Job Administration

These environment variables are concerned with details about the
running of DataStage parallel jobs.
APT_CHECKPOINT_DIR
under the Parallel branch. By default, when running a job, DataStage
stores state information in the current working directory. Use
APT_CHECKPOINT_DIR to specify another directory.

General Job Administration Environment Variables
APT_CLOBBER_OUTPUT
under the Parallel branch. By default, if an output file or data set
already exists, DataStage issues an error and stops before overwriting
it, notifying you of the name conflict. Setting this variable to any value
permits DataStage to overwrite existing files or data sets without a
warning message.
APT_CONFIG_FILE
under the Parallel branch. Sets the path name of the configuration file.
(You may want to include this as a job parameter, so that you can
specify the configuration file at job run time).
APT_DISABLE_COMBINATION
under the Parallel branch. Globally disables operator combining.
Operator combining is DataStage’s default behavior, in which two or
more (in fact any number of) operators within a step are combined
into one process where possible.
You may need to disable combining to facilitate debugging. Note that
disabling combining generates more UNIX processes, and hence
requires more system resources and memory. It also disables internal
optimizations for job efficiency and run times.
APT_EXECUTION_MODE
under the Parallel branch. By default, the execution mode is parallel,
with multiple processes. Set this variable to one of the following
values to run an application in sequential execution mode:
ONE_PROCESS one-process mode
MANY_PROCESS many-process mode
NO_SERIALIZE many-process mode, without serialization
In ONE_PROCESS mode:
The application executes in a single UNIX process. You need only
run a single debugger session and can set breakpoints anywhere
in your code.
Data is partitioned according to the number of nodes defined in
the configuration file.

Environment Variables General Job Administration
Each operator is run as a subroutine and is called the number of

times appropriate for the number of partitions on which it must
operate.
In MANY_PROCESS mode the framework forks a new process for
each instance of each operator and waits for it to complete rather than
calling operators as subroutines.
In both cases, the step is run entirely on the Conductor node rather
than spread across the configuration.
NO_SERIALIZE mode is similar to MANY_PROCESS mode, but the
DataStage persistence mechanism is not used to load and save
objects. Turning off persistence may be useful for tracking errors in
derived C++ classes.
APT_ORCHHOME
Must be set by all DataStage Enterprise Edition users to point to the
top-level directory of the DataStage Enterprise Edition installation.
APT_STARTUP_SCRIPT
As part of running an application, DataStage creates a remote shell on
all DataStage processing nodes on which the job runs. By default, the
remote shell is given the same environment as the shell from which
DataStage is invoked. However, you can write an optional startup
shell script to modify the shell configuration of one or more
processing nodes. If a startup script exists, DataStage runs it on
remote shells before running your application.
APT_STARTUP_SCRIPT specifies the script to be run. If it is not
defined, DataStage searches ./startup.apt, $APT_ORCHHOME/etc/
startup.apt and $APT_ORCHHOME/etc/startup, in that order.
APT_NO_STARTUP_SCRIPT disables running the startup script.
APT_NO_STARTUP_SCRIPT
Prevents DataStage from executing a startup script. By default, this
variable is not set, and DataStage runs the startup script. If this
variable is set, DataStage ignores the startup script. This may be
useful when debugging a startup script. See also
APT_STARTUP_SCRIPT.

Job Monitoring Environment Variables
APT_STARTUP_STATUS
Set thi sto cause messages to be generated as parallel job startup
moves from phase to phase. This can be useful as a diagnostic if
parallel job startup is failing.
APT_THIN_SCORE
Setting this variable decreases the memory usage of steps with 100
operator instances or more by a noticable amount. To use this
optimization, set APT_THIN_SCORE=1 in your environment. There are
no performance benefits in setting this variable unless you are
running out of real memory at some point in your flow or the
additional memory is useful for sorting or buffering. This variable
does not affect any specific operators which consume large amounts
of memory, but improves general parallel job memory handling.
Job Monitoring
These environment variables are concerned with the Job Monitor on
DataStage.
APT_MONITOR_SIZE
under the Parallel branch. Determines the minimum number of
records the DataStage Job Monitor reports. The default is 5000
records.
APT_MONITOR_TIME
under the Parallel branch. Determines the minimum time interval in
seconds for generating monitor information at runtime. The default is
5 seconds. This variable takes precedence over APT_MONITOR_SIZE.
APT_NO_JOBMON
Turn off job monitoring entirely.
APT_PERFORMANCE_DATA
Set this variable to turn on performance data output generation.
APT_PERFORMANCE_DATA can be either set with no value, or be set

Environment Variables Miscellaneous
to a valid path which will be used as the default location for

performance data output.
Miscellaneous
APT_COPY_TRANSFORM_OPERATOR
If set, distributes the shared object file of the sub-level transform
operator and the shared object file of user-defined functions (not
extern functions) via distribute-component in a non-NFS MPP.
APT_DATE_CENTURY_BREAK_YEAR
Four digit year which marks the century two-digit dates belong to. It is
set to 1900 by defult.
APT_IMPEXP_ALLOW_ZERO_LENGTH_FIXED_NULL
When set, allows zero length null_field value with fixed length fields.
This should be used with care as poorly formatted data will cause
incorrect results. By default a zero length null_field value will cause
an error.
APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS
When set, DataStage will reject any string or ustring fields read that
go over their fixed size. By default these records are truncated.
APT_INSERT_COPY_BEFORE_MODIFY
When defined, turns on automatic insertion of a copy operator before
any modify operator (WARNING: if this variable is not set and the
operator immediately preceding 'modify' in the data flow uses a
modify adapter, the 'modify' operator will be removed from the data
flow).
Only set this if you write your own custom operators AND use modify
within those operators.
APT_OLD_BOUNDED_LENGTH
Some parallel datasets generated with DataStage 7.0.1 and later
releases require more disk space when the columns are of type

Miscellaneous Environment Variables
VarChar when compared to 7.0. This is due to changes added for

performance improvements for bounded length VarChars in 7.0.1.
Set APT_OLD_BOUNDED_LENGTH to any value to revert to pre-7.0.1
storage behavior when using bounded length varchars. Setting this
variable can have adverse performance effects. The preferred and
more performant solution is to use unbounded length VarChars (don't
set any length) for columns where the maximum length is rarely used,
rather than set this environment variable.
APT_OPERATOR_REGISTRY_PATH
Used to locate operator .apt files, which define what operators are
available and which libraries they are found in.
APT_PM_NO_SHARED_MEMORY
By default, shared memory is used for local connections. If this
variable is set, named pipes rather than shared memory are used for
local connections. If both APT_PM_NO_NAMED_PIPES and
APT_PM_NO_SHARED_MEMORY are set, then TCP sockets are used
for local connections.
APT_PM_NO_NAMED_PIPES
Specifies not to use named pipes for local connections. Named pipes
will still be used in other areas of DataStage, including subprocs and
setting up of the shared memory transport protocol in the process
manager.
APT_PM_SOFT_KILL_WAIT
Delay between SIGINT and SIGKILL during abornal job shutdown.
Gives time for processes to run cleanups if they catch SIGINT. Defaults
to ZERO.
APT_PM_STARTUP_CONCURRENCY
Setting this to a small integer determines the number of simultaneous
section leader startups to be allowed. Setting this to 1 forces
sequential startup. The default is defined by SOMAXCONN in sys/
socket.h (currently 5 for Solaris, 10 for AIX).

Environment Variables Miscellaneous
APT_RECORD_COUNTS
Causes DataStage to print, for each operator Player, the number of
records consumed by getRecord() and produced by putRecord().
Abandoned input records are not necessarily accounted for. Buffer
operators do not print this information.
APT_SAVE_SCORE
Sets the name and path of the file used by the performance monitor to
hold temporary score data. The path must be visible from the host
machine. The performance monitor creates this file, therefore it need
not exist when you set this variable.
APT_SHOW_COMPONENT_CALLS
This forces DataStage to display messages at job check time as to
which user overloadable functions (such as checkConfig and
describeOperator) are being called. This will not produce output at
runtime and is not guaranteed to be a complete list of all user-
overloadable functions being called, but an effort is made to keep this
synchronized with any new virtual functions provided.
APT_STACK_TRACE
This variable controls the number of lines printed for stack traces. The
values are:
unset. 10 lines printed
0. infinite lines printed
N. N lines printed
none. no stack trace
The last setting can be used to disable stack traces entirely.
APT_WRITE_DS_VERSION
By default, DataStage saves data sets in the Orchestrate Version 4.1
format. APT_WRITE_DS_VERSION lets you save data sets in formats
compatible with previous versions of Orchestrate.
The values of APT_WRITE_DS_VERSION are:
v3_0. Orchestrate Version 3.0
v3. Orchestrate Version 3.1.x
v4. Orchestrate Version 4.0

Network Environment Variables
v4_0_3. Orchestrate Version 4.0.3 and later versions up to but not

including Version 4.1
v4_1. Orchestrate Version 4.1 and later versions through and
including Version 4.6
OSH_PRELOAD_LIBS
Specifies a colon-separated list of names of libraries to be loaded
before any other processing. Libraries containing custom operators
must be assigned to this variable or they must be registered. For
example, in Korn shell syntax:
$ export OSH_PRELOAD_LIBS="orchlib1:orchlib2:mylib1"
Network
These environment variables are concerned with the operation of
DataStage parallel jobs over a network.
APT_IO_MAXIMUM_OUTSTANDING
Sets the amount of memory, in bytes, allocated to a DataStage job on
every physical node for network communications. The default value is
2097152 (2MB).
When you are executing many partitions on a single physical node,
this number may need to be increased.
APT_IOMGR_CONNECT_ATTEMPTS
Sets the number of attempts for a TCP connect in case of a connection
failure. This is necessary only for jobs with a high degree of
parallelism in an MPP environment. The default value is 2 attempts (1
retry after an initial failure).
APT_PM_CONDUCTOR_HOSTNAME
The network name of the processing node from which you invoke a
job should be included in the configuration file as either a node or a
fastname. If the network name is not included in the configuration file,
DataStage users must set the environment variable
APT_PM_CONDUCTOR_HOSTNAME to the name of the node invoking
the DataStage job.

Environment Variables NLS Support
APT_PM_NO_TCPIP
This turns off use of UNIX sockets to communicate between player
processes at runtime. If the job is being run in an MPP (non-shared
memory) environment, do not set this variable, as UNIX sockets are
your only communications option.
APT_PM_NODE_TIMEOUT
This controls the number of seconds that the conductor will wait for a
section leader to start up and load a score before deciding that
something has failed. The default for starting a section leader process
is 30. The default for loading a score is 120.
APT_PM_SHOWRSH
Displays a trace message for every call to RSH.
APT_PM_USE_RSH_LOCALLY
If set, startup will use rsh even on the conductor node.
NLS Support
These environment variables are concerned with DataStage’s
implementation of NLS.
Warning You should not change the settings of any of these
environment variables other than APT_COLLATION
_STRENGTH if NLS is enabled on your server.
APT_COLLATION_SEQUENCE
This variable is used to specify the global collation sequence to be
used by sorts, compares, etc. This value can be overriden at the stage
level.
APT_COLLATION_STRENGTH
Set this to specify the defines the specifics of the collation algorithm.
This can be used to ignore accents, punctuation or other details.
It is set to one of Identical, Primary, Secondary, Tertiary, or Quartenary.
Setting it to Default unsets the environment variable. For an
explanation of possible settings, see:

Oracle Support Environment Variables
http://oss.software.ibm.com/icu/userguide/Collate_Concepts.html
APT_ENGLISH_MESSAGES
If set to 1, outputs every message issued with its English equivalent.
APT_IMPEXP_CHARSET
Controls the character encoding of ustring data imported and
exported to and from DataStage, and the record and field properties
applied to ustring fields. Its syntax is:
APT_IMPEXP_CHARSET icu_character_set
APT_INPUT_CHARSET
Controls the character encoding of data input to schema and
configuration files. Its syntax is:
APT_INPUT_CHARSET icu_character_set
APT_OS_CHARSET
Controls the character encoding DataStage uses for operating system
data such as the names of created files and the parameters to system
calls. Its syntax is:
APT_OS_CHARSET icu_character_set
APT_OUTPUT_CHARSET
Controls the character encoding of DataStage output messages and
operators like peek that use the error logging system to output data
input to the osh parser. Its syntax is:
APT_OUTPUT_CHARSET icu_character_set
APT_STRING_CHARSET
Controls the character encoding DataStage uses when performing
automatic conversions between string and ustring fields. Its syntax is:
APT_STRING_CHARSET icu_character_set
Oracle Support
These environment variables are concerned with the interaction
between DataStage and Oracle databases.

Environment Variables Oracle Support
APT_ORACLE_LOAD_DELIMITED
If this is defined, the orawrite operator creates delimited records when
loading into Oracle sqlldr. This method preserves leading and trailing
blanks within string fields (VARCHARS in the database). The value of
this variable is used as the delimiter. If this is defined without a value,
the default delimiter is a comma. Note that you cannot load a string
which has embedded double quotes if you use this.
APT_ORACLE_LOAD_OPTIONS
You can use the environment variable APT_ORACLE_LOAD_OPTIONS
to control the options that are included in the Oracle load control
file.You can load a table with indexes without using the Index Mode or
Disable Constraints properties by setting the
APT_ORACLE_LOAD_OPTIONS environment variable appropriately.
You need to set the Direct option and/or the PARALLEL option to
FALSE, for example:
APT_ORACLE_LOAD_OPTIONS='OPTIONS(DIRECT=FALSE,PARALLEL=TRUE)'
In this example the stage would still run in parallel, however, since
DIRECT is set to FALSE, the conventional path mode rather than the
direct path mode would be used.
If loading index organized tables (IOTs), you should not set both
DIRECT and PARALLEL to true as direct parallel path load is not
allowed for IOTs.
For more details about loading Oracle tables with indexes, see
"Loading Tables" in Parallel Job Developer’s Guide.
APT_ORACLE_NO_OPS
Set this if you do not have Oracle Parallel server installed on an AIX
system. It disables the OPS checking mechanism on the Oracle
Enterprise stage.
APT_ORACLE_PRESERVE_BLANKS
Set this to set the PRESERVE BLANKS option in the control file. This
preserves leading and trailing spaces. When PRESERVE BLANKS is
not set Oracle removes the spaces and considers fields with only
spaces to be NULL values.

Partitioning Environment Variables
APT_ORA_IGNORE_CONFIG_FILE_PARALLELISM
By default DataStage determines the number of processing nodes
available for a parallel write to Oracle from the configuration file. Set
APT_ORA_IGNORE_CONFIG_FILE_PARALLELISM to use the number
of data files in the destination table’s tablespace instead.
APT_ORA_WRITE_FILES
Set this to prevent the invocation of the Oracle loader when write
mode is selceted on an Oracle Enterprise destination stage. Instead,
the sqlldr commands are written to a file, the name of which is
specified by this environment variable. The file can be invoked once
the job has finished to run the loaders sequentially. This can be useful
in tracking down export and pipe-safety issues related to the loader.
APT_ORAUPSERT_COMMIT_TIME_INTERVAL
These two environment variables work together to specify how often
target rows are committed when using the Upsert method to write to
Oracle.
Commits are made whenever the time interval period has passed or
the row interval is reached, whichever comes first. By default,
commits are made every 2 seconds or 5000 rows.
Partitioning
The following environment variables are concerned with how
DataStage automatically partitions data.
APT_NO_PART_INSERTION
DataStage automatically inserts partition components in your
application to optimize the performance of the stages in your job. Set
this variable to prevent this automatic insertion.
APT_PARTITION_COUNT
Read only. DataStage sets this environment variable to the number of
partitions of a stage. The number is based both on information listed
in the configuration file and on any constraints applied to the stage.
The number of partitions is the degree of parallelism of a stage. For

Environment Variables Reading and Writing Files
example, if a stage executes on two processing nodes,

APT_PARTITION_COUNT is set to 2.
You can access the environment variable APT_PARTITION_COUNT to
determine the number of partitions of the stage from within:
an operator wrapper
a shell script called from a wrapper
getenv() in C++ code
sysget() in the SAS language.
APT_PARTITION_NUMBER
Read only. On each partition, DataStage sets this environment
variable to the index number (0, 1, ...) of this partition within the stage.
A subprocess can then examine this variable when determining which
partition of an input file it should handle.
Reading and Writing Files

These environment variables are concerned with reading and writing
files.
APT_DELIMITED_READ_SIZE
By default, the DataStage will read ahead 500 bytes to get the next
delimiter. For streaming inputs (socket, FIFO, etc.) this is sub-optimal,
since the DataStage may block (and not output any records).
DataStage, when reading a delimited record, will read this many bytes
(minimum legal value for this is 2) instead of 500. If a delimiter is NOT
available within N bytes, N will be incremented by a factor of 2 (when
this environment variable is not set, this changes to 4).
APT_FILE_IMPORT_BUFFER_SIZE
The value in kilobytes of the buffer for reading in files. The default is
128 (i.e., 128 KB). It can be set to values from 8 upward, but is clamped
to a minimum value of 8. That is, if you set it to a value less than 8,
then 8 is used. Tune this upward for long-latency files (typically from
heavily loaded file servers).

Reporting Environment Variables
APT_FILE_EXPORT_BUFFER_SIZE
The value in kilobytes of the buffer for writing to files. The default is
128 (i.e., 128 KB). It can be set to values from 8 upward, but is clamped
to a minimum value of 8. That is, if you set it to a value less than 8,
then 8 is used. Tune this upward for long-latency files (typically from
heavily loaded file servers).
APT_IMPORT_PATTERN_USES_FILESET
When this is set, DataStage will turn any file pattern into a fileset
before processing the files. This allows the files to be processed in
parallel as apposed to sequentially. By default file pattern will cat the
files together to be used as the input.
APT_MAX_DELIMITED_READ_SIZE
By default, when reading, DataStage will read ahead 500 bytes to get
the next delimiter. If it is not found, DataStage looks ahead
4*500=2000 (1500 more) bytes, and so on (4X) up to 100,000 bytes.
This variable controls the upper bound which is by default 100,000
bytes. Note that this variable should be used instead of
APT_DELIMITED_READ_SIZE when a larger than 500 bytes read-
ahead is desired.
APT_PREVIOUS_FINAL_DELIMITER_COMPATIBLE
Set this to revert to the pre-release 7.5 behavior of the final delimiter
whereby, when writing data, a space is inserted after every field in a
record including the last one. (The new behavior is that the a space is
written after every field except the last one).
APT_STRING_PADCHAR
Overrides the pad character of 0x0 (ASCII null), used by default when
DataStage extends, or pads, a string field to a fixed length.
Reporting
These environment variables are concerned with various aspects of
DataStage jobs reporting their progress.

Environment Variables Reporting
APT_DUMP_SCORE
under the Parallel ➤ Reporting branch. Configures DataStage to print
a report showing the operators, processes, and data sets in a running
job.
APT_ERROR_CONFIGURATION
Controls the format of DataStage output messages.
Warning Changing these settings can seriously interfere with
DataStage logging.
This variable’s value is a comma-separated list of keywords (see table
below). Each keyword enables a corresponding portion of the
message. To disable that portion of the message, precede it with a “!”.
Default formats of messages displayed by DataStage include the
keywords severity, moduleId, errorIndex, timestamp, opid, and
message.
The following table lists keywords, the length (in characters) of the
associated components in the message, and the keyword’s meaning.
The characters "##" precede all messages. The keyword lengthprefix
appears in three locations in the table. This single keyword controls
the display of all length prefixes.
Keyword Length Meaning

severity 1 Severity indication: F, E, W, or I.
vseverity 7 Verbose description of error severity (Fatal,

Error, Warning, Information).
jobid 3 The job identifier of the job. This allows you to

identify multiple jobrunning at once. The
default job identifier is 0.
moduleId 4 The module identifier. For DataStage-defined

messages, this value is a four byte string
beginning with T. For user-defined messages
written to the error log, this string is USER. For
all outputs from a subprocess, the string is
USBP.
errorIndex 6 The index of the message specified at the time

the message was written to the error log.
timestamp 13 The message time stamp. This component

consists of the string "HH:MM:SS(SEQ)", at the
time the message was written to the error log.
Messages generated in the same second have
ordered sequence numbers.

Reporting Environment Variables
Keyword Length Meaning

ipaddr 15 The IP address of the processing node
generating the message. This 15-character
string is in octet form, with individual octets
zero filled, for example, 104.032.007.100.
lengthprefix 2 Length in bytes of the following field.
nodename variable The node name of the processing node

generating the message.
lengthprefix 2 Length in bytes of the following field.
opid variable The string <main_program> for error messages

originating in your main program (outside of a
step or within the
APT_Operator::describeOperator()
override). The string <node_nodename>
representing system error messages
originating on a node, where nodename is the
name of the node. The operator originator
identifier, represented by
"ident, partition_number", for errors originating
within a step. This component identifies the
instance of the operator that generated the
message. ident is the operator name (with the
operator index in parenthesis if there is more
than one instance of it). partition_number
defines the partition number of the
operator issuing the message.
lengthprefix 5 Length, in bytes, of the following field.

Maximum length is 15 KB.
message variable Error text.
1 Newline character
APT_MSG_FILELINE
under the Parallel ➤ Reporting branch. Set this to have DataStage log
extra internal information for parallel jobs.
APT_PM_PLAYER_MEMORY
under the Parallel ➤ Reporting branch. Setting this variable causes
each player process to report the process heap memory allocation in
the job log when returning.

Environment Variables Reporting
APT_PM_PLAYER_TIMING
under the Parallel ➤ Reporting branch. Setting this variable causes
each player process to report its call and return in the job log. The
message with the return is annotated with CPU times for the player
process.
APT_RECORD_COUNTS
under the Parallel ➤ Reporting branch. Causes DataStage to print to
the job log, for each operator player, the number of records input and
output. Abandoned input records are not necessarily accounted for.
Buffer operators do not print this information.
OSH_DUMP
under the Parallel ➤ Reporting branch. If set, it causes DataStage to
put a verbose description of a job in the job log before attempting to
execute it.
OSH_ECHO
echo its job specification to the job log after the shell has expanded all
arguments.
OSH_EXPLAIN
place a terse description of the job in the job log before attempting to
run it.
OSH_PRINT_SCHEMAS
print the record schema of all data sets and the interface schema of all
operators in the job log.

SAS Support Environment Variables
SAS Support
These environment variables are concerned with DataStage
interaction with SAS.
APT_HASH_TO_SASHASH
The DataStage hash partitioner contains support for hashing SAS
data. In addition, DataStage provides the sashash partitioner which
uses an alternative non-standard hashing algorithm. Setting the
APT_HASH_TO_SASHASH environment variable causes all
appropriate instances of hash to be replaced by sashash. If the
APT_NO_SAS_TRANSFORMS environment variable is set,
APT_HASH_TO_SASHASH has no affect.
APT_NO_SASOUT_INSERT
This variable selectively disables the sasout operator insertions. It
maintains the other SAS-specific transformations.
APT_NO_SAS_TRANSFORMS
DataStage automatically performs certain types of SAS-specific
component transformations, such as inserting an sasout operator and
substituting sasRoundRobin for RoundRobin. Setting the
APT_NO_SAS_TRANSFORMS variable prevents DataStage from
making these transformations.
APT_SAS_ACCEPT_ERROR
When a SAS procedure causes SAS to exit with an error, this variable
prevents the SAS-interface operator from terminating. The default
behavior is for DataStage to terminate the operator with an error.
APT_SAS_CHARSET
When the -sas_cs option of a SAS-interface operator is not set and a
SAS-interface operator encounters a ustring, DataStage interrogates
this variable to determine what character set to use. If this variable is
not set, but APT_SAS_CHARSET_ABORT is set, the operator will abort;
otherwise the -impexp_charset option or the APT_IMPEXP_CHARSET
environment variable is accessed. Its syntax is:
APT_SAS_CHARSET icu_character_set | SAS_DBCSLANG

Environment Variables SAS Support
APT_SAS_CHARSET_ABORT
Causes a SAS-interface operator to abort if DataStage encounters a
ustring in the schema and neither the -sas_cs option nor the
APT_SAS_CHARSET environment variable is set.
APT_SAS_COMMAND
Overrides the $PATH directory for SAS with an absolute path to the
basic SAS executable. An example path is:
/usr/local/sas/sas8.2/sas
APT_SASINT_COMMAND
Overrides the $PATH directory for SAS with an absolute path to the
International SAS executable. An example path is:
/usr/local/sas/sas8.2int/dbcs/sas
APT_SAS_DEBUG
Set this to set debug in the SAS process coupled to the SAS stage.
Messages appear in the SAS log, which may then be copied into the
DataStage log. Use APT_SAS_DEBUG=1, APT_SAS_DEBUG_IO=1, and
APT_SAS_DEBUG_VERBOSE=1 to get all debug messages.
APT_SAS_DEBUG_IO
Set this to set input/output debug in the SAS process coupled to the
SAS stage. Messages appear in the SAS log, which may then be
copied into the DataStage log.
APT_SAS_DEBUG_LEVEL
Its syntax is:
APT_SAS_DEBUG_LEVEL=[0-3]
Specifies the level of debugging messages to output from the SAS

driver. The values of 1, 2, and 3 duplicate the output for the -debug
option of the SAS operator:
no, yes, and verbose.
APT_SAS_DEBUG_VERBOSE
Set this to set verbose debug in the SAS process coupled to the SAS
stage. Messages appear in the SAS log, which may then be copied
into the DataStage log.

Sorting Environment Variables
APT_SAS_NO_PSDS_USTRING
Set this to prevent DataStage from automatically converting SAS char
types to ustrings in an SAS parallel data set.
APT_SAS_S_ARGUMENT
By default, DataStage executes SAS with -s 0. When this variable is
set, its value is be used instead of 0. Consult the SAS documentation
for details.
APT_SAS_SCHEMASOURCE_DUMP
When using SAS Schema Source, sauses the command line to be
written to the log when executing SAS. You use it to inspect the data
contained in a -schemaSource. Set this if you ar egetting an error
when specifying the SAS data set containing the schema source.
APT_SAS_SHOW_INFO
Displays the standard SAS output from an import or export
transaction. The SAS output is normally deleted since a transaction is
usually successful.
APT_SAS_TRUNCATION
Its syntax is:
APT_SAS_TRUNCATION ABORT | NULL | TRUNCATE
Because a ustring of n characters does not fit into n characters of a

SAS char value, the ustring value must be truncated beyond the space
pad characters and \0.
The sasin and sas operators use this variable to determine how to
truncate a ustring value to fit into a SAS char field. TRUNCATE, which
is the default, causes the ustring to be truncated; ABORT causes the
operator to abort; and NULL exports a null field. For NULL and
TRUNCATE, the first five occurrences for each column cause an
information message to be issued to the log.
Sorting
The following environment variables are concerned with how
DataStage automatically sorts data.

Environment Variables Teradata Support
APT_NO_SORT_INSERTION
DataStage automatically inserts sort components in your job to
optimize the performance of the operators in your data flow. Set this
variable to prevent this automatic insertion.
APT_SORT_INSERTION_CHECK_ONLY
When sorts are inserted automatically by DataStage, if this is set, the
sorts will just check that the order is correct, they won't actually sort.
This is a better alternative to shutting partitioning and sorting off
insertion off using APT_NO_PART_INSERTION and
APT_NO_SORT_INSERTION.
Teradata Support
The following environment variables are concerned with DataStage
interaction with Teradata databases.
APT_TERA_64K_BUFFERS
DataStage assumes that the terawrite operator writes to buffers
whose maximum size is 32 KB. Enable the use of 64 KB buffers by
setting this variable. The default is that it is not set.
APT_TERA_NO_ERR_CLEANUP
Setting this variable prevents removal of error tables and the partially
written target table of a terawrite operation that has not successfully
completed. Set this variable for diagnostic purposes only. In some
cases, setting this variable forces completion of an unsuccessful write
operation.
APT_TERA_NO_SQL_CONVERSION
Set this to prevent the SQL statements you are generating from being
converted to the character set specified for your stage (character sets
can be specified at project, job, or stage level). The SQL statements
are converted to LATIN1 instead.

Transport Blocks Environment Variables
APT_TERA_NO_PERM_CHECKS
Set this to bypass permission checking on the several system tables
that need to be readable for the load process. This can speed up the
start time of the load process slightly.
APT_TERA_SYNC_DATABASE
Specifies the database used for the terasync table. By default, the
database used for the terasync table is specified as part of
APT_TERA_SYNC_USER. If you want the database to be different, set
this variable. You must then give APT_TERA_SYNC_USER read and
write permission for this database.
APT_TERA_SYNC_PASSWORD
Specifies the password for the user identified by
APT_TERA_SYNC_USER.
APT_TERA_SYNC_USER
Specifies the user that creates and writes to the terasync table.
Transport Blocks
The following environment variables are all concerned with the block
size used for the internal transfer of data as jobs run. Some of the
settings only apply to fixed length records The following variables are
used only for fixed-length records.:
APT_MAX_TRANSPORT_BLOCK_SIZE
APT_AUTO_TRANSPORT_BLOCK_SIZE
APT_AUTO_TRANSPORT_BLOCK_SIZE
under the Parallel category. When set, Orchestrate calculates the block
size for transferring data internally as jobs run. It uses this algorithm:

Environment Variables Guide to Setting Environment Variables
if (recordSize * APT_LATENCY_COEFFICIENT
< APT_MIN_TRANSPORT_BLOCK_SIZE)
blockSize = minAllowedBlockSize
else if (recordSize * APT_LATENCY_COEFFICIENT
> APT_MAX_TRANSPORT_BLOCK_SIZE)
blockSize = maxAllowedBlockSize
else blockSize = recordSize * APT_LATENCY_COEFFICIENT
Specifies the number of writes to a block which transfers data
between players. This variable allows you to control the latency of
data flow through a step. The default value is 5. Specify a value of 0 to
have a record transported immediately. This is only used for fixed
length records.
Note Many operators have a built-in latency and are not affected
by this variable.
Specify the default block size for transferring data between players. It
defaults to 131072 (128 KB).
Specify the minimum and maximum allowable block size for
transferring data between players.
APT_MIN_TRANSPORT_BLOCK_SIZE cannot be less than 8192 which
is its default value. APT_MAX_TRANSPORT_BLOCK_SIZE cannot be
greater than 1048576 which is its default value. These variables are
only meaningful when used in combination with
APT_LATENCY_COEFFICIENT and
APT_AUTO_TRANSPORT_BLOCK_SIZE.
Guide to Setting Environment Variables

This section gives some guide as to which environment variables
should be set in what circumstances.
Environment Variable Settings for all Jobs

We recommend that you set the following environment variables for
all jobs:

Guide to Setting Environment Variables Environment Variables
APT_CONFIG_FILE (see page 6-16).

APT_DUMP_SCORE (see page 6-29).
APT_RECORD_COUNTS (see page 6-31)
Optional Environment Variable Settings

We recommend setting the following environment variables as
needed on a per-job basis. These variables can be used to turn the
performance of a particular job flow, to assist in debugging, and to
change the default behavior of specific parallel job stages.
Performance Tuning
APT_BUFFER_MAXIMUM_MEMORY (see page 6-7).
APT_BUFFER_FREE_RUN (see page 6-6)
TMPDIR. This defaults to /tmp. It is used for miscellaneous internal
temporary data, including FIFO queues and Transformer
temporary storage. As a minor optimization, it can be better to
ensure that it is set to a file system separate to the DataStage
install directory.
Job Flow Debugging

OSH_PRINT_SCHEMAS (see page 6-31).
APT_DISABLE_COMBINATION (see page 6-16).
APT_PM_PLAYER_TIMING (see page 6-31).
APT_PM_PLAYER_MEMORY (see page 6-30).
APT_BUFFERING_POLICY (set to FORCE_BUFFERING – see
page 6-7).
Job Flow Design

APT_STRING_PADCHAR (see page 6-28).

7
DataStage Development Kit (Job
Control Interfaces)
DataStage provides a range of methods that enable you to run

DataStage server or parallel jobs directly on the server, without using
the DataStage Director. The methods are:
C/C++ API (the DataStage development kit)
DataStage BASIC calls
Command line Interface commands (CLI)
DataStage macros
These methods can be used in different situations as follows:
API. Using the API you can build a self-contained program that can
run anywhere on your system, provided that it can connect to a
DataStage server across the network.
BASIC. Programs built using the DataStage BASIC interface can be
run from any DataStage server on the network. You can use this
interface to define jobs that run and control other jobs. The
controlling job can be run from the Director client like any other
job, or directly on the server machine from the TCL prompt. (Job
sequences provide another way of producing control jobs – see
DataStage Designer Guide for details.)
CLI. The CLI can be used from the command line of any DataStage
server on the network. Using this method, you can run jobs on
other servers too.
Macros. A set of macros can be used in job designs or in BASIC
programs. These are mostly used to retrieve information about
other jobs.

DataStage Development Kit DataStage Development Kit (Job Control Interfaces)
DataStage Development Kit

The DataStage Development Kit provides the DataStage API, a C or
C++ application programming interface.
This section gives general information about using the DataStage API.
Specific information about API functions is in "API Functions" on
page 7-4.
A listing for an example program which uses the API is in Appendix A.
The dsapi.h Header File

DataStage API provides a header file that should be included with all
API programs. The header file includes prototypes for all DataStage
API functions. It is located in the directory $DSHOME/include Their
format depends on which tokens you have defined:
If the _STDC_ or WIN32 tokens are defined, the prototypes are in
ANSI C style.
If the _cplusplus token is defined, the prototypes are in C++
format with the declarations surrounded by:
extern "C" {…}
Otherwise the prototypes are in Kernighan and Ritchie format.
Data Structures, Result Data, and Threads

DataStage API functions return information about objects as pointers
to data items. This is either done directly, or indirectly by setting
pointers in the elements of a data structure that is provided by the
caller.
Each thread within a calling application is allocated a separate storage
area. Each call to a DataStage API routine overwrites any existing
contents of this data area with the results of the call, and returns a
pointer into the area for the requested data.
For example, the DSGetProjectList function obtains a list of
DataStage projects, and the DSGetProjectInfo function obtains a list
of jobs within a project. When the DSGetProjectList function is
called it retrieves the list of projects, stores it in the thread’s data area,
and returns a pointer to this area. If the same thread then calls
DSGetProjectInfo, the job list is retrieved and stored in the thread’s
data area, overwriting the project list. The job list pointer in the
supplied data structure references the thread data area.
This means that if the results of a DataStage API function need to be
reused later, the application should make its own copy of the data

DataStage Development Kit (Job Control Interfaces) DataStage Development Kit
before making a new DataStage API call. Alternatively, the calls can be
used in multiple threads.
DataStage API stores errors for each thread: a call to the
DSGetLastError function returns the last error generated within the
calling thread.
Writing DataStage API Programs

Your application should use the DataStage API functions in a logical
order to ensure that connections are opened and closed correctly, and
jobs are run effectively. The following procedure suggests an outline
for the program logic to follow, and which functions to use at each
step:
1 If required, set the server name, user name, and password to use
for connecting to DataStage (DSSetServerParams).
2 Obtain the list of valid projects (DSGetProjectList).
3 Open a project (DSOpenProject).
4 Obtain a list of jobs (DSGetProjectInfo).
5 Open one or more jobs (DSOpenJob).
6 List the job parameters (DSGetParamInfo).
7 Lock the job (DSLockJob).
8 Set the job’s parameters and limits (DSSetJobLimit,
DSSetParam).
9 Start the job running (DSRunJob).
10 Poll for the job or wait for job completion (DSWaitForJob,
DSStopJob, DSGetJobInfo).
11 Unlock the job (DSUnlockJob).
12 Display a summary of the job’s log entries
(DSFindFirstLogEntry, DSFindNextLogEntry).
13 Display details of specific log events (DSGetNewestLogId,
DSGetLogEntry).
14 Examine and display details of job stages (DSGetJobInfo – stage
list, DSGetStageInfo).
15 Examine and display details of links within active stages
(DSGetStageInfo – link list, DSGetLinkInfo).
16 Close all open jobs (DSCloseJob).
17 Detach from the project (DSCloseProject).

Building a DataStage API Application

Everything you need to create an application that uses the DataStage
API is in a subdirectory called dsdk (DataStage Development Kit) in
the Ascential\DataStage installation directory on the server machine.
To build an application that uses the DataStage API:
1 Write the program, including the dsapi.h header file in all source
modules that uses the DataStage API.
2 Compile the code. Ensure that the WIN32 token is defined. (This
happens automatically in the Microsoft Visual C/C++ compiler
environment.)
3 Link the application, including vmdsapi.lib, in the list of libraries to
be included.
Redistributing Applications
If you intend to run your DataStage API application on a computer
where DataStage Server is installed, you do not need to include
DataStage API DLLs or libraries as these are installed as part of
DataStage Server.
If you want to run the application from a computer used as a
DataStage client, you should redistribute the following library with
your application:
vmdsapi.dll
If you intend to run the program from a computer that has neither
DataStage Server nor any DataStage client installed, in addition to the
library mentioned above, you should also redistribute the following:
dsclnt32.dll
dsrpc32.dll
You should locate these files where they will be in the search path of
any user who uses the application, for example, in the
%SystemRoot%\System32 directory.
API Functions
This section details the functions provided in the DataStage API.
These functions are described in alphabetical order. The following
table briefly describes the functions categorized by usage:
Usage Function Description

Accessing DSCloseProject Closes a project that was opened with
projects DSOpenProject.

DataStage Development Kit (Job Control Interfaces) DataStage Development Kit

DSGetProjectList Retrieves a list of all projects on the
server.
DSGetProjectInfo Retrieves a list of jobs in a project.
DSOpenProject Opens a project.
DSSetServerParams Sets the server name, user name, and

password to use for a job.
Accessing jobs DSCloseJob Closes a job that was opened with

DSOpenJob.
DSGetJobInfo Retrieves information about a job, such

as the date and time of the last run,
parameter names, and so on.
DSLockJob Locks a job prior to setting job

parameters or starting a job run.
DSOpenJob Opens a job.
DSRunJob Runs a job.
DSStopJob Aborts a running job.
DSUnlockJob Unlocks a job, enabling other processes

to use it.
DSWaitForJob Waits until a job has completed.
Accessing job DSGetParamInfo Retrieves information about a job

parameters parameter.
DSSetJobLimit Sets row processing and warning limits

for a job.
DSSetParam Sets job parameter values.
Accessing DSGetStageInfo Retrieves information about a stage

stages within a job.
Accessing links DSGetLinkInfo Retrieves information about a link of an

active stage within a job.
Accessing log DSFindFirstLogEntry Retrieves entries in a log that meet the

entries specified criteria.
DSFindNextLogEntry Finds the next log entry that meets the

criteria specified in
DSFindFirstLogEntry.
DSGetLogEntry Retrieves the specified log entry.
DSGetNewestLogId Retrieves the newest entry in the log.
DSLogEvent Adds a new entry to the log.


Administering DSAddEnvVar Adds a new environment variable.
Projects and
jobs
DSAddProject Add a project.
DSDeleteEnvVar Delete an environment variable.
DSDeleteProject Delete a project.
DSListEnvVars List environment variables.
DSListProjectProperties List the properties of a project.
DSSetEnvVar Set an environment variable.
DSSetProjectProperty Set property for a project.
Handling DSGetLastError Retrieves the last error code value

errors generated by the calling thread.
DSGetLastErrorMsg Retrieves the text of the last reported

error.

DataStage Development Kit (Job Control Interfaces) DSAddEnvVar
DSAddEnvVar
Add an environment variable to the specified project. It is added to the
User Defined category.
Syntax
int DSAddEnvVar(
DSPROJECT hProject,
char *EnvVarName,
char *Type,
char *PromptText,
char *Value
);
Parameters
hProject is the value returned from DSOpenProject
EnvVarName is the name of the environment variable
Type is DSA_ENVVAR_TYPE_STRING for string type environment
variables or DSA_ENVVAR_TYPE_ENCRYPTED for encrypted
environment variables.
PromptText is the user-visible text describing the environment
variable.
Value is value to set the environment variable to or “”.
Return Values
If the function succeeds, then the return value is DSJE_NOERROR
If the function fails, then the return value is one of the following:
DSJE_BADENVVARNAME invalid environment variable name
DSJE_BADENVVARTYPE invalid type
DSJE_BADENVVARPROMPT no prompt supplied
DSJE_READENVVARDEFNS failed to read environment variable
definitions
DSJE_READENVVARVALUES failed to read environment variable
values
DSJE_WRITEENVVARDEFNS failed to write environment variable
definitions
DSJE_WRITEENVVARVALUES failed to write environment variable
values
DSJE_DUPENVVARNAME environment variable already exists

DSAddEnvVar DataStage Development Kit (Job Control Interfaces)
DSJE_ENCODEFAILED failed to encode an encrypted value
Remarks
To use this method, the program needs to have previously attached to
a project using DSOpenProject. This returns a handle to the project,
hProject.

DataStage Development Kit (Job Control Interfaces) DSAddProject
DSAddProject
Creates a new project. The user who runs the code containing this
function must be a DataStage Administrator.
Syntax
int DSAddProject(
char *ProjectName,
char *ProjectLocation
);
Parameters
ProjectName is the name of the project to create.
ProjectLocation is the pathname of the directory to create the project
in. To create a project in the default project directory (server install
path/Projects/projectName), this argument should be set to “”.
Return Values
DSJE_NOTADMINUSER user is not an administrator
DSJE_ISADMINFAILED failed to determine whether user is
an administrator
DSJE_BADPROJNAME invalid project name supplied
DSJE_GETDEFAULTPATHFAILED failed to determine default
project directory
DSJE_BADPROJLOCATION invalid pathname supplied
DSJE_INVALIDPROJECTLOCATION invalid pathname supplied
DSJE_OPENFAILED failed to open UV.ACCOUNT file
DSJE_READUFAILED failed to lock project create lock
record
DSJE_ADDPROJECTBLOCKED another user is adding a project
DSJE_ADDPROJECTFAILED failed to add project
DSJE_LICENSEPROJECTFAILED failed to license project
DSJE_RELEASEFAILED failed to release project create lock
record

DSCloseJob DataStage Development Kit (Job Control Interfaces)
DSCloseJob
Closes a job that was opened using DSOpenJob.
Syntax
int DSCloseJob(
DSJOB JobHandle
);
Parameter
JobHandle is the value returned from DSOpenJob.
Return Values
If the function succeeds, the return value is DSJE_NOERROR.
If the function fails, the return value is:
DSJE_BADHANDLEInvalid JobHandle.
Remarks
If the job is locked when DSCloseJob is called, it is unlocked.
If the job is running when DSCloseJob is called, the job is allowed to
finish, and the function returns a value of DSJE_NOERROR
immediately.

DataStage Development Kit (Job Control Interfaces) DSCloseProject
DSCloseProject
Closes a project that was opened using the DSOpenProject function.
Syntax
int DSCloseProject(
DSPROJECT ProjectHandle
);
Parameter
ProjectHandle is the value returned from DSOpenProject.
Return Value
This function always returns a value of DSJE_NOERROR.
Remarks
Any open jobs in the project are closed, running jobs are allowed to
finish, and the function returns immediately.

DSDeleteEnvVar DataStage Development Kit (Job Control Interfaces)
DSDeleteEnvVar
Delete a user-defined environment variable in a specified project.
Syntax
int DSDeleteEnvVar(
DSPROJECT hProject,
char *EnvVar
);
Parameters
EnvVarName is the name of the environment variable
Return Values
definitions
values
DSJE_BADENVVAR environment variable does not exist
DSJE_WRITEENVVARDEFNS failed to write environment variable
definitions
values
DSJE_NOTUSERDEFINED environment variable is not user-
defined and therefore cannot be deleted

DataStage Development Kit (Job Control Interfaces) DSDeleteProject
DSDeleteProject
Deletes a project. The user who runs the code containing this function
must be a DataStage Administrator. Note that any jobs scheduled to
be run that are included in this project will also be deleted.
Syntax
int DSDeleteProject(
char *ProjectName
);
Parameter
ProjectName is the name of the project to delete.
Return Value
DSJE_ISADMINFAILED failed to determine whether user is an
administrator
DSJE_OPENFAILED failed to open UV.ACCOUNT file
DSJE_READUFAILED failed to lock project record
DSJE_RELEASEFAILED failed to release project record
DSJE_LISTSCHEDULEFAILED failed to get list of scheduled jobs
for project
DSJE_CLEARSCHEDULEFAILED failed to clear scheduled jobs for
project
DSJE_DELETEPROJECTBLOCKED project locked by another user
DSJE_NOTAPROJECT failed to log to project
DSJE_ACCOUNTPATHFAILED failed to get account path
DSJE_LOGTOFAILED failed to log to UV account
DSJE_DELPROJFAILED failed to delete project definition
DSJE_DELPROJFILESFAILED failed to delete project files

DSFindFirstLogEntry DataStage Development Kit (Job Control Interfaces)
DSFindFirstLogEntry
Retrieves all the log entries that meet the specified criteria, and writes
the first entry to a data structure. Subsequent log entries can then be
read using the DSFindNextLogEntry function.
Syntax
int DSFindFirstLogEntry(
DSJOB JobHandle,
int EventType,
time_t StartTime,
time_t EndTime,
int MaxNumber,
DSLOGEVENT *Event
);
Parameters
EventType is one of the following keys:
This key… Retrieves this type of message…

DSJ_LOGINFO Information
DSJ_LOGWARNING Warning
DSJ_LOGFATAL Fatal
DSJ_LOGREJECT Transformer row rejection
DSJ_LOGSTARTED Job started
DSJ_LOGRESET Job reset
DSJ_LOGBATCH Batch control
DSJ_LOGOTHER All other log types
DSJ_LOGANY Any type of event
StartTime limits the returned log events to those that occurred on or

after the specified date and time. Set this value to 0 to return the
earliest event.
EndTime limits the returned log events to those that occurred before
the specified date and time. Set this value to 0 to return all entries up
to the most recent.
MaxNumber specifies the maximum number of log entries to retrieve,
starting from the latest.

DataStage Development Kit (Job Control Interfaces) DSFindFirstLogEntry
Event is a pointer to a data structure to use to hold the first retrieved

log entry.
Return Values
If the function succeeds, the return value is DSJE_NOERROR, and
summary details of the first log entry are written to Event.
If the function fails, the return value is one of the following:
Token Description
DSJE_NOMORE There are no events matching the filter criteria.
DSJE_NO_MEMORY Failed to allocate memory for results from server.
DSJE_BADHANDLE Invalid JobHandle.
DSJE_BADTYPE Invalid EventType value.
DSJE_BADTIME Invalid StartTime or EndTime value.
DSJE_BADVALUE Invalid MaxNumber value.
Remarks
The retrieved log entries are cached for retrieval by subsequent calls
to DSFindNextLogEntry. Any cached log entries that are not
processed by a call to DSFindNextLogEntry are discarded at the
next DSFindFirstLogEntry call (for any job), or when the project is
closed.
Note The log entries are cached by project handle. Multiple
threads using the same open project handle must
coordinate access to DSFindFirstLogEntry and
DSFindNextLogEntry.

DSFindNextLogEntry DataStage Development Kit (Job Control Interfaces)
DSFindNextLogEntry
Retrieves the next log entry from the cache.
Syntax
int DSFindNextLogEntry(
DSJOB JobHandle,
DSLOGEVENT *Event
);
Parameters
Event is a pointer to a data structure to use to hold the next log entry.
Return Values
If the function succeeds, the return value is DSJE_NOERROR and
summary details of the next available log entry are written to Event.
Token Description
DSJE_NOMORE All events matching the filter criteria have been
returned.
DSJE_SERVER_ERROR Internal error. The DataStage Server returned invalid

data.
Remarks
This function retrieves the next log entry from the cache of entries
produced by a call to DSFindFirstLogEntry.
Note The log entries are cached by project handle. Multiple
threads using the same open project handle must
coordinate access to DSFindFirstLogEntry and
DSFindNextLogEntry.

DataStage Development Kit (Job Control Interfaces) DSGetProjectList
DSGetProjectList
Obtains information reported at the end of execution of certain
parallel stages. The information collected, and available to be
interrogated, is specified at design time. For example, transformer
stage information is specified in the Triggers tab of the Transformer
stage Properties dialog box.
Syntax
int DSGetCustInfo(
DSJOB JobHandle,
char *StageName,
char *CustinfoName
int InfoType,
DSSTAGEINFO *ReturnInfo
);
Parameters
StageName is a pointer to a null-terminated string specifying the
name of the stage to be interrogated.
CustinfoName is a pointer to a null-terminated string specifiying the
name of the variable to be interrogated (as set up on the Triggers
tab).
InfoType is one of the following keys:
This key… Returns this information…

DSJ_CUSTINFOVALUE The value of the specified variable.
DSJ_CUSTINFODESC Description of the variable.
Return Values
Token Description
DSJE_NOT_AVAILABLE There are no instances of the requested information in
the stage.
DSJE_BADSTAGE StageName does not refer to a known stage in the job.

DSGetProjectList DataStage Development Kit (Job Control Interfaces)
Token Description
DSJE_BADCUSTINFO CustinfoName does not refer to a known custinfo item.
DSJE_BADTYPE Invalid InfoType.

DataStage Development Kit (Job Control Interfaces) DSGetJobInfo
DSGetJobInfo
Retrieves information about the status of a job.
Syntax
int DSGetJobInfo(
DSJOB JobHandle,
int InfoType,
DSJOBINFO *ReturnInfo
);
Parameters
InfoType is a key indicating the information to be returned and can
have any of the following values:

DSJ_JOBSTATUS The current status of the job.
DSJ_JOBNAME The name of the job referenced by

JobHandle.
DSJ_JOBCONTROLLER The name of the job controlling the job

referenced by JobHandle.
DSJ_JOBSTARTTIMESTAMP The date and time when the job started.
DSJ_JOBWAVENO The wave number of last or current run.
DSJ_JOBDESC The Job Description specified in the Job

Properties dialog box.
DSJ_JOBFULLDESSC The Full Description specified in the Job

DSJ_JOBDMISERVICE Set to true if this is a web service job.
DSJ_JOBMULTIINVOKABLE Set to true if this job supports multiple

invocations.
DSJ_PARAMLIST A list of job parameter names. Separated

by nulls.
DSJ_STAGELIST A list of active stages in the job. Separated

by nulls.
DSJ_USERSTATUS The value, if any, set as the user status by

the job.
DSJ_JOBCONTROL Whether a stop request has been issued

for the job referenced by JobHandle.

DSGetJobInfo DataStage Development Kit (Job Control Interfaces)

DSJ_JOBPID Process id of DSD.RUN process.
DSJ_JOBLASTTIMESTAMP The date and time when the job last

finished.
DSJ_JOBINVOCATIONS List of job invocation ids. The ids are

separated by nulls.
DSJ_JOBINTERIMSTATUS The status of a job after it has run all

stages and controlled jobs, but before it
has attempted to run an after-job
subroutine. (Designed to be used by an
after-job subroutine to get the status of the
current job.)
DSJ_JOBINVOCATIONID Invocation name of the job referenced by

JobHandle.
DSJ_JOBDESC A description of the job.
DSJ_STAGELIST2 A list of passive stages in the job.

Separated by nulls.
DSJ_JOBELAPSED The elapsed time of the job in seconds.
ReturnInfo is a pointer to a DSJOBINFO data structure where the

requested information is stored. The DSJOBINFO data structure
contains a union with an element for each of the possible return
values from the call to DSGetJobInfo. For more information, see
"Data Structures" on page 7-62.
Return Values
Token Description
the job.
Remarks
For controlled jobs, this function can be used either before or after a
call to DSRunJob.

DataStage Development Kit (Job Control Interfaces) DSGetLastError
DSGetLastError
Returns the calling thread’s last error code value.
Syntax
int DSGetLastError(void);
Return Values
The return value is the last error code value. The "Return Values"
section of each reference page notes the conditions under which the
function sets the last error code.
Remarks
Use DSGetLastError immediately after any function whose return
value on failure might contain useful data, otherwise a later,
successful function might reset the value back to 0 (DSJE_NOERROR).
Note Multiple threads do not overwrite each other’s error codes.

DSGetLastErrorMsg DataStage Development Kit (Job Control Interfaces)
DSGetLastErrorMsg
Retrieves the text of the last reported error from the DataStage server.
Syntax
char *DSGetLastErrorMsg(
DSPROJECT ProjectHandle
);
Parameter
ProjectHandle is either the value returned from DSOpenProject or
NULL.
Return Values
The return value is a pointer to a series of null-terminated strings, one
for each line of the error message associated with the last error
generated by the DataStage Server in response to a DataStage API
function call. Use DSGetLastError to determine what the error
number is.
The following example shows the buffer contents with <null>
representing the terminating NULL character:
line1<null>line2<null>line3<null><null>
The DSGetLastErrorMsg function returns NULL if there is no error

message.
Rermarks
If ProjectHandle is NULL, this function retrieves the error message
associated with the last call to DSOpenProject or
DSGetProjectList, otherwise it returns the last message associated
with the specified project.
The error text is cleared following a call to DSGetLastErrorMsg.
Note The text retrieved by a call to DSGetLastErrorMsg relates
to the last error generated by the server and not necessarily
the last error reported back to a thread using DataStage API.
Multiple threads using DataStage API must cooperate in
order to obtain the correct error message text.

DataStage Development Kit (Job Control Interfaces) DSGetLinkInfo
DSGetLinkInfo
Retrieves information relating to a specific link of the specified active
stage of a job.
Syntax
int DSGetLinkInfo(
DSJOB JobHandle,
char *StageName,
char *LinkName,
int InfoType,
DSLINKINFO *ReturnInfo
);
Parameters
StageName is a pointer to a null-terminated character string
specifying the name of the active stage to be interrogated.
LinkName is a pointer to a null-terminated character string specifying
the name of a link (input or output) attached to the stage.
InfoType is a key indicating the information to be returned and is one
of the following values:
Value Description
DSJ_LINKLASTERR Last error message reported by the link.
DSJ_LINKNAME Name of the link.
DSJ_LINKROWCOUNT Number of rows that have passed down the link.
DSJ_LINKSQLSTATE SQLSTATE value from last error message.
DSJ_LINKDBMSCODE DBMSCODE value from last error message.
DSJ_LINKDESC Description of the link.
DSJ_LINKSTAGE Name of the stage at the other end of the link.
DSJ_INSTROWCOUNT Null-separated list of rowcounts, one per instance for

parallel jobs.
ReturnInfo is a pointer to a DSJOBINFO data structure where the

requested information is stored. The DSJOBINFO data structure
contains a union with an element for each of the possible return
values from the call to DSGetLinkInfo. For more information, see
"Data Structures" on page 7-62.

DSGetLinkInfo DataStage Development Kit (Job Control Interfaces)
Return Value
Token Description
DSJE_NOT_AVAILABLE There is no instance of the requested information
available.
DSJE_BADHANDLE JobHandle was invalid.
DSJE_BADTYPE InfoType was unrecognized.
DSJE_BADLINK LinkName does not refer to a known link for the stage in
question.
Remarks
This function can be used either before or after a call to DSRunJob.

DataStage Development Kit (Job Control Interfaces) DSGetLogEntry
DSGetLogEntry
Retrieves detailed information about a specific entry in a job log.
Syntax
int DSGetLogEntry(
DSJOB JobHandle,
int EventId,
DSLOGDETAIL *Event
);
Parameters
EventId is the identifier for the event to be retrieved, see "Remarks"
Event is a pointer to a data structure to hold details of the log entry.
Return Values
If the function succeeds, the return value is DSJE_NOERROR and the
event structure contains the details of the requested event.
Token Description
DSJE_SERVER_ERROR Internal error. DataStage server returned invalid data.
DSJE_BADEVENTID Invalid event if for a specified job.
Remarks
Entries in the log file are numbered sequentially starting from 0. The
latest event ID can be obtained through a call to
DSGetNewestLogId. When a log is cleared, there always remains a
single entry saying when the log was cleared.

DSGetNewestLogId DataStage Development Kit (Job Control Interfaces)
DSGetNewestLogId
Obtains the identifier of the newest entry in the jobs log.
Syntax
int DSGetNewestLogId(
DSJOB JobHandle,
int EventType
);
Parameters
EventType is a key specifying the type of log entry whose identifier
you want to retrieve and can be one of the following:
This key… Retrieves this type of log entry…

DSJ_LOGFATAL Fatal
DSJ_LOGOTHER Any other log event type
DSJ_LOGANY Any type of event
Return Values
If the function succeeds, the return value is the positive identifier of
the most recent entry of the requested type in the job log file.
If the function fails, the return value is –1. Use DSGetLastError to
retrieve one of the following error codes:
Token Description

DataStage Development Kit (Job Control Interfaces) DSGetNewestLogId
Remarks
Use this function to determine the ID of the latest entry in a log file
before starting a job run. Once the job has started or finished, it is then
possible to determine which entries have been added by the job run.

DSGetParamInfo DataStage Development Kit (Job Control Interfaces)
DSGetParamInfo
Retrieves information about a particular parameter within a job.
Syntax
int DSGetParamInfo(
DSJOB JobHandle,
char *ParamName,
DSPARAMINFO *ReturnInfo
);
Parameters
ParamName is a pointer to a null-terminated string specifying the
name of the parameter to be interrogated.
ReturnInfo is a pointer to a DSPARAMINFO data structure where the
requested information is stored. For more information, see "Data
Structures" on page 7-62.
Return Values
Token Description
DSJE_SERVER_ERROR Internal error. DataStage Server returned invalid data.
Remarks
Unlike the other information retrieval functions, DSGetParamInfo
returns all the information relating to the specified item in a single
call. The DSPARAMINFO data structure contains all the information
required to request a new parameter value from a user and partially
validate it. See "Data Structures" on page 7-62.
This function can be used either before or after a DSRunJob call has
been issued:
If called after a successful call to DSRunJob, the information
retrieved refers to that run of the job.

DataStage Development Kit (Job Control Interfaces) DSGetParamInfo
If called before a call to DSRunJob, the information retrieved

refers to any previous run of the job, and not to any call to
DSSetParam that may have been issued.

DSGetProjectInfo DataStage Development Kit (Job Control Interfaces)
DSGetProjectInfo
Obtains a list of jobs in a project.
Syntax
int DSGetProjectInfo(
DSPROJECT ProjectHandle,
int InfoType,
DSPROJECTINFO *ReturnInfo
);
Parameters
InfoType is a key indicating the information to be returned.
This key… Retrieves this type of log entry…

DSJ_JOBLIST Lists all jobs within the project.
DSJ_PROJECTNAME Name of current project.
DSJ_HOSTNAME Host name of the server.
ReturnInfo is a pointer to a DSPROJECTINFO data structure where

the requested information is stored.
Return Values
Token Description
DSJE_NOT_AVAILABLE There are no compiled jobs defined within the project.
Remarks
The DSPROJECTINFO data structure contains a union with an
element for each of the possible return values from a call to
DSGetProjectInfo.
Note The returned list contains the names of all jobs known to
the project, whether they can be opened or not.
.

DataStage Development Kit (Job Control Interfaces) DSGetProjectList
DSGetProjectList
Obtains a list of all projects on the host system
Syntax
char* DSGetProjectList(void);
Return Values
If the function succeeds, the return value is a pointer to a series of
null-terminated strings, one for each project on the host system,
ending with a second null character. The following example shows the
buffer contents with <null> representing the terminating null
character:
project1<null>project2<null><null>
If the function fails, the return value is NULL. And the DSGetLast-
Error function retrieves the following error code:
DSJE_SERVER_ERRORUnexpected/unknown server error occurred.
Remarks
This function can be called before any other DataStage API function.
Note DSGetProjectList opens, uses, and closes its own
communications link with the server, so it may take some
time to retrieve the project list.

DSGetReposInfo DataStage Development Kit (Job Control Interfaces)
DSGetReposInfo
Provides searching capabilities for design-time objects.
Syntax
int DSGetReposInfo (
DSPROJECT hProject,
int ObjectType,
int InfoType,
const char *SearchCriteria,
const char *StartingCategory,
int Subcategories,
DSREPOSINFO *ReturnInfo
);
Parameters
hProject is the value returned from DSOpenProject for the project
whose jobs you want to search.
ObjectType must currently be set to DSS_JOBS to indicate that you
want to search for jobs.
InfoType is one or more of the following keys:

DSS_JOB_ALL Lists all jobs
DSS_JOB_SERVER Lists all server jobs
DSS_JOB_PARALLEL Lists all parallel jobs
DSS_JOB_MAINFRAME Lists all mainframe jobs
DSS_JOB_SEQUENCE Lists all job sequences
SearchCriteria is the name to match against. Partial name matching

can be used, with multiple * characters used as wild cards anywhere
in the search string.
StartingCategory is the category to start the search in. If no category
name is supplied, or a NULL or empty string, then the root category is
assumed.
SearchSubcategories can have one of two values: 1 (TRUE) and 0
(FALSE). These define whether the search is to include subcategories.
ReturnInfo is a pointer to a structure containing the required return
information (see "DSREPOSINFO" on page 7-75).

DataStage Development Kit (Job Control Interfaces) DSGetReposInfo
Return Value
On success, DSGetReposInfo returns the number of objects that
have been found.
On failure an error code is returned as follows:
DSJE_BADTYPE ObjectType or InfoType values was not
recognised
DSJE_REPERROR An error occurred while trying to access the
reposotory. Call DSGetLastErrorMsg to get the error message
associated with the error code
DSJE_NO_DATASTAGE The attached project does not appear to
be a valid DataStage project

DSGetReposUsage DataStage Development Kit (Job Control Interfaces)
DSGetReposUsage
Returns a list of objects based on the required relationship.
Syntax
int DSGetReposUsage(
DSPROJECT hProject,
int RelationshipType,
const char *ObjectName,
int Recursive,
DSREPOSUSAGE *ReturnInfo
);
Parameters
whose jobs you want to search.
RelationshipType is one of the following keys:

DSS_JOB_USES_JOB Return a list of jobs that the specified job uses.
DSS_JOB_USEDBY_JOB Return a list of jobs the specified job is used by.
DSS_JOB_HASSOURCE_ Return a list of jobs that use the specified table in as a

DRSTABLE source in a DRS Stage.
DSS_JOB_HASTARGET_ Return a list of jobs that use the specified table as a

DRSTABLE target in a DRS Stage.
DSS_JOB_HASSOURCEO Returns a list of jobs that use the specified table as a

RTARGET_DRSTABLE source or target of a DRS Stage.
ObjectName specifies the job or table, and varies according to which

RelationshipType is specified:
for DSS_JOB_USES_JOB and DSS_JOB_USEDBY_JOB
relationships, the job name (without category qualification)
should be given.
for remaining relationships, the fully qualified table name should
be given.
For the DRS Stage table definition relationships, partial matching
of the table name using * characters as wild cards is allowed.
Multiple wildcard characters can be used
Recursive is used by the DSS_JOB_USES_JOB and
DSS_JOB_USEDBY_JOB relationships. It can have 2 values, 1 ( TRUE)
and 0 ( FALSE). If set to TRUE, then for each job found that uses the

DataStage Development Kit (Job Control Interfaces) DSGetReposUsage
ObjectName, the jobs that that job is used in are found and so on. For
all other relationship types the parameter is ignored.
ReturnInfo is a pointer to a structure containing the returned values
(see"DSREPOSUSAGE" on page 7-76). The order in which jobs appear
in the ReturnInfo structure is defined by the RelationshipType. For the
DSS_JOB_USES_JOB RelationshipType, the jobs will apear in the
order in which they appear in the jobs dependency list. This list is on
the Dependencies tab on the Job Properties dialog.
Return Value
On success, DSGetReposUsage returns the number of objects that
have been found.
On failure an error code is returned as follows:
DSJE_REPERROR An error occurred while trying to access the
repository. Call nDSGetLastErrorMsg to get the error
message associated with the error code.
DSJE_NO_DATASTAGE The attached project does not appear
to be a valid DataStage project.
DSJE_UNKNOWN_JOBNAME When the RelationshipType is
DSS_JOB_USES_JOB or DSS_JOB_USEDBY_JOB the supplied job
name cannot be found in the project.

DSGetStageInfo DataStage Development Kit (Job Control Interfaces)
DSGetStageInfo
Obtains information about a particular stage within a job.
Syntax
int DSGetStageInfo(
DSJOB JobHandle,
char *StageName,
int InfoType,
);
Parameters

DSJ_LINKLIST Null-separated list of names of links in
stage.
DSJ_STAGELASTERR Last error message reported from any link

of the stage.
DSJ_STAGENAME Stage name.
DSJ_STAGETYPE Stage type name.
DSJ_STAGEINROWNUM Primary links input row number.
DSJ_VARLIST Null-separated list of stage variable

names in the stage.
DSJ_STAGESTARTTIMESTAMP Date and time when stage started.
DSJ_STAGEENDTIMESTAMP Date and time when stage finished.
DSJ_STAGEDESC Stage description (from stage properties)
DSJ_STAGEINST Null-separated list of instance ids (parallel

jobs).
DSJ_STAGECPU List of CPU time in seconds.
DSJ_LINKTYPES Null-separated list of link types.
DSJ_STAGEELAPSED Elapsed time in seconds.
DSJ_STAGEPID Null-separated list of process ids.
DSJ_STAGESTATUS Stage status.

DataStage Development Kit (Job Control Interfaces) DSGetStageInfo

DSJ_CUSTINFOLIST Null-separated list of custinfo items.
ReturnInfo is a pointer to a DSSTAGEINFO data structure where the

requested information is stored. See "Data Structures" on page 7-62.
Return Values
Token Description
the stage.
Remarks
This function can be used either before or after a DSRunJob function
has been issued.
The DSSTAGEINFO data structure contains a union with an element
for each of the possible return values from the call to
DSGetStageInfo.

DSGetProjectList DataStage Development Kit (Job Control Interfaces)
DSGetProjectList
Obtains information about variables used in transformer stages.
Syntax
int DSGetVarInfo(
DSJOB JobHandle,
char *StageName,
char *VarName
int InfoType,
);
Parameters
VarName is a pointer to a null-terminated string specifiying the name
of the variable to be interrogated.

DSJ_VARVALUE The value of the specified variable.
DSJ_VARDESC Description of the variable.
Return Values
Token Description
the stage.
DSJE_BADVAR VarName does not refer to a known variable in the job.

DataStage Development Kit (Job Control Interfaces) DSListEnvVars
DSListEnvVars
Obtain a list of environment variables and their values in a specified
project.
Syntax
char *DSListEnvVars(
DSPROJECT hProject
);
Parameter
whose environment variables you want to list.
Return Values
null-terminated strings, one for each environment variable, ending
with a second null character. Each string is of the format
EnvVarName=EnvVarValue.
If the function fails, the return value is NULL and the DSGetLastError
function can be used to retrieve an error code as follows:
definitions
values
DSJE.ISPARALLELLICENCED failed to determine if Enterprise
Edition installed
Remarks
hProject.
Environment variables in the parallel category will only be listed if
Enterprise Edition is installed.

DSListProjectProperties DataStage Development Kit (Job Control Interfaces)
DSListProjectProperties
Obtain a list of the values of project properties for specified project.
Properties supported are:
Whether generated OSH is visible in parallel jobs.
Whether runtime column propagation is enabled in parallel jobs.
The base directory name for parallel jobs.
Advanced runtime options for parallel jobs.
Custom deployment commands for parallel jobs.
Deployment job template directory.
Whether job administration is enabled in the DataStage Director
or not.
Syntax
char *DSListProjectProperties(
DSPROJECT hProject
);
Parameter
whose properties you want to list.
Return Values
null-terminated strings, one for each variable, ending with a second
null character. Each string is of the format
PropertyName=PropertyValue where PropertyName will be one of the
following:
This key… Indicates this property…

DSA_OSHVISIBLEFLAG Generated OSH is visible in
parallel jobs. Enterprise Edition
only.
DSA_PRJ_RTCP_ENABLED Runtime column propagation is

enabled in parallel jobs. Enterprise
Edition only.
DSA_PRJ_PX_ADVANCED_RUNTIME_OPTS Specifies advanced runtime

properties for parallel jobs.
Enterprise Edition only.

DataStage Development Kit (Job Control Interfaces) DSListProjectProperties

DSA_PRJ_PX_BASEDIR Specifies the base directory for
parallel jobs. Enterprise Edition
only.
DSA_PRJ_PX_DEPLOY_JOBDIR_TEMPLATE Specifies the deployment directory

template for parallel jobs.
DSA_PRJ_PX_DEPLOY_CUSTOM_ACTION Specifies custom deployment

commands for parallel jobs. Value
is the commands. Enterprise
Edition only.
DSA_PRJ_JOBADMIN_ENABLED Job administration commands are

enabled in the DataStage Director
for jobs in this project.
DSA_PRJ_PX_DEPLOY_GENERATE_XML Generation of XML reports is

enabled for Parallel job
deployment packages.
These tokens are defined in dsapi.h (see "The dsapi.h Header File" on
page 7-2).
If the function fails, the return value is NULL and the DSGetLastError
function can be used to retrieve one of the following error code:
DSJE_READPROJPROPERTY failed to read property
DSJE_ISPARALLELLICENCED failed to determine if Enterprise
Edition installed
DSJE_OSHVISIBLEFLAG failed to get value for OSHVisible
Remarks
hProject.
If Enterprise Edition is not installed, only the setting of the
DSA_PRJ_JOBADMIN_ENABLED will be returned.
DSListProjectProperties

DSLockJob DataStage Development Kit (Job Control Interfaces)
DSLockJob
Locks a job. This function must be called before setting a job’s run
parameters or starting a job run.
Syntax
int DSLockJob(
DSJOB JobHandle
);
Parameter
Return Values
Token Description
Remarks
Locking a job prevents any other process from modifying the job
details or status. This function must be called before any call of
DSSetJobLimit, DSSetParam, or DSRunJob.
If you try to lock a job you already have locked, the call succeeds. If
you have the same job open on several DataStage API handles,
locking the job on one handle locks the job on all the handles.

DataStage Development Kit (Job Control Interfaces) DSLogEvent
DSLogEvent
Adds a new entry to a job log file.
Syntax
int DSLogEvent(
DSJOB JobHandle,
int EventType,
char *Reserved,
char *Message
);
Parameters
EventType is one of the following keys specifying the type of event to
be logged:
This key… Specifies this type of event…

Reserved is reserved for future use, and should be specified as null.

Message points to a null-terminated character string specifying the
text of the message to be logged.
Return Values
Token Description
Remarks
Messages that contain more that one line of text should contain a
newline character (\n) to indicate the end of a line.

DSMakeJobReport DataStage Development Kit (Job Control Interfaces)
DSMakeJobReport
Generates a report describing the complete status of a valid attached
job.
Syntax
int DSMakeJobReport(
DSJOB JobHandle,
int ReportType,
char *LineSeparator,
DSREPORTINFO *ReturnInfo
);
Parameters
ReportType is one of the following values specifying the type of report
to be generated:
This value… Specifies this type of report…

0 Basic, text string containing start/end time, time elapsed and
status of job.
1 Stage/link detail. As basic report, but also contains information

about individual stages and links within the job.
2 Text string containing full XML report.
By default the generated XML will not contain a <?xml-stylesheet?>

processing instruction. If a stylesheet is required, specify a
RetportLevel of 2 and append the name of the required stylesheet
URL, i.e., 2:styleSheetURL. This inserts a processing instruction into
the generated XML of the form:
<?xml-stylesheet type=text/xsl” href=”styleSheetURL”?>
LineSeparator points to a null-terminated character string specifying

the line separator in the report. Special values recognised are:
"CRLF" => CHAR(13):CHAR(10)
"LF" => CHAR(10)
"CR" => CHAR(13)
The default is CRLF if on Windows, else LF.
Return Values

DataStage Development Kit (Job Control Interfaces) DSOpenJob
DSOpenJob
Opens a job. This function must be called before any other function
that manipulates the job.
Syntax
DSJOB DSOpenJob(
DSPROJECT ProjectHandle,
char *JobName
);
Parameters
JobName is a pointer to a null-terminated string that specifies the
name of the job that is to be opened. This may be in either of the
following formats:
job Finds the latest version of the job.

job%Reln.n.n Finds a particular release of the job on a
development system.
Return Values
If the function succeeds, the return value is a handle to the job.
If the function fails, the return value is NULL. Use DSGetLastError to
retrieve one of the following:
Token Description
DSJE_OPENFAIL Server failed to open job.
DSJE_NO_MEMORY Memory allocation failure.
Remarks
The DSOpenJob function must be used to return a job handle before
a job can be addressed by any of the DataStage API functions. You can
gain exclusive access to the job by locking it with DSLockJob.
The same job may be opened more than once and each call to
DSOpenJob will return a unique job handle. Each handle must be
separately closed.

DSOpenProject DataStage Development Kit (Job Control Interfaces)
DSOpenProject
Opens a project. It must be called before any other DataStage API
function, except DSGetProjectList or DSGetLastError.
Syntax
DSPROJECT DSOpenProject(
char *ProjectName
);
Parameter
ProjectName is a pointer to a null-terminated string that specifies the
name of the project to open.
Return Values
If the function succeeds, the return value is a handle to the project.
If the function fails, the return value is NULL. Use DSGetLastError to
retrieve one of the following:
Token Description
DSJE_BAD_VERSION The DataStage server is an older version than
the DataStage API.
DSJE_INCOMPATIBLE_SERVER The DataStage Server is either older or newer

than that supported by this version of
DataStage API.
DSJE_SERVER_ERROR Internal error. DataStage Server returned

invalid data.
DSJE_BADPROJECT Invalid project name.
DSJE_NO_DATASTAGE DataStage is not correctly installed on the

server system.
Remarks
The DSGetProjectList function can return the name of a project that
does not contain valid DataStage jobs, but this is detected when
DSOpenProject is called. A process can only have one project open
at a time.

DataStage Development Kit (Job Control Interfaces) DSRunJob
DSRunJob
Starts a job run.
Syntax
int DSRunJob(
DSJOB JobHandle,
int RunMode
);
Parameters
JobHandle is a value returned from DSOpenJob.
RunMode is a key determining the run mode and should be one of the
following values:
This key… Indicates this action…

DSJ_RUNNORMAL Start a job run.
DSJ_RUNRESET Reset the job.
DSJ_RUNVALIDATE Validate the job.
Return Values
Token Description
DSJE_BADSTATE Job is not in the right state (must be
compiled and not running).
DSJE_BADTYPE RunMode is not recognized.
DSJE_SERVER_ERROR Internal error. DataStage Server returned
invalid data.
Remarks
The job specified by JobHandle must be locked, using DSLockJob,
before the DSRunJob function is called.

DSRunJob DataStage Development Kit (Job Control Interfaces)
If no limits were set by calling DSSetJobLimit, the default limits are

used.

DataStage Development Kit (Job Control Interfaces) DSSetEnvVar
DSSetEnvVar
Set the value for an environment variable in a specified project.
Syntax
int DSSetEnvVar(
DSPROJECT hProject,
char *EnvVarName,
char *Value
);
Parameters
hProject is the value returned from DSOpenProject.
EnvVarName is the name of the environment variable.
Value is the value to set the environment variable to.
Return Values
definitions
values
DSJE_BADENVVAR environment variable does not exist
values
DSJE_ENCODEFAILED failed to encode an encrypted value
DSJE_BADBOOLEANVALUE invalid value given for a boolean
environment variable
DSJE_BADNUMERICVALUE invalid value given for an integer
environment variable
DSJE_BADLISTVALUE invalid value given for an
environment variable with a fixed list of values
DSJE_PXNOTINSTALLED Environment variable is specific to
Enterprise Edition which is not installed
Edition installed

DSSetEnvVar DataStage Development Kit (Job Control Interfaces)
Remarks
You can only set values for environment variables in the parallel
category if Enterprise Edition is installed.
If setting a list type environment variable (for example,
APT_EXECUTION _MODE) , then you should set it to one of the
permissable internal values, rather than one of the list members as
they are shown in the DataStage Administrator client. For example, if
you wanted to set APT_EXECUTION_MODE so that parallel jobs
executed in one process mode, you would set the environment
variable value to ‘ONE_PROCESS’, not ‘One process’ as offered in the
Administrator client. Internal values are given in "Environment
Variables," in the Parallel Job Advanced Developer’s Guide.
If you are setting a boolean type environment variable, set teh value
to 1 for TRUE and 0 for FALSE.

DataStage Development Kit (Job Control Interfaces) DSSetGenerateOpMetaData
DSSetGenerateOpMetaData
Use this to specify whether the job generates operational meta data or
not. This overrides the default setting for the project. In order to
generate operational meta data the Process MetaBroker must be
installed on your DataStage machine.
Syntax
int DSSetGenerateOpMetaData (
JobHandle,
value
);
Parameters
value is TRUE (1) to generate operational meta data, FALSE (0) to not
generate operational meta data.
Return Values
Token Description
DSJE_BADTYPE value is not recognized.

DSSetJobLimit DataStage Development Kit (Job Control Interfaces)
DSSetJobLimit
Sets row or warning limits for a job.
Syntax
int DSSetJobLimit(
DSJOB JobHandle,
int LimitType,
int LimitValue
);
Parameters
LimitType is one of the following keys specifying the type of limit:
This key… Specifies this type of limit…

DSJ_LIMITWARN Job to be stopped after LimitValue warning events.
DSJ_LIMITROWS Stages to be limited to LimitValue rows.
LimitValue is the value to set the limit to.
Return Values
Token Description
DSJE_BADSTATE Job is not in the right state (compiled, not running).
DSJE_BADTYPE LimitType is not the name of a known limiting condition.
DSJE_BADVALUE LimitValue is not appropriate for the limiting condition

type.
Remarks
before the DSSetJobLimit function is called.

DataStage Development Kit (Job Control Interfaces) DSSetJobLimit
Any job limits that are not set explicitly before a run will use the
default values. Make two calls to DSSetJobLimit in order to set both
types of limit.
Set the value to 0 to indicate that there should be no limit for the job.

DSSetParam DataStage Development Kit (Job Control Interfaces)
DSSetParam
Sets job parameter values before running a job. Any parameter that is
not explicitly set uses the default value.
Syntax
int DSSetParam(
DSJOB JobHandle,
char *ParamName,
DSPARAM *Param
);
Parameters
ParamName is a pointer to a null-terminated string that specifies the
name of the parameter to set.
Param is a pointer to a structure that specifies the name, type, and
value of the parameter to set.
Note The type specified in Param need not match the type
specified for the parameter in the job definition, but it must
be possible to convert it. For example, if the job defines the
parameter as a string, it can be set by specifying it as an
integer. However, it will cause an error with unpredictable
results if the parameter is defined in the job as an integer
and a nonnumeric string is passed by DSSetParam.
Return Values
Token Description
DSJE_BADSTATE Job is not in the right state (compiled, not running).
DSJE_BADPARAM Param does not reference a known parameter of the job.
DSJE_BADTYPE Param does not specify a valid parameter type.
DSJE_BADVALUE Param does not specify a value that is appropriate for the
parameter type as specified in the job definition.

DataStage Development Kit (Job Control Interfaces) DSSetParam
Remarks
before the DSSetParam function is called.

DSSetProjectProperty DataStage Development Kit (Job Control Interfaces)
DSSetProjectProperty
Sets the value of a property in a specified project. The user who runs
the code containing this function must be a DataStage Administrator.
Syntax
int DSSetProjectProperty(
DSPROJECT hProject,
char *Property,
char *Value
);
Parameters
Propertyis the name of the property to set. The following properties
are supported:

DSA_OSHVISIBLEFLAG Generated OSH is visible in
parallel jobs, Value is 0 for false or
1 for true. Enterprise Edition only.
DSA_PRJ_RTCP_ENABLED Runtime column propagation is

enabled in parallel jobs, Value is 0
for false or 1 for true. Enterprise
Edition only.
DSA_PRJ_PX_ADVANCED_RUNTIME_OPTS Specifies advanced runtime

properties for parallel jobs, Value is
the advanced properties to set.
DSA_PRJ_PX_BASEDIR Specifies the base directory for

parallel jobs. Value is the base
directory. Enterprise Edition only.
DSA_PRJ_PX_DEPLOY_JOBDIR_TEMPLATE Specifies the deployment directory

template for parallel jobs. Value is
the deployment directory
template. Enterprise Edition only.
DSA_PRJ_PX_DEPLOY_CUSTOM_ACTION Specifies custom deployment

commands for parallel jobs. Value
is the commands. Enterprise
Edition only
DSA_PRJ_JOBADMIN_ENABLED Job administration commands are

enabled in the DataStage Director
for jobs in this project. Value is 0
for false or 1 for true.

DataStage Development Kit (Job Control Interfaces) DSSetProjectProperty

DSA_PRJ_PX_DEPLOY_GENERATE_XML Generation of XML reports is
enabled for Parallel job
deployment packages. Value is 0
for false or 1 for true.
Return Values
DSJE_ISADMINFAILED failed to determine whether user is
an administrator
DSJE_READPROJPROPERTY failed to read property
DSJE_WRITEPROJPROPERTY failed to write property
DSJE_PROPNOTSUPPORTED property not supported
DSJE_BADPROPERTY unknown property name
DSJE_BADPROPVALUE invalid value for this property
DSJE_PXNOTINSTALLED Enterprise Edition not installed
Edition installed
DSJE_OSHVISIBLEFLAG failed to set value for OSHVisible
Remarks
hProject.

DSSetServerParams DataStage Development Kit (Job Control Interfaces)
DSSetServerParams
Sets the logon parameters to use for opening a project or retrieving a
project list.
Syntax
void DSSetServerParams(
char *ServerName,
char *UserName,
char *Password
);
Parameters
ServerName is a pointer to either a null-terminated character string
specifying the name of the server to connect to, or NULL.
UserName is a pointer to either a null-terminated character string
specifying the user name to use for the server session, or NULL.
Password is a pointer to either a null-terminated character string
specifying the password for the user specified in UserName, or NULL.
Return Values
This function has no return value.
Remarks
By default, DSOpenProject and DSGetProjectList attempt to
connect to a DataStage Server on the same computer as the client
process, then create a server process that runs with the same user
identification and access rights as the client process.
DSSetServerParams overrides this behavior and allows you to
specify a different server, user name, and password.
Calls to DSSetServerParams are not cumulative. All parameter
values, including NULL pointers, are used to set the parameters to be
used on the subsequent DSOpenProject or DSGetProjectList call.

DataStage Development Kit (Job Control Interfaces) DSStopJob
DSStopJob
Aborts a running job.
Syntax
int DSStopJob(
DSJOB JobHandle
);
Parameter
Return Values
DSJE_BADHANDLEInvalid JobHandle.
Remarks
The DSStopJob function should be used only after a DSRunJob
function has been issued. The stop request is sent regardless of the
job’s current status. To ascertain if the job has stopped, use the
DSWaitForJob function or the DSJobStatus macro.

DSUnlockJob DataStage Development Kit (Job Control Interfaces)
DSUnlockJob
Unlocks a job, preventing any further manipulation of the job’s run
state and freeing it for other processes to use.
Syntax
int DSUnlockJob(
DSJOB JobHandle
);
Parameter
Return Values
If the function succeeds, the return value is DSJ_NOERROR.
Remarks
The DSUnlockJob function returns immediately without waiting for
the job to finish. Attempting to unlock a job that is not locked does not
cause an error. If you have the same job open on several handles,
unlocking the job on one handle unlocks it on all handles.

DataStage Development Kit (Job Control Interfaces) DSWaitForJob
DSWaitForJob
Waits to the completion of a job run.
Syntax
int DSWaitForJob(
DSJOB JobHandle
);
Parameter
Return Values
Token Description
DSJE_WRONGJOB Job for this JobHandle was not started from a call to
DSRunJob by the current process.
DSJE_TIMEOUT Job appears not to have started after waiting for a

reasonable length of time. (About 30 minutes.)
Remarks
This function is only valid if the current job has issued a DSRunJob
call on the given JobHandle. It returns if the job was started since the
last DSRunJob, and has since finished. The finishing status can be
found by calling DSGetJobInfo.

Data Structures DataStage Development Kit (Job Control Interfaces)
Data Structures
The DataStage API uses the data structures described in this section
to hold data passed to, or returned from, functions. (See"Data
Structures, Result Data, and Threads" on page 7-2). The data
structures are summarized below, with full descriptions in the
following sections:
This data Holds this type of data… And is used by

structure… this function…
DSCUSTINFO Custinfo items from certain types of DSGetCustinfo
parallel stage
DSJOBINFO Information about a DataStage job DSGetJobInfo
DSLINKINFO Information about a link to or from an DSGetLinkInfo

active stage in a job, that is, a stage
that is not a data source or
destination
DSLOGDETAIL Full details of an entry in a job log file DSGetLogEntry
DSLOGEVENT Details of an entry in a job log file DSLogEvent,

DSFindFirstLogEntry,
DSFindNextLogEntry
DSPARAM The type and value of a job parameter DSSetParam
DSPARAMINFO Further information about a job DSGetParamInfo

parameter, such as its default value
and a description
DSPROJECTINFO A list of jobs in the project DSGetProjectInfo
DSREPOSINGFO A list of design time jobs DSGetReposInfo
DSREPOSUSGE A list of design time jobs satisfying a DSGetReposUsage

relationship
DSSTAGEINFO Information about a stage in a job DSGetStageInfo
DSVARINFO Information about stage variables in DSGetVarInfo

transformer stages

DataStage Development Kit (Job Control Interfaces) DSCUSTINFO
DSCUSTINFO
The DSCUSTINFO structure represents various information values
about a link to or from an active stage within a DataStage job.
Syntax
typedef struct _DSCUSTINFO {
int infoType:/
union {
char *custinfoValue;
char *custinfoDesc;
} info;
} DSCUSTINFO;
Members
infoType is a key indicating the type of information and is one of the
following values:
This key… Indicates this information…

DSJ_CUSTINFOVALUE The value of the specified custinfo item.
DSJ_CUSTINFODESC The description of the specified custinfo item.

DSJOBINFO DataStage Development Kit (Job Control Interfaces)
DSJOBINFO
The DSJOBINFO structure represents information values about a
DataStage job.
Syntax
typedef struct _DSJOBINFO {
int infoType;
union {
int jobStatus;
char *jobController;
time_t jobStartTime;
int jobWaveNumber;
char *userStatus;
char *paramList;
char *stageList;
char *jobname;
int jobcontrol;
int jobPid;
time_t jobLastTime;
char *jobInvocations;
int jobInterimStatus;
char *jobInvocationid;
char *jobDesc;
char *stageList2;
char *jobElapsed;
char *jobFullDesc;
int jobDMIService;
int jobMultiInvokable;
} info;
} DSJOBINFO;
Members
infoType is one of the following keys indicating the type of
information:

DSJ_JOBSTATUS The current status of the job.
DSJ_JOBNAME Name of job referenced by JobHandle
DSJ_JOBCONTROLLER The name of the controlling job.
DSJ_JOBSTARTTIMESTAMP The date and time when the job started.
DSJ_JOBWAVENO Wave number of the current (or last) job run.
DSJ_PARAMLIST A list of the names of the job’s parameters.

Separated by nulls.

DataStage Development Kit (Job Control Interfaces) DSJOBINFO

DSJ_STAGELIST A list of active stages in the job. Separated by
nulls.
DSJ_USERSTATUS The status reported by the job itself as defined in

the job’s design.
DSJ_JOBCONTROL Whether a stop request has been issued for the

job.
DSJ_JOBPID Process id of DSD.RUN process.
DSJ_JOBLASTTIMESTAMP The date and time on the server when the job last
finished.
DSJ_JOBINVOCATIONS List of job invocation ids. Separated by nulls.
DSJ_JOBINTERIMSTATUS Current Interim status of the job.
DSJ_JOBINVOVATIONID Invocation name of the job referenced.
DSJ_JOBDESC A description of the job.
DSJ_STAGELIST2 A list of passive stages in the job. Separated by

nulls.
DSJ_JOBELAPSED The elapsed time of the job in seconds.
DSJ_JOBFULLDESSC The Full Description specified in the Job

DSJ_JOBDMISERVICE Set to true if this is a web service job.
DSJ_JOBMULTIINVOKABLE Set to true if this job supports multiple

invocations.
jobStatus is returned when infoType is set to DSJ_JOBSTATUS. Its

value is one of the following keys:
This key… Indicates this status…

DSJS_RUNNING Job running.
DSJS_RUNOK Job finished a normal run with no warnings.
DSJS_RUNWARN Job finished a normal run with warnings.
DSJS_RUNFAILED Job finished a normal run with a fatal error.
DSJS_VALOK Job finished a validation run with no warnings.
DSJS_VALWARN Job finished a validation run with warnings.
DSJS_VALFAILED Job failed a validation run.
DSJS_RESET Job finished a reset run.
DSJS_CRASHED Job was stopped by some indeterminate action.

DSJOBINFO DataStage Development Kit (Job Control Interfaces)
This key… Indicates this status…

DSJS_STOPPED Job was stopped by operator intervention (can’t tell
run type).
DSJS_NOTRUNNABLE Job has not been compiled.
DSJS_NOTRUNNING Any other status. Job was stopped by operator

intervention (can’t tell run type).
jobController is the name of the job controlling the job reference and
is returned when infoType is set to DSJ_JOBCONTROLLER. Note that
this may be several job names, separated by periods, if the job is
controlled by a job which is itself controlled, and so on.
jobStartTime is the date and time when the last or current job run
started and is returned when infoType is set to
DSJ_JOBSTARTTIMESTAMP.
jobWaveNumber is the wave number of the last or current job run and
is returned when infoType is set to DSJ_JOBWAVENO.
userStatus is the value, if any, set by the job as its user defined status,
and is returned when infoType is set to DSJ_USERSTATUS.
paramList is a pointer to a buffer that contains a series of null-
terminated strings, one for each job parameter name, that ends with a
second null character. It is returned when infoType is set to
DSJ_PARAMLIST. The following example shows the buffer contents
with <null> representing the terminating null character:
first<null>second<null><null>
stageList is a pointer to a buffer that contains a series of null-

terminated strings, one for each stage in the job, that ends with a
second null character. It is returned when infoType is set to
DSJ_STAGELIST. The following example shows the buffer contents
with <null> representing the terminating null character:

DataStage Development Kit (Job Control Interfaces) DSLINKINFO
DSLINKINFO
The DSLINKINFO structure represents various information values
Syntax
typedef struct _DSLINKINFO {
int infoType:/
union {
DSLOGDETAIL lastError;
int rowCount;
char *linkName;
char *linkSQLState;
char *linkDBMSCode;
char *linkDesc;
char *linkedStage;
char *rowCountList;
} info;
} DSLINKINFO;
Members
following values:

DSJ_LINKLASTERR The last error message reported from a link.
DSJ_LINKNAME Actual name of link.
DSJ_LINKROWCOUNT The number of rows that have been passed down a link.
DSJ_LINKSQLSTATE SQLSTATE value from last error message.
DSJ_LINKDBMSCODE DBMSCODE value from last error message.
DSJ_LINKDESC Description of the link.
DSJ_LINKSTAGE Name of the stage at the other end of the link.
DSJ_INSTROWCOUNT Comma-separated list of rowcounts, one per

instance (parallel jobs)
lastError is a data structure containing the error log entry for the last
error message reported from a link and is returned when infoType is
set to DSJ_LINKLASTERR.
rowCount is the number of rows that have been passed down a link so
far and is returned when infoType is set to DSJ_LINKROWCOUNT.

DSLOGDETAIL DataStage Development Kit (Job Control Interfaces)
DSLOGDETAIL
The DSLOGDETAIL structure represents detailed information for a
single entry from a job log file.
Syntax
typedef struct _DSLOGDETAIL {
int eventId;
time_t timestamp;
int type;
char *reserved;
char *fullMessage;
} DSLOGDETAIL;
Members
eventId is a a number, 0 or greater, that uniquely identifies the log
entry for the job.
timestamp is the date and time at which the entry was added to the
job log file.
type is a key indicting the type of the event, and is one of the following
values:
This key… Indicates this type of log entry…

DSJ_LOGFATAL Fatal error
DSJ_LOGOTHER Any other type of log entry
reserved is reserved for future use with a later release of DataStage.

fullMessage is the full description of the log entry.

DataStage Development Kit (Job Control Interfaces) DSLOGEVENT
DSLOGEVENT
The DSLOGEVENT structure represents the summary information
for a single entry from a job’s event log.
Syntax
typedef struct _DSLOGEVENT {
int eventId;
time_t timestamp;
int type;
char *message;
} DSLOGEVENT;
Members
eventId is a a number, 0 or greater, that uniquely identifies the log
entry for the job.
timestamp is the date and time at which the entry was added to the
job log file.
type is a key indicating the type of the event, and is one of the
following values:
This key… Indicates this type of log entry…

DSJ_LOGFATAL Fatal error
DSJ_LOGOTHER Any other type of log entry
message is the first line of the description of the log entry.

DSPARAM DataStage Development Kit (Job Control Interfaces)
DSPARAM
The DSPARAM structure represents information about the type and
value of a DataStage job parameter.
Syntax
typedef struct _DSPARAM {
int paramType;
union {
char *pString;
char *pEncrypt;
int pInt;
float pFloat;
char *pPath;
char *pListValue;
char *pDate;
char *pTime;
} paramValue;
} DSPARAM;
Members
paramType is a key specifying the type of the job parameter. Possible
values are as follows:
This key… Indicates this type of parameter…

DSJ_PARAMTYPE_STRING A character string.
DSJ_PARAMTYPE_ENCRYPTED An encrypted character string (for example, a

password).
DSJ_PARAMTYPE_INTEGER An integer.
DSJ_PARAMTYPE_FLOAT A floating-point number.
DSJ_PARAMTYPE_PATHNAME A file system pathname.
DDSJ_PARAMTYPE_LIST A character string specifying one of the values

from an enumerated list.
DDSJ_PARAMTYPE_DATE A date in the format YYYY-MM-DD.
DSJ_PARAMTYPE_TIME A time in the format HH:MM:SS.
pString is a null-terminated character string that is returned when

paramType is set to DSJ_PARAMTYPE_STRING.
pEncrypt is a null-terminated character string that is returned when
paramType is set to DSJ_PARAMTYPE_ENCRYPTED. The string
should be in plain text form when passed to or from DataStage API
where it is encrypted. The application using the DataStage API should

DataStage Development Kit (Job Control Interfaces) DSPARAM
present this type of parameter in a suitable display format, for

example, an asterisk for each character of the string rather than the
character itself.
pInt is an integer and is returned when paramType is set to
DSJ_PARAMTYPE_INTEGER.
pFloat is a floating-point number and is returned when paramType is
set to DSJ_PARAMTYPE_FLOAT.
pPath is a null-terminated character string specifying a file system
pathname and is returned when paramType is set to
DSJ_PARAMTYPE_PATHNAME.
Note This parameter does not need to specify a valid pathname
on the server. Interpretation and validation of the pathname
is performed by the job.
pListValue is a null-terminated character string specifying one of the
possible values from an enumerated list and is returned when
paramType is set to DDSJ_PARAMTYPE_LIST.
pDate is a null-terminated character string specifying a date in the
format YYYY-MM-DD and is returned when paramType is set to
DSJ_PARAMTYPE_DATE.
pTime is a null-terminated character string specifying a time in the
format HH:MM:SS and is returned when paramType is set to
DSJ_PARAMTYPE_TIME.

DSPARAMINFO DataStage Development Kit (Job Control Interfaces)
DSPARAMINFO
The DSPARAMINFO structure represents information values about a
parameter of a DataStage job.
Syntax
typedef struct _DSPARAMINFO {
DSPARAM defaultValue;
char *helpText;
char *paramPrompt;
int paramType;
DSPARAM desDefaultValue;
char *listValues;
char *desListValues;
int promptAtRun;
} DSPARAMINFO;
Members
defaultValue is the default value, if any, for the parameter.
helpText is a description, if any, for the parameter.
paramPrompt is the prompt, if any, for the parameter.
paramType is a key specifying the type of the job parameter. Possible
values are as follows:
This key… Indicates this type of parameter…

DSJ_PARAMTYPE_STRING A character string.
DSJ_PARAMTYPE_ENCRYPTED An encrypted character string (for example, a

password).
DSJ_PARAMTYPE_INTEGER An integer.
DSJ_PARAMTYPE_FLOAT A floating-point number.
DSJ_PARAMTYPE_PATHNAME A file system pathname.
DDSJ_PARAMTYPE_LIST A character string specifying one of the values

from an enumerated list.
DDSJ_PARAMTYPE_DATE A date in the format YYYY-MM-DD.
DSJ_PARAMTYPE_TIME A time in the format HH:MM:SS.
desDefaultValue is the default value set for the parameter by the job’s
designer.

DataStage Development Kit (Job Control Interfaces) DSPARAMINFO
Note Default values can be changed by the DataStage

administrator, so a value may not be the current value for
the job.
listValues is a pointer to a buffer that receives a series of null-
terminated strings, one for each valid string that can be used as the
parameter value, ending with a second null character as shown in the
following example (<null> represents the terminating null character):
desListValues is a pointer to a buffer containing the default list of

values set for the parameter by the job’s designer. The buffer contains
a series of null-terminated strings, one for each valid string that can be
used as the parameter value, that ends with a second null character.
The following example shows the buffer contents with <null>
representing the terminating null character:
Note Default values can be changed by the DataStage

administrator, so a value may not be the current value for
the job.
promptAtRun is either 0 (False) or 1 (True). 1 indicates that the
operator is prompted for a value for this parameter whenever the job
is run; 0 indicates that there is no prompting.

DSPROJECTINFO DataStage Development Kit (Job Control Interfaces)
DSPROJECTINFO
The DSPROJECTINFO structure represents information values for a
DataStage project.
Syntax
typedef struct _DSPROJECTINFO {
int infoType;
union {
char *jobList;
} info;
} DSPROJECTINFO;
Members
infoType is a key value indicating the type of information to retrieve.
Possible values are as follows.

DSJ_JOBLIST List of jobs in project.
DSJ_PROJECTNAME Name of current project.
DSJ_HOSTNAME Host name of the server.
jobList is a pointer to a buffer that contains a series of null-terminated

strings, one for each job in the project, and ending with a second null
character, as shown in the following example (<null> represents the
terminating null character):

DataStage Development Kit (Job Control Interfaces) DSREPOSINFO
DSREPOSINFO
The DSREPOSINFO structure gives information about design-time
objects that have been searched for.
Syntax
struct _DSREPOSJOBINFO;
typedef struct _DSREPOSJOBINFO DSREPOSJOBINFO;
struct _DSREPOSJOBINFO
{
char* jobname; /* Includes category */
int jobtype; /* InfoType constant */
DSREPOSJOBINFO* nextjob; /* ptr next job or NULL */
};
typedef struct _DSREPOSINFO
{
int infoType;
union
{
DSREPOSJOBINFO* jobs; /*linkedlist of found jobs */
} info;
} DSREPOSINFO;
Members

DSS_JOBS List of jobs.
jobs is a pointer to a structure linked to another structure, or

terminated with a null. There is one structure for each job returned.
Each structure contains the job name (including category) and the job
type as follows:

DSS_JOB_SERVER Server job
DSS_JOB_PARALLEL Parallel job
DSS_JOB_MAINFRAME Mainframe job
DSS_JOB_SEQUENCE Job sequence

DSREPOSUSAGE DataStage Development Kit (Job Control Interfaces)
DSREPOSUSAGE
The DSREPOSUSAGE structure gives information about objects
meeting a specified relationship.
DSREPOSUSAGE
Syntax
struct _DSREPOSUSAGEJOB;
typedef struct _DSREPOSUSAGEJOB DSREPOSUSAGEJOB;
struct _DSREPOSUSAGEJOB
{
char *jobname; /* Job and cat name */
int jobtype; /* type of job */
DSREPOSUSAGEJOB *nextjob; /* next sibling job */
DSREPOSUSAGEJOB *childjob;
};
typedef struct _DSREPOSUSAGE
{
int infoType;
union
{
DSREPOSUSAGEJOB *jobs; /*linkedlist of jobs*/
} info
} DSREPOSUSAGE;
Members

DSS_JOBS List of jobs.
jobs is a pointer to a structure linked to another structure, or

terminated with a null. There is one structure for each job returned.
Each structure contains the job name (including category) and the job
type as follows:

DSS_JOB_SERVER Server job
DSS_JOB_PARALLEL Parallel job
DSS_JOB_MAINFRAME Mainframe job
DSS_JOB_SEQUENCE Job sequence

DataStage Development Kit (Job Control Interfaces) DSSTAGEINFO
DSSTAGEINFO
The DSSTAGEINFO structure represents various information values
about an active stage within a DataStage job.
Syntax
typedef struct _DSSTAGEINFO {
int infoType;
union {
DSLOGDETAIL lastError;
char *typeName;
int inRowNum;
char *linkList;
char *stagename;
char *varlist;
char *stageStartTime;
char *stageEndTime;
char *linkTypes;
char *stageDesc;
char *instList;
char *cpuList;
time_t stageElapsed;
char *pidList;
int stageStatus;
char *custInfoList
} info;
} DSSTAGEINFO;
Members
infoType is a key indicating the information to be returned and is one
of the following:

DSJ_LINKLIST Null-separated list of link names.
DSJ_STAGELASTERR The last error message generated from any

link in the stage.
DSJ_STAGENAME Name of stage.
DSJ_STAGETYPE The stage type name, for example,

Transformer or BeforeJob.
DSJ_STAGEINROWNUM The primary link’s input row number.
DSJ_VARLIST List of stage variable names.
DSJ_STAGESTARTTIME-STAMP Date and time when stage started.
DSJ_STAGEENDTIME-STAMP Date and time when stage finished.
DSJ_STAGEDESC Stage description (from stage properties)

DSSTAGEINFO DataStage Development Kit (Job Control Interfaces)

DSJ_STAGEINST Null-separated list of instance ids (parallel
jobs).
DSJ_STAGECPU Null-separated list of CPU time in seconds
DSJ_LINKTYPES Null-separated list of link types.
DSJ_STAGEELAPSED Elapsed time in seconds.
DSJ_STAGEPID Null-separated list of process ids.
DSJ_STAGESTATUS Stage status.
DSJ_CUSTINFOLIST Null-separated list of custinfo item names.
lastError is a data structure containing the error message for the last
error (if any) reported from any link of the stage. It is returned when
infoType is set to DSJ_STAGELASTERR.
typeName is the stage type name and is returned when infoType is set
to DSJ_STAGETYPE.
inRowNum is the primary link’s input row number and is returned
when infoType is set to DSJ_STAGEINROWNUM.
linkList is a pointer to a buffer that contains a series of null-terminated
strings, one for each link in the stage, ending with a second null
character, as shown in the following example (<null> represents the
terminating null character):

DataStage Development Kit (Job Control Interfaces) DSLINKINFO
DSLINKINFO
The DSLINKINFO structure represents various information values
Syntax
typedef struct _DSVARINFO {
int infoType:/
union {
char *varValue;
char *varDesc;
} info;
} DSVARINFO;
Members
following values:

DSJ_VARVALUE The value of the specified variable.
DSJ_VARDESC The description of the specified

variable.

Error Codes DataStage Development Kit (Job Control Interfaces)
Error Codes
The following table lists DataStage API error codes in alphabetical
order:
Table 7-1 API Error Codes

Error Token Code Description
DSJE_ACCOUNTPATHFAILED -140 Failed to get account path.
DSJE_ADDPROJECTBLOCKED -134 Another user is adding a project.
DSJE_ADDPROJECTFAILED -135 Failed to add project.
DSJEBADBOOLEANVALUE -118 Invalid value given for a boolean

DSJE_BADENVVAR -116 Environment variable does not

exist.
DSJE_BADENVVARNAME -108 Invalid environment variable

name.
DSJE_BADENVVARTYPE -109 Invalid environment variable

type.
DSJE_BADENVVARPROMPT -110 No prompt supplied.
DSJE_BADHANDLE –1 Invalid JobHandle.
DSJE_BADLINK –9 LinkName does not refer to a

known link for the stage in
question.
DSJE_BADLISTVALUE -120 Invalid value given for a list

DSJE_BADNAME –12 Invalid project name.
DSJE_BADNUMERICVALUE -119 Invalid value given for an

integer environment variable.
DSJE_BADPARAM –3 ParamName is not a parameter

name in the job.
DSJE_BADPROJECT –1002 ProjectName is not a known

DataStage project.
DSJE_BADPROJLOCATION -130 Invalid pathname supplied.
DSJE_BADPROJNAME -128 Invalid project name supplied.
DSJE_BADPROPERTY -104 Unknown property name.
DSJE_BADPROPVALUE -106 Invalid value for this property.
DSJE_BADSTAGE –7 StageName does not refer to a

known stage in the job.

DataStage Development Kit (Job Control Interfaces) Error Codes

DSJE_BADSTATE –2 Job is not in the right state
(compiled, not running).
DSJE_BADTIME –13 Invalid StartTime or EndTime

value.
DSJE_BADTYPE –5 Information or event type was

unrecognized.
DSJE_BAD_VERSION –1008 The DataStage server does not

support this version of the
DataStage API.
DSJE_BADVALUE –4 Invalid MaxNumber value.
DSJE_CLEARSCHEDULEFAILED -127 Failed to clear scheduled jobs for

project.
DSJE_DECRYPTERR –15 Failed to decrypt encrypted

values.
DSJE_DELETEPROJECTBLOCKED -138 Project locked by another user.
DSJE_DELPROJFAILED -124 Failed to delete project

definition.
DSJE_DELPROJFILESFAILED -125 Failed to delete project files.
DSJE_DUPENVVARNAME -115 Environment variable being

added already exists.
DSJE_ENCODEFAILED -123 Failed to encode an encrypted

value.
DSJE_GETDEFAULTPATHFAILED -129 Failed to determine default

project directory.
DSJE_INCOMPATIBLE_SERVER –1009 The server version is

incompatible with this version
of the DataStage API.
DSJE_ISADMINFAILED -101 Failed to determine whether

user is an administrator.
DSJE_ISPARALLELLICENCED -122 Failed to determine if Enterprise

Edition installed.
DSJE_INVALIDPROJECTLOCATION -131 Invalid pathname supplied.
DSJE_JOBDELETED –11 The job has been deleted.
DSJE_JOBLOCKED –10 The job is locked by another

process.
DSJE_LICENSEPROJECTFAILED -136 Failed to license project.


DSJE_LISTSCEDULEFAILED -126 Failed to get list of scheduled
jobs for project.
DSJE_LOGTOFAILED -141 Failed to log to UV account.
DSJE_NOACCESS –16 Cannot get values, default

values or design default values
for any job except the current
job.
DSJE_NO_DATASTAGE –1003 DataStage is not installed on the

server system.
DSJE_NOERROR 0 No DataStage API error has

occurred.
DSJE_NO_MEMORY –1005 Failed to allocate dynamic

memory.
DSJE_NOMORE –1001 All events matching the filter

criteria have been returned.
DSJE_NOTADMINUSER -100 User is not an administrator.
DSJE_NOTAPROJECT -139 Failed to log to project.
DSJE_NOT_AVAILABLE –1007 The requested information was

not found.
DSJE_NOTINSTAGE –8 Internal server error.
DSJE_NOTUSERDEFINED -117 Environment variable is not

user-defined and therefore
cannot be deleted.
DSJE_OPENFAIL -1004 The attempt to open the job

failed – perhaps it has not been
compiled.
DSJE_OPENFAILED -132 Failed to open UV.ACCOUNT

file.
DSJE_OSHVISIBLEFLAG -107 Failed to get value for

OSHVisible.
DSJE_PROPNOTSUPPORTED -105 Unsupported property.
DSJE_PXNOTINSTALLED -121 Environment variable is specific

to Enterprise Edition which is
not installed.
DSJE_READENVVARDEFNS -111 Failed to read environment

variable definitions.
DSJE_READENVVARVALUES -112 Failed to read environment

variable values.


DSJE_READPROJPROPERTY -102 Failed to read property.
DSJE_READUFAILED -133 Failed to lock project create lock

record.
DSJE_RELEASEFAILED -137 Failed to release project create

lock record.
DSJE_REPERROR –99 General server error.
DSJE_SERVER_ERROR –1006 An unexpected or unknown

error occurred in the DataStage
server engine.
DSJE_TIMEOUT –14 The job appears not to have

started after waiting for a
reasonable length of time.
(About 30 minutes.)
DSJE_UNKNOWN_JOBNAME -201 The supplied jobname cannot be

found in the project.
DSJE_WRITEENVVARDEFNS -113 Failed to write environment

DSJE_WRITEENVVARVALUES -114 Failed to write environment

variable values.
DSJE_WRITEPROJPROPERTY -103 Property not supported.
DSJE_WRONGJOB –6 Job for this JobHandle was not

started from a call to
DSRunJob by the current
process.
The following table lists DataStage API error codes in numerical order:
Code Error Token Description

0 DSJE_NOERROR No DataStage API error has
occurred.
–1 DSJE_BADHANDLE Invalid JobHandle.
–2 DSJE_BADSTATE Job is not in the right state

(compiled, not running).
–3 DSJE_BADPARAM ParamName is not a parameter

name in the job.
–4 DSJE_BADVALUE Invalid MaxNumber value.
–5 DSJE_BADTYPE Information or event type was

unrecognized.


–6 DSJE_WRONGJOB Job for this JobHandle was not
started from a call to DSRunJob
by the current process.
–7 DSJE_BADSTAGE StageName does not refer to a

known stage in the job.
–8 DSJE_NOTINSTAGE Internal server error.
–9 DSJE_BADLINK LinkName does not refer to a

known link for the stage in
question.
–10 DSJE_JOBLOCKED The job is locked by another

process.
–11 DSJE_JOBDELETED The job has been deleted.
–12 DSJE_BADNAME Invalid project name.
–13 DSJE_BADTIME Invalid StartTime or EndTime

value.
–14 DSJE_TIMEOUT The job appears not to have started

after waiting for a reasonable
length of time. (About 30 minutes.)
–15 DSJE_DECRYPTERR Failed to decrypt encrypted values.
–16 DSJE_NOACCESS Cannot get values, default values or

design default values for any job
except the current job.
–99 DSJE_REPERROR General server error.
-100 DSJE_NOTADMINUSER User is not an administrator.
-101 DSJE_ISADMINFAILED Failed to determine whether user is

an administrator.
-102 DSJE_READPROJPROPERTY Failed to read property.
-103 DSJE_WRITEPROJPROPERTY Property not supported.
-104 DSJE_BADPROPERTY Unknown property name.
-105 DSJE_PROPNOTSUPPORTED Unsupported property.
-106 DSJE_BADPROPVALUE Invalid value for this property.
-107 DSJE_OSHVISIBLEFLAG Failed to get value for OSHVisible.
-108 DSJE_BADENVVARNAME Invalid environment variable name.
-109 DSJE_BADENVVARTYPE Invalid environment variable type.
-110 DSJE_BADENVVARPROMPT No prompt supplied.


-111 DSJE_READENVVARDEFNS Failed to read environment variable
definitions.
-112 DSJE_READENVVARVALUES Failed to read environment variable

values.
-113 DSJE_WRITEENVVARDEFNS Failed to write environment

-114 DSJE_WRITEENVVARVALUES Failed to write environment

variable values.
-115 DSJE_DUPENVVARNAME Environment variable being added

already exists.
-116 DSJE_BADENVVAR Environment variable does not

exist.
-117 DSJE_NOTUSERDEFINED Environment variable is not user-

defined and therefore cannot be
deleted.
-118 DSJE_BADBOOLEANVALUE Invalid value given for a boolean

-119 DSJE_BADNUMERICVALUE Invalid value given for an integer

-120 DSJE_BADLISTVALUE Invalid value given for a list

-121 DSJE_PXNOTINSTALLED Environment variable is specific to

Enterprise Edition which is not
installed.
-122 DSJE_ISPARALLELLICENCED Failed to determine if Enterprise

Edition installed.
-123 DSJE_ENCODEFAILED Failed to encode an encrypted

value.
-124 DSJE_DELPROJFAILED Failed to delete project definition.
-125 DSJE_DELPROJFILESFAILED Failed to delete project files.
-126 DSJE_LISTSCHEDULEFAILED Failed to get list of scheduled jobs

for project.
-127 DSJE_CLEARSCHEDULEFAILED Failed to clear scheduled jobs for

project.
-128 DSJE_BADPROJNAME Invalid project name supplied.
-129 DSJE_GETDEFAULTPATHFAILED Failed to determine default project

directory.
-130 DSJE_BADPROJLOCATION Invalid pathname supplied.
-131 DSJE_INVALIDPROJECTLOCATION Invalid pathname supplied.


-132 DSJE_OPENFAILED Failed to open UV.ACCOUNT file.
-133 DSJE_READUFAILED Failed to lock project create lock

record.
-134 DSJE_ADDPROJECTBLOCKED Another user is adding a project.
-135 DSJE_ADDPROJECTFAILED Failed to add project.
-136 DSJE_LICENSEPROJECTFAILED Failed to license project.
-137 DSJE_RELEASEFAILED Failed to release project create lock

record.
-138 DSJE_DELETEPROJECTBLOCKED Project locked by another user.
-139 DSJE_NOTAPROJECT Failed to log to project.
-140 DSJE_ACCOUNTPATHFAILED Failed to get account path.
-141 DSJE_LOGTOFAILED Failed to log to UV account.
-201 DSJE_UNKNOWN_JOBNAME The supplied jobname cannot be

found in the project.
–1001 DSJE_NOMORE All events matching the filter

criteria have been returned.
–1002 DSJE_BADPROJECT ProjectName is not a known

DataStage project.
–1003 DSJE_NO_DATASTAGE DataStage is not installed on the

server system.
–1004 DSJE_OPENFAIL The attempt to open the job failed –

perhaps it has not been compiled.
–1005 DSJE_NO_MEMORY Failed to allocate dynamic memory.
–1006 DSJE_SERVER_ERROR An unexpected or unknown error

occurred in the DataStage server
engine.
–1007 DSJE_NOT_AVAILABLE The requested information was not

found.
–1008 DSJE_BAD_VERSION The DataStage server does not

support this version of the
DataStage API.
–1009 DSJE_INCOMPATIBLE_SERVER The server version is incompatible

with this version of the DataStage
API.

DataStage Development Kit (Job Control Interfaces) DataStage BASIC Interface
The following table lists some common errors that may be returned
from the lower-level communication layers:
Table 7-2 API Communication Layer Error Codes

Error Description
Number
39121 The DataStage server license has expired.
39134 The DataStage server user limit has been reached.
80011 Incorrect system name or invalid user name or password provided.
80019 Password has expired.
DataStage BASIC Interface

These functions can be used in a job control routine, which is defined
as part of a job’s properties and allows other jobs to be run and be
controlled from the first job. Some of the functions can also be used
for getting status information on the current job; these are useful in
active stage expressions and before- and after-stage subroutines.
These functions are also described in Chapter 18,"BASIC
Programming,"
Table 7-3 BASIC Functions

To do this… Use this…
Specify the job you want to control DSAttachJob
Set parameters for the job you want to DSSetParam

control
Set limits for the job you want to control DSSetJobLimit
Request that a job is run DSRunJob
Wait for a called job to finish DSWaitForJob
Get information from certain parallel DSGetCustInfo

stages.
Get information about the current DSGetProjectInfo

project
Get information about the controlled job DSGetJobInfo

or current job
Get information about a stage in the DSGetStageInfo

controlled job or current job

DataStage BASIC Interface DataStage Development Kit (Job Control Interfaces)

Get information about a link in a DSGetLinkInfo
controlled job or current job
Get information about a controlled job’s DSGetParamInfo

parameters
Get the log event from the job log DSGetLogEntry
Get a number of log events on the DSGetLogSummary

specified subject from the job log
Get the newest log event, of a specified DSGetNewestLogId

type, from the job log
Log an event to the job log of a different DSLogEvent

job
Stop a controlled job DSStopJob
Return a job handle previously obtained DSDetachJob

from DSAttachJob
Log a fatal error message in a job's log DSLogFatal

file and aborts the job.
Log an information message in a job's DSLogInfo

log file.
Put an info message in the job log of a DSLogToController

job controlling current job.
Log a warning message in a job's log DSLogWarn

file.
Generate a string describing the DSMakeJobReport

complete status of a valid attached job.
Insert arguments into the message DSMakeMsg

template.
Ensure a job is in the correct state to be DSPrepareJob

run or validated.
Interface to system send mail facility. DSSendMail
Log a warning message to a job log file. DSTransformError
Convert a job control status or error DSTranslateCode

code into an explanatory text message.
Suspend a job until a named file either DSWaitForFile

exists or does not exist.
Checks if a BASIC routine is cataloged, DSCheckRoutine

either in VOC as a callable item, or in the
catalog space.

DataStage Development Kit (Job Control Interfaces) DataStage BASIC Interface

Execute a DOS or DataStage Engine DSExecute
command from a befor/after subroutine.
Set a status message for a job to return DSSetUserStatus

as a termination message when it
finishes
Specifies whether a job should generate DSSetGenerateOpMetaData

operational meta data as it runs. This
overrides the default setting for the
project.

DSAttachJob DataStage Development Kit (Job Control Interfaces)
DSAttachJob
Attaches to a job in order to run it in job control sequence. A handle is
returned which is used for addressing the job. There can only be one
handle open for a particular job at any one time.
Syntax
JobHandle = DSAttachJob (JobName, ErrorMode)
JobHandle is the name of a variable to hold the return value which is

subsequently used by any other function or routine when referring to
the job. Do not assume that this value is an integer.
JobName is a string giving the name of the job to be attached to.
ErrorMode is a value specifying how other routines using the handle
should report errors. It is one of:
DSJ.ERRFATAL Log a fatal message and abort the controlling job
(default).
DSJ.ERRWARNINGLog a warning message but carry on.
DSJ.ERRNONENo message logged - caller takes full
responsibility (failure of DSAttachJob itself will be logged,
however).
Remarks
A job cannot attach to itself.
The JobName parameter can specify either an exact version of the job
in the form job%Reln.n.n, or the latest version of the job in the form
job. If a controlling job is itself released, you will get the latest
released version of job. If the controlling job is a development
version, you will get the latest development version of job.
Example
This is an example of attaching to Release 11 of the job Qsales:
Qsales_handle = DSAttachJob ("Qsales%Rel1",
➥ DSJ.ERRWARN)

DataStage Development Kit (Job Control Interfaces) DSCheckRoutine
DSCheckRoutine
Checks if a BASIC routine is cataloged, either in the VOC as a callable
item, or in the catalog space.
Syntax
Found = DSCheckRoutine(RoutineName)
RoutineName is the name of BASIC routine to check.

Found Boolean. @False if RoutineName not findable, else @True.
Example
rtn$ok = DSCheckRoutine(“DSU.DSSendMail”)
If(NOT(rtn$ok)) Then
* error handling here
End.

DSDetachJob DataStage Development Kit (Job Control Interfaces)
DSDetachJob
Gives back a JobHandle acquired by DSAttachJob if no further
control of a job is required (allowing another job to become its
controller). It is not necessary to call this function, otherwise any
attached jobs will always be detached automatically when the
controlling job finishes.
Syntax
ErrCode = DSDetachJob (JobHandle)
JobHandle is the handle for the job as derived from DSAttachJob.

ErrCode is 0 if DSStopJob is successful, otherwise it may be the
following:
DSJE.BADHANDLE Invalid JobHandle.
The only possible error is an attempt to close DSJ.ME. Otherwise, the
call always succeeds.
Example
The following command detaches the handle for the job qsales:
Deterr = DSDetachJob (qsales_handle)

DataStage Development Kit (Job Control Interfaces) DSExecute
DSExecute
Executes a DOS or DataStage Engine command from a before/after
subroutine.
Syntax
Call DSExecute (ShellType, Command, Output, SystemReturnCode)
ShellType (input) specifies the type of command you want to execute

and is either NT or UV (for DataStage Engine).
Command (input) is the command to execute. Command should not
prompt for input when it is executed.
Output (output) is any output from the command. Each line of output
is separated by a field mark, @FM. Output is added to the job log file
as an information message.
SystemReturnCode (output) is a code indicating the success of the
command. A value of 0 means the command executed successfully. A
value of 1 (for a DOS command) indicates that the command was not
found. Any other value is a specific exit code from the command.
Remarks
Do not use DSExecute from a transform; the overhead of running a
command for each row processed by a stage will degrade
performance of the job.

DSGetCustInfo DataStage Development Kit (Job Control Interfaces)
DSGetCustInfo
Obtains information reported at the end of execution of certain
parallel stages. The information collected, and available to be
interrogated, is specified at design time. For example, transformer
stage information is specified in the Triggers tab of the Transformer
stage Properties dialog box.
Syntax
Result = DSGetCustInfo (JobHandle, StageName, CustInfoName, InfoType)
JobHandle is the handle for the job as derived from DSAttachJob, or

it may be DSJ.ME to refer to the current job.
StageName is the name of the stage to be interrogated. It may also be
DSJ.ME to refer to the current stage if necessary.
VarName is the name of the variable to be interrogated.
InfoType specifies the information required and can be one of:
DSJ.CUSTINFOVALUE
DSJ.CUSTINFODESC
Result depends on the specified InfoType, as follows:
DSJ.CUSTINFOVALUE String - the value of the specified
custinfo item.
DSJ.CUSTINFODESC String - description of the specified
custinfo item.
Result may also return an error condition as follows:
DSJE.BADHANDLE JobHandle was invalid.
DSJE.BADTYPEInfoType was unrecognized.
DSJE.NOTINSTAGE StageName was DSJ.ME and the caller is
not running within a stage.
DSJE.BADSTAGE StageName does not refer to a known stage in
the job.
DSJE.BADCUSTINFOCustInfoName does not refer to a known
custinfo item.

DataStage Development Kit (Job Control Interfaces) DSIPCPageProps
DSIPCPageProps
Returns the size (in KB) of the Send/Recieve buffer of an IPC (or Web
Service) stage.
Syntax
Result = DSGetIPCStageProps (JobName, StageName)
or
Call DSGetIPCStageProps (Result, JobName, StageName)
JobName is the name of the job in the current project for which
information is required. If JobName does not exist in the current
project, Result will be set to an empty string.
StageName is the name of an IPC stage in the specified job for which
information is required. If StageName does not exist, or is not an IPC
stage within JobName, Result will be set to an empty string.
Result is an array containing the following fields:
the size (in kilobytes) of the Send/Receive buffer of the IPC (or Web
Service) stage StageName within JobName.
the seconds timeout value of the IPC (or Web Service) stage
StageName within JobName.
Example
The following returns the size and timeout of the stage “IPC1” in the
job “testjob”:
buffersize = DSGetIPCStageProps (testjob, IPC1)

DSGetJobInfo
Provides a method of obtaining information about a job, which can be
used generally as well as for job control. It can refer to the current job
or a controlled job, depending on the value of JobHandle.
Syntax
Result = DSGetJobInfo (JobHandle, InfoType)

DSJ.JOBSTATUS
DSJ.JOBNAME
DSJ.JOBCONTROLLER
DSJ.JOBSTARTTIMESTAMP
DSJ.JOBWAVENO
DSJ.PARAMLIST
DSJ.STAGELIST
DSJ.USERSTATUS
DSJ.JOBCONTROL
DSJ.JOBPID
DSJ.JPBLASTTIMESTAMP
DSJ.JOBINVOCATIONS
DSJ.JOBINTERIMSTATUS
DSJ.JOBINVOCATIONID
DSJ.JOBDESC
DSJ.JOBFULLDESC
DSJ.STAGELIST2
DSJ.JOBELAPSED
DSJ.JOBEOTCOUNT
DSJ.JOBEOTTIMESTAMP
DSJ.JOBRTISERVICE
DSJ.JOBMULTIINVOKABLE
DSJ.JOBFULLSTAGELIST


DSJ.JOBSTATUS Integer. Current status of job overall. Possible
statuses that can be returned are currently divided into two
categories:
Firstly, a job that is in progress is identified by:
DSJS.RESETJob finished a reset run.
DSJS.RUNFAILEDJob finished a normal run with a fatal
error.
DSJS.RUNNINGJob running - this is the only status that
means the job is actually running.
Secondly, jobs that are not running may have the following
statuses:
DSJS.RUNOKJob finished a normal run with no
warnings.
DSJS.RUNWARNJob finished a normal run with
warnings.
DSJS.STOPPEDJob was stopped by operator
intervention (can't tell run type).
DSJS.VALFAILEDJob failed a validation run.
DSJS.VALOKJob finished a validation run with no
warnings.
DSJS.VALWARNJob finished a validation run with
warnings.
DSJ.JOBNAME String. Actual name of the job referenced by the
job handle.
DSJ.JOBCONTROLLER String. Name of the job controlling the
job referenced by the job handle. Note that this may be several job
names separated by periods if the job is controlled by a job which
is itself controlled, etc.
DSJ.JOBSTARTTIMESTAMP String. Date and time when the job
started on the server in the form YYYY-MM-
DD HH:NN:SShh:nn:ss.
DSJ.JOBWAVENO Integer. Wave number of last or current run.
DSJ.PARAMLIST. Returns a comma-separated list of parameter
names.
DSJ.STAGELIST. Returns a comma-separated list of active stage
names.

DSJ.USERSTATUS String. Whatever the job's last call of

DSSetUserStatus last recorded, else the empty string.
DSJ.JOBCONTROL Integer. Current job control status, i.e.,
whether a stop request has been issued for the job.
DSJ. JOBPID Integer. Job process id.
DSJ.JOBLASTTIMESTAMP String. Date and time when the job
last finished a run on the server in the form YYYY-MM-
DD HH:NN:SS.
DSJ.JOBINVOCATIONS. Returns a comma-separated list of
Invocation IDs.
DSJ.JOBINTERIMSTATUS. Returns the status of a job after it
has run all stages and controlled jobs, but before it has attempted
to run an after-job subroutine. (Designed to be used by an after-job
subroutine to get the status of the current job).
DSJ.JOBINVOCATIONID. Returns the invocation ID of the
specified job (used in the DSJobInvocationId macro in a job
design to access the invocation ID by which the job is invoked).
DSJ.STAGELIST2. Returns a comma-separated list of passive
stage names.
DSJ.JOBELAPSED String. The elapsed time of the job in
seconds.
DSJ.JOBDESC string. The Job Description specified in the Job
DSJ.JOBFULLDESSC string. The Full Description specified in
the Job Properties dialog box.
DSJ.JOBRTISERVICE integer. Set to true if this is a web service
job.
DSJ.JOBMULTIINVOKABLE integer. Set to true if this job
supports multiple invocations
DSJ.JOBEOTCOUNT integer. Count of EndOfTransmission
blocks processed by this job so far.
DSJ.JOBEOTTIMESTAMP timestamp. Date/time of the last
EndOfTransmission block processed by this job.
DSJ.FULLSTAGELIST. Returns a comma-separated list of all
stage names.
Result may also return error conditions as follows:
DSJE.BADTYPEInfoType was unrecognized.

Remarks
When referring to a controlled job, DSGetJobInfo can be used either
before or after a DSRunJob has been issued. Any status returned
following a successful call to DSRunJob is guaranteed to relate to
that run of the job.
Examples
The following command requests the job status of the job qsales:
q_status = DSGetJobInfo(qsales_handle, DSJ.JOBSTATUS)
The following command requests the actual name of the current job:
whatname = DSGetJobInfo (DSJ.ME, DSJ.JOBNAME)

DSGetJobMetaBag DataStage Development Kit (Job Control Interfaces)
DSGetJobMetaBag
Returns a dynamic array containing the MetaBag properties
associated with the named job.
Syntax
Result = DSGetJobMetaBag(JobName, Owner)
or
Call DSGetJobMetaBag(Result, JobName, Owner)
project Result will be set to an empty string.
Owner is an owner name whose metabag properties are to be
returned. If Owner is not a valid owner within the current job, Result
will be set to an empty string. If Owner is an empty string, a field
mark delimited string of metabag property owners within the current
job will be returned in Result.
Result returns a dynamic array of metabag property sets, as follows:
RESULT<1> = MetaPropertyName01 @VM MetaPropertyValue01
RESULT<..> = MetaPropertyName.. @VM MetaPropertyValue..
RESULT<N>= MetaPropertyNameN @VM MetaPropertyValueN
Example
The following returns the metabag properties for owner mbowner in
the job “testjob”:
linksmdata = DSGetJobMetaBag (testjob, mbowner)

DataStage Development Kit (Job Control Interfaces) DSGetLinkInfo
DSGetLinkInfo
Provides a method of obtaining information about a link on an active
stage, which can be used generally as well as for job control. This
routine may reference either a controlled job or the current job,
depending on the value of JobHandle.
Syntax
Result = DSGetLinkInfo (JobHandle, StageName, LinkName, InfoType)

it can be DSJ.ME to refer to the current job.
StageName is the name of the active stage to be interrogated. May
also be DSJ.ME to refer to the current stage if necessary.
LinkName is the name of a link (input or output) attached to the stage.
May also be DSJ.ME to refer to current link (e.g. when used in a
Transformer expression or transform function called from link code).
DSJ.LINKLASTERR
DSJ.LINKNAME
DSJ.LINKROWCOUNT
DSJ.LINKSQLSTATE
DSJ.LINKDBMSCODE
DSJ.LINKDESC
DSJ.LINKSTAGE
DSJ.INSTROWCOUNT
DSJ.LINKEOTROWCOUNT
DSJ.LINKLASTERR String – last error message (if any) reported
from the link in question.
DSJ.LINKNAME String – returns the name of the link, most
useful when used with JobHandle = DSJ.ME and StageName =
DSJ.ME and LinkName = DSJ.ME to discover your own name.
DSJ.LINKROWCOUNT Integer – number of rows that have
passed down a link so far.
DSJ.LINKSQLSTATE – the SQL state for the last error occurring
on this link.
DSJ.LINKDBMSCODE – the DBMS code for the last error
occurring on this link.

DSGetLinkInfo DataStage Development Kit (Job Control Interfaces)
DSJ.LINKDESC – description of the link.

DSJ.LINKSTAGE – name of the stage at the other end of the link.
DSJ.INSTROWCOUNT – comma-separated list of rowcounts,
one per instance (parallel jobs)
DSJ.LINKEOTROWCOUNT – row count since last
EndOfTransmission block.
DSJE.BADTYPE InfoType was unrecognized.
DSJE.BADSTAGE StageName does not refer to a known stage
in the job.
DSJE.BADLINK LinkName does not refer to a known link
for the stage in question.
Remarks
When referring to a controlled job, DSGetLinkInfo can be used
either before or after a DSRunJob has been issued. Any status
returned following a successful call to DSRunJob is guaranteed to
relate to that run of the job.
Example
The following command requests the number of rows that have
passed down the order_feed link in the loader stage of the job qsales:
link_status = DSGetLinkInfo(qsales_handle, "loader",
➥ "order_feed", DSJ.LINKROWCOUNT)

DataStage Development Kit (Job Control Interfaces) DSGetLinkMetaData
DSGetLinkMetaData
Returns a dynamic array containing the column metadata of the
specified stage.
Syntax
Result = DSGetLinkMetaData(JobName, LinkName)
or
Call DSGetLinkMetaData(Result, JobName, LinkName)
information is required. If the JobName does not exist in the current
project then the function will return an empty string.
LinkName is the name of the link in the specified job for which
information is required. If the LinkName does not exist in the specified
job then the function will return an empty string.
Result returns a dynamic array of nine fields, each field will contain N
values where N is the number of columns on the link.
Result<1,1…N> is the column name
Result<2,1…N> is 1 for primary key columns otherwise 0
Result<3,1…N> is the column sql type. See ODBC.H.
Result<4,1…N> is the column precision
Result<5,1…N> is the column scale
Result<6,1…N> is the column desiplay width
Result<7,1…N> is 1 for nullable columns otherwise 0
Result<8,1…N> is the column descriptions
Result<9,1…N> is the column derivation
Example
The following returns the meta data of the link ilink1 in the job
“testjob”:
linksmdata = DSGetLinkMetaData (testjob, ilink1)

DSGetLogEntry DataStage Development Kit (Job Control Interfaces)
DSGetLogEntry
Reads the full event details given in EventId.
Syntax
EventDetail = DSGetLogEntry (JobHandle, EventId)

EventId is an integer that identifies the specific log event for which
details are required. This is obtained using the DSGetNewestLogId
function.
EventDetail is a string containing substrings separated by \. The
substrings are as follows:
Substring1Timestamp in form YYYY-MM-DD HH:NN:SS
Substring2User information
Substring3EventType – see DSGetNewestLogId
Substring4 – n Event message
If any of the following errors are found, they are reported via a fatal
log event:
DSJE.BADVALUE Error accessing EventId.
Example
The following commands first get the EventID for the required log
event and then reads full event details of the log event identified by
LatestLogid into the string LatestEventString:
latestlogid =
➥ DSGetNewestLogId(qsales_handle,DSJ.LOGANY)
LatestEventString =
➥ DSGetLogEntry(qsales_handle,latestlogid)

DataStage Development Kit (Job Control Interfaces) DSGetLogSummary
DSGetLogSummary
Returns a list of short log event details. The details returned are
determined by the setting of some filters. (Care should be taken with
the setting of the filters, otherwise a large amount of information can
be returned.)
Syntax
SummaryArray = DSGetLogSummary (JobHandle, EventType, StartTime,
EndTime, MaxNumber)

EventType is the type of event logged and is one of:
DSJ.LOGINFO Information message
DSJ.LOGWARNING Warning message
DSJ.LOGFATAL Fatal error
DSJ.LOGREJECT Reject link was active
DSJ.LOGSTARTED Job started
DSJ.LOGRESET Log was reset
DSJ.LOGANY Any category (the default)
StartTime is a string in the form YYYY-MM-DD HH:NN:SS or YYYY-
MM-DD.
EndTime is a string in the form YYYY-MM-DD HH:NN:SS or YYYY-MM-
DD.
MaxNumber is an integer that restricts the number of events to return.
0 means no restriction. Use this setting with caution.
SummaryArray is a dynamic array of fields separated by @FM. Each
field comprises a number of substrings separated by \, where each
field represents a separate event, with the substrings as follows:
Substring1EventId as per DSGetLogEntry
Substring2Timestamp in form YYYY-MM-DD
HH:NN:SS
Substring3EventType – see DSGetNewestLogId
Substring4 – n Event message
If any of the following errors are found, they are reported via a fatal
log event:
DSJE.BADTYPE Invalid EventType.

DSGetLogSummary DataStage Development Kit (Job Control Interfaces)
DSJE.BADTIME Invalid StartTime or EndTime.

DSJE.BADVALUE Invalid MaxNumber.
Example
The following command produces an array of reject link active events
recorded for the qsales job between 18th August 1998, and 18th
September 1998, up to a maximum of MAXREJ entries:
RejEntries = DSGetLogSummary (qsales_handle,
➥ DSJ.LOGREJECT, "1998-08-18 00:00:00", "1998-09-18
➥ 00:00:00", MAXREJ)

DataStage Development Kit (Job Control Interfaces) DSGetNewestLogId
DSGetNewestLogId
Gets the ID of the most recent log event in a particular category, or in
any category.
Syntax
EventId = DSGetNewestLogId (JobHandle, EventType)

DSJ.LOGFATAL Fatal error
DSJ.LOGREJECT Reject link was active
DSJ.LOGSTARTED Job started
DSJ.LOGRESET Log was reset
DSJ.LOGANY Any category (the default)
EventId is an integer that identifies the specific log event. EventId can
also be returned as an integer, in which case it contains an error code
as follows:
DSJE.BADTYPE Invalid EventType.
Example
The following command obtains an ID for the most recent warning
message in the log for the qsales job:
Warnid = DSGetNewestLogId (qsales_handle,
➥ DSJ.LOGWARNING)

DSGetParamInfo
Provides a method of obtaining information about a parameter, which
can be used generally as well as for job control. This routine may
reference either a controlled job or the current job, depending on the
value of JobHandle.
Syntax
Result = DSGetParamInfo (JobHandle, ParamName, InfoType)

ParamName is the name of the parameter to be interrogated.
InfoType specifies the information required and may be one of:
DSJ.PARAMDEFAULT
DSJ.PARAMHELPTEXT
DSJ.PARAMPROMPT
DSJ.PARAMTYPE
DSJ.PARAMVALUE
DSJ.PARAMDES.DEFAULT
DSJ.PARAMLISTVALUES
DSJ.PARAMDES.LISTVALUES
DSJ.PARAMPROMPT.AT.RUN
DSJ.PARAMDEFAULT String – Current default value for the
parameter in question. See also DSJ.PARAMDES.DEFAULT.
DSJ.PARAMHELPTEXT String – Help text (if any) for the
parameter in question.
DSJ.PARAMPROMPT String – Prompt (if any) for the parameter
in question.
DSJ.PARAMTYPE Integer – Describes the type of validation test
that should be performed on any value being set for this
parameter. Is one of:
DSJ.PARAMTYPE.STRING
DSJ.PARAMTYPE.ENCRYPTED
DSJ.PARAMTYPE.INTEGER

DataStage Development Kit (Job Control Interfaces) DSGetParamInfo
DSJ.PARAMTYPE.FLOAT (the parameter may contain periods

and E)
DSJ.PARAMTYPE.PATHNAME
DSJ.PARAMTYPE.LIST (should be a string of Tab-separated
strings)
DSJ.PARAMTYPE.DATE (should be a string in form YYYY-MM-
DD)
DSJ.PARAMTYPE.TIME (should be a string in form HH:MM)
DSJ.PARAMVALUE String – Current value of the parameter for
the running job or the last job run if the job is finished.
DSJ.PARAMDES.DEFAULT String – Original default value of the
parameter - may differ from DSJ.PARAMDEFAULT if the latter
has been changed by an administrator since the job was installed.
DSJ.PARAMLISTVALUES String – Tab-separated list of allowed
values for the parameter. See also
DSJ.PARAMDES.LISTVALUES.
DSJ.PARAMDES.LISTVALUES String – Original Tab-separated
list of allowed values for the parameter – may differ from
DSJ.PARAMLISTVALUES if the latter has been changed by an
administrator since the job was installed.
DSJ.PROMPT.AT.RUN String – 1 means the parameter is to be
prompted for when the job is run; anything else means it is not
(DSJ.PARAMDEFAULT String to be used directly).
DSJE.BADPARAM ParamName is not a parameter name in the
job.
DSJE.BADTYPE InfoType was unrecognized.
Remarks
When referring to a controlled job, DSGetParamInfo can be used
Example
The following command requests the default value of the quarter
parameter for the qsales job:

Qs_quarter = DSGetparamInfo(qsales_handle, "quarter",

➥ DSJ.PARAMDEFAULT)

DataStage Development Kit (Job Control Interfaces) DSGetProjectInfo
DSGetProjectInfo
Provides a method of obtaining information about the current project.
Syntax
Result = DSGetProjectInfo (InfoType)

DSJ.JOBLIST
DSJ.PROJECTNAME
DSJ.HOSTNAME
DSJ.JOBLIST String - comma-separated list of names of all jobs
known to the project (whether the jobs are currently attached or
not).
DSJ.PROJECTNAME String - name of the current project.
DSJ.HOSTNAME String - the host name of the server holding
the current project.
– DSJE.BADTYPE InfoType was unrecognized.

DSGetStageInfo
Provides a method of obtaining information about a stage, which can
be used generally as well as for job control. It can refer to the current
job, or a controlled job, depending on the value of JobHandle.
Syntax
Result = DSGetStageInfo (JobHandle, StageName, InfoType)

InfoType specifies the information required and may be one of:
DSJ.LINKLIST
DSJ.STAGELASTERR
DSJ.STAGENAME
DSJ.STAGETYPE
DSJ.STAGEINROWNUM
DSJ.VARLIST
DSJ.STAGESTARTTIMESTAMP
DSJ.STAGEENDTIMESTAMP
DSJ.STAGEDESC
DSJ.STAGEINST
DSJ.STAGECPU
DSJ.LINKTYPES
DSJ.STAGEELAPSED
DSJ.STAGEPID
DSJ.STAGESTATUS
DSJ.STAGEEOTCOUNT
DSJ.STAGEEOTTIMESTAMP
DSJ.CUSTINFOLIST
DSJ.STAGEEOTSTART
DSJ.LINKLIST – comma-separated list of link names in the stage.

DataStage Development Kit (Job Control Interfaces) DSGetStageInfo
DSJ.STAGELASTERR String – last error message (if any)

reported from any link of the stage in question.
DSJ.STAGENAME String – most useful when used with
JobHandle = DSJ.ME and StageName = DSJ.ME to discover
your own name.
DSJ.STAGETYPE String – the stage type name (e.g.
"Transformer", "BeforeJob").
DSJ. STAGEINROWNUM Integer – the primary link's input row
number.
DSJ.VARLIST – comma-separated list of stage variable names.
DSJ.STAGESTARTTIMESTAMP – date/time that stage started
executing in the form YYY-MM-DD HH:NN:SS.
DSJ.STAGEENDTIMESTAMP – date/time that stage finished
executing in the form YYY-MM-DD HH:NN:SS.
DSJ.STAGEDESC – stage description.
DSJ.STAGEINST – comma-separated list of instance ids (parallel
jobs).
DSJ.STAGECPU – list of CPU times in seconds.
DSJ.LINKTYPES – comma-separated list of link types.
DSJ.STAGEELAPSED – elapsed time in seconds.
DSJ.STAGEPID – comma-separated list of process ids.
DSJ.STAGESTATUS – stage status.
DSJ.STAGEEOTCOUNT – Count of EndOfTransmission blocks
processed by this stage so far.
DSJ.STAGEEOTTIMESTAMP – Data/time of last
EndOfTransmission block received by this stage.
DSJ.CUSTINFOLIST – custom information generated by stages
(parallel jobs).
DSJ.STAGEEOTSTART – row count at start of current
EndOfTransmission block.
DSJE.BADTYPEI nfoType was unrecognized.
the job.

Remarks
When referring to a controlled job, DSGetStageInfo can be used
Example
The following command requests the last error message for the
loader stage of the job qsales:
stage_status = DSGetStageInfo(qsales_handle, "loader",
➥ DSJ.STAGELASTERR)

DataStage Development Kit (Job Control Interfaces) DSGetStageLinks
DSGetStageLinks
Returns a field mark delimited list containing the names of all of the
input/output links of the specified stage.
Syntax
Result = DSGetStageLinks(JobName, StageName, Key)
or
Call DSGetStageLinks(Result, JobName, StageName, Key)
project, then the function will return an empty string.
StageName is the name of the stage in the specified job for which
information is required. If the StageName does not exist in the
specified job then the function will return an empty string.
Key depending on the value of Key the returned list will contain all of
the stages links (Key=0), only the stage’s input links (Key=1) or only
the stage’s output links (Key=2).
Result returns a field mark delimited list containing the names of the
links.
Example
The following returns a list of all the input links on the stage called
“join1” in the job “testjob”:
linkslist = DSGetStageLinks (testjob, join1, 1)

DSGetStagesOfType DataStage Development Kit (Job Control Interfaces)
DSGetStagesOfType
Returns a field mark delimited list containing the names of all of the
stages of the specified type in a named job..
Syntax
Result = DSGetStagesOfType (JobName, StageType)
or
Call DSGetStagesOfType (Result, JobName, StageType)
project then the function will return an empty string.
StageType is the name of the stage type, as shown by the Manager
stage type properties form eg CTransformerStage or ORAOCI8. If the
StageType does not exist in the current project or there are no stages
of that type in the specifed job, then the function will return an empty
string.
Result returns a field mark delimited list containing the names of all of
the stages of the specified type in a named job.
Example
The following returns a list of all the aggregator stages in the parallel
job “testjob”:
stagelist = DSGetStagesOfType (testjob, PxAggregator)

DataStage Development Kit (Job Control Interfaces) DSGetStagesTypes
DSGetStagesTypes
Returns a field mark delimited string of all active and passive stage
types that exist within a named job..
Syntax
Result = DSGetStageTypes(JobName )
or
Call DSGetStageTypes(Result, JobName )
project, Result will be set to an empty string.
Result is a sorted, field mark delimited string of stage types within
JobName.
Example
The following returns a list of all the types of stage in the job
“testjob”:
stagetypelist = DSGetStagesOfType (testjob)

DSGetProjectInfo DataStage Development Kit (Job Control Interfaces)
DSGetProjectInfo
Provides a method of obtaining information about variables used in
transformer stages.
Syntax
Result = DSGetVarInfo (JobHandle, StageName, VarName, InfoType)

VarName is the name of the variable to be interrogated.
DSJ.VARVALUE
DSJ.VARDESCRIPTION
DSJ.VARVALUE String - the value of the specified variable.
DSJ.VARDESCRIPTION String - description of the specified
variable.
DSJE.BADTYPE InfoType was not recognized.
DSJE.BADVAR VarName was not recognized.
the job.

DataStage Development Kit (Job Control Interfaces) DSLogEvent
DSLogEvent
Logs an event message to a job other than the current one. (Use
DSLogInfo, DSLogFatal, or DSLogWarn to log an event to the
current job.)
Syntax
ErrCode = DSLogEvent (JobHandle, EventType, EventMsg)

EventMsg is a string containing the event message.
ErrCode is 0 if there is no error. Otherwise it contains one of the
following errors:
DSJE.BADHANDLEInvalid JobHandle.
DSJE.BADTYPEInvalid EventType (particularly note that you
cannot place a fatal message in another job’s log).
Example
The following command, when included in the msales job, adds the
message “monthly sales complete” to the log for the qsales job:
Logerror = DsLogEvent (qsales_handle, DSJ.LOGINFO,
➥ "monthly sales complete")

DSLogFatal DataStage Development Kit (Job Control Interfaces)
DSLogFatal
Logs a fatal error message in a job's log file and aborts the job.
Syntax
Call DSLogFatal (Message, CallingProgName)
Message (input) is the warning message you want to log. Message is

automatically prefixed with the name of the current stage and the
calling before/after subroutine.
CallingProgName (input) is the name of the before/after subroutine
that calls the DSLogFatal subroutine.
Remarks
DSLogFatal writes the fatal error message to the job log file and
aborts the job. DSLogFatal never returns to the calling before/after
subroutine, so it should be used with caution. If a job stops with a fatal
error, it must be reset using the DataStage Director before it can be
rerun.
In a before/after subroutine, it is better to log a warning message
(using DSLogWarn) and exit with a nonzero error code, which allows
DataStage to stop the job cleanly.
DSLogFatal should not be used in a transform. Use
DSTransformError instead.
Example
Call DSLogFatal("Cannot open file", "MyRoutine")

DataStage Development Kit (Job Control Interfaces) DSLogInfo
DSLogInfo
Logs an information message in a job's log file.
Syntax
Call DSLogInfo (Message, CallingProgName)
Message (input) is the information message you want to log. Message

is automatically prefixed with the name of the current stage and the
calling program.
CallingProgName (input) is the name of the transform or before/after
subroutine that calls the DSLogInfo subroutine.
Remarks
DSLogInfo writes the message text to the job log file as an
information message and returns to the calling routine or transform. If
DSLogInfo is called during the test phase for a newly created routine
in the DataStage Manager, the two arguments are displayed in the
results window.
Unlimited information messages can be written to the job log file.
However, if a lot of messages are produced the job may run slowly
and the DataStage Director may take some time to display the job log
file.
Example
Call DSLogInfo("Transforming: ":Arg1, "MyTransform")

DSLogToController DataStage Development Kit (Job Control Interfaces)
DSLogToController
This routine may be used to put an info message in the log file of the
job controlling this job, if any. If there isn't one, the call is just ignored.
Syntax
Call DSLogToController(MsgString)
MsgString is the text to be logged. The log event is of type

Information.
Remarks
If the current job is not under control, a silent exit is performed.
Example
Call DSLogToController(“This is logged to parent”)

DataStage Development Kit (Job Control Interfaces) DSLogWarn
DSLogWarn
Logs a warning message in a job's log file.
Syntax
Call DSLogWarn (Message, CallingProgName)

calling before/after subroutine.
CallingProgName (input) is the name of the before/after subroutine
that calls the DSLogWarn subroutine.
Remarks
DSLogWarn writes the message to the job log file as a warning and
returns to the calling before/after subroutine. If the job has a warning
limit defined for it, when the number of warnings reaches that limit,
the call does not return and the job is aborted.
DSLogWarn should not be used in a transform. Use
DSTransformError instead.
Example
If InputArg > 100 Then
Call DSLogWarn("Input must be =< 100; received
":InputArg,"MyRoutine")
End Else
* Carry on processing unless the job aborts
End

DSMakeJobReport DataStage Development Kit (Job Control Interfaces)
DSMakeJobReport
Generates a report describing the complete status of a valid attached
job.
Syntax
ReportText = DSMakeJobReport(JobHandle, ReportLevel, LineSeparator)
JobHandle is the string as returned from DSAttachJob.

ReportLevel specifies the type of report and is one of the following:
0 – basic report. Text string containing start/end time, time elapsed
and status of job.
1 – stage/link detail. As basic report, but also contains information
2 – text string containing full XML report.
LineSeparator is the string used to separate lines of the report. Special

values recognised are:
"CRLF" => CHAR(13):CHAR(10)
"LF" => CHAR(10)
"CR" => CHAR(13)
The default is CRLF if on Windows, else LF.
Remarks
If a bad job handle is given, or any other error is encountered,
information is added to the ReportText.
Example
h$ = DSAttachJob(“MyJob”, DSJ.ERRNONE)
rpt$ = DSMakeJobReport(h$,0,”CRLF”)

DataStage Development Kit (Job Control Interfaces) DSMakeMsg
DSMakeMsg
Insert arguments into a message template. Optionally, it will look up a
template ID in the standard DataStage messages file, and use any
returned message template instead of that given to the routine.
DSMakeMsg
Syntax
FullText = DSMakeMsg(Template, ArgList)
FullText is the message with parameters substituted

Template is the message template, in which %1, %2 etc. are to be
substituted with values from the equivalent position in ArgList. If the
template string starts with a number followed by "\", that is assumed
to be part of a message id to be looked up in the DataStage message
file.
Note: If an argument token is followed by "[E]", the value of that
argument is assumed to be a job control error code, and an
explanation of it will be inserted in place of "[E]". (See the
DSTranslateCode function.)
ArgList is the dynamic array, one field per argument to be substituted.
Remarks
This routine is called from job control code created by the
JobSequence Generator. It is basically an interlude to call
DSRMessage which hides any runtime includes.
It will also perform local job parameter substitution in the message
text. That is, if called from within a job, it looks for substrings such as
"#xyz#" and replaces them with the value of the job parameter named
"xyz".
Example
t$ = DSMakeMsg(“Error calling DSAttachJob(%1)<L>%2”,
➥jb$:@FM:DSGetLastErrorMsg())

DSPrepareJob DataStage Development Kit (Job Control Interfaces)
DSPrepareJob
Used to ensure that a compiled job is in the correct state to be run or
validated.
Syntax
JobHandle = DSPrepareJob(JobHandle)
JobHandle is the handle, as returned from DSAttachJob(), of the job

to be prepared.
JobHandle is either the original handle or a new one. If returned as 0,
an error occurred and a message is logged.
Example
h$ = DSPrepareJob(h$)

DataStage Development Kit (Job Control Interfaces) DSRunJob
DSRunJob
Starts a job running. Note that this call is asynchronous; the request is
passed to the run-time engine, but you are not informed of its
progress.
Syntax
ErrCode = DSRunJob (JobHandle, RunMode)

RunMode is the name of the mode the job is to be run in and is one of:
DSJ.RUNNORMAL (Default) Standard job run.
DSJ.RUNRESET Job is to be reset.
DSJ.RUNVALIDATE Job is to be validated only.
ErrCode is 0 if DSRunJob is successful, otherwise it is one of the
following negative integers:
DSJE.BADSTATEJob is not in the right state (compiled, not
running).
DSJE.BADTYPE RunMode is not a known mode.
Remarks
If the controlling job is running in validate mode, then any calls of
DSRunJob will act as if RunMode was DSJ.RUNVALIDATE,
regardless of the actual setting.
A job in validate mode will run its JobControl routine (if any) rather
than just check for its existence, as is the case for before/after routines.
This allows you to examine the log of what jobs it started up in
validate mode.
After a call of DSRunJob, the controlled job’s handle is unloaded. If
you require to run the same job again, you must use DSDetachJob
and DSAttachJob to set a new handle. Note that you will also need
to use DSWaitForJob, as you cannot attach to a job while it is
running.
Example
The following command starts the job qsales in standard mode:
RunErr = DSRunJob(qsales_handle, DSJ.RUNNORMAL)

DSSendMail DataStage Development Kit (Job Control Interfaces)
DSSendMail
This routine is an interface to a sendmail program that is assumed to
exist somewhere in the search path of the current user (on the server).
It hides the different call interfaces to various sendmail programs, and
provides a simple interface for sending text. For example:
Syntax
Reply = DSSendMail(Parameters)
Parameters is a set of name:value parameters, separated by either a

mark character or "\n".
Currently recognized names (case-insensitive) are:
"From"Mail address of sender, e.g. Me@SomeWhere.com
Can only be left blank if the local template file does not contain a
"%from%" token.
"To" Mail address of recipient, e.g. You@ElseWhere.com
Can only be left blank if the local template file does not contain a
"%to%" token.
"Subject" Something to put in the subject line of the message.
Refers to the "%subject%" token. If left as "", a standard subject
line will be created, along the lines of "From DataStage job:
jobname"
"Server" Name of host through which the mail should be sent.
May be omitted on systems (such as Unix) where the SMTP host
name can be and is set up externally, in which case the local
template file presumably will not contain a "%server%" token.
"Body" Message body.
Can be omitted. An empty message will be sent. If used, it must be
the last parameter, to allow for getting multiple lines into the
message, using "\n" for line breaks. Refers to the "%body%" token.
Note The text of the body may contain the tokens "%report% or
%fullreport% anywhere within it, which will cause a report
on the current job status to be inserted at that point. A full
report contains stage and link information as well as job
status.
Reply. Possible replies are:
DSJE.NOERROR (0) OK

DataStage Development Kit (Job Control Interfaces) DSSendMail
DSJE.NOPARAM Parameter name missing - field does not look

like 'name:value'
DSJE.NOTEMPLATE Cannot find template file
DSJE.BADTEMPLATE Error in template file
Remarks
The routine looks for a local file, in the current project directory, with a
well-known name. That is, a template to describe exactly how to run
the local sendmail command.
Example
code = DSSendMail("From:me@here\nTo:You@there\nSubject:Hi
ya\nBody:Line1\nLine2")

DSSetGenerateOpMetaData DataStage Development Kit (Job Control Interfaces)
DSSetGenerateOpMetaData
Use this to specify whether the job generates operational meta data or
not. This overrides the default setting for the project. In order to
generate operational meta data the Process MetaBroker must be
installed on your DataStage machine.
Syntax
ErrCode = DSSetGenerateOpMetaData (JobHandle, value)

value is TRUE to generate operational meta data, FALSE to not
generate operational meta data.
ErrCode is 0 if DSRunJob is successful, otherwise it is one of the
DSJE.BADTYPE value is wrong.
Example
The following command causes the job qsales to generate operational
meta data whatever the project default specifies:
GenErr = DSSetGenerateOpMetaData(qsales_handle, TRUE)

DataStage Development Kit (Job Control Interfaces) DSSetJobLimit
DSSetJobLimit
By default a controlled job inherits any row or warning limits from the
controlling job. These can, however, be overridden using the
DSSetJobLimit function.
Syntax
ErrCode = DSSetJobLimit (JobHandle, LimitType, LimitValue)

LimitType is the name of the limit to be applied to the running job and
is one of:
DSJ.LIMITWARN Job to be stopped after LimitValue warning
events.
DSJ.LIMITROWS Stages to be limited to LimitValue rows.
LimitValue is an integer specifying the value to set the limit to. Set this
to 0 to specify unlimited warnings.
ErrCode is 0 if DSSetJobLimit is successful, otherwise it is one of the
running).
DSJE.BADTYPELimitType is not a known limiting condition.
DSJE.BADVALUELimitValue is not appropriate for the limiting
condition type.
Example
The following command sets a limit of 10 warnings on the qsales job
before it is stopped:
LimitErr = DSSetJobLimit(qsales_handle,
➥ DSJ.LIMITWARN, 10)

DSSetParam DataStage Development Kit (Job Control Interfaces)
DSSetParam
Specifies job parameter values before running a job. Any parameter
not set will be defaulted.
Syntax
ErrCode = DSSetParam (JobHandle, ParamName, ParamValue)

ParamName is a string giving the name of the parameter.
ParamValue is a string giving the value for the parameter.
ErrCode is 0 if DSSetParam is successful, otherwise it is one of the
running).
DSJE.BADPARAMParamName is not a known parameter of the
job.
DSJE.BADVALUEParamValue is not appropriate for that
parameter type.
Example
The following commands set the quarter parameter to 1 and the
startdate parameter to 1/1/97 for the qsales job:
paramerr = DSSetParam (qsales_handle, "quarter", "1")
paramerr = DSSetParam (qsales_handle, "startdate",
➥ "1997-01-01")

DataStage Development Kit (Job Control Interfaces) DSSetUserStatus
DSSetUserStatus
Applies only to the current job, and does not take a JobHandle
parameter. It can be used by any job in either a JobControl or After
routine to set a termination code for interrogation by another job. In
fact, the code may be set at any point in the job, and the last setting is
the one that will be picked up at any time. So to be certain of getting
the actual termination code for a job the caller should use
DSWaitForJob and DSGetJobInfo first, checking for a successful
finishing status.
This routine is defined as a subroutine not a function because there
are no possible errors.
Syntax
Call DSSetUserStatus (UserStatus)
UserStatus String is any user-defined termination message. The string

will be logged as part of a suitable "Control" event in the calling job’s
log, and stored for retrieval by DSGetJobInfo, overwriting any
previous stored string.
This string should not be a negative integer, otherwise it may be
indistinguishable from an internal error in DSGetJobInfo calls.
Example
The following command sets a termination code of “sales job done”:
Call DSSetUserStatus("sales job done")

DSStopJob DataStage Development Kit (Job Control Interfaces)
DSStopJob
This routine should only be used after a DSRunJob has been issued.
It immediately sends a stop request to the run-time engine. The call is
asynchronous. If you need to know that the job has actually stopped,
you must call DSWaitForJob or use the Sleep statement and poll for
DSGetJobStatus. Note that the stop request gets sent regardless of
the job's current status.
Syntax
ErrCode = DSStopJob (JobHandle)

ErrCode is 0 if DSStopJob is successful, otherwise it may be the
following:
Example
The following command requests that the qsales job is stopped:
stoperr = DSStopJob(qsales_handle)

DataStage Development Kit (Job Control Interfaces) DSTransformError
DSTransformError
Logs a warning message to a job log file. This function is called from
transforms only.
Syntax
Call DSTransformError (Message, TransformName)

calling transform.
TransformName (input) is the name of the transform that calls the
DSTransformError subroutine.
Remarks
DSTransformError writes the message (and other information) to
the job log file as a warning and returns to the transform. If the job has
a warning limit defined for it, when the number of warnings reaches
that limit, the call does not return and the job is aborted.
In addition to the warning message, DSTransformError logs the
values of all columns in the current rows for all input and output links
connected to the current stage.
Example
Function MySqrt(Arg1)
If Arg1 < 0 Then
Call DSTransformError("Negative value:"Arg1, "MySqrt")
Return("0") ;*transform produces 0 in this case
End
Result = Sqrt(Arg1) ;* else return the square root
Return(Result)

DSTranslateCode DataStage Development Kit (Job Control Interfaces)
DSTranslateCode
Converts a job control status or error code into an explanatory text
message.
Syntax
Ans = DSTranslateCode(Code)
Code is:
If Code > 0, it's assumed to be a job status.
If Code < 0, it's assumed to be an error code.
(0 should never be passed in, and will return "no error")
Ans is the message associated with the code.
Remarks
If Code is not recognized, then Ans will report it.
Example
code$ = DSGetLastErrorMsg()
ans$ = DSTranslateCode(code$)

DataStage Development Kit (Job Control Interfaces) DSWaitForFile
DSWaitForFile
Suspend a job until a named file either exists or does not exist.
Syntax
Reply = DSWaitForFile(Parameters)
Parameters is the full path of file to wait on. No check is made as to

whether this is a reasonable path (for example, whether all directories
in the path exist). A path name starting with "-", indicates a flag to
check the non-existence of the path. It is not part of the path name.
Parameters may also end in the form " timeout:NNNN" (or
"timeout=NNNN") This indicates a non-default time to wait before
giving up. There are several possible formats, case-insensitive:
nnn number of seconds to wait (from now)
nnnS ditto
nnnM number of minutes to wait (from now)
nnnH number of hours to wait (from now)
nn:nn:nn wait until this time in 24HH:NN:SS. If this or nn:nn time
has passed, will wait till next day.
The default timeout is the same as "12H".
The format may optionally terminate "/nn", indicating a poll delay
time in seconds. If omitted, a default poll time is used.
Reply may be:
DSJE.NOERROR (0) OK - file now exists or does not exist,
depending on flag.
DSJE.BADTIME Unrecognized Timeout format
DSJE.NOFILEPATH File path missing
DSJE.TIMEOUT Waited too long
Examples
Reply = DSWaitForFile("C:\ftp\incoming.txt timeout:2H")
(wait 7200 seconds for file on C: to exist before it gives up.)

Reply = DSWaitForFile("-incoming.txt timeout=15:00")
(wait until 3 pm for file in local directory to NOT exist.)

Reply = DSWaitForFile("incoming.txt timeout:3600/60")
(wait 1 hour for a local file to exist, looking once a minute.)

DSWaitForJob DataStage Development Kit (Job Control Interfaces)
DSWaitForJob
This function is only valid if the current job has issued a DSRunJob
on the given JobHandle(s). It returns if the/a job has started since the
last DSRunJob has since finished.
Syntax
ErrCode = DSWaitForJob (JobHandle)
JobHandle is the string returned from DSAttachJob. If commas are

contained, it's a comma-delimited set of job handles, representing a
list of jobs that are all to be waited for.
ErrCode is 0 if no error, else possible error values (<0) are:
DSJE.WRONGJOB Job for this JobHandle was not run from
within this job.
ErrCode is >0 => handle of the job that finished from a multi-job wait.
Remarks
DSWaitForJob will wait for either a single job or multiple jobs.
Example
To wait for the return of the qsales job:
WaitErr = DSWaitForJob(qsales_handle)

DataStage Development Kit (Job Control Interfaces) Job Status Macros
Job Status Macros

A number of macros are provided in the JOBCONTROL.H file to
facilitate getting information about the current job, and links and
stages belonging to the current job. These macros provide the
functionality of using the DataStage BASIC DSGetProjectInfo,
DSGetJobInfo, DSGetStageInfo, and DSGetLinkInfo functions
with the DSJ.ME token as the JobHandle and can be used in all active
stages and before/after subroutines. The macros provide the
functionality for all the possible InfoType arguments for the
DSGet…Info functions.
The available macros are:
DSHostName
DSProjectName
DSJobStatus
DSJobName
DSJobController
DSJobStartDate
DSJobStartTime
DSJobWaveNo
DSJobInvocations
DSJobInvocationID
DSStageName
DSStageLastErr
DSStageType
DSStageInRowNum
DSStageVarList
DSLinkRowCount
DSLinkLastErr
DSLinkName
For example, to obtain the name of the current job:
MyName = DSJobName
To obtain the full current stage name:

MyName = DSJobName : "." : DSStageName
In addition, the following macros are provided to manipulate

Transformer stage variables:

Command Line Interface DataStage Development Kit (Job Control Interfaces)
DSGetVar(VarName) returns the current value of the named

stage variable. If the current stage does not have a stage variable
called VarName, then "" is returned and an error message is
logged. If the named stage variable is defined but has not been
initialized, the "" is returned and an error message is logged.
DSSetVar(VarName, VarValue) sets the value of the named stage
variable. If the current stage does not have a stage variable called
VarName, then an error message is logged.
Command Line Interface

The DataStage CLI comprises three groups of commands, those
concerned with running DataStage Jobs, those concerned with
administering DataStage projects, and those concerned with
searching jobs.
Commands for Controlling DataStage Jobs

These command options give you access to the same functionality as
the DataStage API functions described on page 7-4 or the BASIC
functions described on page 7-87. There is a single command, dsjob,
with a large range of options. These options are described in the
following topics:
The logon clause
Starting a job
Stopping a job
Listing projects, jobs, stages, links, and parameters
Setting an alias for a job
Retrieving information
Accessing log files
Importing job executables
Generating a report
All output from the dsjob command is in plain text without column
headings on lists, or any other sort of description. This enables the
command to be used in shell or batch scripts without extra
processing.
The DataStage CLI returns a completion code of 0 to the operating
system upon successful execution, or one of the DataStage API error
codes on failure. See "Error Codes" on page 7-80. The return code is

DataStage Development Kit (Job Control Interfaces) Command Line Interface
also printed to the standard error stream in all cases. On UNIX

servers, a code of 255 is returned if the error code is negative or
greater than 254, to see the “real” return code in these cases, capture
and process the standard error stream.
The Logon Clause

By default, the DataStage CLI connects to the DataStage server engine
on the local system using the user name and password of the user
invoking the command. You can specify a different server, user name,
or password using the logon clause, which is equivalent to the API
DSSetServerParams function. Its syntax is as follows:
[ –server servername ][ –user username ][ –password password ]
servername specifies a different server to log on to.
username specifies a different user name to use when logging on.
password specifies a different password to use when logging on.
You can also specify these details in a file using the following syntax:
[ –file filename servername ]
servername specifies the server for which the file contains login
details.
filename is the name of the file containing login details. The file
should contain the following information:
servername, username, password
You can use the logon clause with any dsjob command.
Starting a Job
You can start, stop, validate, and reset jobs using the –run option.
dsjob –run
[ –mode [ NORMAL | RESET | VALIDATE ] ]
[ –param name=value ]
[ –warn n ]
[ –rows n ]
[ –wait ]
[ –stop ]
[ –jobstatus]
[–userstatus]
[–local]
[–opmetadata [TRUE | FALSE]]
[-disableprjhandler]

[-disablejobhandler]
[useid] project job|job_id
–mode specifies the type of job run. NORMAL starts a job run, RESET
resets the job and VALIDATE validates the job. If –mode is not
specified, a normal job run is started.
–param specifies a parameter value to pass to the job. The value is in
the format name=value, where name is the parameter name, and
value is the value to be set. If you use this to pass a value of an
environment variable for a job (as you may do for parallel jobs), you
need to quote the environment variable and its value, for example -
param '$APT_CONFIG_FILE=chris.apt' otherwise the current
value of the environment variable will be used.
–warn n sets warning limits to the value specified by n (equivalent to
the DSSetJobLimit function used with DSJ_LIMITWARN specified as
the LimitType parameter).
–rows n sets row limits to the value specified by n (equivalent to the
DSSetJobLimit function used with DSJ_LIMITROWS specified as the
LimitType parameter).
–wait waits for the job to complete (equivalent to the DSWaitForJob
function).
–stop terminates a running job (equivalent to the DSStopJob
function).
–jobstatus waits for the job to complete, then returns an exit code
derived from the job status.
–userstatus waits for the job to complete, then returns an exit code
derived from the user status if that status is defined. The user status is
a string, and it is converted to an integer exit code. The exit code 0
indicates that the job completed without an error, but that the user
status string could not be converted. If a job returns a negative user
status value, it is interpreted as an error.
-local use this when running a DataStage job from withing a
shellscript on a UNIX server. Provided the script is run in the project
directory, the job will pick up the settings for any environment
variables set in the script and any setting specific to the user
environment.
-opmetadata use this to have the job generate operational meta data
as it runs. If MetaStage, or the Process Meta Data MetaBroker, is not
installed on the machine, then the option has no effect. If you specify
TRUE, operational meta data is generated, whatever the default
setting for the project. If you specify FALSE, the job will not generate
operational meta data, whatever the default setting for the project.

-disableprjhandler use this to disable any error message handler

that has been set on a project wide basis (see "Managing Message
Handlers," in DataStage Manager Guide for a description of message
handlers).
-disablejobhandler use this to disable any error message handler
that has been set for this job (see "Managing Message Handlers," in
DataStage Manager Guide for a description of message handlers).
useid specify this if you intend to use a job alias (jobid) rather than a
job name (job) to identify the job.
project is the name of the project containing the job.
job is the name of the job. To run a job invocation, use the format
job.invocation_id.
job_id is an alias for the job that has been set using the dsjob -jobid
command (see page 7-145).
Stopping a Job
You can stop a job using the –stop option.
dsjob –stop [useid] project job|job_id
–stop terminates a running job (equivalent to the DSStopJob

function).
project is the name of the project containing the job.
job is the name of the job. To stop a job invocation, use the format
job.invocation_id.
command (see page 7-145)
Listing Projects, Jobs, Stages, Links, and Parameters

You can list projects, jobs, stages, links, and job parameters using the
dsjob command. The different versions of the syntax are described in
the following sections.
Listing Projects
The following syntax displays a list of all known projects on the
server:
dsjob –lprojects
This syntax is equivalent to the DSGetProjectList function.

Listing Jobs
The following syntax displays a list of all jobs in the specified project:
dsjob –ljobs project
project is the name of the project containing the jobs to list.

This syntax is equivalent to the DSGetProjectInfo function.
Listing Stages
The following syntax displays a list of all stages in a job:
dsjob –lstages [useid] project job|job_id
project is the name of the project containing job.
job is the name of the job containing the stages to list. To identify a
job invocation, use the format job.invocation_id.
command (see page 7-145)
This syntax is equivalent to the DSGetJobInfo function with
DSJ_STAGELIST specified as the InfoType parameter.
Listing Links
The following syntax displays a list of all the links to or from a stage:
dsjob –llinks [useid] project job|job_id stage
job is the name of the job containing stage. To identify a job
invocation, use the format job.invocation_id.
stage is the name of the stage containing the links to list.
This syntax is equivalent to the DSGetStageInfo function with
DSJ_LINKLIST specified as the InfoType parameter.
Listing Parameters
The following syntax display a list of all the parameters in a job and
their values:
dsjob –lparams [useid] project job|job_id

job is the name of the job whose parameters are to be listed. To
identify a job invocation, use the format job.invocation_id.
This syntax is equivalent to the DSGetJobInfo function with
DSJ_PARAMLIST specified as the InfoType parameter.
Listing Invocations
The following syntax displays a list of the invocations of a job:
dsjob -linvocations
Setting an Alias for a Job

The dsjob command can be used to specify your own ID for a
DataStage job. Other commands can then use that alias to refer to the
job.
dsjob –jobid [my_ID] project job
my_ID is the alias you want to set for the job. If you omit my_ID, the
command will return the current alias for the specified job. An alias
must be unique within the project, if the alias already exists an error
message is displayed
job is the name of the job. To identify a job invocation, use the format
job.invocation_id.
Retrieving Information
The dsjob command can be used to retrieve and display the available
information about specific projects, jobs, stages, or links. The different
versions of the syntax are described in the following sections.
Displaying Job Information

The following syntax displays the available information about a
specified job:
dsjob –jobinfo [useid] project job|job_id

job is the name of the job. To identify a job invocation, use the format
job.invocation_id.
The following information is displayed:
The current status of the job
The name of any controlling job for the job
The date and time when the job started
The wave number of the last or current run (internal DataStage
reference number)
User status
This syntax is equivalent to the DSGetJobInfo function.
Displaying Stage Information

The following syntax displays all the available information about a
stage:
dsjob –stageinfo [useid] project job|job_id stage
stage is the name of the stage.
The last error message reported from any link to or from the stage
The stage type name, for example, Transformer or Aggregator
The primary links input row number
This syntax is equivalent to the DSGetStageInfo function.
Displaying Link Information

The following syntax displays information about a specified link to or
from a stage:
dsjob –linkinfo [useid] project job|job_id stage link

stage is the name of the stage containing link.
link is the name of the stage.
The last error message reported by the link
The number of rows that have passed down a link
This syntax is equivalent to the DSGetLinkInfo function.
Displaying Parameter Information

This syntax displays information about the specified parameter:
dsjob –paraminfo [useid] project job|job_id param
job is the name of the job containing parameter. To identify a job
parameter is the name of the parameter.
The parameter type
The parameter value
Help text for the parameter that was provided by the job’s
designer
Whether the value should be prompted for
The default value that was specified by the job’s designer
Any list of values
The list of values provided by the job’s designer
This syntax is equivalent to the DSGetParamInfo function.

Accessing Log Files

The dsjob command can be used to add entries to a job’s log file, or
retrieve and display specific log entries. The different versions of the
syntax are described in the following sections.
Adding a Log Entry

The following syntax adds an entry to the specified log file. The text
for the entry is taken from standard input to the terminal, ending with
Ctrl-D.
dsjob –log [ –info | –warn ] [useid] project job|job_id
–info specifies an information message. This is the default if no log

entry type is specified.
–warn specifies a warning message.
job is the name of the job that the log entry refers to. To identify a job
This syntax is equivalent to the DSLogEvent function.
Displaying a Short Log Entry

The following syntax displays a summary of entries in a job log file:
dsjob –logsum [–type type] [ –max n ] [useid] project job|job_id
–type type specifies the type of log entry to retrieve. If –type type is
not specified, all the entries are retrieved. type can be one of the
following options:
This Retrieves this type of log entry…

option…
INFO Information.
WARNING Warning.
FATAL Fatal error.
REJECT Rejected rows from a Transformer stage.
STARTED All control logs.
RESET Job reset.

This Retrieves this type of log entry…

option…
BATCH Batch control.
ANY All entries of any type. This is the default if type is not specified.
–max n limits the number of entries retrieved to n.

project is the project containing job.
job is the job whose log entries are to be retrieved. To identify a job
Displaying a Specific Log Entry

The following syntax displays the specified entry in a job log file:
dsjob –logdetail [useid] project job|job_id entry
entry is the event number assigned to the entry. The first entry in the
file is 0.
This syntax is equivalent to the DSGetLogEntry function.
Identifying the Newest Entry

The following syntax displays the ID of the newest log entry of the
specified type:
dsjob –lognewest [useid] project job|job_id type

type can be one of the following options:
This option… Retrieves this type of log entry…

INFO Information
WARNING Warning
FATAL Fatal error
REJECT Rejected rows from a Transformer stage
STARTED Job started
RESET Job reset
BATCH Batch
This syntax is equivalent to the DSGetNewestLogId function.
Importing Job Executables

The dsjob command can be used to import job executables from a
DSX file into a specified project. Note that this command is only
available on UNIX servers.
dsjob –import project DSXfilename [-OVERWRITE] [-JOB[S] jobname …] |
[-LIST]
project is the project to import into.

DSXfilename is the DSX file containing the job executables.
-OVERWRITE specifies that any existing jobs in the project with the
same name will be overwritten.
-JOB[S] jobname specifies that one or more named job executables
should be imported (otherwise all the executable in the DSX file are
imported).
-LIST causes DataStage to list the executables in a DSX file rather
than import them.
For details of how to export job executables to a DSX file see
DataStage Manager Guide.
Generating a Report
The dsjob command can be used to generate an XML format report
containing job, stage, and link information.
dsjob –report [useid] project job|jobid [report_type]

job specifies the job to be reported on by job name. To identify a job
report_type is one of the following:
BASIC – Text string containing start/end time, time elapsed and
status of job.
DETAIL – As basic report, but also contains information about
individual stages and links within the job.
LIST – Text string containing full XML report.
The generated report is written to stdout.

This syntax is equivalent to the DSMakeJobReport
function.DETAIL – As basic report, but also contains information
LIST – Text string containing full XML report.
Commands for Administering DataStage

There is a single command, dsadmin, with a large range of options.
These options are described in the following topics:
The logon clause
Creating a project
Deleting a project
Enabling/Disabling the display of generated OSH in parallel jobs.
Enabling/Disabling runtime column propagation in parallel jobs.
Enabling/Disabling the availability of job administartion features
in the DataStage Director for a particular project.
Setting the advanced runtime options for parallel jobs.
Setting the base directory name for parallel jobs.

Setting the deployed job template directory for parallel jobs.

Setting custom deployment options for parallel jobs.
Creating a new environment variable.
Deleting an environment variable.
Setting the value of an environment variable
Listing projects on a server.
Listing project properties.
Listing environment variables.
The Logon Clause

details.
You can use the logon clause with any dsadmin command.
Creating a Project
The dsadmin command can be used for creating projects. You need to
have DataStage administrator status in order to use this command:
dsadmin -createproject ProjectName [-location ProjectLocation]
ProjectName is the name of the project.

-location ProjectLocation is the location of the project in the form of a
pathname.

If no location is specified, the project is created in the Projects

directory in the server install directory.
Deleting a Project
The dsadmin command can be used for deleting existing projects.
You need to have DataStage administrator status in order to use this
command:
dsadmin -deleteproject ProjectName
ProjectName is the project to be deleted.
Enabling/Disabling OSH Display

The dsadmin command can be used for enabling or disabling the
display of generated OSH in parallel jobs. You need to have DataStage
administrator status in order to use this command:
dsadmin -oshvisible TRUE | FALSE ProjectName
Note Although this command requires a project name, this

setting applies to ALL projects on the server
This command is only available for Enterprise Edition.
Enabling/Disabling Runtime Column Propagation

The dsadmin command can be used for enabling or disabling runtime
column propagation in parallel jobs in a particular project. You need to
dsadmin -enablercp TRUE | FALSE ProjectName
ProjectName is the project whose parallel jobs are to have runtime

column propagation enabled or disabled.
Enabling/Disabling Job Administration from the

Director
The dsadmin command can be used for enabling or disabling the job
administration features in the DataStage Director for jobs in a
particular project. You need to have DataStage administrator status in
order to use this command:
dsadmin -enablejobadmin TRUE | FALSE ProjectName
ProjectName is the project for which job administration in the Director

will be enabled or disabled.

Enabling/Disabling Generation of XML Report

This option is only relevant for Parallel jobs being compiled into a
deployment package. The deployment package can include a job
report in XML format, and this command enables or disables the
generation of this report.
dsadmin -enablegeneratexml TRUE | FALSE ProjectName
ProjectName is the project whose parallel jobs are to have XML

reports enabled or disabled.
Enabling/Disabling Advanced Runtime Properties

The dsadmin command can be used for setting advanced runtime
properties for parallel jobs in a particular project. You need to have
DataStage administrator status in order to use this command:
dsadmin -advancedruntime "AdvancedRuntimeOptions" ProjectName
ProjectName is the project whose parallel jobs will have the specified
advanced runtime options set.
AdvancedRuntimeOptions is the value to set the property to and must
be quoted.
To unset the properties repeat the command with an empty string, for
example:
dsadmin -advancedruntime "“ myproject
Setting the Base Directory

The dsadmin command can be used for setting the base directory for
parallel jobs in a particular project. You need to have DataStage
administrator status in order to use this command:
dsadmin -basedirectory BaseDirectoryName ProjectName
ProjectName is the project whose parallel jobs the base directory is

being set for.
BaseDirectoryName is the value to set the property to.
Setting the Deployment Directory Template

The dsadmin command can be used for setting the deployment
directory template for parallel jobs in a particular project. You need to

dsadmin -deploymentdirectory DirectoryTemplate ProjectName
ProjectName is the project whose parallel jobs are having the

deployment directory template defined.
DirectoryTemplate is the value to set the property to.
Setting Custom Deployment Options

The dsadmin command can be used for setting the custom
deployment options for parallel jobs in a particular project. You need
to have DataStage administrator status in order to use this command:
dsadmin -customdeployment “Commands” ProjectName
ProjectName is the project whose parallel jobs the custom

deployment options are being set for.
Commands is the value to set the property to and must be quoted.
Adding an Environment Variable

The dsadmin command can be used for creating a new environment
variable in a particular project. The environment variable is added to
the “User Defined” category
dsadmin -envadd EnvVarName -type STRING | ENCRYPTED
-prompt "PromptText" [-value "Value"] ProjectName
EnvVarName is the name of the environment variable being created.

-type specified the type of the environment variable and should be set
to either STRING or ENCRYPTED.
-prompt PromptText is the prompt to be associated with this
environment value The PromptText must be quoted as it can contain
spaces
-value Value is the value for the new environment variable. Value
must be quoted. If this is not given, the value for the environment
variable will need to be set using the dsadmin –envset command.
ProjectName is the project to which the environment variable is being
added.
Deleting an Environment Variable

The dsadmin command can be used for deleting an environment
variable in a particular project. It is not possible to delete a built-in
environment variables.

dsadmin -envdelete EnvVarName ProjectName
EnvVarName is the environment variable being deleted.

ProjectName is the project the environment variable is being deleted
from.
Setting the Value of an Environment Variable

The dsadmin command can be used for setting the value of an
environment variable in a particular project. If setting a list type
environment variable (for example, APT_EXECUTION _MODE) , then
you should set it to one of the permissable internal values, rather than
one of the list members as they are shown in the DataStage
Administrator client. For example, if you wanted to set
APT_EXECUTION_MODE so that parallel jobs executed in one process
mode, you would set the environment variable value to
‘ONE_PROCESS’, not ‘One process’ as offered in the Administrator
client. Internal values are given in "Environment Variables," in the
Parallel Job Advanced Developer’s Guide.
If you are setting a boolean type environment variable, set the value
to 1 for TRUE and 0 for FALSE.
If you are using $ENV to set the value of an environment variable to its
current setting in the environment, then you should use single quotes
to ensure that it picks up the correct value (e.g., dsadmin -envset NEW3
-value '$ENV' dstage).
dsadmin -envset EnvVarName -value "Value" ProjectName
EnvVarName is the environment variable whose value is being set.

-value “Value” is the value for the environment variable and must be
quoted.
ProjectName is the project for which the environment variable is
being set.
Listing Projects
The dsadmin command can be used for listing the projects on a
server.
dsadmin -listprojects
Lists all the projects on the server.
Listing Properties
The dsadmin command can be used for listing the properties of a
project. The following properties are listed:

Whether generated OSH is visible in parallel jobs.

Whether runtime column propagation is enabled in parallel jobs.
The base directory name for parallel jobs.
Advanced runtime options for parallel jobs.
Custom deployment commands for parallel jobs.
Deployed job directory template.
Whether job administration is enabled in the DataStage Director
or not.
The parallel job properties will only be listed if Enterprise Edition is
enabled.
dsadmin -listproperties ProjectName
ProjectName is the project for which the properties are to be listed.
Listing Environment Variables

The dsadmin command can be used for listing the environment
variables in a project.
dsadmin -listenv ProjectName
Commands for Searching Jobs

There is a single command, dssearch, that allows you to search for
jobs and usage information about jobs. The command has a number
of options described in the following sections:
The logon clause.
Search for jobs whose name match, or partially match, the
supplied text.
Search for jobs used by job sequences.
Search for job sequences that use the specified job.
Search for jobs that use the specified database table in a DRS
stage.
Note The dsjob command has an option for listing the jobs in a
project, but this is aimed at jobs that are compiled and
ready to run (or reset and run). The dssearch command
searchs design time information to allow you to search jobs
currently under development.

The Logon Clause

details.
You can use the logon clause with any dssearch command.
Search for Jobs by Name

The dssearch command can be used for searching for a job by name.
dssearch -ljobs -matches [-sub[categories]
[-c[ategory] CategoryName]
[-j[obtype] s[erver] |p[arallel] |
m[ainframe] | seq[uence] ]
[-oc[ategory]]
[-oj[obtype]]
ProjectName [JobName]
JobName is the full or partial name of the job to search for. For partial
job name matching, any number of * characters can be used for
missing parts using wildcard matching. If the job name is not
supplied, then all jobs will be returned.
ProjectName the project containing the job specified by JobName.
-ljobs indicates that jobs are to be searched.
-matches indicates that jobs are to be searched by name.
- sub[categories] optionally indicates that the the supplied starting
category and all subcategories will be searched. When no category is
specified and the subcategories option is specified, the whole project
is searched.

- c[ategory] CategoryName optionally restricts the search to the

named category.
- j[obtype] s[erver] |p[arallel] |m[ainframe] | seq[uence] optionally
restricts the search by type of job. You can specify multiple jobtypes
by separating the types with a single space, e.g., -jobtype server
parallel.
- oc[ategory] specify this to have all the jobs found in the full
category path returned.
-oj[obtype] specify this to have the jobtype, as well as the job name,
reported.
For example:
dssearch –ljobs –matches –subcategories sub –oc –oj dstage Job*
returns all the jobs in the project called dstage project that start with
the text “Job”. The order of the returned jobs is based on category
order, and then job type within the category. An example result might
look like:
Job1,server job
Job2,sequence job
Category1\Job3,server job
Category1\Job4,server job
Category1\SubCategory1\Job5,sequence job
Category2\SubCategory2\Job6,server job
If –ocategory is defined, the job’s category appears on the line before

the job name. If –ojobtype is defined, the job type appears after the
job name with a comma separating it from the job name.
Search for Job Sequences that Use Specified Job

Use this option to search for job sequences which use the specified
job. In this context, a job sequence can also include a job with job
control code that runs other jobs, provided the other jobs have been
specified as dependencies (see "Job Control Routines" in DataStage
Designer Guide).
dssearch –ljobs –usedby [-r[ecursive]]
[-oc[ategory]]
[-oj[obtype]]
ProjectName JobName
JobName is the full name of the job whose inclusion in job sequences
you are searching. Remember, job names are unique throughout the
whole project, so do not have to be qualified by category.
ProjectName the project containing the job specified by JobName.

–usedby finds where the supplied job is used. In other words it finds
all jobs that have the supplied job in its dependency list.
-recursive takes each sequence job that the supplied job is in, and
recursively searches for sequence jobs that in turn contain them.
-ocategory specifies that, for each job found, the full category path
should be output.
-ojobtype specifies that, for each job found, its job type is output after
a comma.
Each sequence job found is output on a new line. When the –
recursive option is supplied, the tab character is used to show the
tree of jobs being used by other jobs. When the –ojobtype option is
supplied, the job type is output after each job name, separated from
the name by a comma.
For example:
dssearch –ljobs –usedby –r –oc –oj dstage JobE
An example output from this command-line shows that JobE is used

in JobD and JobH. JobD is used in JobB. JobB is used in
ContainingJob:
CategoryA\CategoryB\JobD,sequence job
CategoryA\JobB,sequence job
ContainingJob,sequence job
CategoryC\JobH,server job
Search for Jobs used in a Specified Job Sequence

Use this option to search for jobs which are used by a specified job
sequence. In this context, a job sequence can also include a job with
job control code that runs other jobs, provided the other jobs have
been specified as dependencies (see "Job Control Routines" in
DataStage Designer Guide).
dssearch –ljobs –uses [-r[ecursive]]
[-oc[ategory]]
[-oj[obtype]]
ProjectName JobName

JobName is the full name of the sequence job whose jobs our are
searching for. Partial job name matches are not allowed. Remember,
job names are unique throughout the whole project, so do not have to
be qualified by category.
ProjectName the project containing the sequence job specified by
JobName.
-recursive specifies that, if any of the jobs found are themselves
sequence job, these will be searched as well.

will be output.
-ojobtype specifies that, for each job found, its job type should be
output.
Each job found is output on a new line. When the –recursive option is
supplied, the tab character is used to show the tree of jobs using other
jobs. When the –ojobtype option is supplied, the job type is output
after each job name, separated from the name by a comma.
For example:
dssearch –ljobs –uses -recursive –ocategory dstage ContainingJob
An example output from this command shows that JobA and JobB
jobs are used by the sequence ContainingJob. The jobs named JobC
and JobD are used by the job named JobB. JobE is a job that is used
by JobD.
CategoryA\JobA
CategoryA\JobB
CategoryA\CategoryB\JobC
CategoryA\CategoryB\JobD
CategoryA\JobE
Search for Job Using Specified Table Definition

Use this option to search for jobs that use the specified table
definition.
dssearch –ljobs –usesdrstable
[-s[ource] | -t[arget]]
[-oc[ategory]]
[-oj[obtype]]
ProjectName TableName
TableName is the full or partial name of the table to search for. For
partial table name matching, any number of * characters can be used
for missing parts using wildcard matching. All jobs containing a DRS
stage that accesses this table will be returned by the search.
ProjectName is the project containing the jobs being searched.
-usesdrstable indicates that jobs are to be searched for DSR stages
containing the table.
-source specify this if you only want jobs returned that use the table
as a source
-target specify this if you only want jobs returned that use the table
as a target.
will be output.

XML Schemas and Sample Stylesheets DataStage Development Kit (Job Control Interfaces)
-ojobtype specifies that, for each job found, its job type should be
output.
For example:
dssearch –ljobs –usesdrstable -source dstage customers
returns a list of jobs containing DRS stages that use the customers
table as a data source.
dssearch –ljobs –usesdrstable –s –oc –oj dstage *.Person
This command-line returns all the jobs, with category and job type,
containing DRS Stages with a source table whose name ends in
“.Person”. A possible result could be:
CategoryA\DRSJobA,server job
CategoryB\DRSJobB,server job
XML Schemas and Sample Stylesheets

You can generate an XML report giving information about a job using
the following methods:
DSMakeJobReport API function (see page 7-44)
DSMakeJobReport BASIC function (see page 7-124)
dsjob command (see page 7-150)
DataStage provides the following files to assist in the handling of
generated XML reports:
DSReportSchema.xsd. An XML schema document that fully
describes the structure of the XML job report documents.
DSmonitor.xsl. An example XSLT stylesheet that creates an html
web page similar to the Director Monitor view from the XML
report.
DSwaterfall.xsl. An example XSLT stylesheet that creates an
html web page showing a waterfall report describing how data
flowed between all the processes in the job from the XML report.
The files are all located in the DataStage client directory (\Program
Files\Ascential\DataStage).
You can embed a reference to a stylesheet when you create the report
using any of the commands listed above. Once the report is generated
you can view it in an Internet browser.
Alternatively you can use an xslt processor such as saxon or msxsl to
convert an already generated report. For example:
java - jar saxon.jar jobreport.xml DSMonitor.xsl > jobmonitor.htm

DataStage Development Kit (Job Control Interfaces) XML Schemas and Sample Stylesheets
Would generate an html file called jobmonitor.htm from the report

jobreport.xml, while:
maxsl jobreport.xml DSwaterfall.xsl > jobwaterfall.htm
Would generate an html file called jobwaterfall.htm from the report

jobreport.xml.

XML Schemas and Sample Stylesheets DataStage Development Kit (Job Control Interfaces)

A
Header Files
DataStage comes with a range of header files that you can include in
code when you are defining a Build stage. The following sections list
the header files and the classes and macros that they contain. See the
header files themselves for more details about available functionality.
C++ Classes – Sorted By Header File

apt_ framework/ accessorbase. h
APT_ AccessorBase
APT_ AccessorTarget
APT_ InputAccessorBase
APT_ InputAccessorInterface
APT_ OutputAccessorBase
APT_ OutputAccessorInterface
apt_ framework/ adapter. h
APT_ AdapterBase
APT_ ModifyAdapter
APT_ TransferAdapter
APT_ ViewAdapter
apt_ framework/ collector. h
APT_ Collector
apt_ framework/ composite. h
APT_ CompositeOperator
Parallel Job Advanced Developer’s Guide A-1

C++ Classes – Sorted By Header File Header Files
apt_ framework/ config. h

APT_ Config
APT_ Node
APT_ NodeResource
APT_ NodeSet
apt_ framework/ cursor. h
APT_ CursorBase
APT_ InputCursor
APT_ OutputCursor
apt_ framework/ dataset. h
APT_ DataSet
apt_ framework/ fieldsel. h
APT_ FieldSelector
apt_ framework/ fifocon. h
APT_ FifoConnection
apt_ framework/ gsubproc. h
APT_ GeneralSubprocessConnection
APT_ GeneralSubprocessOperator
apt_ framework/ impexp/ impexp_ function. h
APT_ GFImportExport
apt_ framework/ operator. h
APT_ Operator
apt_ framework/ partitioner. h
APT_ Partitioner
APT_ RawField
apt_ framework/ schema. h
APT_ Schema
APT_ SchemaAggregate
APT_ SchemaField
APT_ SchemaFieldList
APT_ SchemaLengthSpec
apt_ framework/ step. h
APT_ Step
A-2 Parallel Job Advanced Developer’s Guide

Header Files C++ Classes – Sorted By Header File
apt_ framework/ subcursor. h

APT_ InputSubCursor
APT_ OutputSubCursor
APT_ SubCursorBase
apt_ framework/ tagaccessor. h
APT_ InputTagAccessor
APT_ OutputTagAccessor
APT_ ScopeAccessorTarget
APT_ TagAccessor
apt_ framework/ type/ basic/ float. h
APT_ InputAccessorToDFloat
APT_ InputAccessorToSFloat
APT_ OutputAccessorToDFloat
APT_ OutputAccessorToSFloat
apt_ framework/ type/ basic/ integer. h
APT_ InputAccessorToInt16
APT_ InputAccessorToUInt16
APT_ OutputAccessorToInt16
APT_ OutputAccessorToUInt16
apt_ framework/ type/ basic/ raw. h
APT_ InputAccessorToRawField
APT_ OutputAccessorToRawField
APT_ RawFieldDescriptor
apt_ framework/ type/ conversion. h
APT_ FieldConversion
APT_ FieldConversionRegistry

C++ Classes – Sorted By Header File Header Files
apt_ framework/ type/ date/ date. h

APT_ DateDescriptor
APT_ InputAccessorToDate
APT_ OutputAccessorToDate
apt_ framework/ type/ decimal/ decimal. h
APT_ DecimalDescriptor
APT_ InputAccessorToDecimal
APT_ OutputAccessorToDecimal
apt_ framework/ type/ descriptor. h
APT_ FieldTypeDescriptor
APT_ FieldTypeRegistry
apt_ framework/ type/ function. h
APT_ GenericFunction
APT_ GenericFunctionRegistry
APT_ GFComparison
APT_ GFEquality
APT_ GFPrint
apt_ framework/ type/ protocol. h
APT_ BaseOffsetFieldProtocol
APT_ EmbeddedFieldProtocol
APT_ FieldProtocol
APT_ PrefixedFieldProtocol
APT_ TraversedFieldProtocol
apt_ framework/ type/ time/ time. h
APT_ TimeDescriptor
APT_ InputAccessorToTime
APT_ OutputAccessorToTime
apt_ framework/ type/ timestamp/ timestamp. h
APT_ TimeStampDescriptor
APT_ InputAccessorToTimeStamp
APT_ OutputAccessorToTimeStamp
apt_ framework/ utils/fieldlist. h
APT_ FieldList
apt_ util/archive. h
APT_ Archive
APT_ FileArchive
APT_ MemoryArchive

Header Files C++ Classes – Sorted By Header File
apt_ util/ argvcheck. h

APT_ ArgvProcessor
apt_util/basicstring.h
APT_BasicString
apt_ util/ date. h
APT_ Date
apt_ util/ dbinterface. h
APT_ DataBaseDriver
APT_ DataBaseSource
APT_ DBColumnDescriptor
apt_ util/ decimal. h
APT_ Decimal
apt_ util/ endian. h
APT_ ByteOrder
apt_ util/ env_ flag. h
APT_ EnvironmentFlag
apt_ util/ errind. h
APT_ Error
apt_ util/ errlog. h
APT_ ErrorLog
apt_ util/ errorconfig. h
APT_ ErrorConfiguration
apt_ util/ fast_ alloc. h
APT_ FixedSizeAllocator
APT_ VariableSizeAllocator
apt_ util/ fileset. h
APT_ FileSet
apt_ util/ identifier. h
APT_ Identifier
apt_ util/keygroup. h
APT_ KeyGroup

C++ Macros – Sorted By Header File Header Files
apt_ util/locator. h
APT_ Locator
apt_ util/persist. h
APT_ Persistent
apt_ util/proplist. h
APT_ Property
APT_ PropertyList
apt_ util/random. h
APT_ RandomNumberGenerator
apt_ util/rtti. h
APT_ TypeInfo
apt_ util/ string. h
APT_ String
APT_ StringAccum
apt_ util/ time. h
APT_ Time
APT_ TimeStamp
apt_util/ustring.h
APT_UString
C++ Macros – Sorted By Header File

apt_ framework/ accessorbase. h
APT_ DECLARE_ ACCESSORS()
APT_ IMPLEMENT_ ACCESSORS()
apt_ framework/ osh_ name. h
APT_ DEFINE_ OSH_ NAME()
APT_ REGISTER_ OPERATOR()
apt_ framework/ type/ basic/ conversions_ default. h
APT_ DECLARE_ DEFAULT_ CONVERSION()
APT_ DECLARE_ DEFAULT_ CONVERSION_ WARN()
apt_ framework/ type/ protocol. h
APT_ OFFSET_ OF()

Header Files C++ Macros – Sorted By Header File
apt_ util/ archive. h

APT_ DIRECTIONAL_ SERIALIZATION()
apt_ util/assert. h
APT_ ASSERT()
APT_ DETAIL_ FATAL()
APT_ DETAIL_ FATAL_ LONG()
APT_ MSG_ ASSERT()
APT_ USER_ REQUIRE()
APT_ USER_ REQUIRE_ LONG()
apt_ util/condition. h
CONST_ CAST()
REINTERPRET_ CAST()
apt_ util/errlog. h
APT_ APPEND_ LOG()
APT_ DUMP_ LOG()
APT_ PREPEND_ LOG()
apt_ util/ exception. h
APT_ DECLARE_ EXCEPTION()
APT_ IMPLEMENT_ EXCEPTION()
apt_ util/ fast_ alloc. h
APT_ DECLARE_ NEW_ AND_ DELETE()
apt_ util/ logmsg. h
APT_ DETAIL_ LOGMSG()
APT_ DETAIL_ LOGMSG_ LONG()
APT_ DETAIL_ LOGMSG_ VERYLONG()
apt_ util/ persist. h
APT_ DECLARE_ ABSTRACT_ PERSISTENT()
APT_ DECLARE_ PERSISTENT()
APT_ DIRECTIONAL_ POINTER_ SERIALIZATION()
APT_ IMPLEMENT_ ABSTRACT_ PERSISTENT()
APT_ IMPLEMENT_ ABSTRACT_ PERSISTENT_ V()
APT_ IMPLEMENT_ NESTED_ PERSISTENT()
APT_ IMPLEMENT_ PERSISTENT()
APT_ IMPLEMENT_ PERSISTENT_ V()
apt_ util/ rtti. h
APT_ DECLARE_ RTTI()
APT_ DYNAMIC_ TYPE()

C++ Macros – Sorted By Header File Header Files
APT_ IMPLEMENT_ RTTI_ BASE()

APT_ IMPLEMENT_ RTTI_ BEGIN()
APT_ IMPLEMENT_ RTTI_ END()
APT_ IMPLEMENT_ RTTI_ NOBASE()
APT_ IMPLEMENT_ RTTI_ ONEBASE()
APT_ NAME_ FROM_ TYPE()
APT_ PTR_ CAST()
APT_ STATIC_ TYPE()
APT_ TYPE_ INFO()

Index
Symbols APT_IMPORT_PATTERN_USES_FILESET 6–28

_cplusplus token 7–2 APT_INPUT_CHARSET 6–24
_STDC_ token 7–2 APT_IO_MAP/APT_IO_NOMAP and
APT_BUFFERIO_MAP/
A APT_BUFFERIO_NOMAP 6–15
administration APT_IO_MAXIMUM_OUTSTANDING 6–22
C API functions 7–6 APT_IOMGR_CONNECT_ATTEMPTS 6–22
API 7–2 APT_LATENCY_COEFFICIENT 6–37
APT_AUTO_TRANSPORT_BLOCK_SIZE 6–36 APT_LINKER 6–9
APT_BUFFER_DISK_WRITE_INCREMENT 6–7, APT_LINKOPT 6–9
6–14 APT_MAX_TRANSPORT_BLOCK_SIZE/
APT_BUFFER_FREE_RUN 6–6 APT_MIN_TRANSPORT_BLOCK_SIZE 6–37
APT_BUFFER_MAXIMUM_TIMEOUT 6–7 APT_MONITOR_SIZE 6–18
APT_BUFFERING_POLICY 6–7 APT_MONITOR_TIME 6–18
APT_CHECKPOINT_DIR 6–15 APT_MSG_FILELINE 6–30
APT_CLOBBER_OUTPUT 6–16 APT_NO_PART_INSERTION 6–26
APT_COLLATION_STRENGTH 6–23 APT_NO_SAS_TRANSFORMS 6–34
APT_COMPILEOPT 6–9 APT_NO_SORT_INSERTION 6–35
APT_COMPILER 6–9 APT_NO_STARTUP_SCRIPT 6–17
APT_CONFIG_FILE 6–16 APT_OLD_BOUNDED_LENGTH 6–19
APT_CONSISTENT_BUFFERIO_SIZE 6–14 APT_OPERATOR_REGISTRY_PATH 6–20
APT_DATE_CENTURY_BREAK_YEAR 6–19 APT_ORA_IGNORE_CONFIG_FILE_PARALLELIS
APT_DB2INSTANCE_HOME 6–10 M 6–26
APT_DB2READ_LOCK_TABLE 6–10 APT_ORA_WRITE_FILES 6–26
APT_DBNAME 6–10 APT_ORACLE_NO_OPS 6–25
APT_DEBUG_OPERATOR 6–11 APT_ORACLE_PRESERVE_BLANKS 6–25
APT_DEBUG_PARTITION 6–11 APT_ORAUPSERT_COMMIT_ROW_INTERVAL 6
APT_DEBUG_STEP 6–12 –26
APT_DEFAULT_TRANSPORT_BLOCK_SIZE 6–3 APT_ORAUPSERT_COMMIT_TIME_INTERVAL 6
7 –26
APT_DISABLE_COMBINATION 6–16 APT_ORCHHOME 6–17
APT_DUMP_SCORE 6–29 APT_OS_CHARSET 6–24
APT_ERROR_CONFIGURATION 6–29 APT_OUTPUT_CHARSET 6–24
APT_EXECUTION_MODE 6–12, 6–16 APT_PARTITION_COUNT 6–26
APT_FILE_EXPORT_BUFFER_SIZE 6–28 APT_PARTITION_NUMBER 6–27
APT_FILE_IMPORT_BUFFER_SIZE 6–27 APT_PM_CONDUCTOR_HOSTNAME 6–22
APT_IMPEXP_CHARSET 6–24 APT_PM_DBX 6–13
Parallel Job Advanced Developer’s Guide Index-1

Index
APT_PM_NO_NAMED_PIPES 6–20 description 7–64

APT_PM_NO_TCPIP 6–23 how used 7–2
APT_PM_PLAYER_MEMORY 6–30 summary of usage 7–62
APT_PM_PLAYER_TIMING 6–31 DataStage API
APT_PM_XLDB 6–13 building applications that use 7–4
APT_PM_XTERM 6–13 header file 7–2
APT_PREVIOUS_FINAL_DELIMITER_COMPATIB programming logic example 7–3
LE 6–28 redistributing programs 7–4
APT_RDBMS_COMMIT_ROWS 6–10 DataStage CLI
APT_RECORD_COUNTS 6–21, 6–31 completion codes 7–140
APT_SAS_ACCEPT_ERROR 6–32 logon clause 7–141, 7–152, 7–158
APT_SAS_CHARSET 6–32 overview 7–140
APT_SAS_CHARSET_ABORT 6–33 using to run jobs 7–141
APT_SAS_DEBUG 6–33 DataStage Development Kit 7–2
APT_SAS_DEBUG_IO 6–33 API functions 7–4
APT_SAS_DEBUG_LEVEL 6–33 command line interface 7–140
APT_SAS_NO_PSDS_USTRING 6–34 data structures 7–62
APT_SAS_S_ARGUMENT 6–34 dsjob command 7–141
APT_SAS_SCHEMASOURCE_DUMP 6–34 error codes 7–80
APT_SAVE_SCORE 6–21 job status macros 7–140
APT_SHOW_COMPONENT_CALLS 6–21 writing DataStage API programs 7–3
APT_STACK_TRACE 6–21 DataStage server engine 7–141, 7–152, 7–158
APT_STARTUP_SCRIPT 6–17 DB2DBDFT 6–10
APT_STARTUP_STATUS 6–18 DLLs 7–4
APT_STRING_CHARSET 6–24 documentation conventions iv
APT_STRING_PADCHAR 6–28 dsadmin command
APT_TERA_64K_BUFFERS 6–35 description 7–151
APT_TERA_NO_ERR_CLEANUP 6–35 dsapi.h header file
APT_TERA_NO_PERM_CHECKS 6–36 description 7–2
APT_TERA_NO_SQL_CONVERSION 6–35 including 7–4
APT_TERA_SYNC_DATABASE 6–36 DSCloseJob function 7–10
APT_TERA_SYNC_PASSWORD 6–36 DSCloseProject function 7–11
APT_TERA_SYNC_USER 6–36 DSCUSTINFO data structure 7–63
APT_THIN_SCORE 6–18 DSDetachJob function 7–92
APT_WRITE_DS_VERSION 6–21 dsdk directory 7–4
DSExecute subroutione 7–93
B DSFindFirstLogEntry function 7–14
batch log entries 7–149 DSFindNextLogEntry function 7–14, 7–16
build stage macros 5–21 DSGetCustInfo function 7–94
build stages 5–1 DSGetIPCPageProps function 7–95
DSGetJobInfo function 7–19, 7–96
C and controlled jobs 7–20
command line interface 7–140 DSGetJobMetaBag function 7–100
commands DSGetLastError function 7–21
dsadmin 7–151 DSGetLastErrorMsg function 7–22
dsjob 7–140 DSGetLinkInfo function 7–23, 7–101
custom stages 5–1 DSGetLinkMetaData function 7–103
DSGetLogEntry function 7–25, 7–104
D DSGetLogSummary function 7–105
data structures DSGetNewestLogId function 7–26, 7–107
DSGetParamInfo function 7–28, 7–108
Index-2 Parallel Job Advanced Developer’s Guide

Index
DSGetProjectInfo function 7–29, 7–111 and DSGetStageInfo 7–37

DSGetProjectList function 7–31 DSStageInRowNum macro 7–139
DSGetStageInfo function 7–36, 7–112 DSStageLastErr macro 7–139
DSGetStageLinks function 7–115 DSStageName macro 7–139
DSGetStagesOfType function 7–116 DSStageType macro 7–139
DSGetStageTypes function 7–117 DSStageVarList macro 7–139
DSGetVarInfo function 7–17, 7–38, 7–118 DSStopJob function 7–59, 7–134
DSHostName macro 7–139 DSTransformError function 7–135
dsjob command DSUnlockJob function 7–60
description 7–140 DSVARINFO data structure 7–79
DSJobController macro 7–139 DSWaitForJob function 7–61
DSJOBINFO data structure 7–64
and DSGetJobInfo 7–20 E
and DSGetLinkInfo 7–23 error codes 7–80
DSJobInvocationID macro 7–139 errors
DSJobInvocations macro 7–139 and DataStage API 7–80
DSJobName macro 7–139 functions used for handling 7–6
DSJobStartDate macro 7–139 retrieving message text 7–22
DSJobStartTime macro 7–139 retrieving values for 7–21
DSJobStatus macro 7–139 Event Type parameter 7–14
DSJobWaveNo macro 7–139 example build stage 5–25
DSLINKINFO data structure 7–67
DSLinkLastErr macro 7–139 F
DSLinkName macro 7–139 fatal error log entries 7–148
DSLinkRowCount macro 7–139 functions, table of 7–4
DSLockJob function 7–42
DSLOGDETAIL data structure 7–68 I
DSLOGEVENT data structure 7–69 information log entries 7–148
DSLogEvent function 7–43, 7–119
DSLogFatal function 7–120 J
DSLogInfo function 7–121 job control interface 7–1
DSLogWarn function 7–123 job handle 7–45
DSMakeJobReport function 7–44 job parameters
DSOpenJob function 7–45 displaying information about 7–147
DSOpenProject function 7–46 functions used for accessing 7–5
DSPARAM data structure 7–70 listing 7–144
DSPARAMINFO data structure 7–72 retrieving information about 7–28
and DSGetParamInfo 7–28 setting 7–54
DSPROJECTINFO data structure 7–74 job status macros 7–140
and DSGetProjectInfo 7–30 jobs
DSProjectName macro 7–139 closing 7–10
DSREPOSINFO data structure 7–75 displaying information about 7–145
DSREPOSUSAGE data structure 7–76 functions used for accessing 7–5
DSRunJob function 7–47 listing 7–29, 7–143
DSSetGenerateOpMetaData function 7–50, locking 7–42
7–130 opening 7–45
DSSetJobLimit function 7–50, 7–52, 7–130 resetting 7–47, 7–141
DSSetParam function 7–54, 7–132 retrieving status of 7–19
DSSetServerParams function 7–58 running 7–47, 7–141
DSSetUserStatus subroutine 7–133 stopping 7–59, 7–143
DSSTAGEINFO data structure 7–77

Index
unlocking 7–60 projects

validating 7–47, 7–141 closing 7–11
waiting for completion 7–61 functions used for accessing 7–4
listing 7–31, 7–143
L opening 7–46
library files 7–4 PT_DEBUG_SIGNALS 6–11
limits 7–52
links R
displaying information about 7–146 redistributable files 7–4
functions used for accessing 7–5 rejected rows 7–148
listing 7–143 result data
retrieving information about 7–23 reusing 7–2
log entries storing 7–2
adding 7–43, 7–148 row limits 7–52, 7–142
batch control 7–149
fatal error 7–148 S
finding newest 7–26, 7–149 server names, setting 7–58
functions used for accessing 7–5 stages
job reset 7–148 displaying information about 7–146
job started 7–148 functions used for accessing 7–5
new lines in 7–43 listing 7–143
rejected rows 7–148 retrieving information about 7–36
retrieving 7–14, 7–16
retrieving specific 7–25, 7–149 T
types of 7–14 threads
warning 7–148 and DSFindFirstLogEntry 7–16
logon clause 7–141, 7–152, 7–158 and DSFindNextLogEntry 7–16
and DSGetLastErrorMsg 7–22
M and error storage 7–3
macros, job status 7–140 and errors 7–21
and log entries 7–15
N and result data 7–2
new lines in log entries 7–43 using multiple 7–3
tokens
O _cplusplus 7–2
OSH_BUILDOP_CODE 6–8 _STDC_ 7–2
OSH_BUILDOP_HEADER 6–8 WIN32 7–2
OSH_BUILDOP_OBJECT 6–8
OSH_BUILDOP_XLC_BIN 6–8 U
OSH_CBUILDOP_XLC_BIN 6–9 user names, setting 7–58
OSH_DUMP 6–31
OSH_ECHO 6–31 V
OSH_EXPLAIN 6–31 vmdsapi.dll 7–4
OSH_PRELOAD_LIBS 6–22 vmdsapi.lib library 7–4
OSH_PRINT_SCHEMAS 6–31
OSH_STDOUT_MSG 6–22 W
warning limits 7–52, 7–142
P warnings 7–148
parameters, see job parameters WIN32 token 7–2
passwords, setting 7–58 wrapped stages 5–1

Index
writing
DataStage API programs 7–3

Index

Datastage Parallel Job Advanced Developers Guide

Uploaded by

Datastage Parallel Job Advanced Developers Guide

Uploaded by

Ascential DataStage

for Ascential DataStage™ Enterprise Edition

Parallel Job Advanced Developer’s

Part No. 00D-030DS751

Ascential DataStage™ is a powerful software suite that is used to

Organization of This Manual

Parallel Job Advanced Developer’s Guide iii

 Chapter 4 explains the link buffering used by parallel jobs in detail.

UPPERCASE In syntax, uppercase indicates BASIC statements and functions

Italic In syntax, italic indicates information that you supply. In text,

Plain In text, plain indicates Windows commands and options, file

[] Brackets enclose optional items. Do not type the brackets unless

{} Braces enclose nonoptional items from which you must select

➤ A right arrow between menu commands indicates you should

iv Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide v

vi Parallel Job Advanced Developer’s Guide

How to Use this Guide

Parallel Job Advanced Developer’s Guide vii

viii Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide ix

x Parallel Job Advanced Developer’s Guide

Job Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18

Parallel Job Advanced Developer’s Guide xi

NLS Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23

xii Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xiii

Teradata Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-35

xiv Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xv

xvi Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xvii

xviii Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xix

xx Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xxi

xxii Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide xxiii

xxiv Parallel Job Advanced Developer’s Guide

Parallel Job Advanced Developer’s Guide 1-1

1-2 Parallel Job Advanced Developer’s Guide

DataStage Designer Interface

Parallel Job Advanced Developer’s Guide 2-1

Processing Large Volumes of Data

2-2 Parallel Job Advanced Developer’s Guide

Designing for Good Performance

Avoid unnecessary type conversions.

Use Transformer stages sparingly and wisely

Increase Sort performance where possible

Parallel Job Advanced Developer’s Guide 2-3

Remove Unneeded Columns

Avoid reading from sequential files using the Same

2-4 Parallel Job Advanced Developer’s Guide

Default and Explicit Type Conversions

Parallel Job Advanced Developer’s Guide 2-5

conversion is required, a blank square indicates that conversion is not

2-6 Parallel Job Advanced Developer’s Guide

 The environment variable APT_STRING_PADCHAR can be used to

Using Transformer Stages

Parallel Job Advanced Developer’s Guide 2-7

Using Sequential File Stages

Using Database Stages

2-8 Parallel Job Advanced Developer’s Guide

management policies on the database. If you want to create a table on

Database Sparse Lookup vs. Join

DB2 Database Tips

Parallel Job Advanced Developer’s Guide 2-9

Write vs. Load

Oracle Database Tips

Loading and Indexes

2-10 Parallel Job Advanced Developer’s Guide

PARALLEL=FALSE). This allows the loading of indexed tables without

Teradata Database Tips

Parallel Job Advanced Developer’s Guide 2-11

2-12 Parallel Job Advanced Developer’s Guide

This chapter is intended to help resolve any performance problems. It

Chapter 4 explains the link buffering used by parallel jobs in detail.

The environment variable APT_STRING_PADCHAR can be used to

Enable the APT_DUMP_SCORE and APT_RECORD_COUNTS

When importing fixed-length data, the Number of Readers per

Throughput is preferable to overhead. The goal of the DataStage

AUTOMATIC_BUFFERING. Buffer a data set only if necessary to

APT_BUFFER_MAXIMUM_MEMORY. Specifies the maximum