Datastage - Job Sequence Invocation & Control
Datastage - Job Sequence Invocation & Control
Datastage - Job Sequence Invocation & Control
Options
Run through the DS client Director menu
Command line interface DSJob Commands
Used directly or within a OS shell or batch script
DSJob available with client installation
DataStage API
callable through a C/C++NotProgram
covered as part of this course
Distribute DLLs and header files to enable remote execution without a DS Client
Through other DataStage Executable Components - which have to be in turn invoked
through any of the listed means
DataStage BASIC Job Control
Written in DataStage Basic Script
Embedded as a Job Control script within the job definition OR
Not covered as part of this course
Called as a Server Routine OR
In Parallel Jobs, wrapped into a Basis Transform Stage
Invoked as an activity within a Sequence Job
which has to be in turn invoked through any of the listed means
August 7, 2021 2
Command Line Interface
Use dsjob for controlling DataStage jobs
Options available are:
Logon
Starting a job
Stopping a job
Listing projects, jobs, stages, links and parameters
Setting an alias for job
Retrieving information
Accessing log files
Importing job executables
Generating a report
CLI commands returns status code to OS
Use dsjob – run to start, stop, validate or reset DataStage jobs
August 7, 2021 3
Job Sequences
Job Sequences …
Specifies the sequence of Server Jobs to run (control flow)
Contains control information, like specifies different courses of action depending on
Job’s status (success or failure)
Consists of Activity Stages and Triggers
Supports parameters for Activity Stages and also for Job Sequences
Restartable with the help of Checkpoint information (maintained by DS)
Supports automatic Exception Handling
Activit
y
Trigger
August 7, 2021 4
Creating Job Sequences
Creating Job Sequences …
Select File New
August 7, 2021 5
Job Sequence Development
Developing a Job Sequence involves …
Adding Activity Stages
Linking Stages
Setting up Properties for each Activity
Setting up Trigger Information for each
Activity
Compiling the Job Sequence
Running the Job Sequence
Repository
August 7, 2021 6
Activity Stages
Activity Stages …
August 7, 2021 7
Activity Stages
Activity Stages …
Start Loop and End Loop – Together these two stages allows to implement For…Next
or For…Each loop
Terminator – Allows to ensure the sequence is stopped cleanly for certain situations
August 7, 2021 8
Triggers
Triggers …
Provides control information to Stage Activities
Specifies different courses of action depending on Job’s status (success or failure)
Trigger names must be unique for each Activity
August 7, 2021 9
Scenario for a Job Sequence 1
Scenario …
3 Dimension Load Jobs and 2 Fact Load Jobs
Fact Load Jobs to start after all Dimension Load Jobs completes successfully
If any Dimension Load Jobs fail, terminate all running Jobs
August 7, 2021 10
Sample Job Sequence 1
Sample Job Sequence 1 …
Executes
Dimension_Load_3 Waits for the
Job successful
completion and
triggers other Jobs
Terminates the
Sequence if
Dimension_Load_
3 fails
August 7, 2021 11
Job Activity
Job Activity …
Executes a DataStage Job
Job
selection
Other options:
Reset then run,
Validate only,
Reset only
Trigger
types
Supply
parameters
to Job
August 7, 2021 12
Sequencer and Terminator Activities
August 7, 2021 13
Scenario for a Job Sequence 2
Scenario …
5 input files are available in a folder with the same layout
Single Server Job available to sort a input file
Wait for a trigger to start the Job
Send a message to a computer after Job completion (success or failure)
Handle exception
Wait-For-File Activity – Wait for the trigger file before executing the Job
Start and End Loop Activity – Create For…Next loop to process 5 files
August 7, 2021 14
Sample Job Sequence 2
Sample Job Sequence 2 …
For…Next Loop to
execute Sort Job
for 5 input files
Executes Sort
Job
When any failure
occurs, control is
transferred here
August 7, 2021 15
Wait-For-File & Start and End Loop Activities
Wait-For-File Activity … Start and End Loop Activity …
Waits for a specified file to appear or Implements For…Next or For…Each
disappear loop
Appear option – does not delete the file Current value of counter stored in
after finding it stage_label.$Counter
Numeric
Loop
List Loop
August 7, 2021 16
ExecCommand and Exception Handler Activities
ExecCommand Activity … Exception Handler Activity …
Specifies OS command to execute No input triggers and sends one
For command failures, enabling Log unconditional output trigger
warnings or Automatically handle failed Variables $ErrSource, $ErrNumber,
activities in Job Sequence properties $ErrMessage can be passed to
records warnings in Director Log subsequent activities
Enable Automatically handle failed
activities in Job Sequence properties to
invoke exception handler
Message
outputted
in Director
Log
August 7, 2021 17
Other Activity stages
Routine
Specifies a routine from the Repository (but not transforms)
Routine arguments can be accessed by subsequent activities
Email Notification
– Specifies the email notification to be sent using SMTP
Email template file dssendmail_template.txt under Projects folder allows to create
different email templates for different projects
Nested Conditions
– Allows branch the execution of sequence based on condition
Example: If today is weekday execute weekday_Job else Weekend_Job
User Variable
– Allows to define global variables within a sequence
For example, the activity can be used to set Job parameters
August 7, 2021 18
Job Sequence Properties
Job Sequence Properties…
Select Edit Job Properties
Specifies Display
parameter s code
s
Enables
Director to
run multiple
instances
Records
checkpoint
information
for DataStage
to recover
the Sequence
August 7, 2021 19