Advanced Queuing Internals: Julian Dyke Independent Consultant

Advanced Queuing
Internals
Julian Dyke
Independent Consultant
Web Version - November 2008

1 © 2008 Julian Dyke juliandyke.com
Agenda
 Introduction
 Single Consumer Queues
 Multiple Consumer Queues
 Recipients
 Subscribers
 Exception Queues
 Array Payloads
 Buffered Messages
 Spilled Messages
 Performance

Introduction
Advanced Queuing
 Advanced Queuing
 Introduced in Oracle 8.0
 Extended and enhanced in most subsequent versions
 Supports Oracle Streams in Oracle 9.2 and above
 Supports buffered messages in Oracle 10.2 and above
 Allows messages to be enqueued and dequeued from queues that are

managed by the database
 Each queue is associated with a queue table

 Properties of queue table specify behaviour of associated queues
 Each queue has a payload which can be:

 RAW - only messages of type RAW can be enqueued
 Object type- only messages of the specified type can be enqueued
 ANYDATA - messages with any object type can be enqueued

Introduction
Advanced Queuing
 By default messages are dequeued in the order they are enqueued
 Default behaviour can be overridden in several ways
 Messages can be persistent or buffered

 Persistent messages
 Stored in queue table
 Survive an instance restart
 Buffered messages
 Stored in SGA
 Can be spilled to queue table
 Lost during instance restart
 Messages can be immediate or on-commit

 Immediate messages are committed immediately when they are
enqueued/dequeued
 On-commit messages are committed with the enqueuing transaction
 Buffered messages can only be immediate.

Introduction
Advanced Queuing
 Queue tables can be created for single or multiple consumers
 Messages in single consumer queue tables can only be dequeued once
 Messages in multiple consumer queue tables can be dequeued multiple
times by multiple consumers
 Multiple consumer queue tables can be associated with

 Multiple recipients
 Multiple subcribers
 Subscribers can:
 Specify rules to control which messages they dequene
 Specify transformations to be performed against dequeued data

Advanced Queuing
Introduction
 Sessions can listen for messages on multiple queues
 Session is notified when a message arrives on any of the target queues
 Queue messages can be propagated from one queue to another

 In the same database
 In different databases
 Propagation can be immediate or at specified intervals
 Transformation converts payload from one object type to another

 Queue data can be transformed when messages are:
 Enqueued
 Propagated
 Dequeued
 Messages must be transformed using DBMS_TRANSFORM API

Introduction
Payloads
 Queue payloads can be
 RAW
 Abstract data types
 ANYDATA
 Abstract data types

 Maximum number of attributes is limited to 900
 For example:
CREATE TYPE type1 AS OBJECT

(
c1 NUMBER,
c2 NUMBER,
c3 NUMBER
);
/

Single Consumer Queues
Queue Tables
 Every queue must be associated with a queue table
 Queue table defines properties of queue
 Reported in DBA_QUEUE_TABLES
 Managed using DBMS_AQADM
 For example:
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');
 By default single consumer queues will be created

 Each queue table can contain multiple queues
 Queues inherit properties of queue table
 Each queue table block contains blocks for one queue

Queue Table Columns (RAW Payload)
 Queue table QT1 contains the following columns:
Col# Intcol# Column Name Data Type Col# Intcol# Column Name Data Type
1 1 Q_NAME VARCHAR2(30) 16 16 DEQ_TIME TIMESTAMP(6)
2 2 MSG_ID RAW(16) 17 17 DEQ_UID VARCHAR2(30)
3 3 CORRID VARCHAR2(128) 18 18 DEQ_TID VARCHAR2(30)
4 4 PRIORITY NUMBER 19 19 RETRY_COUNT NUMBER
5 5 STATE NUMBER 20 20 EXCEPTION_QSCHEMA VARCHAR2(30)
6 6 DELAY TIMESTAMP(6) 21 21 EXCEPTION_QUEUE VARCHAR2(30)
7 7 EXPIRATION NUMBER 22 22 STEP_NO NUMBER
8 8 TIME_MANAGER_INFO TIMESTAMP(6) 23 23 RECIPIENT_KEY NUMBER
9 9 LOCAL_ORDER_NO NUMBER 24 24 DEQUEUE_MSG_ID RAW(16)
10 10 CHAIN_NO NUMBER 25 25 SENDER_NAME VARCHAR2(30)
11 11 CSCN NUMBER 26 26 SENDER_ADDRESS VARCHAR2(1024)
12 12 DSCN NUMBER 27 27 SENDER_PROTOCOL NUMBER
13 13 ENQ_TIME TIMESTAMP(6) 28 28 USER_DATA BLOB
14 14 ENQ_UID VARCHAR2(30) 29 29 USER_PROP SYS.ANYDATA
15 15 ENQ_TID VARCHAR2(30)

Queue Table Columns (Object Payload)
 Queue table QT1 contains the following columns:
Col# Intcol# Column Name Data Type Col# Intcol# Column Name Data Type
1 1 Q_NAME VARCHAR2(30) 17 17 DEQ_UID VARCHAR2(30)
2 2 MSG_ID RAW(16) 18 18 DEQ_TID VARCHAR2(30)
3 3 CORRID VARCHAR2(128) 19 19 RETRY_COUNT NUMBER
4 4 PRIORITY NUMBER 20 20 EXCEPTION_QSCHEMA VARCHAR2(30)
5 5 STATE NUMBER 21 21 EXCEPTION_QUEUE VARCHAR2(30)
6 6 DELAY TIMESTAMP(6) 22 22 STEP_NO NUMBER
7 7 EXPIRATION NUMBER 23 23 RECIPIENT_KEY NUMBER
8 8 TIME_MANAGER_INFO TIMESTAMP(6) 24 24 DEQUEUE_MSG_ID RAW(16)
9 9 LOCAL_ORDER_NO NUMBER 25 25 SENDER_NAME VARCHAR2(30)
10 10 CHAIN_NO NUMBER 26 26 SENDER_ADDRESS VARCHAR2(1024)
11 11 CSCN NUMBER 27 27 SENDER_PROTOCOL NUMBER
12 12 DSCN NUMBER 28 28 USER_DATA TYPE1
13 13 ENQ_TIME TIMESTAMP(6) 28 29 SYS_NC00029$ NUMBER
14 14 ENQ_UID VARCHAR2(30) 28 30 SYS_NC00030$ NUMBER
15 15 ENQ_TID VARCHAR2(30) 28 31 SYS_NC00031$ NUMBER
16 16 DEQ_TIME TIMESTAMP(6) 29 32 USER_PROP SYS.ANYDATA

Database Objects (RAW payload)
 The following objects will be created (object IDs and constraint IDs will vary):
Object Object Name Object Type

ID
70581 QT1 TABLE
70582 SYS_LOB0000070581C00028$ LOB
$
70583 SYS_IL0000070581C00028$$ LOB INDEX
70584 SYS_LOB0000070581C00029$$ LOB
70585 SYS_IL0000070581C00029$$ LOB INDEX
70586 SYS_C009433 INDEX
70587 AQ$_QT1_T INDEX
70588 AQ$_QT1_I INDEX
70589 QT70581_BUFFER VIEW
70590 AQ$QT1 VIEW
70591 AQ$_QT1_F VIEW
 70592 are
LOB columns AQ$_QT1_E QUEUE
used for USER_DATA and USER_PROP columns
Database Objects (Object Payload)
Object Object Name Object Type

ID
70581 QT1 TABLE
70582 SYS_LOB0000070581C00032$ LOB
$
70583 SYS_IL0000070581C00032$$ LOB INDEX
70584 SYS_C009433 INDEX
70585 AQ$_QT1_T INDEX
70586 AQ$_QT1_I INDEX
70587 QT70581_BUFFER VIEW
70589 AQ$QT1 VIEW
70590 AQ$_QT1_F VIEW
 70591 is used
LOB column AQ$_QT1_E
for USER_PROP columnQUEUE

Index Columns
 The following indexes will be created by default (constraint IDs will vary):
Index Name Column Column Name

#
SYS_C00943 1 MSGID
6
AQ$_QT1_I 1 Q_NAME
2 STATE
3 ENQ_TIME
4 STEP_NO
5 CHAIN_NO
6 LOCAL_ORDER_NO
AQ$_QT1_T 1 TIME_MANAGER_INF
O

Sort Lists
 The columns indexed by AQ$_QTI are determined by the SORT_LIST
parameter.
 Possible values are
 enq_time (default)
DBMS_AQADM.CREATE_QUEUE_TABLE
 priority ('QT1','TYPE1',SORT_LIST=>"priority,enq_time");
 priority,enq_time
 enq_time,priority
 Must be defined when queue table is created
 Cannot be subsequently altered
Column enq_time priority priority,enq_time enq_time,priority

#
1 Q_NAME Q_NAME Q_NAME Q_NAME
2 STATE STATE STATE STATE
3 ENQ_TIME PRIORITY PRIORITY ENQ_TIME
4 STEP_NO CHAIN_NO ENQ_TIME STEP_NO
5 CHAIN_NO LOCAL_ORDER_NO STEP_NO PRIORITY
6 LOCAL_ORDER_N CHAIN_NO CHAIN_NO
O
1 7 © 2008 Julian Dyke LOCAL_ORDER_NO juliandyke.com
LOCAL_ORDER_N
Views
 Two views are created for each queue table
 For example for QT1 (object ID =70581):
 QT<object_id>_BUFFER
 e.g. QT70581_BUFFER
 based on X$BUFFER2
 AQ$_<queue_table_name>_F
 e,g. AQ$_QT1_F
 based on QT1 and ALL_DEQUEUE_QUEUES

Views
 Two views are created for each queue table
 For example for QT1 (object ID =70581)

 QT70581_BUFFER
 AQ$_QT1_F
 QT70581_BUFFER is defined as follows:
SELECT
"ADDR", "INDX", "INST_ID", "OBJNO", "QUEUE_ID", "MSGID", "CORRID",
"SEQUENCE_NUM", "MSG_NUM", "STATE", "PRIORITY", "EXPIRATION",
"ENQ_TIME", "ENQ_UID", "ENQ_USER_NAME", "RETRY_COUNT",
"SENDER_NAME", "SENDER_ADDRESS", "SENDER_PROTOCOL",
"DEQUEUE_MSGID", "SRCSEQUENCE_NUM", "SUBSCRIBER_ID",
"EXCEPTIONQ_SCHEMA", "EXCEPTIONQ_NAME"
FROM X$BUFFER2
WHERE objno = 70581;

Views
 AQ$_QT1_F is defined as follows:
SELECT
qt.q_name Q_NAME, qt.rowid ROW_ID, qt.msgid MSGID, qt.corrid CORRID,
qt.priority PRIORITY, qt.state STATE, qt.delay DELAY, qt.expiration EXPIRATION,
qt.enq_time ENQ_TIME, qt.enq_uid ENQ_UID, qt.enq_tid ENQ_TID,
qt.deq_time DEQ_TIME, qt.deq_uid DEQ_UID, qt.deq_tid DEQ_TID,
qt.retry_count RETRY_COUNT, qt.exception_qschema EXCEPTION_QSCHEMA,
qt.exception_queue EXCEPTION_QUEUE, qt.cscn CSCN, qt.dscn DSCN,
qt.chain_no CHAIN_NO, qt.local_order_no LOCAL_ORDER_NO,
qt.time_manager_info TIME_MANAGER_INFO, qt.step_no STEP_NO,
qt.user_data USER_DATA , qt.sender_name SENDER_NAME,
qt.sender_address SENDER_ADDRESS, qt.sender_protocol SENDER_PROTOCOL,
qt.dequeue_msgid DEQUEUE_MSGID, 'PERSISTENT' DELIVERY_MODE,
0 SEQUENCE_NUM, 0 MSG_NUM, qo.qid QUEUE_ID,
qt.user_prop USER_PROP
FROM
"QT1" qt,
ALL_DEQUEUE_QUEUES qo
WHERE qt.q_name = qo.name
AND qo.owner = 'US01'
WITH READ ONLY;

Queues
 Every queue must be associated with a queue table
 Queue table must exist before queue can be created
DBMS_AQADM.CREATE_QUEUE
(queue_name => 'Q1',queue_table => 'QT1');
 This statement might create the following object:

Object ID Object Name Object
Type
70793 Q1 QUEUE
 Every queue has a type which can be:

 NORMAL (default)
 EXCEPTION
 NON PERSISTENT
 Non persistent queues are deprecated in Oracle 10.2

 Use buffered messages instead

Enqueue
 The following code enqueues a message of TYPE1 on a single consumer
queue:
DECLARE
message TYPE1;
msgprop dbms_aq.message_properties_t;
enqopt dbms_aq.enqueue_options_t;
enq_msgid RAW(16);
BEGIN
message := new TYPE1 (10001,20001,30001);
msgprop.expiration :=DBMS_AQ.NEVER
dbms_aq.enqueue
(
queue_name => 'Q1',
enqueue_options => enqopt,
message_properties => msgprop,
payload => message,
msgid => enq_msgid
);
END;

Enqueue
 The enqueue processes executes the following recursive statement:
insert into "US01"."QT1" (q_name, msgid, corrid, priority, state, delay, expiration,
time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid,
retry_count, exception_qschema, exception_queue, recipient_key, dequeue_msgid,
user_data, sender_name, sender_address, sender_protocol, user_prop, cscn, dscn)
values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, 0, :15,:16, :17, :18, :19, :20, :21,
:22, :23, :24, :25)
 In Oracle 11.1 this statement uses the LOAD TABLE CONVENTIONAL operation
STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL (cr=1 pr=5 pw=5
time=0 us)')
 For this statement the following objects are modified

Object ID Object Name Object
Type
70581 QT1 TABLE
70586 SYS_C009433 INDEX
70588 AQ$_QT1_1 INDEX

Dequeue
 The following code dequeues a message of TYPE1 from a single consumer
queue:
DECLARE
message TYPE1;
deqopt dbms_aq.dequeue_options_t;
deq_msgid RAW(16);
BEGIN
dbms_aq.dequeue
(
queue_name => 'Q1',
dequeue_options => deqopt,
payload => message,
msgid => deq_msgid
);
END;

Dequeue
 The dequeue processes executes the following recursive statement:
select /*+ FIRST_ROWS(1) */ tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay,

tab.expiration, tab.retry_count, tab.exception_qschema, tab.exception_queue, tab.chain_no,
tab.local_order_no, tab.enq_time, tab.time_manager_info, tab.state, tab.enq_tid,
tab.step_no, tab.sender_name, tab.sender_address, tab.sender_protocol,
tab.dequeue_msgid, tab.user_prop, tab.user_data
from "US01"."QT1" tab
where q_name = :1 and (state = :2 )
order by q_name, state, enq_time, step_no, chain_no, local_order_no
for update skip locked
 The statement selects all rows in the queue specified by :1 with a state of :2
 The FIRST_ROWS(1) hint is used to optimize the plan
 The statement locks any rows to be deleted
 This will generate undo/redo
 The statement uses the FOR UPDATE SKIP LOCKED clause to skip any rows
still locked by ongoing transactions

Dequeue
 Execution plan for SELECT FOR UPDATE statement is:
STAT #3 id=1 cnt=1 pid=0 pos=1 obj=0 op='FOR UPDATE (cr=7 pr=2 pw=2 time=0 us)'
STAT #3 id=2 cnt=1 pid=1 pos=1 obj=0 op='SORT ORDER BY (cr=7 pr=0 pw=0 time=0 us
cost=4 size=2759 card=1)'
STAT #3 id=3 cnt=1 pid=2 pos=1 obj=70581 op='TABLE ACCESS FULL QT1 (cr=7 pr=0
pw=0 time=0 us cost=3 size=2759 card=1)'
 As queue grows, object statistics must be gathered to

 ensure AQ$_QT1_I index is used
 prevent full table scans on QT1
 Rows identified by SELECT FOR UPDATE are deleted using:
delete /*+ CACHE_CB("QT9") */ from "US01"."QT1" where rowid = :1
 Execution plan for DELETE statement is:

STAT #7 id=1 cnt=1 pid=0 pos=1 obj=0 op='DELETE QT1 (cr=1 pr=2 pw=2 time=0 us)'
STAT #7 id=2 cnt=1 pid=1 pos=1 obj=70581 op='TABLE ACCESS BY USER ROWID QT1
(cr=1 pr=0)'

Queues
Exception Queues
 A default exception queue is created for each queue table
 Exception messages will be moved to default queue unless a user-defined
exception queue has been specified when the message is enqueued
 For example to create a user-defined exception queue
(queue_name => 'Q1',queue_table => 'QT1');
(queue_name => 'Q1E',queue_table => 'QT1'
queue_type => DBMS_AQADM.EXCEPTION_QUEUE);
Object ID Object Name Object Queue Type

Type
70793 Q1 QUEUE NORMAL
70794 Q1E QUEUE EXCEPTION
 To check number of rows in each queue:
SELECT q_name, COUNT(*)
FROM qt1
GROUP BY q_name;

Queues
Exception Queues
 Exceptions will be written to user-defined exception queue if it is specified
during enqueue operation
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_enqopt dbms_aq.enqueue_options_t;
l_enq_msgid RAW(16);
BEGIN
l_payload := new TYPE1 (10001,20001,30001);
l_msgprop.expiration := 60;
l_msgprop.exception_queue := 'Q1E';
dbms_aq.enqueue
(
queue_name => 'Q1',
enqueue_options => l_enqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_enq_msgid
);
END;
 Message will expire after 60 seconds
 Expired message will be move to exception queue Q1E by queue monitor

Multiple Consumer Queues
Introduction
 There are two ways to use multiple consumer queues
 Multiple Recipients
 Multiple Subscribers
 The same queue definitions are used for both examples:
BEGIN
dbms_aqadm.create_queue_table
('QT3','TYPE1',multiple_consumers=>TRUE)
dbms_aqadm.create_queue ('Q3','QT3');
dbms_aqadm.start_queue ('Q3');
END;
/

Database Objects
DBMS_AQADM.CREATE_QUEUE_TABLE
('QT1','TYPE1',MULTIPLE_CONSUMERS=>TRUE);
Object ID Object Name Object Object ID Object Name Object

Type Type
70756 QT1 TABLE 70767 AQ$_QT1_H TABLE
70757 SYS_LOB0000070581C00032$ LOB 70768 SYS_IOT_TOP_70767 INDEX
$
70769 AQ$_QT1_G TABLE
70758 SYS_IL0000070581C00032$$ LOB INDEX
70770 SYS_IOT_OVER_70769 TABLE
70759 SYS_C009457 INDEX
70771 SYS_IOT_TOP_70769 INDEX
70760 AQ$_QT1_S TABLE
70772 AQ$_QT1_I TABLE
70761 SYS_C009460 INDEX
70762 AQ$_QT1_N SEQUENCE
70763 AQ$QT1_S VIEW 70774 QT70756_BUFFER VIEW
70764 AQ$_QT1_V EVAL CTXT 70775 AQ$QT1 VIEW
70765 AQ$_QT1_T TABLE 70776 AQ$_QT1_F VIEW
70766 SYS_IOT_TOP_70765 INDEX 70777 AQ$_QT1_E QUEUE

Tables
 AQ$_<queue_table_name>_T e.g AQ$_QT3_T
 IOT used queue monitor to manage timed operations
 Single consumer queues use TIME_MANAGER_INFO column only
 AQ$_<queue_table_name>_I
 IOT that maintains state for dequeue operations
 One row per message per recipient/subscriber
 AQ$_<queue_table_name>_S
 Heap table containing information about subscribers
 AQ$_<queue_table_name>_H
 IOT used to store dequeue history
 One row per message per recipient/subscriber
 AQ$_<queue_table_name>_G
 IOT correlating messages to subscriber signatures

Tables
 AQ$_<queue_table_name>_T
 IOT used queue monitor to manage timed operations
 e.g. AQ$_QT3_T
Column Data Type

Name
NEXT_DATE TIMESTAMP
TXN_ID VARCHAR2(30)
MSGID RAW(16)
ACTION NUMBER
 First 3 columns form primary key
 Values for the ACTION column include:

 0 - delay
 1 - expiration
 2 - delay
 Single consumer queues use TIME_MANAGER_INFO column only

Tables
 AQ$_<queue_table_name>_I
 IOT that maintains state for dequeue operations
Column Name Data Type

SUBSCRIBER NUMBER
NAME VARCHAR2(30)
QUEUE# NUMBER
MSG_ENQ_TIME TIMESTAMP
MSG_STEP_NO NUMBER
MSG_CHAIN_NO NUMBER
MSG_LOCAL_ORDER_N NUMBER
O
MSG_ID RAW(16)
HINT ROWID
SPARE RAW(16)
 First eight columns form primary key
 HINT and SPARE columns are stored in IOT overflow segment

Tables
 AQ$_<queue_table_name>_S
 Heap table containing information about subscribers

SUBSCRIBER_ID NUMBER
QUEUE_NAME VARCHAR2(30)
NAME VARCHAR2(30)
ADDRESS VARCHAR2(1024
)
PROTOCOL NUMBER
SUBSCRIBER_TYPE NUMBER
RULE_NAME VARCHAR2(30)
TRANS_NAME VARCHAR2(65)
RULESET_NAME VARCHAR2(65)
NEGATIVE_RULESET_NAME VARCHAR2(65)
CREATION_TIME TIMESTAMP(6)
MODIFICATION_TIME TIMESTAMP(6)
DELETION_TIME TIMESTAMP(6)
3 © 2008 Julian Dyke

SCN_AT_REMOVE NUMBER
juliandyke.com
Tables
 AQ$_<queue_table_name>_H
 IOT used to store dequeue history

MSGID RAW(16)
SUBSCRIBER# NUMBER
NAME VARCHAR2(30)
ADDRESS# NUMBER
DEQUEUE_TIME TIMESTAMP
TRANSACTION_ID VARCHAR2(30)
DEQUEUE_USER VARCHAR2(30)
PROPAGATED_MSGID RAW(16)
RETRY_COUNT NUMBER
HINT ROWID
SPARE RAW(16)
 First four columns form primary key
 No IOT overflow segment
Tables
 AQ$_<queue_table_name>_G
 IOT correlating messages to subscriber signatures
Column Data Type

Name
NAME VARCHAR2(30)
ADDRESS# NUMBER
SIGN SYS.AQ$_SIG_PRO
P
DBS_SIGN SYS.AQ$_SIG_PRO
 All columns form primary key P

Indexes
 By default six indexes are created for each queue table. For example:
Index Name Index Type # Table Name

Columns
SYS_C009457 NORMAL 1 QT3
SYS_C009460 NORMAL 1 AQ$_QT3_S
SYS_IOT_TOP_7076 IOT 3 AQ$_QT3_T
5
SYS_IOT_TOP_7076 IOT 4 AQ$_QT3_H
7
SYS_IOT_TOP_7076 IOT 4 AQ$_QT3_
 Index9 columns for NORMAL indexes are: G
SYS_IOT_TOP_7077 IOT 8 AQ$_QT3_I
2 Index Name Column Column Name
#
SYS_C00945 1 MSGID
7
SYS_C00946 1 SUBSCRIBER_ID
 Index columns
0 for IOT indexes are shown on previous slides

Views
 Three views are created for each queue table
 For example for QT3 (object ID = 70756)
 QT70756_BUFFER
 AQ$_QT3_F
 AQ$QT3
 <queue_object_Id>_BUFFER e.g QT70756_BUFFER

 Similar for single and multiple consumers
 AQ$_<queue_table_name>_F e.g AQ$_QT3_F

 Similar for single and multiple consumers
 AQ$<queue_table_name> views e.g. AQ$QT3

 Based on:
 Queue table (QT3)
 History IOT (AQ$_QT3_H)
 Subscriber table (AQ$_QT3_S)

Views
 AQ$<queue_table_name> views (AQ$QT3) are based on:
 Queue table (QT3)
 History IOT (AQ$_QT3_H)
 Subscriber table (AQ$_QT3_S)
 Abbreviated definition is as follows:
SELECT
<column_list>
FROM
"QT8" qt,
"AQ$_QT8_H" h,
"AQ$_QT8_S" s
WHERE qt.msgid = h.msgid
AND ((h.subscriber# != 0 AND h.subscriber# = s.subscriber_id)
OR (h.subscriber# = 0 AND h.address# = s.subscriber_id))
AND (qt.state != 7 OR qt.state != 9)
WITH READ ONLY;
 Best view to understand current state of queue for all subscribers

Views
 AQ$QT3 contains the following columns
Column Name Data Type Column Name Data Type
QUEUE VARCHAR2(30) RETRY_COUNT NUMBER
MSG_ID RAW(16) EXCEPTION_QUEUE_OWNER VARCHAR2(30)
CORR_ID VARCHAR2(128) EXCEPTION_QUEUE VARCHAR2(30)
MSG_PRIORITY NUMBER USER_DATA TYPE1
MSG_STATE VARCHAR2(16) PROPAGATED_MSGID RAW(16)
DELAY DATE SENDER_NAME VARCHAR2(30)
DELAY_TIMESTAM TIMESTAMP(6) SENDER_ADDRESS VARCHAR2(1024)
P SENDER_PROTOCOL NUMBER
EXPIRATION NUMBER ORIGINAL_MSGID RAW(16)
ENQ_TIME DATE ORIGINAL_QUEUE_NAME VARCHAR2(30)
ENQ_TIMESTAMP TIMESTAMP(6) ORIGINAL_QUEUE_OWNER VARCHAR2(30)
ENQ_USER_ID VARCHAR2(30) EXPIRATION_REASON VARCHAR2(31)
ENQ_TXN_ID VARCHAR2(30) CONSUMER_NAME VARCHAR2(30)
DEQ_TIME DATE ADDRESS VARCHAR2(1024)
DEQ_TIMESTAMP TIMESTAMP(6) PROTOCOL NUMBER
DEQ_USER_ID VARCHAR2(30)
DEQ_TXN_ID VARCHAR2(30)
Recipients
 The following code enqueues a message for three named recipients
DECLARE
l_payload type1;
l_recipient_list dbms_aq.aq$_recipient_list_t;
BEGIN
l_recipient_list(1) := sys.aq$_agent ('CONSUMER1',NULL,NULL);
l_msgprop.recipient_list := l_recipient_list;
l_msgprop.expiration := DBMS_AQ.NEVER;
dbms_aq.enqueue
(
queue_name => 'Q3',
);
END;
Recipients
 A recipient list is constructed using AQ$_AGENT objects
 In the example all recipients are in the local database
 The enqueue operation performs the following actions:

 Inserts one row in the queue table (QT3)
 Inserts three rows in the queue status table (AQ$_QT3_I)
 Inserts three rows in the queue history table (AQ$_QT3_H)

Recipients
 The following code dequeues a message for one of the named recipients
DECLARE
l_payload TYPE1;
l_deqopt dbms_aq.dequeue_options_t;
l_deq_msgid RAW(16);
BEGIN
l_deqopt.consumer_name := 'CONSUMER2';
dbms_aq.dequeue
(
queue_name => 'Q3',
dequeue_options => l_deqopt,
msgid => l_deq_msgid
);
END;
 Notes
 A consumer name MUST be specified
 The message must have been enqueued specifically for that consumer

Recipients
 The dequeue operation performs the following actions
 Deletes one row from the queue status IOT (AQ$_QT3_I)
 Updates the following columns in one row of the queue history IOT
(AQ$_QT3_H)
 DEQUEUE_TIME
 TRANSACTION_ID
 DEQUEUE_USER
 Inserts one row into the queue timer table (AQ$_QT3_T)
 The queue monitor (QMNC) process asynchronously checks the timer table
(AQ$_QT3_T) for actions
 If any actions are found these are sent to the queue monitor slaves (Q001,
Q002 etc)
 When last recipient has dequeued message, queue monitor slaves perform
the following actions
 Delete all rows for message in queue history table (AQ$_QT3_H)
 Delete row in queue table (QT3) for message

Subscribers
 Subscribers must exist for the queue before messages can be enqueued
 The following code creates two subscribers for queue Q3
DECLARE
l_subscriber sys.aq$_agent;
BEGIN
l_subscriber := sys.aq$_agent ('SUBSCRIBER1',NULL,NULL);
DBMS_AQADM.ADD_SUBSCRIBER
(
queue_name => 'Q3',
subscriber => l_subscriber
);
(
queue_name => 'Q3',
);
END;
 Creating a subscriber inserts one row in the AQ$_QT3_S table

Subscribers
 The following code enqueues ten messages on Q3
DECLARE
l_payload TYPE1;
BEGIN
FOR f IN 1..10
LOOP
l_payload := new TYPE1 (10000 + f,20000 + f,30000 + f);
dbms_aq.enqueue
(
queue_name => 'Q3',
);
END LOOP;
END;

Subscribers
 The following code dequeues a message from Q3 for SUBSCRIBER1
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE1;
BEGIN
l_deqopt.consumer_name := 'SUBSCRIBER1';
dbms_aq.dequeue
(
queue_name => 'Q3',
);
DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1));
END;

Subscribers
 Subscribers can subsequently be added and deleted dynamically
DECLARE
l_subscriber sys.aq$_agent;
BEGIN
(
queue_name => 'Q3',
);
DBMS_AQADM.REMOVE_SUBSCRIBER
(
queue_name => 'Q3',
);
END;
 New subscribers will only be allowed to dequeue messages that have been enqueued after the subscriber was added

Array Payloads
Introduction
 Payload of a queue can optionally be a VARRAY of object types
 For example:
CREATE OR REPLACE TYPE type2 AS VARRAY (10) OF type1;

/
CREATE OR REPLACE TYPE type3 AS OBJECT (c1 type2);
/
 Queue table can be created with a TYPE3 payload

 It is not possible to create a queue table with a TYPE2 payload
 For example:
DBMS_AQADM.CREATE_QUEUE ('Q3','QT3');
DBMS_AQADM.START_QUEUE ('Q3');

Array Payloads
Enqueue
DECLARE
l_payload TYPE3;
enqopt dbms_aq.enqueue_options_t;
enq_msgid RAW(16);
BEGIN
l_payload := new TYPE3 (TYPE2 (
TYPE1 (10001,20001,30001),
TYPE1 (10002,20002,30002),
TYPE1 (10003,20003,30003),
TYPE1 (10004,20004,30004)
));
msgprop.expiration := DBMS_AQ.NEVER;
dbms_aq.enqueue
(
queue_name => 'Q4',
enqueue_options => enqopt,
msgid => enq_msgid
);
END;

Array Payloads
Dequeue
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE3;
deqopt dbms_aq.dequeue_options_t;
deq_msgid RAW(16);
BEGIN
dbms_aq.dequeue
(
queue_name => 'Q4',
dequeue_options => deqopt,
msgid => deq_msgid
);
FOR i IN 1..message.c1.COUNT
LOOP
DBMS_OUTPUT.PUT ('C1 = '||TO_CHAR (l_payload.c1(i).c1)||' ');
DBMS_OUTPUT.PUT ('C2 = '||TO_CHAR (l_payload.c1(i).c2)||' ');
DBMS_OUTPUT.PUT ('C3 = '||TO_CHAR (l_payload.c1(i).c3));
DBMS_OUTPUT.NEW_LINE ();
END LOOP;
END;

Buffered Messages
Introduction
 In Oracle 10.2 and above messages can be buffered in the SGA
 Messages will not be written to database immediately
 Messages are spillled to database if:
 Number of messages exceeds threshold value
 Messages not dequeued within 10 minutes
 Buffered messages
 Are much faster than persistent queues
 Do not guarantee reliability
 Cannot form part of a transaction
 Do not support (Oracle 11.1)
 Message retention / delay
 Transaction grouping
 Array enqueue / dequeue
 Message export / import

Buffered Messages
Introduction
 Buffering is specified at message level
 Queues can contain both persistent and buffered messages
 Payload can be ADT, XML, ANYDATA or RAW
 Support for LOB payloads is restricted
 The following definitions are used with the examples in this section
BEGIN
dbms_aqadm.create_queue_table ('QT1','TYPE1')
dbms_aqadm.create_queue ('Q1','QT1');
dbms_aqadm.start_queue ('Q1');
END;
 Note that all queue tables support buffered messages

 No additional attributes are specified for the queue or the queue table

Buffered Messages
Enqueue
 The following code enqueues a buffered message
DECLARE
l_payload TYPE1;
BEGIN
l_enqopt.visibility := DBMS_AQ.IMMEDIATE;
l_enqopt.delivery_mode := DBMS_AQ.BUFFERED;
dbms_aq.enqueue
(
queue_name => 'Q1',
);
END;

Buffered Messages
Dequeue
 The following code dequeues a buffered message:
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE1;
BEGIN
l_deqopt.visibility := DBMS_AQ.IMMEDIATE;
l_deqopt.delivery_mode := DBMS_AQ.BUFFERED;
dbms_aq.dequeue
(
queue_name => 'Q1',
);
END;
Buffered Messages
Memory Usage
 Memory is allocated from the Streams Pool
 The following table shows the amount of streams pool memory required to
enqueue 5101 messages with the TYPE1 payload:
Befor After Befor After

e e
kodpaih3 0 10,324,448 recov_kgqmsub 336 504
kwqbsinfy:mpr 480 2,448,480 kwqbsinfy:cco 332 332
image handles 84 428,512 kwqbsinfy:sta 208 312
kwqbsinfy:bms 72 387,692 spilled:kwqbl 216 288
kggmem_fl_1 44 224,444 fixed allocation callback 256 256
kggbt_alloc_block 2,072 88,060 kgqmsub 144 216
Sender info 14,140 19,796 deqtree_kgqmctx 136 192
recov_kgqbtctx 12,288 16,384 substree_kgqmctx 120 160
kwqbcqini:spilledovermsgs 2,952 3,936 time manager index 120 160
kwqbsinfy:bqg 1,236 1,648 msgtree_kgqmctx 120 160
recov_kggmctx 924 1,232 name_kgqmsub 32 48

Buffered Messages
Database Objects
 Additional database objects are created the first time a buffered message is
enqueued on a queue table
 This will cause elapsed time of first enqueue operation to be high
 For example the following objects might be created
Object ID Object Name Object Type

72638 AQ$_QT3_P TABLE
72639 SYS_LOB0000072638C00032$ LOB
$
72640 SYS_IL0000072638C00032$$ LOB
INDEX
72641 SYS_C0010003 INDEX
 The enqueuing session also creates a service for the queue
 For example SYS$US01.Q3.TEST where
 US01 is the queue owner
 Q3 is the queue name
 TEST is the database name

Buffered Messages
Database Objects
 AQ$_<table_queue_name>_P contains the following columns
Column Name Data Type Column Name Data Type

Q_NAME VARCHAR2(30) DEQ_TIME DATE
MSGID RAW(16) DEQ_UID VARCHAR2(30)
CORRID VARCHAR2(128 DEQ_TID VARCHAR2(30)
) RETRY_COUNT NUMBER
PRIORITY NUMBER EXCEPTION_QSCHEMA VARCHAR2(30)
STATE VARCHAR2(16) EXCEPTION_QUEUE VARCHAR2(30)
DELAY DATE STEP_NO NUMBER
EXPIRATION NUMBER RECIPIENT_KEY NUMBER
TIME_MANAGER_INF TIMESTAMP(6) DEQUEUE_MSGID RAW(16)
O
SENDER_NAME VARCHAR2(30)
LOCAL_ORDER_NO NUMBER
SENDER_ADDRESS VARCHAR2(1024
CHAIN_NO NUMBER )
CSCN NUMBER SENDER_PROTOCOL NUMBER
DSCN NUMBER USER_DATA TYPE1
ENQ_TIME DATE USER_PROP SYS.ANYDATA
ENQ_UID VARCHAR2(30)
ENQ_TID VARCHAR2(30)
Buffered Messages
Database Objects
 The AQ$_<queue_table_name>_P table has one primary key index on
 Q_NAME
 MSGID
 Two view definitions are also updated when the first buffered message is
enqueued:
 AQ$<queue_table_name>
 e.g. AQ$QT3
 reports all messages in persistent and buffered queues
 AQ$_<queue_table_name>_F
 e.g. AQ$_QT3_F
 reports all messages that have not yet been dequeued in both
persistent and buffered queues

Buffered Messages
Database Objects
 The queue monitor slaves write spilled messages to
AQ$_<queue_table_name>_P
 Rows are inserted individually; no array operation is used
 For example
INSERT INTO "us01"."aq$_qt3_p"

(
q_name, msgid, corrid, priority,state, delay, expiration, time_manager_info,
local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid, retry_count,
exception_qschema, exception_queue, recipient_key, dequeue_msgid,
user_data, sender_name, sender_address, sender_protocol, dscn, cscn
)
VALUES
(:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,0,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24)
 Messages are asynchronously deleted from AQ$_<queue_table_name>_P by

queue monitor slaves
 Messages are deleted using an array size of 32
 For example
DELETE FROM us01.aq$_qt24_p WHERE q_name = :1 AND msgid = :2

Buffered Messages
Database Objects
 AQ$_<table_queue_name>_D contains the following columns
Column Data Type

Name
OID NUMBER
MSGNUM NUMBER
MSGID RAW(16)
SUB NUMBER
SEQNUM NUMBER
RSUBS SYS.AQ$_RECIPIENT
 The RSUBS column is stored asS a LOB

Buffered Messages
Spillage
 If flow control is enabled then number of buffered messages that can be
enqueued on any queue is limited
 Subsequent attempts to enqueue messages will be rejected
 Set _BUFQ_STOP_FLOW_CONTROL parameter to TRUE to disable flow

control completely
 Limited to
 5000 buffered messages
 15000 captured messages
 Can be overridden in 10.2.0.3 by applying Patch 5093060 and setting

 Event 10867 for buffered messages (level is # messages)
 Event 10868 for captured messages (level is # messages)
 Can be fixed in 10.2.0.4 onwards by setting:

 _BUFFERED_PUBLISHER_FLOW_CONTROL_THRESHOLD
 _CAPTURED_PUBLISHER_FLOW_CONTROL_THRESHOLD

Buffered Messages
Database Objects
 For a multiple consumer queue the following objects will be created when the
first buffered message is enqueued:
Object ID Object Name Object Type

72638 AQ$_QT3_P TABLE
72639 SYS_LOB0000072638C00032$$ LOB
72640 SYS_IL0000072638C00032$$ LOB
INDEX
72641 SYS_C0010003 INDEX
72642 AQ$_QT3_D TABLE
72643 SYS_IOT_OVER_72642 TABLE
72644 SYS_LOB0000072642C00006$$ LOB
72645 SYS_IL0000072642C00006$$ INDEX

Performance
Elapsed Times
Enqueue Dequeue
No Commit Commit No Commit Commit
Single Consumer
PERSISTENT 4.77 10.99 5.75 9.62
ON COMMIT
Single Consumer
PERSISTENT 10.80 11.41 8.77 9.78
IMMEDIATE
Single Consumer
BUFFERED 2.32 2.60 1.53 2.13
IMMEDIATE
Single Consumer
PERSISTENT
ON COMMIT 0.66 1.38 1.00 1.35
VARRAY(10) OF
TYPE1
Multi Consumer
PERSISTENT
6.40 14.45 6.36 11.20
ON COMMIT
2 recipients
Multi Consumer
PERSISTENT
6.02 14.59 6.54 11.40
ON COMMIT
2 subscribers
10000 TYPE1 messages enqueued then 10000 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5
6 © 2008 Julian Dyke

x86
juliandyke.com
Performance
Redo Generation
Enqueue Dequeue
No Commit Commit No Commit Commit
Single Consumer
PERSISTENT 9223 15491 10806 15521
ON COMMIT
Single Consumer
PERSISTENT 15165 15485 14936 14904
IMMEDIATE
Single Consumer
BUFFERED 0 0 0 0
IMMEDIATE
Single Consumer
PERSISTENT
ON COMMIT 1211 1831 1381 1832
VARRAY(10) OF
TYPE1
Multi Consumer
PERSISTENT
16459 23324 8102 12521
ON COMMIT
2 recipients
Multi Consumer
PERSISTENT
15832 23404 7934 12953
ON COMMIT
2 subscribers
10 TYPE1 messages enqueued then 10 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86

Conclusion
 Several single queues may be more efficient than
 Multiple recipients
 Multiple subscribers
 Use ON_COMMIT visibility where possible

 No transaction overhead for queuing operations
 Reduces undo / redo generation
 IMMEDIATE is much more expensive
 Buffered messages give best performance

 Provided they do not spill regularly
 Array payloads are very efficient

 Message overhead is reduced

Thank you for your interest
[email protected]

Advanced Queuing Internals: Julian Dyke Independent Consultant

Uploaded by

Copyright:

Available Formats

Advanced Queuing Internals: Julian Dyke Independent Consultant

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Queuing Internals: Julian Dyke Independent Consultant

Uploaded by

Copyright:

Available Formats

Advanced Queuing

Web Version - November 2008

2 © 2008 Julian Dyke juliandyke.com

 Allows messages to be enqueued and dequeued from queues that are

 Each queue is associated with a queue table

 Each queue has a payload which can be:

3 © 2008 Julian Dyke juliandyke.com

 Messages can be persistent or buffered

 Messages can be immediate or on-commit

4 © 2008 Julian Dyke juliandyke.com

 Multiple consumer queue tables can be associated with

5 © 2008 Julian Dyke juliandyke.com

 Queue messages can be propagated from one queue to another

 Transformation converts payload from one object type to another

6 © 2008 Julian Dyke juliandyke.com

 Abstract data types

CREATE TYPE type1 AS OBJECT

7 © 2008 Julian Dyke juliandyke.com

 By default single consumer queues will be created

8 © 2008 Julian Dyke juliandyke.com

9 © 2008 Julian Dyke juliandyke.com

1 © 2008 Julian Dyke juliandyke.com

Object Object Name Object Type

Object Object Name Object Type

1 © 2008 Julian Dyke juliandyke.com

Index Name Column Column Name

1 © 2008 Julian Dyke juliandyke.com

Column enq_time priority priority,enq_time enq_time,priority

 For example for QT1 (object ID =70581):

1 © 2008 Julian Dyke juliandyke.com

 For example for QT1 (object ID =70581)

 QT70581_BUFFER is defined as follows:

1 © 2008 Julian Dyke juliandyke.com

1 © 2008 Julian Dyke juliandyke.com

 This statement might create the following object:

 Every queue has a type which can be:

 Non persistent queues are deprecated in Oracle 10.2

1 © 2008 Julian Dyke juliandyke.com

1 © 2008 Julian Dyke juliandyke.com

 For this statement the following objects are modified

2 © 2008 Julian Dyke juliandyke.com

2 © 2008 Julian Dyke juliandyke.com

select /*+ FIRST_ROWS(1) */ tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay,

2 © 2008 Julian Dyke juliandyke.com

 As queue grows, object statistics must be gathered to

 Rows identified by SELECT FOR UPDATE are deleted using:

delete /*+ CACHE_CB("QT9") */ from "US01"."QT1" where rowid = :1

 Execution plan for DELETE statement is:

2 © 2008 Julian Dyke juliandyke.com

 For example to create a user-defined exception queue

Object ID Object Name Object Queue Type

2 © 2008 Julian Dyke juliandyke.com

2 © 2008 Julian Dyke juliandyke.com

 The same queue definitions are used for both examples:

2 © 2008 Julian Dyke juliandyke.com

Object ID Object Name Object Object ID Object Name Object

70764 AQ$_QT1_V EVAL CTXT 70775 AQ$QT1 VIEW

70765 AQ$_QT1_T TABLE 70776 AQ$_QT1_F VIEW

70766 SYS_IOT_TOP_70765 INDEX 70777 AQ$_QT1_E QUEUE

select /+ FIRST_ROWS(1) / tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay,

delete /+ CACHE_CB("QT9") / from "US01"."QT1" where rowid = :1