Using Sybase IQ Effectively

Richard Soundy
Business Development EMEA
[email protected]
 Sybase IQ

• In these presentations I concentrate on

3 areas
• IQ Indexes
• Index Advisor and Workload Monitor
• In Line Monitoring (new in 15)
 Indexes
• Typical RDBMS uses B-tree index
– usually a separate structure in addition to data
– consists of data pages with values and
pointers
– Sybase (and others) has special type which
contains pointers and data pages
• B-tree indexes are expensive
– consume space and time (to build)
– require maintenance after data refresh
– not useful with low cardinality data columns
 B-Tree – an example Let’s assume a table with
key values from 1 to 8 – and
build the B-Tree

The decision tree is now very

simple. The question asked is “Is
the required value less than or equal
Index Levels - Keys to the value in the tree?”
If it is take the left fork else go right
4

2 6
1 3 5 7
1 2 3 4 5 6 7 8
Leaf Level (Data)
 Sybase IQ Indexes
• Indexes are the data (a data access method)
– no separate data store
• A column is likely to have multiple indexes
– always has at least one index
• Index selection for a column is based upon
– Number of discrete values (cardinality)
– Usage in queries
– To some extent, the column’s data type
 Index Types -1
• Default Index (aka Fast Projection or FP)
– Raw data compressed on disk
– comes in 4 flavours
• Low Fast (LF)
– Bit map index
• High Non Group (HNG)
– Bit-wise index
• High Group (HG)
– G-Array (relative of a B-tree)
– Comes in 3 flavours
 Index Types - 2
• Compare Index (CMP)
– Column Compare Index
• Word Index (WD)
– Sort of “Free Text” Index
• DATE
• TIME
• DTTM (date time index)
 Default Index - FP
• Created automatically by Create Table
• Used for
– Ad-Hoc joins
– String searches
– Certain calculations
– Projection of data
• This index cannot be created or dropped
 Disk Compression
• Current FP (default) index supports:
• Raw data values (compressed on disk)
• 1-byte lookup table compression for up to
255 distinct values (compressed on disk
and in memory)
• 2-byte lookup table compression for up to
65K distinct values (compressed on disk
and in memory)
• 3-byte lookup table compression for up to
16m distinct values (compressed on disk
and in memory)
 Creating FP in IQ 15 (upgrade?)
• 3-byte with many distincts or wide
datatype requires large cache to
support
• Eligible existing columns can be
upgraded using a stored procedure
• This is sp_iqrebuildindex (as for the
older migration when FP(2) arrived)
 Disk Compression – Managing FPs
• 16M (16,777,216) values is the upper limit
for distinct values in a column for the new
compression, but the practical limit will
usually be lower
• Options will allow users to tune memory
usage
• Supported for most data types
• except LOBs and columns less than 4
bytes wide
Distinct Count
IQ Fast Project Indexes
Column Data Column Data Column Data
Size = 1 byte Size = 2 bytes Size = 3 bytes
Column Data
Size > 3 bytes
<257 values PFP/Flat FP(1) FP(1) FP(1)
257-65,536 Not Possible PFP/Flat PFP/Flat FP(2)
65,537-16,777,216 Not Possible No Possible PFP/Flat FP(3)
>16,777,216 Not Possible Not Possible Not Possible FP(3)

A PFP is a Projectable FP Index either an FP(1), FP(2) or

FP(3)
The system will (unless otherwise specified) generate a Flat
FP Index in these conditions
Raw Data Lookup Pages 1 Data Pages
Raw Data
Color 2 1 byte Column
Data Values
Red 1 102 3
Red 1 FP(1)
<256 Data Values
Blue 2 412
Blue Values 3 A
Green 3 662 1
Green C
3 Lookup Pages
Red >256 <65,536 2 V
values Column
01 A
A A 204
Color 02
01 230 03 C C 46
Red
01 V V 298
FP(2)
>65,536 <16M Blue 02 943 03
values 01
Green 03 117 PFP
03 Projectable Data Pages
>16M values
02 FP Index
A
Color C
Color 001 Note that the “Tokens” are the
V
Red Red 001 943 002 same as the data. We can
project directly from the Data A
Blue 003
Blue 002 845 Pages without reference to the V
001
FP(3) Lookup Pages. A
Green 003
Green 003 481 Only works for 1, 2 and 3 byte
Flat FP V
Red 001 datatypes C
003
002
 Enumerated FP handling
• An enumerated FP is an:
– FP(1), FP(2), FP(3), PFP(1,2,3)
• Options that affect FP Rollover
– Minimize_Storage
• Causes all columns to maintain min enumerated FP size
– FP_Lookup_Size
• Determines 1 or 2 byte FP lookup storage size in KB
– FP_Lookup_Size_PPM
• Restricts FP lookup storage size as parts-per-million of main
memory
• New Stored Procedure sp_iqindexmetadata
– Details the specified index or indexes
– Shows the “type” of FP index generated
Maximum Unique Values in FP(1 and 2)
• The lookup table for a 1-byte FP will never be
larger than 1 page (of whatever size)
• For a 2-byte FP the lookup table will be larger
(depending upon the data type size)
– For a bigint of 8 bytes, 65536 entries take only
around 1+ MB.
– For a max varchar of 255 bytes, 65536 entries take
no more than 18 MB.
• It is critical to keep the entire ByteStore (lookup
table) in memory for performance reasons
Column Data Size vs. FP_LOOKUP_SIZE
Maximum Unique Values in FP(3)
FP_LOOKUP_SIZE Lookup Pages
(128K/256K/512K)
4 byte 8 byte 32 byte 64 byte 128 byte 256 byte

1 MB 8/4/2 87,381 65,536 26,214 14,563 7,710 3,986

4 MB 32 / 16 / 8 349,525 262,144 104,857 58,254 30,840 15,947

8 MB 64 / 32 /16 699,050 5424,288 209,715 116,508 61,680 31,895

16 MB 128 / 64 / 32 1,298,101 1,048,576 419,430 233,016 123,361 63,791

32 MB 256 / 128 / 64 2,796,202 2,097,152 838,860 466,033 246,723 127,583

64 MB 512 / 256 / 128 5,592,405 4,194,304 1,677,721 932,067 493,447 255,166

128 MB 1,024 / 512 / 256 11,184,810 8,388,608 3,355,443 1,864,135 986,895 510,333

256 MB 2,048 / 1,024 / 512 16,777,216 16,777,216 7,610,886 3,728,270 1,973,790 1,020667

Calculation is: Max Unique Values = FP_LOOKUP_SIZE / (Column Size + Cardinality)

Note: Consider CASE, remember ‘A’ does not equal ‘a’

 IQ Unique Constraint - 1
• Create Table department
(dept_id char(4) Not Null IQ Unique(200),
emp_lname varchar(25) Not Null IQ Unique(75000),

• Improves load processing

– IQ will construct 1-byte FP index for dept_id
– Won’t try either a 1-byte or 2-byte FP on
emp_lname
• Too many values for either
 IQ Unique Constraint - 2
• In terms of query performance the 1-byte,
2-byte and 3-byte FP indexes can speed up
the server
• The first method is that searching an FP (of
any type) can be done in parallel, which
may be faster than a so-called fast index
(LF and HG)
 IQ Unique Constraint - 3
• In terms of query performance the 1-byte,
2-byte and 3-byte FP indexes can speed up
the server
• The second speed up is that for a LIKE
operator the 1-byte, 2-byte and 3-byte FP
can be faster than an HNG index
– Based on the width of the search predicate
– And the width of the FP index (1, 2 or 3 bytes)
 FP Indexes – Flip-over - 1
• During load, IQ-M will try to build Flat FP first,
unless otherwise specified
• If you specify a low cardinality 1-byte FP at the
start (by using IQ UNIQUE or setting
Minimize_Storage on) then this will
– Resort to a 2-byte FP after 256 values… then …
– Resort to a 3-byte FP after 65,526 values… then …
– Resort to Flat FP after approx. 16m values or n
lookup pages (see two slides on…)
 FP Indexes – Flip-over - 2
• If you specify a 2-byte FP at the start (by using IQ
UNIQUE) then this will
– Resort to a 3-byte FP after 65,526 values… then …
– Resort to Flat FP after approx. 16m values or n lookup
pages (see two slides on…)
• If you specify a 3-byte FP at the start (by using IQ
UNIQUE) then this will
– Resort to Flat FP after approx. 65526 values or n lookup
pages (see two slides on…)
• If you set IQ UNIQUE to greater than 16M (or have
Minimize_Storage off) then the index will be flat
 FP Index Growth
• Although an FP can, as the cardinality grows,
change
– 1-byte FP>2-byte FP
– 2-byte FP>3-byte FP
– 3-byte FP>Flat FP
– it can never revert
• There is a cost in conversion – either way
– The more rows you have loading in the column
when the index flips, the lager the cost in time
• The only backwards conversion is to drop the
column (not the index), recreate and reload
(Expensive)
 Limit for 2 and 3 byte FP - 1
FP_Lookup_Size Def 32767
• This option controls the number of discrete values
that a 2-byte FP can contain, as a maximum Note
this is in Kbytes
• This is used for constraining the size of a 1- or 2-
byte FP
• By default a 2-byte FP or 3-byte FP to Flat FP flip
will occur when the lookup table grows beyond 32
MB
• Remember you must keep the entire ByteStore in
memory, or performance will suffer
 Use of 1 and 2-byte FP Indexes
• Access Paths
• We can perform more operations on the lookup
table of the FP – instead of the other indexes.
– LIKE (simple and Complex)
– Simple Predicate
• YEAR(column_name) > ‘1995’
• SQRT(column_name) < 100
• DATEPART(hh,column_name) between 10 and 12
• Also the resulting “scan” is performed in
parallel (if required)
 CASE - 1
• CASE
– CASE RESPECT is the fastest
– There is a 10-20% hit going to CASE IGNORE
• Implications to FP Indexes
– Regardless of RESPECT vs. IGNORE all
enumerated FP indexes store all the binary
values for the data
– So ABC, abc, Abc, Abc are all stored even
for CASE IGNORE
 CASE - 2
• Remember because we store all
“values” we can flip the FP where we
might not want to
• A solution to this is to set the server in
CASE RESPECT (because it is faster)
• Then use an ETL tool to rtrim() and
ucase() or lcase() all of the incoming
character data
 CASE - 3
• The HG index stores data in what is
called “conditioned” mode.
– For a CASE IGNORE database there is
only one entry per logical value
• ABC = abc = Abc etc.
– For a CASE RESPECT database there has
to be one entry per value
• ABC != abc != Abc etc.
 CASE – 4
• For an LF index we hold partially
conditioned values
– For CASE RESPECT and CASE IGNORE all
values have a bit-map
– This can be wasteful on space
• The reason for having this to allow for
some rare cases (some group by’s) where
we still project values from an LF index
 Low Fast Index
• Traditional Bit Map for Low Cardinality
– Less than 10,000 unique values in a
column
– This is hard-coded, exceeding 9,999
values will rollback the load/insert/update
– Can be unique
 Low Fast Index
• Required for performance involving
– Joins
– Group by
– MIN, MAX, functions
– Where clause predicates
• Equality / Inequality, Ranges, IN lists
 Bitmap for "Yes" or "No" field

What are bitmaps? row-id YES NO

Bitmaps are representations 1 0 1

for each value in a field 2 0 1
True = 1 3 0 1
False = 0 4 1 0
Bit position corresponds to a fixed row ID 5 0 1
For each discrete value there is one 6 1 0
bitmap – the length of which is number
7 1 0
of rows in the table
8 0 1
9 0 1
10 1 0
11 1 0
12 1 0
 Digression on Bitmaps - 1
• If there are 7,000,000 rows in the table
each bitmap will be 7,000,000 bits long
• This could be 1,000,000 bytes (almost a
megabyte)
• If there are 1000 possible values in the
column this could mean 1 Gbyte
(approx.) for the column
• Is this correct ?
 Digressions on Bitmaps - 2
• Yes, sort of…
• In IQ-M there are 4 ways of holding a
bitmap page
• The conditions for the 4 types of page are
– All Zero Bitmap
– Few 1s
– 20-80% 1s
– Almost all 1s
– All 1 bitmap
 Bitmap “Types”
• An all Zero bitmap page is not stored
– just an entry in the block map
• An all 1 bitmap page is also not stored
– a similar entry in the block map
• For the 20-80% 1s there is a real bitmap
• For the nearly all 1s or nearly all zero
pages the data in Run Length Encoded
 Run Length Encoding

• Used when there is a very sparse set in

bits set (or not set)
• Very efficient on storage

1-50,90,102-135,1090-4573,7833,
9011-11430,...
 Bitmap Indexes
B itm ap In d ex fo r ST AT E

ro w-id
1
AK
0
AL
0
AR
0
CA
1
CO
0
DC
0
DE
0
FL
0
...
2 0 0 0 0 0 0 0 1
3 0 1 0 0 0 0 0 0
4 1 0 0 0 0 0 0 0
5 0 1 0 0 0 0 0 0

Each unique value has it’s own bitmap

Designed for incremental additions of rows
Query:
select count(*) from customers where state =‘AL’
 High Group Index - 1
• High Cardinality data columns
– More than 1000 unique values
• Can enforce uniqueness
– Special internal structure for unique HG
indexes
– Automatically created by Create Table for
columns with UNIQUE or PRIMARY KEY
constraint (regardless of cardinality)
 High Group Index - 2

B-Tree
Index

a b c

1,2

New Blocks can be added

3,5,6 into the Linked List
 High Group Index - 3
Much faster for load
Much faster for Skewed data
B-Tree
give bitmaps directly to optimizer
Index

a B(ptr) C=4

1,2

When a page is completely filled

With one value the array is
1011010101001
1010001001001
converted to a bitmap
0010100101010
 High Group Index - 4
• Required for performance on High
Cardinality columns used for:
– Joins
– Select Distinct, Count Distinct
– Group By
• Takes up the most space in the database
• Requires the longest time to load/delete
• Cannot be used with certain data types
 High Group Transition - 1
• The point at which a High Group G-Array
transitions from a list of rowids to a bitmap is
dependant upon the IQ Page Size:
Page Size Transition Point
64 Kbytes 4,096 rowids
128 Kbytes 8,192 rowids
256 Kbytes 16,384 rowids
512 Kbytes 32,768 rowids
This is quite important when calculating how large indexes are
likely to grow, and hence potentially which index to use for a
given column (datatype/cardinality).
 High Group Transition - 2
• Remember when the G-Array is a list or rowids
it is, at most, one page long for each value. A
bitmap is a lot bigger (for larger tables).
Page Size # rows in a 1 page
bitmap
64 Kbytes 640,000 rowids
128 Kbytes 1.2 M rowids
256 Kbytes 2.4 M rowids
512 Kbytes 5.1 M rowids

So as you can see - if you are running on a 64 KB database, and

have a 2 million row table - when the G-Array entry flips to a
bitmap – it will grow the G-Array part of the index size by 3 pages
(for each and every value that flips).
 Specialised High Group - 1
• When a column is created (or altered) to
have the following constraints
– UNIQUE
– Primary Key
• Then the column has a unique HG
index created automatically
• This is a HG without a G-Array
 Specialised High Group - 2
• Provided the combination of two or more columns
is unique, then you can generate a unique (or non-
unique) multi-column HG index on product of the
two columns
• You may still generate other indexes on the base
columns
• The Multicolumn HG is used for:
– Referential Integrity
– Join Processing
• The next slide details the new uses in Sybase IQ
15.0
 Sybase IQ 15.0 – HG Usage - 1
– Multicolumn Ordered Projection
• Multicolumn index can eliminate sorts
– Many more predicates can be executed in the
indexes
• IQ is usually more efficient when it can execute
WHERE clause predicates in IQ’s indexes
• In 15.0 allows more complex expressions to be
executed in the indexes
SELECT COUNT(*) FROM table1
WHERE col1 = 10 and col2 = 50 and col3 = ‘ABC’
Create HG index Indexname_HG on table1(col1, col2, col3)
 Rules for “Order By”
• If an ORDER BY has more than one
column then a Multi Column (HG) Index
will be used if…
• All projected columns + all ORDERing
columns (except ROWID()) exist within an
index
• The ORDERing keys must match the leading
HG columns in order
• If more than one Multi Column Index
matches the above the one with the lowest
cardinality (distinct counts) will be used
 Assume that there is a multicolumn HG index on x, y and z
CREATE HG INDEX t1_xyz_HG on t1(x, y, z)

SELECT x, y, z FROM If a query has an ORDER BY clause, and the ORDER BY column list is a
prefix
t1 of a multicolumn index where all columns referenced in the SELECT list are
ORDER BY x, y present in a multicolumn index, then the multicolumn index performs vertical
projection

SELECT power(x, 2), x + y, sin(z) FROM t1

ORDER BY x, y If expressions exist on base columns in the SELECT list, and all the
columns
referenced in all the expressions are present in the multicolumn index,
then the query will use a multicolumn index;

SELECT ROWID(t1) + x, z FROM t1

ORDER BY x, y, z If the ROWID() function is in the SELECT list expressions,
multicolumn indexes will be used.
SELECT x, y FROM t1
ORDER BY x, y, z, ROWID(t1) If ROWID() is present at the end of an ORDER BY list,
and if the columns of that list—except for ROWID()—use
multicolumn indexes in the exact order, multicolumn indexes
will be used
 MC HG Indexes - A Warning
• However - remember
• HG index size and load performance is
dependent upon the width of the entire index
• This means that M-C HGs are (relatively) slow
to load and they can be big to very big…
• Even with the IQ 15.0 Load Improvements we
still need to consider the use of MC HG
Indexes in Time-Critical Loading Applications
 High Non Group Index (HNG)
• Bit-Wise Index
– data stored as binary
– vertically partitioned
– patented by Sybase
– cannot be unique
– cannot be used with certain data types
• Used for
– range searches for all cardinality columns
– aggregation (sum and average functions)
 HNG - High Card Bit-Wise Index

Sales in binary form S ales in b in ary fo rm

Sales 8 bit 4 bit 2 bit 1 bit 8 b it 4 b it 2 b it 1 b it

6 0 1 1 0 0 1 1 0
9 1 0 0 1 1 0 0 1
5 0 1 0 1 0 1 0 1
11 1 0 1 1 1 0 1 1
9 1 0 0 1 1 0 0 1
3 0 0 1 1 0 0 1 1
7 0 1 1 1 0 1 1 1
12 1 1 0 0 1 1 0 0
Data with large number of values stored in binary form
Data sliced vertically - each bit position can be manipulated separately
Many bit positions are either all on or all off so no storage space is required
with compression
Sales 8 bit 4 bit 2 bit 1 bit

6 0 1 1 0 For the query:

9 1 0 0 1 select sum(sales) from customers
5 0 1 0 1
11 1 0 1 1 ASIQ performs the sum
9 1 0 0 1 as follows :
3 0 0 1 1
7 0 1 1 1
12 1 1 0 0

#1bits on*1 + #2bits on*2 + #4bits on*4 +#8bits on*8

(6*1) + (4*2) +(4*4)+ (4*8) = 62

 HNG Indexes with Other Indexes
• Any Cardinality columns also need an
HNG Index
– Columns used with aggregates (sum, avg)
– Range searches
– Root String searches
• example: where cust_name like “Stan%”
• all other string searches will use the FP index
 Compare (CMP) Index
• The CMP index is used for “comparing” 2
columns in the same table
• It is really just 3 bitmaps
– A “less than” bitmap
– An “Equal to” bitmap
– A “Greater than” bitmap
• Performance of t1.col1 > t1.col2 is
substantially improved
• Load times are only marginally affected (<1%)
 Word (WD) Index
• This is a specialised index that indexes each
and every “word” in a column
• Used for char() varchar() and long varchar()
• Slightly faster to load than an equivalent
column HG index
• Accessed by the “contains” verb
where t1.col1 contains (‘Richard’, ‘Soundy’)
 Word (WD) Index
• From IQ 12.5 the WRD index supports the like
clause – but it is only accelerated with the WRD
index if the token is delimited

• Like “Richard” -> Handled by HG or LF

• Like “%Richard%” -> Handled by FP
• Like “Richard%” -> Handled by HNG/HG/LF
(Range query)
• Like “%Richard%”-> Handled by WRD
(Note spaces)
 Word Index Use
• The delimiters and the length of the entries
can be set for the WD index during index
creation time.
• By default the delimiters are all the ASCII
characters not defined as number or
alphabetic
• The limit is the max size of each entry (word
length)
CREATE WD INDEX earnings_wd ON
earnings_report_table(earn_col) DELIMITED BY ‘:;.’ LIMIT 25
 DATE, TIME and DTTM Indexes - 1
• There are 3 “period” indexes
• These are the DATE, TIME and DTTM
indexes.
• These are not automatically generated, they
require a manual create index command for
each required index.
• Each index has a complex internal structure
that is shown on the next page(s)
 DATE, TIME and DTTM Indexes - 2
Day of Quarter of Week of
Index Day Month Year Hour Minute Sec.
Week Year Year

DTTM         

DATE         

TIME         

The components of the index are persistent bitmaps,

similar in construction to an LF Bitmap structure
 DATE, TIME and DTTM Indexes - 3
• The Query Engine is fully conversant with the
new indexes and DATEPART(), DATEDIFF()
etc. will work incredibly fast in comparison
with earlier Sybase IQ implementations.
• Also it should be noted that the Query Engine
will use these indexes even for syntax such
as below:
Where datepart(month, date_column) = ‘Jan’
 Indexing Strategy
• Some columns may now need 2 or 3 (or
even more )indexes
– Possible Combinations of “basic” indexes:
• FP + LF
• FP + LF + HNG
• FP + HG
• FP + HG + HNG
• Unlikely Combination
– FP + LF + HG (the HG will always be used in
preference to the LF)
 Assigning Indexes
• Every column has the Default Index (FP)
– Create Table command builds this index
– cannot be dropped

• No additional indexes needed with

– columns used only for Projection (Select list)
– columns with ‘bit’ datatype
 High Group and Low Fast
• Regardless of cardinality put an HG (or LF) on:
• Any Primary Key (if not defined in create or alter
table – where it is created automatically)
• Any UNIQUE column (if not defined in create or
alter table – where it is created automatically)
• Any column used in a join
• Why do the above?
– It is better and generally faster – especially on Low
Cardinality Columns
– For a LC column the HG may be smaller than the LF
• This is a change from previous thoughts
 High Group – Warning - 1
• A High Group Index is the most complex index
• It takes the longest to load (and delete)
• The load time is directly proportional to the
length or width of the data component
– Smaller (in size not cardinality) data is better
– Cardinality is also an issue the higher the number of
“new” values in an incremental load the slower the
load can get
– Can the data be broken into more than one column?
 Low Fast or High Group
• If a column is not
– A Primary Key
– UNIQUE
– used in a join (ad-hoc or Join Index)
• But is still used for more than just a
projection column it will require another
one or two indexes
 Low Fast or High Group
• Put an HG or an LF (depending on
Cardinality) on the following column types
– used in SELECT statement with functions:
• min, max, count, count(distinct)
– used in WHERE clause predicates:
• equality / inequality
• IN list
• EXISTS
– used in GROUP BY
– used in ORDER BY
 Additional Indexes
• The 1000 cardinality break point is not a fixed
limit
• Absolute limit is cardinality of 9,999
• There are no hard and fast rules
– A 5,000 card LF may work faster in one case
– In an other an 800 card HG may be faster
• Remember an LF is much faster to load data
into than an HG
• Although for larger cardinality an LF may well
be larger on disk – and hence slower for query
operations.
 Regardless of Cardinality
• Build an HNG index on all columns used
– in BETWEEN or RANGE comparisons
– with AVG() or SUM() functions
– root string searches
• Examples:
– where sold_date between ‘1/5/99’ and ‘2/5/99’
– where revenue > 1000
– Select sum(revenue) from ...
– where customer_name like ‘Syb%’
• But remember an enumerated FP may do
the above faster!
 HNG and LF
• It is also suggested that for columns that
are used in Range searches have both an
HNG and an LF on them
• If the LF can be applied (for cardinality
reasons)
• Why? –
– The optimiser can (in certain circumstances)
use either an HNG or an LF to satisfy a range
query – so lets have both – if we can afford the
load and delete overheads
 HNG or LF (?)
• An HNG will be faster than an LF for SUM() anytime
the number of distinct values is larger than the
number of bits in the column’s data type
• An HNG will be faster than an LF for range
predicates whenever the number of distinct values
within the selected range exceeds ~2.5 times the
number of bits in the column’s data type
• For very low cardinality, in a join column a HG will be
the fastest index
 HG vs. LF and HNG
• For very low cardinality (dimension) tables then
an HG may be better than both an LF and an
HNG
• Why?
• The HG may only be 2 pages long (1 for the B-
tree and 1 for the G-Array)
• Whereas the LF will be n pages long (1 for the
B-Tree Identity page and 1 for each value)
• The HNG can also be large (1 page for the B-
Tree Identity Page and 1 for each bit in the
datatype)
 Indexing Tips - 1
• Use IQ Unique constraint in Create Table
– This will improve (usually) load performance
– Cardinality near the critical thresholds is the
most important
• Create Indexes before loading Data
– ASIQ is NOT like ASE – indexes load very
fast, so do all the index loads simultaneously
 Indexing Tips – 2
• Choose Indexes by committee
– If in doubt as to the use of a column ask
everybody!
– If in doubt index everything, you can always drop
indexes later (but beware of load times on wide
HG)
• Specify HG index as UNIQUE
– This information is of great use to the optimiser
– Also the UNIQUE HG is a very efficient index
 One Last thing on Indexes
• There is a bitmap on each and every
column and each and every table called
the Existence Bitmaps
• The row Existence Bitmap tells IQ-M
whether there is a row for that RowId or
not
• The Column Existence Bitmap tells IQ-M
that the column has a value (not NULL)
 A Well Forgotten Fact
• There are 2 reasons IQ-M needs enumerated
indexes (LF, HG, enumerated FP)
• The primary reason we have discussed, that is
to solve the query
• In IQ 12.4 and beyond there is another reason,
to provide the Optimizer with accurate distinct
counts
• The second reason is very important – this
allows the optimizer to very accurately plan how
to solve the query
 Database Administration –
Index Maintenance System Procs
• New System Stored Procedures for the DBA
– To Report Index Fragmentation and Density
– To Rebuild Index(es)
• sp_iqindexfragmentation ( table| index_names )
– Reports on the amount of empty space in index
structures
• sp_iqrowdensity (table | columns )
– Reports internal row fragmentation of FP indexes
 Database Administration –
Index Maintenance
• sp_iqrebuildindex ( table | indexes )
– Rebuild any index that has become
fragmented
– Will recreate optimized FP indexes
• Over time some optimized FP indexes may have
"rolled over" to become a Flat FP
• This utility will rebuild the optimized FP index if
cardinality is within limits
Index Advisor and Workload Monitor

• But, given all the above

• IQ can (and will) tell you if you if:
– You need another index on a column
– Or, you have an index (or any object) that
you do not use
 Index Advisor 1
• This option allows the “display” of the
optimizer “suggestions” as to “missing”
indexes
• Messages sent to
– The Query Plan
– The IQ Message Log
 Index Advisor 2
• Index advisor messages can be accumulated
across the execution of many queries
– Index_Advisor option must be set ON
• New system stored procedure
sp_iqindexadvice displays accumulated
index advice messages
 sp_iqindexadvice ( [resetflag] )

• Retrieves aggregated index advice

messages
– Displays Advice, Number of Instances and
Last Date issued

80 Novemb
 Index Advice Usage
• Index Advice messages persist only in memory
– Number of messages retained is set by the database
option Index_Advisor_Max_Rows
– Index Advice messages may be cleared at anytime
using the reset flag (any nonzero number)
• Example: exec sp_indexadvice (1)
– Messages will also be lost when the IQ server is
shutdown
• You can create an event to capture the output to a
table
 Workload Monitoring
– Workload Monitoring provides 7 system stored
procedures which allow the DBA to monitor
database object usage
• DBA can assess usage of tables, columns and indexes
• These are useful in decisions regarding dropping unused
indexes and objects occupying space in the database

– Like the Index Advisor all data collected by the

Workload Monitor is only kept in memory
• Will be lost at server shutdown if not captured
 Enabling Workload Monitoring
– sp_iqworkmon stored procedure enables
Workload Monitoring, reports on the status of
the monitor and the number of rows collected

sp_iqworkmon (action | mode)

• action: “start” | “stop” | “status”| “reset”

• mode: “index” | “table” | “column” | “all”
 sp_iqworkmon Procedure

Monitor is Off Monitor Enabled Monitor Collecting

Data
 Monitoring Objects - 1
– sp_iqtableuse() - Reports Tables Used in
Queries
• Returns owner, date/time of last access, number of
query references and table unique identifier

– sp_iqcolumnuse() – Reports Columns Used in

Queries
• Returns table name, column name, owner, date/time of
last access, column unique identifier, number of query
accesses
 Monitoring Objects - 2

– sp_iqindexuse() - Reports Indexes Used in

Queries
• Returns index name, table name, owner, index
unique identifier, index type, date/time of last
access, number of metadata/uniqueness
accesses, number of query accesses, and number
of access for Unique or RI checks
 Table Usage

– sp_iqtableuse
 Column Usage
– sp_iqcolumnuse
 Index Activity
– sp_iqindexuse
• NOpt = Number of Metadata/Uniqueness Accesses
• NQry = Number of Query Accesses
• NConstraint = Number of Accesses for Unique or RI
Checks
 Monitoring Unused Objects
– sp_iqunusedtable()
• Returns table name and owner

– sp_iqunusedindex()
• Returns index name, table name, owner, index
type

– sp_iqunusedcolumn()
• Returns table name, column name, owner
 Capturing Workload Monitoring
Statistics - 1
– Like the Index Advisor, you need to
capture the statistics periodically

– Create a table to capture the output,

periodically insert data, and then reset the
workload monitor
• You can use the output of the stored procedure
as input data
 Capturing Workload Monitoring
Statistics - 2
Example: Insert rows about unused indexes
from the stored proc for unused indexes and
then reset the workload monitor statistics

insert mon_table (idx_nm, tab_nm, owner, idx_typ)

select * from sp_iqunusedindex();
commit;
exec sp_iqworkmon reset;
 About Sybase Central (SC)

• SC is system management framework.

• A SC plug-in is a pluggable component
for managing a point product.
• IQ Plug-in 15.0 runs in SC 6.0
• IQ Plug-in 15.0 extends SA Plug-in 11.0
 New Features in IQPlugin 15
• Single Server Management
– DBSpace Management
– Table Partitioning
– Large Result Set Filtering
– Monitoring
– Login Policy
– Remote File System Browsing
 Monitoring in Sybase Central
• “Performance Monitor” tab is completely re-
developed.
• Over 70 statistics grouped by10 categories: CPU
Usage, Memory Usage, Cache Statistics, Connection
Statistics, Thread Usage, Transaction Statistics,
DBSpace Usage, Store I/O Statistics, Network
Statistics, Request Statistics.
• Real-time dynamic charts: Time Series, 2D Bar, 3D
Bar
 Monitoring in Sybase Central
• The monitor (data collection) is automatically
started/stopped upon the show/hide of the
“Performance Monitor” tab.
• Data collection rate is configurable.
• Time window is configurable.
• Chart be can saved as JPEG image.
• Chart is also printable.
Performance Monitor Tab
Change Statistics to Monitor
Time Series Chart With Data Table Collapsed
 Backup Features in IQ 15

• Four types of database backups:

– Full: makes a complete copy of the database
– Incremental: copies all transactions since the
last backup of any type
– Incremental-since-full: copies all transactions
since the last full backup
– Virtual: copies all of the database except for
the table data from the IQ Store. Table data
should be backed up with operating system-
level commands.
 Backup and Restore in IQ 15
• There are four significant changes to backup
and restore:
– An IQ full, incremental-since-full or incremental
backup may be restricted to only the set of read-
write files in the database
– An IQ backup may selectively back up a set of
read-only DBSpaces and/or read-only files
– A restore of a backup may, optionally, restore only
a subset of the DBSpaces and/or files.
– The subset must be READONLY DBSpaces
and/or files
– System tables now contain backup history