AIX On Oracle ARPUG Presentation PDF

Download as pdf or txt
Download as pdf or txt
You are on page 1of 72

Advanced Technical Skills (ATS) North America

Oracle Database on AIX -


Key Ingredients for a Successful Relationship
... and improved communication between
System Administrator, DBA and SAN team

Ralf Schmidt-Dannert
Executive IT Specialist, IBM

9/18/2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Special notices
This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in
other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM
offerings available in your area.
Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions
on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give
you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY
10504-1785 USA.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives
only.
The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or
guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the
results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations
and conditions.
IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions
worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment
type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal
without notice.
IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.
All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this
document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-
available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document
should verify the applicable data for their specific environment.

Revised September 26, 2006

2 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Special notices (cont.)


© IBM Corporation 1994-2012 All rights reserved.
References in this document to IBM products or services do not imply that IBM intends to make them available in every country.
Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at
http://www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other
countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered
trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Cell Broadband Engine and Cell/B.E. are trademarks of Sony Computer Entertainment, Inc., in the United States, other countries, or both and are used under license
therefrom.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.
The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
environmental costs and performance characteristics may vary by customer.
Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does
not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information,
including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or
any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance,
function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here
to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any
user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements
equivalent to the ratios stated here.

3 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Acknowledgements
Dale Martin, IBM ATS
Rebecca Ballough, IBM ATS
Steven Nasypany, IBM ATS
Damir Rubic, IBM ATS

4 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Learning Objectives

Develop a better understanding how the Oracle database


and AIX interact especially in the area of memory.
Be aware of typical “pitfalls” in the context of storage layout
and configuration for Oracle databases on AIX.
Know the AIX tuning parameters and their “best practice”
values for an Oracle database server.
Know what tools are available on AIX to analyze
performance problems.

5 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Agenda
AIX - Oracle: Memory Utilization
– AIX Memory - Overview, kernel tuning parameters, …
– What memory is Oracle using – file cache / computational,
4KB / 64KB / 16MB pages, pinned / un-pinned?
– Tuning for expected workload behavior

AIX - Oracle: CPU Utilization


– How to utilize IBM PowerVM features in an Oracle environment?
– Oracle license implications
– Tuning
AIX - Oracle: I/O
– Storage options, characteristics and comparison
– Database layout on AIX - Best Practice
– Setup and tuning
AIX - Oracle: Network
AIX - Oracle: Miscellaneous
My system doesn't behave - What tools are available in AIX to investigate
the cause?
Where to find AIX/Oracle documentation?

6 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Power 7 Systems Portfolio fully supports Oracle Databases


Power 795
Power 780

Power 770

Power 750

NEW
Power 775

BladeCenter Intel
PS700 / PS701
PS702 / Power7
PS703 / PS704
Power 710
Power 720 Storage HPC
Power 730 Network

Power 740
Power 755
PureSystems
© 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX – Oracle

Memory Utilization

8 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX Physical Memory – pools, pages, page lists


Physical
Automatic (*1)
Automatic Memory
Automatic (*1)
Automatic
psmd proc. Manual
4KB 64KB 16MB 16GB
free psmd free
proc. Manual
free free
4KB 64KB 16MB 16GB
free
Automatic free free free

psmd proc.Automatic
4KB 16MB 16GB
64KB
used psmd proc.
used used 16MB used
4KB
64KB used 16GB
Memory Pool 0 used
used
Memory Pool 1
Paging Space
On Disk
This is a simplified view

(*1) Only when large amounts of memory are requested at once and not enough free pages on 4KB / 64KB free lists.
9 September 2012 © 2012 IBM Corporation
M B used

0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
1 4 :0 2
1 4 :0 2
1 4 :0 2
1 4 :0 3
1 4 :0 3
1 4 :0 3

10
1 4 :0 3
1 4 :0 4
1 4 :0 4
1 4 :0 4
1 4 :0 5
1 4 :0 5
1 4 :0 5
1 4 :0 6
1 4 :0 6

4kb free
1 4 :0 6

4kb used
1 4 :0 6
1 4 :0 7
1 4 :0 7
1 4 :0 7
1 4 :0 8
1 4 :0 8
1 4 :0 8
1 4 :0 9
1 4 :0 9
1 4 :0 9
1 4 :0 9

Time
1 4 :1 0
1 4 :1 0
1 4 :1 0
1 4 :1 1
1 4 :1 1
1 4 :1 1
64kb used
64kb free

1 4 :1 2
Advanced Technical Skills (ATS) North America

1 4 :1 2
1 4 :1 2

September 2012
1 4 :1 2
1 4 :1 3
1 4 :1 3
1 4 :1 3
4K - 64K - 16MB Page Dynamics

1 4 :1 4
1 4 :1 4
1 4 :1 4
1 4 :1 5
1 4 :1 5
1 4 :1 5
1 4 :1 5
1 4 :1 6
4KB_used MB 4KB_free MB 64KB_used MB 64KB_free MB 16MB_usedMB 16MB_freeMB

1 4 :1 6
1 4 :1 6
1 4 :1 7
1 4 :1 7
1 4 :1 7
1 4 :1 8
4KB pages
64KB pages
16MB pages

© 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX Memory Management Concepts


Two primary categories of memory pages: Computational and File System
AIX tries to utilize all of the physical memory available
– What is not required to support computational page demand will tend to be used for file
system cache
Requests for new memory pages are satisfied from the free page list(s)
– Small reserve of free pages maintained by “stealing” Computational or File pages
– AIX uses “demand paging” algorithm – generally not written to paging space until “stolen”
System% Process% FScache%
100

90

80
Free Memory
70
% Physical memory used

60

50
File cache is always
40
4KB memory pages !
30

20

10

0
12:01
12:06
12:12
12:17
12:22
12:28
12:33
12:38
12:44
12:49
12:54
13:00
13:05
13:10
13:16
13:21
13:26
13:32
13:37
13:42
13:48
13:53
13:58
14:04
14:09
14:14
14:20
14:25
14:30
14:36
14:41
14:46
14:52
14:57
15:02
15:08
15:13
15:18
15:24
15:29
15:34
15:40
15:45
15:50
15:56
16:01
16:06
16:12
16:17
16:22
16:28
16:33
16:38
16:44
16:49
16:54
17:00
17:05
17:10
Time

11 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

VMM Page Stealing Process (lrud)


Definitions:
lrud = VMM page stealing process = LRU Daemon (1 per memory pool)
numperm, numclient = # pages currently used for filesystem buffer cache
maxperm, maxclient = target maximum # pages to use for filesystem buffer cache
free pages = # pages immediately available to satisfy new memory requests

vmo Parameters:
minperm% = target min % real memory for filesystem buffer cache
maxperm%, maxclient% = target max % real memory for filesystem buffer cache
minfree = target minimum number of free memory pages
maxfree = number of free memory pages at which lrud stops stealing pages

When does lrud (for a given memory pool and page size) start?
When free pages < minfree (4K and 64K pages)
When (maxclient - numclient) < minfree (4K pages only)

When does lrud stop?


When free pages > maxfree (4K and 64K pages)
When (maxclient – numclient) > maxfree (4K pages only)
12 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

VMM Page Stealing Thresholds (AIX 7.1, 6.1, 5.3)


minfree / maxfree values are per memory pool
– Total system minfree = minfree * # of memory pools
– Total system maxfree = maxfree * # of memory pools

AIX 7.1, 6.1 and 5.3 defaults are acceptable for most workloads
– Consider increasing if vmstat ‘fre’ column frequently approaches zero, or
if “vmstat –s” shows significantly increasing “free frame waits” over time

Suggested starting points if tuning is required:


– minfree >= max(960,(120 x # logical CPUs )) / #mem pools
– maxfree = minfree + ((MAX(maxpgahead, j2_maxPageReadAhead) *
# logical CPUs) / # mem pools)

Example:
10-way LPAR with SMT-2 enabled,
with maxpgahead=8 and j2_maxPageReadAhead=128 and 2 memory pools:
minfree = 1200 = max(960,(120 x 10 x 2) / 2
maxfree = 2480 = 1200 + ((max(128,8) x 10 x 2) / 2)
vmo –p –o minfree=1200 –o maxfree=2480
13 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX System Paging Concepts & Requirements


By default, AIX uses a “demand paging” policy
– For Oracle DB, the goal is ZERO system paging activity
– Filesystem pages written back to filesystem disk (if dirty); never to system paging space
– Unless otherwise specified, computational pages are not written to paging space unless/until they
are stolen by lrud. (*1)
Once written to paging space, pages are not removed from paging space until the
process associated with those pages terminates
– For long running processes (e.g. Oracle DB), even low levels of system paging can result in
significant growth in paging space usage over time
– Paging space should be considered a fail-safe mechanism for providing sufficient time to identify and
correct paging issues, not a license to allow ongoing system paging activity
Paging space allocation Rule-of-Thumb:
Physical Paging
– ½ the physical memory + 4 GB, with the following cap: Memory Space
lower or Max
equal to

Resolve paging issues quickly: 128GB 60GB

Reduce effective minimum file system cache size (minperm) 256GB 100GB

Reduce Oracle SGA or PGA size 512GB 150GB

Add physical memory 1TB 200GB

14 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

JFS2 inode / metadata caches


JFS2 utilizes two caches - one for inodes Unused
and one for metadata
Caches grow in size until maximum size is
reached before cache slots are reused
File cache
Default values are tuned for a file server!
Each entry in the inode cache requires
about 1KB of physical memory
1MB of memory can cache about 1000 files
Process
100%
Configured via ioo parameters:
– j2_inodeCacheSize (Default: 400 = 10%) *1 physical
memory
– j2_metadataCacheSize (Default: 400 = 4%) *1 “System memory”

The current memory use can be


verified via: AIX “pinned”
cat /proc/sys/fs/jfs2/memory_usage 4% *1 Can not
metadata cache: 31186944 metadata cache
be paged
inode cache: 34209792 !
total: 65396736 10% *1 Inode cache
*1 Note: Default values in AIX 7.1 are 200 (5%) , 200 (2%)
15 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Large Segment Aliasing (AKA Terabyte Segment)


Workloads with large memory footprints and low spatial locality may
perform poorly due to Segment Lookaside Buffer (SLB) faults
– May consume up to 20% of total execution time for some workloads
Architectural trend toward smaller SLB sizes can exacerbate SLB
related performance issues:
– POWER6 has 64 SLB entries – 20 for kernel, 44 for user processes – allowing
11GB of accessible memory before incurring SLB faults
– POWER7 has 32 SLB entries – 20 for kernel, 12 for user processes – allowing
3GB of accessible memory before incurring SLB faults
Oracle SGA sizes are typically in the 10s to 100s of Gigabytes
With Large Segment Aliasing, each SLB entry can address up to
1TB of memory
– Supports shared memory (Oracle SGA) addressability for up to 12TB on
POWER7 and up to 44TB on POWER6 without SLB faults
– Enabled by default on AIX 7.1
– Disabled by default on AIX 6.1 TL06+ – May be enabled by setting “vmo”
esid_allocator=1 (Recommended)
16 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Local, Near and Far Memory


Power Systems use a “shared memory” model
– any processor has local access to part of memory

High-end Power Systems (e.g. p770, p780, p795) use multiple building blocks
(CECs) to scale capacity
– Each building block has it’s own set of processor and memory chips
– Building blocks are interconnected via a switched communications fabric

The closer the memory is to the processor accessing it, the faster the memory
access
– Local Memory: Directly attached to the chip’s memory controller
– Near Memory: On an adjacent chip, accessed via intra-node communication paths
– Far Memory: On a different CEC drawer, accessed via inter-node communication paths

Model Local Near Far


Power 710/730 Same Chip Other Chip n/a
Power 720/740 Same Chip Other Chip n/a
Power 750 Same Chip Other Chip n/a
Flex System p260 / p460 Same Chip Other Chip n/a
Power 770/780 Same Chip Other Chip, Same CEC Different CEC
Power 795 Same Chip Other Chip, Same CEC Different CEC

17 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Oracle Memory and Memory Affinity


Oracle SGA is “striped” across all the available memory in the LPAR
– If the LPAR configuration has a combination of near, local and far memory allocated to
it, SGA will be (more or less) evenly spread across all of it
– The greater the number of CECs involved, the greater the likelihood of remote memory
accesses
Oracle PGA for a given process tends to be allocated in the near memory
of the processor that process was running on when the memory was
allocated
– The AIX dispatcher will attempt to maintain affinity between a given process and the
processor that process gets scheduled on
– rsets may (optionally) be used to force affinity to a subset of available processors (e.g.
those on a given chip, or within a given CEC), although this could potentially cause
dispatching delays in heavily loaded environments
“vmo” enhanced_affinity_private (“vmo” restricted parameter)
– The percentage of application data that is to be allocated local, with the remaining
memory to be striped across all available memory in the LPAR
– Default value is 20% in AIX 6.1 TL5 and 40% in AIX 6.1 TL6+ and AIX 7.1
– A value of 100% has been used in a number of Oracle performance benchmarks, but
before changing a restricted parameter, contact IBM support.
18 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

“Socketizing” Workloads (p770,p780,p795)


Don’t over-allocate CPUs
– If a given workload (LPAR) requires <= 16 processors (single CEC), don’t allocate more than
16 processors (vCPU)
– If all the LPARs in a given shared pool require (in aggregate) <= 32 processors (2 CECs),
don’t allocate more than 32 processors (vCPU) to the shared pool
– For Shared Processor LPARs, don’t overallocate vCPUs relative to Entitled Capacity

Don’t over-allocate memory


– May cause processors/memory to be allocated on additional CECs because there wasn’t
sufficient free memory available on the optimal CEC

Help the Hypervisor do its job


– Stay current on Firmware (e.g. AM720_101 or later) to avoid any known CPU/memory
allocation or virtual processor dispatching issues
– Where appropriate, consider LPAR boot order to ensure high priority LPARs get optimal
choice of the available CPUs and memory

Consider the use of rsets (Advanced Tuning – Can lead to unexpected results)
– For example, where heavy application (e.g. java) workload is co-located on DB LPAR
– Or, could potentially affinitize some of the Oracle processes, e.g. system processes, or DB
connections spawned by a given listener
19 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Oracle Server Architecture – Memory Structures

PGA
PGA PGA
RVWR PMON SMON
PGA

System Global Area (SGA) ARC0 Archive


Logs
DB Buffer Cache
Shared Pool Redo Log PGA
Flash-
Buffer
back Log

LGWRn
PGA

DBWRn CHKP
Redo
Logs
User PGA
PGA Control
Files

PGA D000
DB SGA is shared among processes
Files
PGA is private to an individual server or
background process

20 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Memory Usage in an Oracle Database Environment


Computational
Some used for AIX kernel processing
Some used by Oracle/client executable programs
Includes Oracle SGA and PGA memory

File System Cache


May be used for caching or pre-fetching of Oracle .dbf files
– Only for local file system based (non-RAC) environments where Direct I/O
(or Concurrent I/O) is not used
May be used for other Oracle related files
– Archive logs, export/import files, backups, binaries, etc.
May be used for non-Oracle related files
– Application files, system files, etc.

Virtual Memory Management Priorities


Always want to keep computational pages in memory -- System
paging/swapping may degrade Oracle/application performance
– Allocate enough physical memory to support computational footprint
requirement + small file cache
– When necessary, steal filesystem pages, not computational

21 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Parameter Tuning (AIX 7.1, AIX 6.1)

Most AIX 7.1, AIX 6.1 parameters configured by default to be


‘correct’ for most workloads

As of AIX 6.1, many tunables are now classified as ‘Restricted’


– Only change if AIX Support requests it
– Restricted parameters will not be displayed unless the ‘-F’ option
is used for “vmo” or other commands

When migrating from AIX 5.3 to AIX 6.1 or AIX 7.1, existing
parameter override settings in AIX 5.3 will be transferred to
AIX 6.1 or AIX 7.1 environment
– After migration, review/verify parameter values are properly set

22 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Recommended vmo “Starting Points” - Review


Parameter Recommend AIX 7.1 AIX 7.1 AIX 6.1 AIX 6.1 AIX 5.3
Value Default Restricted Default Restricted Default
esid_allocator 1 1 Yes 0 Yes N/A
minperm% 3 3 No 3 No 20
maxperm% 90 90 Yes 90 Yes 80
maxclient% 90 90 Yes 90 Yes 80
strict_maxclient 1 1 Yes 1 Yes 1
strict_maxperm 0 0 Yes 0 Yes 0
lru_file_repage 0 N/A N/A 0 Yes 1 or 0(*1)
lru_poll_interval 10 10 Yes 10 Yes 10
minfree 960 960 No 960 No 960
maxfree 1088(*2) 1088 No 1088 No 1088
page_steal_method 1 1 Yes 1 Yes 0
memory_affinity 1 1 Yes 1 Yes 1
v_pinshm 0 0 No 0 No 0
lgpg_regions 0 0 No 0 No 0
lgpg_size 0 0 No 0 No 0
maxpin% Leave at Default 90(*3) No 80(*3) No 80
vmm_klock_mode 2 (see notes) 2 No 1 Yes N/A
*1 Depending on AIX 5.3 TL level *2 Do not reduce below default *3 Depends on LSA use – LSA active 90, otherwise 80
23 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX Multiple Page Size Support with Oracle


4K
– Typically used on older hardware which does not support 64K pages, or with older Oracle versions
(< 10.2.0.4)

64K – Preferred – most of the “Large Pages” benefit without the issues
– In 10.2.0.4+ (*1), 11g Oracle will automatically use 64k for SGA if supported by hardware
– May also be used for program data, text and stack areas: *1 – with Oracle patch 7226548
# ldedit –btextpsize=64k –bdatapsize=64k –bstackpsize=64k oracle
# export LDR_CNTRL=DATAPSIZE=64K@TEXTPSIZE=64K@STACKPSIZE=64K oracle

16M (Large Pages) – Discouraged – Limited benefit and potential adverse impacts
– May be useful if maximum possible performance is required and Oracle SGA changes are tightly
coordinated with AIX Sysadmin
– If improperly configured, can contribute to severe system paging and kernel panics

16G available with POWER5+ and later & AIX 5.3 TL4+ and later AIX releases
– Cannot be used with Oracle

Also see:
(*1) – MOS note
with Oracle # 372157.1
patch 7226548

24 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Oracle Memory Structures Allocation


9i : Dynamic memory resizing
– db_cache_size (dynamic parameter)
– sga_max_size (static parameter) – maximum size of the SGA for the lifetime of the
instance.
– pga_aggregate_target (dynamic parameter) – specifies the target aggregate PGA
memory available to all server processes attached to the instance; note that this is a
target and not a hard limit!
– additional parameter : db_cache_advice (dynamic parameter) – enables or disables
statistics gathering used for predicting behavior with different cache sizes.

10g : Automatic Shared Memory Management (ASMM)


– sga_target (dynamic) – if set, db_cache_size, shared_pool_size, large_pool_size and
streams_pool_size are dynamically sized
• Minimum values for these pools may optionally be specified - recommended
– Can be dynamically increased up to sga_max_size
– To use ASMM, sga_target must be >0

25 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Oracle Memory Structures Allocation


11g : Automatic Memory Management (AMM)
– memory_target (dynamic parameter) – specifies the total memory size to be
used by the instance SGA and PGA. Exchanges between SGA and PGA are
done according to workload requirements
– If sga_target and pga_aggregate_target are not set, the policy is to give 60%
of memory_target to the SGA and 40% to the PGA
– memory_max_target (static parameter) – specifies the maximum amount of
memory allowed to be used by the Oracle instance
– To use Automatic Memory Management, memory_target must be >0 and
LOCK_SGA=false

See Metalink notes 443746.1 and 452512.1 explaining AMM and these new parameters

AMM dynamic resizing of the shared pool can cause a fair amount of “cursor: pin s”
wait time. One strategy to minimize this is to set minimum sizes for memory areas you
particularly care about.
In addition, you can change the frequency how often AMM analyzes and adjusts the
memory distribution. See: Metalink note: 742599.1 ( _memory_broker_stat_interval)
26 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

SGA_MAX_SIZE and LOCK_SGA implications (11g, 10.2.4.0+)


LOCK_SGA=false Preferred
– Oracle dynamically allocates memory for the SGA only as needed up to the size
specified by SGA_TARGET
– SGA_TARGET may be dynamically increased, up to SGA_MAX_SIZE
– 64K pages automatically used for SGA if supported in the environment. If needed, 4K (or
16M) pages are converted to 64K pages. Down-conversion of 16M pages to 64K pages
is only triggered at DB startup if needed; after startup additional unused 16M pages are
not converted, even if not enough 4K or 64K pages are available!

LOCK_SGA=true Discouraged
– Oracle Automatic Memory Management (AMM) cannot be used
– Oracle pre-allocates all memory as specified by SGA_MAX_SIZE and pins it in memory,
even if it’s not all used (i.e. SGA_TARGET < SGA_MAX_SIZE)
– If sufficient 16M pages are available, those will be used. Otherwise, all the SGA memory
will be allocated from 64K (if supported) or 4K pages (if 64K pages are not supported).
If needed, 4K or 16M pages will be converted to 64K pages, but 16M pages are never
automatically created. Also see comment above re 16M to 64K page conversion!
– If a value for SGA_MAX_SIZE is specified larger than the amount of available memory
for computational pages, the system can become unresponsive due to system paging.
– If the specified SGA_MAX_SIZE is much larger than the currently available pages on the
combined 64K and 16M page free lists, the database startup can fail with error: “IBM AIX
RISC System/6000 Error: 12: Not enough space”. In this case re-try to start the
database.
27 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Memory - Page Sizes and use with Oracle DB


Page type small (*1) medium large huge
Memory page size 4KB 64KB 16MB 16GB

Processor all POWER5+™ or later POWER4™ or later Power5+ or later

AIX 5.3 TL04+ AIX 5.3 TL04+


AIX™ support all AIX 5.3 and later
AIX 6.1, AIX 7.1 AIX 6.1, AIX 7.1
Kernel 32bit & 64bit 64bit 32bit & 64bit 64bit
Restricted no no yes yes
Pageable yes yes no no
Requires User Yes
no no Yes (OS)
Configuration (HMC + OS)
small <-> medium small <-> medium (*3) large -> medium
Automatic conversion no
(AIX6.1+, POWER6+) (AIX6.1+, POWER6+) (AIX6.1+, POWER6+)

> = 10.2.0.4/5 default (*2)


Oracle SGA support <10.2.0.4 default 10g / 11g no
11g default

Activate use for > = 10.2.0.4/5 autom. (*2) lock_sga=true and


automatic n/a
Oracle SGA 11g: automatic user permissions

*1 - Used system wide if Active Memory Sharing (AMS) is used, *2 – with Oracle patch 7226548 for 10gR2
*3 – This conversion is only triggered at database startup, if needed, and not for later memory allocations

28 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX – Oracle

CPU Utilization

29 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Physical, Virtual, Logical Layers

AIX 5.2 AIX 5.3 AIX 5.3 AIX 6.1 AIX 7.1
Micro
LPAR LPAR Micro Micro Partition
Partition SMT-4
SMT=off SMT=on Partition
SMT=on SMT=off L L L L LL
L L LL LL LL LL L L L L L L Logical

V V V V V V V V V Virtual
2.1 Proc. Units 0.8 Proc Units 1.2 Proc Units
2 CPUs 1 CPU Physical
Shared Pool 1 Pool 2
(dedicated) (dedicated)
13 CPU Default Pool 0*

16 CPU SMP Server

Think “PVL “ P=Physical V=Virtual L=Logical (SMT)


* All activated, non-dedicated CPUs are automatically placed into the shared processor pool 0.
Only 2.1+0.8+1.2 = 4.1 processor units of “desired capacity” has been allocated from the pool of 13 CPUs

30 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Virtual Shared Processor Pools – Licensing Benefits

Server with 16 processor cores


POWER6/7 Multiple shared pools: n5 n6 n7 n8 n9

• Can reduce the number of Uncapped Uncapped Uncapped Uncapped Uncapped


software licenses by putting a AIX AIX AIX AIX AIX
limit on the amount of processors OAS OAS OAS
an uncapped partition can use
Oracle DB Oracle DB App 1 App2 QA
• Up to 64 shared pools VP = 5 VP = 4 VP = 4 VP = 6 VP = 3
Ent. = 2.5 Ent. = 1.70 Ent. = 2.00 Ent. = 2.00 Ent. = 1.00

CUoD n1 n2 n3 n4 Virtual Shared pool #1 Virtual Shared pool #2


VIOS VIOS AIX Linux Max Cap: 5 processors Max Cap: 6 processors
Oracle Physical Shared Pool (9 processor cores)
DB
4 0.5 0.5 1 1 1 2 3 4 5 6 7 8 9

Oracle DB cores to license:


• 1 from dedicated partition n3
• 5 from shared CPU pool 1
Oracle DB core – license factors:
=6 POWER6: 1.0
POWER7: 1.0
OAS cores to license:
• 6 from shared CPU pool 2
=6

31 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Virtual Processor - Folding


Dynamically adjusts active Virtual Processors (VPs)
– System consolidates loads onto a minimal number of VPs
• Scheduler computes utilization of VPs every second
– If VPs needed to host physical utilization is less than the current active VP
count, a VP is put to sleep
– If VPs needed are greater than the current active VPs, more are enabled
– On by default in AIX 5.3 TL03 and later
• vpm_xvcpus tunable
• vpm_fold_policy tunable

Increases processor utilization and affinity


– Inactive VPs don’t get dispatched and waste physical CPU cycles
– Fewer VPs can be more accurately dispatched to physical resources by the
Hypervisor with potential for improved processor cache efficiency
When to adjust – Check with IBM support before changing!
– Bursty workloads with short response-time requirements may need sub-second
dispatch latency
• Disable processor folding or manually tune the number of VPs
– # schedo –o vpm_xvcpus=[-1 | N]
– Where N specifies the number of VPs to enable in addition to the number of VPs
needed to consume physical CPU utilization
– A value of “-1” disables CPU folding
32 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

CPU Considerations
Use Simultaneous Multi-Threading (SMT) with AIX 5.3 (or later) on
Power5 (or later) environments
Micropartitioning Guidelines
– Virtual CPUs for specific LPAR <= physical processors in shared CPU pool

CAPPED LAR
• Virtual CPUs should be the nearest integer >= capping limit
UNCAPPED LPAR
• Virtual CPUs should be set to the max peak demand requirement
• Preferably, set Entitlement >= Virtual CPUs / 1.5
DLPAR considerations – CPU_COUNT refers to Logical CPU
Oracle 9i
– Oracle CPU_COUNT does not recognize change in # cpus
– AIX scheduler can still use the added CPUs
Oracle 10g/11g
– Oracle CPU_COUNT recognizes change in number of active CPU
Max CPU_COUNT limited to 3x CPU_COUNT at instance startup. This can limit the amount
of physical CPU resources utilized if you start out with SMT-off and switch to SMT-4 or
dynamically add a large number of VP.
Note: Recommended to set PARALLEL_THREADS_PER_CPU=1 if SMT is active.
33 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX – Oracle

IO

34 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

The AIX IO stack


Application Application memory area caches data to avoid IO

NFS caches file attributes


ASM options Logical file system NFS has a cached filesystem for NFS clients and
Raw disks

uses client pages for cache


Raw LVs

Local FS
NFS Other
JFS / JFS2
JFS and JFS2 cache use extra system RAM
JFS uses persistent pages for cache
VMM
JFS2 uses client pages for cache

LVM (LVM device drivers)


Multi-path IO driver (optional)
Disk Device Drivers Queues exist for both adapters and disks
Adapter Device Drivers Adapter device drivers use DMA for IO
Disk subsystem (optional) Disk subsystems have read and write cache
Disk Disks have memory to store commands/data
Cache Persisted write Cache, as well as read Cache – “I/O complete”
acknowledge sent back to application before data written to physical disk.

IOs can be coalesced (good) or split up (bad) as they go thru the IO stack
35 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

I/O Stack Tuning Options (Device Level)


Disk
queue_depth - maximum # of concurrent active I/Os for an hdisk / hdiskpower; additional
I/O beyond that limit will be queued. Recommended/supported maximum is storage
subsystem dependent.
max_transfer - the maximum allowable I/O transfer size (default is 0x40000 or 256k).
Maximum supported value is storage subsystem dependent. All current technology
supports 1MB I/O size set to 0x100000.

Fiber Channel Disk Adapter (fcsn)


num_cmd_elems - maximum number of outstanding I/Os for an adapter.
set to 1024 or 2048 (within storage subsystem vendor guidelines)
max_xfer_size - Increasing value (to at least 0x200000) will also increase DMA size from
16 MB to 256 MB, but this should only be done after IBM support has directed you to do
so, as it can lead in specific configurations to system stability issues or AIX not being able
to boot.
dyntrk - when set to yes (recommended), allows for immediate re-routing of I/O requests to
an alternative path when a device ID (N_PORT_ID) change has been detected; only
applies to multi-path configurations.
fc_err_recov - when set to “fast_fail” (recommended), if the driver receives an RSCN
notification from the switch, the driver will check to see if the device is still on the fabric and
will flush back outstanding I/Os if the device is no longer found.
To validate / change current parameter settings use: “lsattr”, “chdev”
36 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Data Striping to Avoid I/O Hotspots

Old Wisdom
Isolate files based on function and/or usage
– Manually intensive effort
– Leads to I/O hotspots over time that impact throughput capacity and
performance

New Wisdom
Stripe objects across as many physical disks as possible
– Minimal manual intervention
– Evenly balanced I/O across all available physical components
– Good average I/O response time and object throughput capacity with no
hotspots

Implementation Options:
– ASM and GPFS do this automatically within a given disk group or file system
– Can be implemented with conventional Volume Managers and file systems

37 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Data Layout for Optimal I/O Performance

Example:
1. Use RAID-5 or RAID-10 to create striped LUNs (hdisks)
– Four 3+P (or 2+2) RAID arrays
– 1 (or more) LUNs per RAID array
– Each LUN is spread across 4 drives

Storage
1
HW Striping
RAID-5 vs. RAID-10 Performance Comparison
I/O Profile RAID-5 RAID-10
LUN 1
Sequential Read Excellent Excellent

Sequential Write Excellent (*1) Good LUN 2

Random Read Excellent Excellent LUN 3

Random Write Fair Excellent


LUN 4

*1 – Assumes optimizing SAN storage sub-system!


38 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Data Layout for Optimal I/O Performance

Example…
2. Stripe or spread individual objects across multiple LUNs (hdisks) for
maximum distribution
– Each object is spread across 4 LUNs, each from different array (16 drives)

AIX Storage
Volume (Disk) Group HW Striping
AIX LVM striping with JFS2 or
hdisk 1 LUN 1
ASM Disk Group or
SW Striping

IBM GPFS

hdisk 2 LUN 2

hdisk 3 LUN 3

hdisk 4 LUN 4
2

Note: ASM, AIX LVM or GPFS can not share the same hdisks.

39 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Software Striping with AIX – LV striping & PP spreading


Stripe using Logical Volume (LV)
– Create Logical Volume with the striping option : mklv –S <strip-size> ...
– Oracle recommends a stripe size of a multiple of
db_block_size * db_file_multiblock_read_count (Usually around 1 MB)
– Valid LV Strip sizes:
AIX 7.1, 6.1, 5.3: 4k to 128M in powers of 2.
– For RAW LVs, use AIX Logical Volume 0 offset (9i Release 2 or later)
Use Scalable Volume Groups (VGs), or use “mklv –T O” with Big VGs
Requires AIX APAR IY36656 and Oracle patch (bug 2620053)

PP striping (AKA spreading)


– Create a Volume Group with a 8M,16M or 32M PPsize. (PPsize will be the “Strip
size”)
– Choose a Scalable Volume Group : # mkvg –S –s <PPsize> ...
– Create LV with “Maximum range of physical volume” option to spread PPs on
different hdisk in a Round Robin fashion : # mklv –e x ...
Note:
If you create a JFS2 filesystem on a striped (or PP spreaded) LV, use the INLINE logging option. It
will avoid « hot spot » by creating the jfs2 redo log inside the filesystem itself, which is striped,
instead of using a single PP stored on 1 hdisk. (# crfs –a logname=INLINE …)

40 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Filesystems

AIX Standard Filesystems (FS):


JFS – no longer being enhanced
JFS2 – generally the preferred filesystem
Better for large files/filesystems and for filesystems with large numbers of files.

Mount options:
Buffer Caching (default) – stage data in FS buffer cache
Direct I/O (DIO) – no filesystem caching
Concurrent I/O (CIO) – DIO + no write serialization (JFS2 only)
Release Behind Read (RBR) – memory pages released (available for
stealing) after pages copied to internal buffers
Release Behind Write (RBW) – memory pages released (available for
stealing) after pages written to disk
Release Behind Read / Write (RBRW) – combination of RBR and RBW
No Access Time (NOATIME): do not update last accessed time when file is
accessed
41 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Oracle Server Architecture – Files

PGA
PGA PGA
RVWR PMON SMON
PGA

System Global Area (SGA) ARC0 Archive


Logs

Flash- Shared Pool DB Buffer Redo Log PGA


back Log Cache Buffer

LGWRn
PGA

DBWRn CHKP Redo


Logs

User PGA
PGA Control
Files

PGA D000
DB
Files Oracle
Binaries

42 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Tuning Oracle DB Buffer Cache


Buffer Cache is the primary database I/O avoidance option!
Old Wisdom
If the buffer hit% is > 90% it’s good enough

New Wisdom:
Depending on workload, a higher hit% may provide significant improvements
– For a given workload with a buffer hit% of 98%, a 1% increase (to 99%)
will reduce physical I/O requests by 50%
– Reducing IOPS typically also improves response time for remaining I/Os
– In many cases, adding server memory may be cheaper than adding I/O
subsystem cache memory or short-stroking disks
Evaluate impact of increasing db_cache_size on physical I/O
Monitor for and address potential impact:
– Increased logical read rates and higher peak CPU demand due to
reduced I/O wait time (increase CPU capacity as appropriate)
– System paging due to memory shortage (add physical memory as
necessary)
43 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Oracle Options for Data Storage on AIX

ACFS
(JFS) / JFS2 RAW GPFS ASM
(11.2.0.2)

Database
Files

Redo Log
Files

Control Files

Archive Log
Files

Oracle
Binaries

Unsupported upgrade after 11gR2, or new installs with 11gR2 or later

44 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

JFS/JFS2 environments - Oracle Database Files


Data Base Files (DBF)
I/O size ranges from db_block_size to db_block_size * db_file_multiblock_read_count
Use CIO (or DIO for JFS) or filesystem cache, depending on I/O characteristics
If block size is >=4096, use a filesystem block size of 4096, else use 2048
Redo Log/Control Files
I/O size is always a multiple of 512 bytes
Use CIO (or DIO for JFS) and set filesystem block size (agblksize) to 512
Archive Log and Backup Files
Don’t use CIO or DIO
‘rbrw’ mount option can be advantageous
Flashback Log Files
Writes are sequential, sized as a multiple of db_block_size
By default, dbca will configure a single location for the flash recovery area - for flashback logs,
archive logs, and backup logs
Flashback Log files should use CIO, DIO, or ‘rbrw’ mount
Oracle Binaries
Don’t use CIO or DIO
Use NOATIME to reduce ‘getcwd’ overhead
System Root (/) Filesystem
Use NOATIME to reduce ‘getcwd’ overhead (requires “bosboot”, “reboot”)
45 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Asynchronous I/O for filesystem environments

AIX parameters
aio_minservers = minimum # of AIO server processes
aio_maxservers = maximum # of AIO server processes
aio_maxreqs = maximum # of concurrent AIO requests
“enable” at system restart (not required with AIX 6.1 or AIX 7.1)
aio_server_inactivity = time before idle AIO processes will be terminated
(AIX 6.1 and AIX 7.1 only)

AIX 5.3 settings are often too low for Oracle workloads
> Recommend using AIX 6.1 defaults

Oracle parameters
disk_asynch_io = TRUE
filesystemio_options = {ASYNCH | SETALL}
db_writer_processes (let default)
db_writer_io_slaves (do not set when using AIO)

46 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Asynchronous I/O for filesystem environments…

Monitor Oracle usage:


• Watch alert log and *.trc files in BDUMP directory for warning message:
Warning “lio_listio returned EAGAIN”
This usually indicates that maxservers and/or maxreqs is set too low or that
the IO sub-system is not able to support the IO workload, but check AIX
system log for errors before increasing the number of AIO servers!

Monitor from AIX:


• “pstat –a | grep aios” (for AIO server processes)

• Use “-A” option for NMON (interactive or spreadsheet mode)

• iostat –Aq (AIX 5.3 – ensure AIO is enabled!)


(Note: initial AIX 7.1 and some older AIX 6.1 TL levels do not work correctly)

47 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Asynchronous I/O – Parameter Tuning

AIX 6.1 and AIX 7.1


– Use ioo command to change
– Defaults are good starting points:
aio_minservers = 3 (per logical CPU)
aio_maxservers = 30 (maybe increase to 50) (per logical CPU)
aio_maxreqs = 65536
aio_server_inactivity = 300

AIX 5.3
– Use aioo (or ‘smitty aio’) command to change
– Recommended starting points:
minservers = 3 (Systemwide) (default = 1)
maxservers = 50 (Per CPU) (default = 10)
maxreqs = 65536 (default = 4096)
“enable” at system restart (default = disable)

Note: In AIX 6.1 and AIX 7.1 AIO processes will end when not used,
whereas in AIX 5.3 a started AIO process runs till system is rebooted
48 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Asynchronous IO (AIO) fastpath


Raw Devices or ASM (rhdisk) environments use kernelized or “fastpath” AIO and not AIO processes
Better performance compared to non-fast_path
The AIO parameters discussed earlier do not apply
No aio server processes => “pstat –a | grep aios | wc –l” is not relevant, use “iostat –A” instead

1 3
Application AIX Kernel
Disk
2 4
5

• Raw Devices / ASM :


verify AIO configuration with : AIX 5L: lsattr –El aio0, AIX 6.1, AIX 7.1: ioo –Fa
enable asynchronous IO fast_path. :
AIX 5L : chdev -a fastpath=enable -l aio0 (default since AIX 5.3)
AIX 6.1, AIX 7.1 : ioo –p –o aio_fastpath=1 (default setting)

• JFS2 with CIO and AIX 5.3 TL5+ :


Activate fsfast_path (comparable to fast_path but for JFS2 + CIO)
AIX 5L : adding the following line in /etc/inittab: aioo:2:once:aioo –o fsfastpath=1
AIX 6.1, AIX 7.1 : ioo –p –o aio_fsfastpath=1 (default setting)

Note:
Apply APAR IZ74245 for AIX 6.1 or IZ59538 for AIX 5.3 to fix potential sequential read issue with ASM and “fastpath” IO.
49 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Cached vs. non-Cached (Direct) I/O


File System caching tends to benefit heavily sequential workloads with low write content
due to sequential read ahead.
To enable caching for JFS/JFS2:
Use default filesystem mount options
Set Oracle filesystemio_options=ASYNCH (default)

DIO tends to benefit heavily random access workloads and CIO, in addition to DIO benefits,
tends to benefit heavy update workloads. To disable JFS, JFS2 caching:
In 9i, set filesystemio_options=ASYNCH and use dio (JFS) or cio (JFS2) file system
mount option
In 10g/11g
If Oracle files do not need to be concurrently accessed by external utilities, set
filesystemio_options=SETALL
Otherwise set filesystemio_options=ASYNCH and use dio (JFS) or cio (JFS2) mount
With Oracle 11.2.0.2+ and AIX 6.1+ always use filesystemio_options=SETALL and do
not specify dio / cio FS mount option.

When using DIO/CIO, FS buffer cache is not used. Consider the following Oracle DB
changes:
Increase db_cache_size
Increase db_file_multiblock_read_count if set in init.ora, but see Notes
Read Metalink Note #s 272520.1, 257338.1, 360287.1, 232935.1
50 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX rendev command with ASM


rendev command is used for renaming devices which are listed in ODM
Syntax / Description
– rendev –l <original name> -n <new name>
– The device entry under /dev will be renamed corresponding to <new name>
– Certain devices such as /dev/console, /dev/mem, /dev/null, and others that are identified
only with /dev special files cannot be renamed
– Command will fail for any device that does not have both a Configure and an
Unconfigure method
– Any name that is 15 characters or less and not already used in the system can be used
If used to rename hdisk devices for ASM use, it is recommended that you
keep the "hdisk" prefix, as this will allow the default ASM discovery string
to match the renamed hdisks. Corresponding rhdisk is renamed as well.
Example:
# rendev –l hdisk10 –n hdiskASM10
# ls /dev/*ASM*
/dev/hdiskASM10
/dev/rhdiskASM10

51 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX lkdev command with ASM


The lkdev command locks the specified device. Any attempt to modify
device attributes by using the chdev or chpath command is denied. In
addition, an attempt to delete the specified device or one of its paths from
the ODM by using either the rmdev or rmpath command is denied.
Syntax:
lkdev [ -l <Name> -a | -d [ -c <Text> ] ]
<Name> Name of device to be changed (required)
-a Locks the specified device.
-d Unlocks the specified device.
-c <Text> Specifies a text of up to 64 printable characters with no embedded spaces.

Examples:
– To enable the lock for the hdiskASM10 disk device and create a text label, enter the following
command:
# lkdev -l hdiskASM10 -a -c ASMdisk
– To remove the lock for the hdisk1 disk device and remove the text label, enter the following
command:
# lkdev -l hdiskASM10 -d
Note:
The text label of a locked device can not be changed! Instead, the device needs
to be first unlocked and then locked again with the new text label specified.
52 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX – Oracle

Network

53 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Network parameters (no)


use_isno = 1 means any parameters set at the interface level override parameters
set with ‘no’
– This is the default value; (restricted parameter in AIX 7.1 and AIX 6.1)
If use_isno = 0, any parameters set with ‘no’ override interface-specific parameters
If use_isno = 1, set parameters for each interface using ‘ifconfig’ or ‘chdev’ (check
with: lsattr -E -l en0 –H)
Refer to the following URL for a chart on appropriate interface-specific parameters:
– http://publib.boulder.ibm.com/infocenter/systems/topic/com.ibm.aix.prftungd/doc/prftungd/prftungd.pdf

Generally appropriate parameters for Gigabit Ethernet Oracle interfaces:


– tcp_sendspace = 262144
– tcp_recvspace = 262144
– rfc1323 = 1
Examples:
# no -p -o tcp_sendspace=262144
# no -p -o tcp_recvspace=262144
# no -p -o rfc1323=1

54 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Routing Table Entry Locking


There are 2 alternative locking strategies for Routing Table
entries (rtentry) – simple and complex
– The current default locking strategy is “simple” (*1)

The Simple Performance Lock Analysis Tool (splat) may be used


to monitor rtentry lock performance
The complex locking strategy can improve performance when
there is a lot of activity on Routing Table entries
– Can be enabled by setting rtentry_lock_complex=1 (Default in AIX 7.1)

Example:
> # no -p -o rtentry_lock_complex=1

*1: Parameter not available in AIX 5.3; AIX 7.1 default value is “complex” – value of 1

55 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX – Oracle

Miscellaneous

56 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX - Oracle: Miscellaneous (1/2)

/etc/security/limits
Set to “-1” for everything except “core” for Oracle user, but beware
that the DBA does not mis-configure the SGA size!

sys0 attribute maxuproc


– Should be >= 16384 (verified by installer!)
– maxuproc > 128 + SUM of PROCESSES +
PARALLEL_MAXSERVERS for all DB instances in LPAR

Oracle environment variables:


AIXTHREAD_SCOPE=S

Use 64-bit AIX kernel (32bit kernel only available in AIX 5.3 and earlier)

Time synchronization – use the “-x” flag with xntpd


Edit /etc/rc.tcpip, search for xntpd and add the ‘-x’ to the line for xntpd:
• # Start up Network Time Protocol (NTP) daemon
• start /usr/sbin/xntpd "$src_running" “-x” Add
(lower case x)
57 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

AIX - Oracle: Miscellaneous (2/2)

Oracle hot patching (11.2.0.2 and later)


Online patches (11gR2+) should only be used when the patch needs to
be applied urgently and database downtime cannot be scheduled.
It is strongly recommended to rollback all online patches and replace
them with regular (offline) patches on next instance shutdown
See MOS note 761111.1 for further details
Oracle 11.2.0.2, 11.2.0.3 require USLA Heap patch:
• 13443029 (requires AIX 6.1 TL07 SP2 or AIX 7.1 TL01 SP2)
OR
• 10190759 (disables hot patching)
Further details / latest updates please check:
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102066

58 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

My system doesn't behave

What tools are available in AIX


to investigate the cause?

59 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Performance Monitoring and Tuning Tools


Processes &
CPU Memory I/O Subsystem Network
Threads

vmstat, topas, vmstat, topas, vmstat, topas, netstat, topas, ps, pstat, topas,
nmon, iostat, ps, nmon, ps, lsps, nmon, iostat, nmon, atmstat, nmon,
Status and mpstat, lparstat, sar, ipcs, lparstat lvmstat, lsps, entstat, tokstat, emstat/alstat
Initial Monitor time/timex, lsattr/lsdev, fddistat, nfsstat,
Commands emstat/alstat lspv/lsvg/lslv ifconfig

netpmon svmon, netpmon, fileplace, filemon netpmon, tcpdump svmon, truss,


Monitor filemon kdb, dbx, gprof,
kdb, fuser, prof
Commands

tprof, curt, splat, trace,trcrpt trace, trcrpt iptrace, ipreport, truss, pprof curt,
Trace Level trace, trcrpt trace, trcrpt splat, trace, trcrpt
Commands

schedo, fdpr, vmo, rmss,fdpr, ioo, lvmo, chdev, no, chdev,ifconfig nfso,chdev, fdpr
bindprocessor, chps/mkps migratepv,chlv,
bindintcpu, reorgvg
Tuning tools
nice/renice, setpri

Note: In AIX 5.3 TL09+, AIX 6.1 TL02+, AIX 7.1 releases nmon == topas_nmon and is part of AIX base install!
60 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

NMON
nmon -M -^ -f -d -T -A -s10 -c9999 (as root user!)
– -M: detailed memory information per page size (4K, 64K, 16M, 16G)
– -^: FC adapter statistics
– -f: spreadsheet mode
– -d: disk service times section
– -T: collect TOP and UARG information
– -A: include AIO statistics
– -s 10: 10 second capture interval
– -c 9999: number of intervals to run
To stop the data collection cleanly:
– kill –USR2 <PID of nmon process>
Creates by default a file in the current directory:
<server name>_<date>_<time>.nmon

Note: Try “nmon –h” to get the full list of available options.
61 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

nmon On Demand Recording (ODR)


New function ideal for benchmarks, proof-of-concepts and problem
analysis
Allows “high-resolution” recordings to be made while in monitoring mode
– Records samples at the interactive monitoring rate
– AIX 5.3 TL12, AIX 6.1 TL05 and AIX 7.1
Usage
– Start nmon, use “[“ and “]” brackets to start and end a recording
• Records standard background recording metrics, not just what is on screen.
• You can adjust the recorded sampling interval with
-s [seconds] on startup, interactive options “-” and “+” (<shift> +) do
NOT change ODR interval
– Generates a standard nmon recording of format:
<host>_<YYYYMMDD>_<HHMMSS>.nmon
– Tested with nmon Analyser, and works well

62 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

NMON Analyser – Which Worksheets to look at.


Kernel settings, Disk devices, … :
– BBBx
CPU:
– SYS_SUMM, PROC (RunQueue), TOP; LPAR or CPU_ALL –
depends if LPAR deployed in shared CPU pool or not.
– Try the nmon_analyser option “PIVOT” to see in one graph what
processes are using the CPU; requires TOP data!
Memory:
– MEMUSE, MEMPAGES64K, PAGE, TOP
Disk IO:
– DISK_x, EMC_x, ESS_x, PROCAIO, PROC (Swap-In)
Network:
– NET, NETPACKET, NETSIZE

Tip: Try the “PIVOT” option to see CPU utilization by process type.
63 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Where to find AIX / Oracle


documentation?

64 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Where to find AIX / Oracle documentation?

IBM TechDocs

My Oracle Support (MOS) notes

IBM Redbooks: (www.ibm.com/redbooks)

AIX / PowerVM Wiki

AIX InfoCenter pages on the WEB

65 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

IBM TechDocs - Technical Sales Library


http://www.ibm.com/support/techdocs

Soon to be updated to 2.3


Oracle Architecture and Tuning on AIX v2.20
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883

Configuring IBM TotalStorage for Oracle OLTP Applications


http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100319

Diagnosing Oracle® Database Performance on AIX® Using IBM® NMON and


Oracle Statspack Reports
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101720

Breaking the Oracle I/O Performance Bottleneck


http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3885
Oracle Technology Essential White Papers Regularly Updated!

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101559

There are many more Oracle related white papers – especially


covering Oracle RAC with IBM servers and IBM storage.
66 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Oracle Documentation

My Oracle Support: http://support.oracle.com Regularly Updated!


– 282036.1 - Minimum Software Versions and Patches
Required to Support Oracle Products on IBM Power
Systems

756671.1 - Oracle Recommended Patches -- Oracle


Database
Oracle Reference Manuals:
http://otn.oracle.com/documentation/index.html

Oracle Certification Info (on MOS as well):


http://otn.oracle.com/support/metalink/index.html

67 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX
AIX 5.3 Product Documentation.
– http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.pseries.doc/hardware.htm
AIX 6.1 Product Documentation
– http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp
AIX 7.1 Product Documentation
– http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp
– http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=sg247910 (IBM AIX Version 7.1 Differences Guide)

IBM Wikis
– https://www.ibm.com/developerworks/wikis/dashboard.action
AIX Wiki
– http://www.ibm.com/developerworks/wikis/display/WikiPtype/Home
AIX Performance Tools (nmon, nmon analyser/consolidator, etc)
– http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon
AIX DeveloperWorks
– http://www.ibm.com/developerworks/aix
AIX multiple page supprt
– http://www-03.ibm.com/systems/resources/systems_p_os_aix_whitepapers_multiple_page.pdf
Tuning IBM AIX 5L V5.3 and AIX 6.1 for Oracle Database on POWER systems
– http://www-304.ibm.com/partnerworld/wps/servlet/ContentHandler/whitepaper/aix/oracle/performance_analysis
PowerVM Wiki
– https://www.ibm.com/developerworks/wikis/display/virtualization/Home

68 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX / POWER
AIXpert Blog on Local, Near and Far Memory
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/local_near_far_memory_part
_1_large_power7_boxes_more_local_memory26?lang=en
Oracle Database and 1 TB Segment Aliasing (TD105761)
http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105761
IBM EnergyScale for POWER7 Processor-Based Systems
ftp://public.dhe.ibm.com/common/ssi/ecm/en/pow03039usen/POW03039USEN.PDF
Active Memory Expansion: Overview and Usage Guide
ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/pow03037usen/POW03037USEN.PDF
IBM PowerVM Virtualization Active Memory Sharing
http://www.redbooks.ibm.com/abstracts/redp4470.html?Open
IBM System p Advanced POWER Virtualization (PowerVM) Best Practices
http://www.redbooks.ibm.com/abstracts/redp4194.html?Open
Power Systems Enterprise Servers with PowerVM Virtualization and RAS
http://www.redbooks.ibm.com/abstracts/sg247965.html?Open

69 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

AIX APARs to check


IZ88711: BIND64 CORES WITH -BLAZY OPTION ON AIX61
IZ91983: LOCKL PERFORMANCE ISSUE
IZ94396: THERE IS A TIMING ISSUE BETWEEN THE SYNC DAEMON AND A MAPPED FIL APPLIES TO AIX 6100-06
IZ97088: SMT 4 scheduling fix / improvement
IV10172: WAITPROC IDLE LOOPING CONSUMES CPU
IV06194: SRAD LOAD BALANCING ISSUES ON SHARED LPARS
IV10259: MISCELLANEOUS DISPATCHER/SCHEDULING PERFORMANCE FIXES
IV03903: Address space lock contention issue
IZ76101: Scale Light Load borrowing (Multi-SRAD Scaling issue)
IV11857: Slow startup of AIO processes (workaround: use kernelized AIO, or increase aio_minservers and set
aio_server_inactivity to 86400.
IZ71987 (AIX6.1), IZ67445 (AIX 5.3 TL12): Paging Space Growth May Occur Unexpectedly With 64K (medium) Pages
Enabled
IV11261: SYSTEM CRASH IN AS_FORK_ALIAS IF ESID_ALLOCATOR IS ENABLED
IV23851 (AIX 6.1), IV23859 (AIX 7.1): APPLICATIONS RUN SLOWLY WITH HIGH SYSTEM TIME
IV20880: This problem requires use of shared symbol table and has been seen on AIX 6.1 TL7 and 7.1 TL1 when using
Oracle 11gR2 with on-line patching. This includes versions 11.2.0.2 / 11.2.0.3 with Oracle Patch 13443029.
IV09580 (AIX 6.1 TL07), IV09541 (AIX 7.1 TL01): FILE.ATION OVERFLOW REPORTED IN ERROR WHILE LINKING
LARGE binaries
IV26272: 4K page size working storage page stealing when the 64K pagesize is in a maxpin condition and there are 4K
file cache pages available for stealing.

AIX APARs are available here: http://www-933.ibm.com/support/fixcentral/


70 September 2012 © 2012 IBM Corporation
Advanced Technical Skills (ATS) North America

Important Oracle BUGs (11gR2) to check in context of AIX

Mutex Wait:
Bug 10411618: ADD DIFFERENT WAIT SCHEMES FOR MUTEX WAITS
Note: 11.2.0.2.2 PSU breaks this patch and additional patch (12431716) is required
Master Note: WAITEVENT: "library cache: mutex X" [ID 727400.1]

Bug 12740358 : DBMS_UTILITY.FORMAT_CALL_STACK is Still Slower Than 10G.


This issue can be observed in AIX as a high number of system calls – millions – to functions like “sigaction()”.

Bug 9842771: Wrong SREADTIM and MREADTIM statistics in AUX_STATS$

Bug 12596494: Generally Higher CPU Usage in 11.2.0.2 than 10.2.0.4


MOS 1062676.1 - ORAAGENT or ORAROOTAGENT High Resource (CPU, Memory
etc) Usage
11.2.0.3 RAC shows high CPU usage in ologgerd and heavy write to
$GRID_HOME/crf/db - 9+MB/s. Patch 11.2.0.3.1 GI+PSU resolves this issue

71 September 2012 © 2012 IBM Corporation


Advanced Technical Skills (ATS) North America

Q&A

Presenter email address: dannert <at> us <dot> ibm <dot> com

72 September 2012 © 2012 IBM Corporation

You might also like