Handling Basic Oracle Database Issues

Logging in to SQL Promt from a unix based machine:

1. Login as user Orace: #su - oracle
2. Check for the database pmon processes running:
ps -ef | grep pmon
oracle 650 1 0 2011 ? 00:03:19 ora_pmon_instance1
oracle 1724 1 0 2011 ? 00:03:30 ora_pmon_instance2
grid 12864 1 0 2011 ? 00:01:21 asm_pmon_+ASM1
For logging in to instance1:
1. Set the environment variable
$ . oraenv
ORACLE_SID = [oracle] ? instance1
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11.2.0.3/dbhome_1
The ORACLE_HOME path can be located from /etc/oratab.
How to check the version of the Components of the Database
SQL> COL PRODUCT FORMAT A55
SQL> COL VERSION FORMAT A15
SQL> COL STATUS FORMAT A15
SQL> SELECT * FROM PRODUCT_COMPONENT_VERSION;
PRODUCT

VERSION

STATUS

------------------------------------------------------- --------------- --------------NLSRTL

11.2.0.3.0

Oracle Database 11g Enterprise Edition

Production

11.2.0.3.0

64bit Productio

n
PL/SQL

11.2.0.3.0

TNS for Linux:

11.2.0.3.0

Production
Production

Checking Database status

select instance_name, to_char(startup_time, 'YYYY.MM.DD HH24:MI:SS'), status, database_status,
instance_role from v$instance;
Checking database instance start time
select to_char(startup_time, 'YYYY.MM.DD HH24:MI:SS'), instance_name from gv$instance;

Checking instances running on database

Login as user oracle and set the environment variable.
srvctl status database -d <Instance Name>
srvctl config database -d <Instance Name>
Checking cluster services on Database
#cd /u01/app/11.2.0.3/grid/bin
#./crsctl status res -t -w "NAME co .db"
#crsctl stat res -t
#crsctl stat res -t -w "NAME co <service_name>"
Checking Cluster health:
# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
Restarting the Database
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 304807936 bytes
Fixed Size
Variable Size

2227864 bytes
109052264 bytes

Database Buffers

188743680 bytes

Redo Buffers

4784128 bytes

Database mounted.
Database opened
Metrics Average Users Waiting Counts is at XX.XXX for event class <Class Name>
The wait_class are generally Concurrency, Network, Other, Configuration, Application. Following
command can be used to check the current status of the metrics Avg. Users Waiting counts:
select count(*) from v$session_wait where wait_class='Concurrency';
select count(*) from v$session_wait where wait_class='Network';

select count(*) from v$session_wait where wait_class='Other';

select count(*) from v$session_wait where wait_class='Configuration';
select count(*) from v$session_wait where wait_class='Application';

Metrics Database Time Spent Waiting (%) is at 80.882 for event class <Class Name>

select wait_class,wait_time,seconds_in_wait,state FROM v$session_wait where

wait_class='Configuration';
Metrics Database Wait Time Ratio is at XX.XX
select METRIC_NAME, VALUE from SYS.V_$SYSMETRIC where METRIC_NAME IN ('Database CPU
Time Ratio','Database Wait Time Ratio') AND INTSIZE_CSEC = (select max(INTSIZE_CSEC) from
SYS.V_$SYSMETRIC);
METRIC_NAME

VALUE

---------------------------------------------------------------- ---------Database Wait Time Ratio

59.2883618

Database CPU Time Ratio

40.7116382

Metrics "Current Open Cursors Count" is at 4294967291

select count(*) from v$open_cursor where cursor_type like 'OPEN%';
Metrics "Logons Per Sec"
It represents the number of logons per second during the sample period:
select value, metric_unit,metric_name from v$metric where metric_name = 'Logons Per Sec';
Metrics Buffer Cache Hit Ratio:
SELECT ROUND((1-(phy.value / (cur.value + con.value)))*100,2) "Cache Hit Ratio" FROM v$sysstat
cur, v$sysstat con, v$sysstat phy WHERE cur.name = 'db block gets' AND con.name = 'consistent gets'
AND phy.name = 'physical reads';
The standby database is approximately XXXX seconds behind the primary database

select * from V$DATAGUARD_STATS;

It will display following columns in o/p:

NAME VALUE UNIT TIME_COMPUTED DATUM_TIME
Check for the row where NAME value is "apply lag". The value field in the row will give the lag time
between standby database and primary database.
Check the redo log difference on standby database:
SELECT r.item, sofar, r.units
FROM gv$recovery_progress r
WHERE r.start_time = (SELECT MAX(start_time) from gv$recovery_progress);
ITEM SOFAR UNITS
-------------------------------- ---------- -------------------------------Log Files 9071 Files
Active Apply Rate 134643 KB/sec
Average Apply Rate 792494 KB/sec
Maximum Apply Rate 169175786 KB/sec
Redo Applied 455743288 Megabytes
Last Applied Redo 0 SCN+Time
Active Time 389895 Seconds
Apply Time per Log 34 Seconds
Checkpoint Time per Log 8 Seconds
Elapsed Time 588876 Seconds
Standby Apply Lag 1666 Seconds
11 rows selected.
Check if the MRP process is running and applying the redo log files
select inst_id, process, status MRP_stat, thread#, sequence#, block#
from gv$managed_standby
where process like 'MRP%' or process like 'RFS%' and status != 'IDLE'
order by inst_id,process,thread# ;
INST_ID PROCESS MRP_STAT THREAD# SEQUENCE# BLOCK#
---------- --------- ------------ ---------- ---------- ---------1 MRP0 APPLYING_LOG 4 115056 207

After checking the MRP Process, check the difference between the Last Sequence Received & Last
Sequence Applied. The value should be zero or minimal in ideal situation:

SELECT null dummy, ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received",

APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
FROM
( SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME )
IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#))
ARCH,
( SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME )
IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#))
APPL
WHERE ARCH.THREAD#=APPL.THREAD#
ORDER BY 2;

Checking Table space issue

SELECT Total.name "Tablespace Name",
nvl(Free_space, 0) Free_space,
nvl(total_space-Free_space, 0) Used_space,
total_space,((free_space/total_space)*100) percent_free
FROM
(select tablespace_name, sum(bytes/1024/1024) Free_Space
from sys.dba_free_space
group by tablespace_name
) Free,
(select b.name, sum(bytes/1024/1024) TOTAL_SPACE
from sys.v_$datafile a, sys.v_$tablespace B
where a.ts# = b.ts#
group by b.name
) Total
WHERE Free.Tablespace_name(+) = Total.name
ORDER BY Total.name;
Checking TEMP tablespace:
select TABLESPACE_NAME, sum(BYTES_FREE/(1024*1024*1024)) FREE_SPACE_IN_GB from
V$TEMP_SPACE_HEADER group by TABLESPACE_NAME;
select sum(BYTES_FREE/(1024*1024*1024)) from V$TEMP_SPACE_HEADER;
select * from DBA_TEMP_FREE_SPACE;
Session DBSXXX-XXXX blocking n other sessions for all instances

SELECT blocking_sid, num_blocked FROM ( SELECT blocking_sid, SUM(num_blocked) num_blocked

FROM ( SELECT l.id1, l.id2, MAX(DECODE(l.block, 1, i.instance_name||'-'||l.sid, 2,
i.instance_name||'-'||l.sid, 0 )) blocking_sid, SUM(DECODE(l.request, 0, 0, 1 )) num_blocked FROM
gv$lock l, gv$instance i WHERE ( l.block!= 0 OR l.request > 0 ) AND l.inst_id = i.inst_id GROUP BY
l.id1, l.id2) GROUP BY blocking_sid ORDER BY num_blocked DESC) WHERE num_blocked != 0 ;
Sessions blocked by TX lock
SQL> select * from v$lock where block=1;
ADDR

KADDR

SID TY

ID1

ID2

LMODE

---------------- ---------------- ---------- -- ---------- ---------- ---------REQUEST

CTIME

BLOCK

---------- ---------- ---------00000009B9924140 00000009B99241B8

0

8658

2323 TX

10092563

997536

SQL> select count(*) from v$lock where id1=10092563 and id2=997536;

COUNT(*)
---------2
SQL> select count(*) from v$lock where id1=10092563;
COUNT(*)
---------2

XX sessions are blocked by TM lock

select * from dba_blockers;
select * from gv$session_blockers;
Unlocking the locked DB user account:
SELECT username, account_status FROM dba_users WHERE username= 'DBSNMP';
alter user DBSNMP account unlock;
select os_username,username,to_char(timestamp,'MM-DD-YYYY HH24:MI:SS') as time from (select *
from dba_audit_trail order by timestamp desc) dba_audit_trail_alias WHERE rownum <= 100 ORDER
BY rownum DESC;

Listner response to TNS ping is XXX msecs

Set the oraenv variable.
Use Command:
# tnsping net_service_name count
Note: net_service_name must exist in tnsnames.ora file. The location of this file will be something
like:/
u01/app/11.2.0.3/grid/network/admin or
/u01/app/oracle/product/11.2.0.3/oh1066/network/admin/samples/listener.ora
db_recovery_file_dest_size of XXXXXXX bytes is XX.XX% used and
has XXXXXXX remaining bytes available.
SQL>select name, floor(space_limit / 1024 / 1024/1024) "Size in GB", ceil(space_used / 1024 /
1024/1024) "Used in GB", floor(space_limit / 1024 / 1024/1024) - ceil(space_used / 1024 /
1024/1024) "FREE space in GB" from v$recovery_file_dest;
SQL>SELECT Sum(percent_space_used)-Sum(percent_space_reclaimable)
"PCT_SPACE_USED_NOT_FREEABLE" FROM v$flash_recovery_area_usage;
To increase the FRA size:
SQL>show parameter recovery
SQL>alter system set db_recovery_file_dest_size=50G scope=both;
Delete Old rman backup archives:
$ rman target /
RMAN> crosscheck backup;
RMAN> delete noprompt obsolete;
RMAN> crosscheck archivelog all;
RMAN> delete noprompt archivelog all;
RMAN> exit;
$sync
XX object(s) are invalid in the <schema_owner> schema
COLUMN object_name FORMAT A30
SELECT owner, object_type, object_name, status FROM
ORDER BY owner, object_type, object_name;
OWNER OBJECT_NAME

OBJECT_TYPE

dba_objects WHERE status = 'INVALID'

--------------------------------------------------------------------------------------------------Here OWNER is schema & OBJECT_NAME is object.

alter materialized view VEHICLES.CARSALES_RB_FEATURES_MV compile;
Free space % in Disk group <DISK GROUP NAME>
select name, round(TOTAL_MB/1024) as Total_SPACE,round(free_mb/1024) AS
FREE_SPACE_IN_GB,round(100 * (FREE_MB / TOTAL_MB),2) AS
PERCENTAT_OF_FREE_SPACE_IN_GB from v$asm_diskgroup;
Disk Group XXXX_XX requires rebalance because the space usage imbalance between disks
is high
SQL> select GROUP_NUMBER, DISK_NUMBER ,TOTAL_MB,FREE_MB ,NAME from v$asm_disk WHERE
NAME LIKE 'XXXX_XX%' ORDER BY DISK_NUMBER;
SQL> select g.name "Diskgroup", 100*(max((d.total_mb-d.free_mb)/d.total_mb)-min((d.total_mbd.free_mb)/d.total_mb))/max((d.total_mb-d.free_mb)/d.total_mb) "Imbalance",
100*(max(d.total_mb)-min(d.total_mb))/max(d.total_mb) "Varience",
100*(min(d.free_mb/d.total_mb)) "MinFree", count(*) "DiskCnt", g.type "Type" from &asm_disk d ,
&asm_diskgroup g where d.group_number = g.group_number an 2 d d.group_number <> 0 and
d.state = 'NORMAL' and d.mount_status = 'CACHED' group by g.name , g.type;
Enter value for asm_disk: v$asm_disk_stat
Enter value for asm_diskgroup: v$asm_diskgroup_stat
XX% of user dump area is used.
SQL> show parameter USER_DUMP_DEST;
NAME

TYPE

VALUE

------------------------------------ ----------- -----------------------------user_dump_dest

string

/u01/app/oracle/diag/rdbms/opr143/opr1431/trace

$ df -h /u01/app/oracle/diag/rdbms/opr143/opr1431/trace
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1 99G 72G 22G 77% /u01

Checking job failures

SELECT COUNT(*) FROM dba_jobs WHERE NVL(failures, 0) < > 0;
Retrieve the SQL statement from the top oracle OS processes with the following SQL query
select ses.sid SID,sqa.SQL_TEXT SQL from v$session ses, v$sqlarea sqa, v$process proc where
ses.paddr=proc.addr and ses.sql_hash_value=sqa.hash_value and proc.spid=<OS_process_id>;
Creating a ADRCI package for an incident:

$adrci
ADRCI>show home
ADRCI>set home <path>
ADRCI>show home
ADRCI>show incident
ADRCI>ips pack incident 160348 in /tmp
Getting imageinfo for the Database:
[root@dm01db01 ~]# imageinfo
Kernel version: 2.6.18-274.18.1.0.1.el5 #1 SMP Thu Feb 9 19:07:16 EST 2012 x86_64
Image version: 11.2.3.1.1.120607
Image activated: 2012-12-15 07:39:48 +0100
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
Getting OPatch details:
[grid@dm01db01 OPatch]$ pwd
/u01/app/11.2.0.3/grid/OPatch
[grid@dm01db01 OPatch]$ ./opatch lsinventory -detail > /tmp/Opatch.txt

Finding our Primary Database for a standby database:

show parameter DEST_2
tnsping <value from the above query>

Viewing Oracle Database Parameters

1. To view non-hidden parameters
show parameter <parameter name>
2. To view hidden parameter starting with '_'
SELECT a.ksppinm "Parameter",
b.ksppstvl "Session Value",
c.ksppstvl "Instance Value"
FROM

x$ksppi a,

x$ksppcv b,
x$ksppsv c
WHERE a.indx = b.indx
AND

a.indx = c.indx

AND

a.ksppinm LIKE '<Parameter Name>';

Checking the redo log difference on standby database:

Check for the stand by lag time in seconds:
SQL> SELECT r.item, sofar, r.units FROM gv$recovery_progress r WHERE r.start_time = (SELECT
MAX(start_time) from gv$recovery_progress);
ITEM SOFAR UNITS
-------------------------------- ---------- -------------------------------Log Files 9071 Files
Active Apply Rate 134643 KB/sec
Average Apply Rate 792494 KB/sec
Maximum Apply Rate 169175786 KB/sec
Redo Applied 455743288 Megabytes
Last Applied Redo 0 SCN+Time
Active Time 389895 Seconds
Apply Time per Log 34 Seconds
Checkpoint Time per Log 8 Seconds
Elapsed Time 588876 Seconds
Standby Apply Lag 1666 Seconds
11 rows selected.
Check if the MRP process is running and applying logs:
SQL> select inst_id, process, status MRP_stat, thread#, sequence#, block#
from gv$managed_standby

where process like 'MRP%' or process like 'RFS%' and status != 'IDLE'
order by inst_id,process,thread# ;
INST_ID PROCESS MRP_STAT THREAD# SEQUENCE# BLOCK#
---------- --------- ------------ ---------- ---------- ---------1 MRP0 APPLYING_LOG 4 115056 207
Check for the difference between the Last Sequence Received & Last Sequence Applied
Difference. Monitor the difference closely, if it does not decreases over the time, the MRP
process needs to be restarted.
SQL> SELECT null dummy, ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received",
APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
FROM
( SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME )
IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#))
ARCH,
( SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME )
IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#))
APPL
WHERE ARCH.THREAD#=APPL.THREAD#
ORDER BY 2;
D Thread Last Sequence Received Last Sequence Applied Difference
- ---------- ---------------------- --------------------- ---------1 115057 115056 1
2 120630 120628 2
2 rows selected.

ORA-01013: user requested cancel of current operation:

Cause: The user interrupted an Oracle operation by entering CTRL-C, Control-C, or another canceling
operation. This forces the current operation to end. This is an informational message only.
The most common cause of ORA-01013 is timeouts. Timeouts can be a little more tricky to identify
because they may not be specifically reported in connection with ORA-01013. . If a timeout is causing
your ORA-01013 error in a SQL pass-through query, the properties in the timeout parameter should be
altered to a higher value.
ORA-03111: break received on communication channel
The error could be because of any of the following reasons:

a. Network timeout between application server and database server.

b. SQL Query cross the specified timeout value from web services like tomcat Or JDBC connection.
c. Or if client JDBC Thin driver use lower version to compare the oracle version.