Revision: 1.0
Status: Final Version
April 25, 2001
By
Kevin Reardon
Centers of Expertise
Oracle Corporation
Table of Contents
Introduction ......................................................................................................................... 2
Oracle Net Fundamentals .................................................................................................... 3
Connect Packets .............................................................................................................. 4
1. Bequeathed Connection....................................................................................... 4
2. Redirected Connection ........................................................................................ 5
3. Refused Connection ............................................................................................ 6
Data Packets .................................................................................................................... 6
Marker Packets................................................................................................................ 7
Understanding the Parts of Oracle Net................................................................................ 8
Oracle Net Categories and Layers................................................................................... 8
Outline of an Oracle Net Session Connection................................................................. 9
Locating the Correct Error Codes in the Trace Output ..................................................... 10
Example Trace File ........................................................................................................... 13
Oracle Net Trace File Header........................................................................................ 13
Trace Configuration Information .............................................................................. 14
Parameter Source Information .................................................................................. 14
Log Configuration Information................................................................................. 15
Initializing Oracle Net, Traversing the Name Space and Authenticating the User....... 16
Initializing Oracle Net............................................................................................... 16
Traversing the Name Space....................................................................................... 16
Authenticating the User............................................................................................. 22
Tracking the Flow of Packets........................................................................................ 25
Security Negotiation.................................................................................................. 30
Two-Task Common................................................................................................... 36
Database Login.......................................................................................................... 42
Summary ........................................................................................................................... 46
Oracle Net’s tracing feature is a useful facility for customers, support, and developers to
debug their network problems. Many find, the messages in the trace files are cryptic and
hard to understand. Customers who do not understand the fundamentals of Oracle Net
and need to understand why their client to database communication is failing often read
the Oracle Net trace files to no avail. In trying to determine network problems, the
analyst runs into another problem: how to understand the cryptic Oracle Net trace file.
To better understand and read the trace file, this paper explains the basics of the Oracle
Net protocol. The principles presented here apply to reading and understanding client,
server, or listener trace files as they are of the same format. These principles also hold if
network problem determination needs to be performed on more complex features of the
Oracle database (e.g. replication or database links).
This paper restricts itself to cover the most complex part of the Oracle Net protocol, the
connection process to the database. This is done presenting an example of examining a
client trace file up to the point of logging into the database. Why this point in the trace
file was chosen is because this is where the majority of the features of Oracle Net unfold.
However, the vast majority of the client’s interaction is with the database is transferring
data, but to Oracle Net that is just sending and receiving data. Therefore it will be more
useful, for Oracle Net trace analysis, to detail the connect process.
This paper lightly explains the Oracle Net internals, but is mostly intended to show what
is happening in the trace file. In this paper the following topics are covered:
Oracle Net’s main functions are to first establish connections between an Oracle client
and an Oracle database and second to transfer data between the two. These functions are
performed between a client and a server or, in the case of replication or database links,
between two servers. Most of the activity that involves the features of Oracle Net occurs
in the before logging into the database, when setting up the connection and session to the
Oracle database. Once the connection is established, Oracle Net’s main job is to act as a
data courier for the client and server.
The Oracle Net protocol1 establishes connections and transfers data via various packets.
These packets contain different types of information in them depending on the packet
type. Some of the more common packet types are Connect, Accept, Refuse, Redirect,
Data, and Marker. Here is a list of the Oracle Net packet types. NSPT stands for
Network Session PackeT.
To initially cover these fundamentals, this paper describes several packet types and
groups them in the appropriate roles: connection, transferring data, and interrupting data
transfer. What follows is a description of the packets sent and received by each node for
particular scenarios.
1
NOTE: Oracle Net resides on the Open Systems Interconnection layers of Session, Presentation, and
Application. It is convention to refer to the communication at these layers as to transferring “messages”
and not “packets.” However, to keep consistency with other Oracle documentation, these messages are
refered to as packets in this paper.
1. Bequeathed Connection
When a database connection is requested by an Oracle client process, such as SQL*Plus,
but no service name (or network connection address) is specified, a database is assumed
to be running on the same node as the client. A service name is normally specified by the
user on the command line, login screen, or by setting the TWO_TASK environment
variable.
Since no service name was given, and thus no network address is associated with the
connect request, it is impossible for Oracle Net to know what network address a listener,
dispatcher, or a shadow server process may be listening on, so a brand new dedicated
shadow server process is started. The BEQ adapter starts the shadow server process (the
BEQ adapter is in the native Oracle client application code). The Oracle client then waits
for the shadow process to start and attach itself to the SGA (which is the Oracle Instance
defined in the ORACLE_SID environment variable). If the start-up of the shadow
process is successful, the BEQ adapter uses inter-process communication beween shadow
and client (the actual communication protocol is dependant on the Operating System).
The client then sends a Connect packet2 to the shadow process. The shadow process
sends back an “accept” packet to the client. Once the connection is established, data can
flow between the two processes.
2
NOTE: The connect packet when Bequeathing the connection is NOT a packet that traverses a network.
The Connect can only be bequeathed when the client and server co-exist on the same node.
When an Oracle client process requests a connection with a service name specified,
access to a remote database is assumed. The client process then looks in the
SQLNET.ORA file for the Name Directory Services to be used. In the example
presented in this paper, the client queries an Oracle Names Server, followed by an LDAP
Server, and finally finds the entry in the local service name file TNSNAMES.ORA.
Once the client has the needed name resolution information, it translates the service name
into network protocol specific address information. It then performs a network protocol
connect to the listener at that network address and sends the Oracle Net connect packet.
The listener on the remote database takes in the connect packet, decides if it is to create a
dedicated server shadow process or use an existing dispatcher process, and sends that
process’ address back to the client. The client then resends the connect request directly to
that remote process. Finally, the remote process accepts the connection. Once the
connection is established, data can flow between the two nodes. This connection is
“redirected” rather then “bequeathed” (regardless if the server process is located on the
same or remote node).
When the listener on the remote database does not know about the service being
requested by the client, or if the service is unavailable, the connection cannot be
established and must be refused. A refuse packet is generated and sent by the listener
back to the client. The connection to the database will not be established and the reason
for the refusal will be contained in the refuse packet.
Data Packets
Data packets will be the most common packets during the interaction with the database.
These packets would contain queries and their results, stored procedure calls, and
information or messages that make up the bulk of the session with the database. The
packet type for a data packet is NSPTDA.
NSPTDA packets can be streamed if the network protocol supports this data transfer
technique. TCP/IP supports data streaming through Windowing3. Data streaming can be
seen in an Oracle Net trace by finding the sections where several NSPTDA packets are
being sent with none being received (or the reverse depending on data flow direction).
There are some special cases of data packets, two of which are used for Network Security
and Dead Connection Detection.
Network Security is one special case of data packets occurring during the initial
connection. Network Security negotiation consists of two packets exchanged between
the client and server processes that determine the type of encryption and data integrity
3
Please see “Internetworking with TCP/IP” by Comer for further information on TCP/IP Windowing.
A second special case of a data packet is for Dead Connection Detection. Thse packets
are sent out by the Server to the client to determine if the client is still operational. This
case is called Dead Connection Detection. Configured in the SQLNET.ORA file, they
are sent out by the server to be responded to by the client at a configurable interval.
Marker Packets
The Marker packets are for internal Oracle Net use only. They are used for transferring a
Ctrl-C or Out of Band breaks4 from the client to server, or for transferring Oracle Trace5
information to the server.
4
See the Oracle Net manual for an explanation of Out Of Band (OOB) breaks.
5
See the Oracle8i Designing and Tuning for Performance manual for more information on Oracle Trace.
For example:
As mentioned above, the “nricdt” is the name of the procedure that generated the
message in the trace file. The first 2-3 letters of the procedure name show the component
layer of Oracle Net that executed and placed the entry in the trace file. In this example,
the “nr” of nricdt is the component layer executed. The component layers are as follows:
Once all this is done, Oracle Net has completed its functions and the login into the
database server takes over and the database session is established. The rest of the
conversation between the client and the server, or the server and the server, consists of
NI, NS, and NT calling each other sending data back and forth.
It is very important to know what to look for in the trace file when searching for errors.
Later in this paper, a trace file is analyzed, identifying a few error examples. Not only is
it important to interpret the error codes, but also what is occurring with the Oracle Net
procedures prior to the error. It is very important to not only examine the errors but to
also examine them in context, as this will lead to a correct resolution quickly.
The error information generated is the same across all the trace levels and what process
being traced, be it the Oracle Listener, client or server processes. For enabling Oracle
Net tracing, please refer to the Oracle Networking Manual. When Oracle Net first
encounters an error condition, it calls the procedure ntt2err. This error procedure will
output a line reporting the procedure that was executing, the operation it was in, and the
resultant error codes were either generated or received. Here is a list of possible
operations.
Here is a segment of a trace file to illustrate the error messages that can be generated.
[07-APR-2001 14:19:18] nsrdr: entry
[07-APR-2001 14:19:18] nsrdr: recving a packet
[07-APR-2001 14:19:18] nsprecv: entry Oracle Net enters into a network
[07-APR-2001 14:19:18] nsprecv: reading from transport... transport operation 5 (read) which fails.
[07-APR-2001 14:19:18] nttrd: entry
[07-APR-2001 14:19:37] ntt2err: entry It passes this condition to the NT error
[07-APR-2001 14:19:37] ntt2err: soc 160 error - operation=5, procedure ntt2err. The errors are:
ntresnt[0]=517, ntresnt[1]=54, ntresnt[2]=0
[07-APR-2001 14:19:37] ntt2err: exit ntresnt[0]=517 and ntresnt[0]=54.
[07-APR-2001 14:19:37] nttrd: exit
[07-APR-2001 14:19:37] nsprecv: transport read error
[07-APR-2001 14:19:37] nsprecv: error exit NS Packet receive has interpreted this
[07-APR-2001 14:19:37] nserror: entry condition as a read error and forwards
[07-APR-2001 14:19:37] nserror: nsres: id=0, op=68,
ns=12547, ns2=12560; nt[0]=517, nt[1]=54, nt[2]=0; the conditions to the NS error
ora[0]=0, ora[1]=0, ora[2]=0 procedure nserror. NS then translates
[07-APR-2001 14:19:37] nsrdr: error exit
[07-APR-2001 14:19:37] nsdo: nsctxrnk=0 them to the NS errors:
ns=12547 and ns2=12560.
Every time there is an error condition in Oracle Net, the error code is logged in the trace
file. As in this example, the same error codes are logged several times as the connection
state is moving through the different layers of Oracle Net. Each layer records its own
error codes plus the error codes of the layers below it.
The Network Interface layer is the layer presenting the final error to the user. This error
can be useful in troubleshooting application problems but sometimes only points to a
problem needing further investigation. The network session layer controls the
connection, but the NS error codes may not be the most helpful in determining the real
problem. The more important error messages are the ones at the far right on the lines
shown above. They represent the lower network transport layer error codes and are the
source of the problem with this connection.
The most efficient way to evaluate error codes is to find the lowest level error code
logged. To reiterate from the example above:
[07-APR-2001 14:19:37] nserror: nsres: id=0, op=68,
ns=12547, ns2=12560; nt[0]=517, nt[1]=54, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0
Is interpreted as:
nserror: nsres: id=0, op=68,
ns=12547, TNS-12547 TNS:lost contact
ns2=12560; TNS-12560 TNS:protocol adapter error
nt[0]=517, TNS-517 TNS: Lost contact
nt[1]=54, Operating System protocol error: 54
nt[2]=0; Operating System error: none generated
ora[0]=0, ora[1]=0, ora[2]=0 NS global descriptors: none generated
Here the Operating System error 54 is the actual cause of the NI error ORA-3113 that
was sent to the client process. To locate OS protocol in UNIX, examine the file:
/usr/include/sys/errno.h
WSAECONNRESET 54 10054
Connection reset by peer. An existing connection was forcibly closed by the remote host. This
normally results if the peer application on the remote host is suddenly stopped, the host is
rebooted, or the remote host used a "hard close" on the remote socket. This error may also result
if a connection was broken due to keep-alive activity detecting a failure while one or more
operations are in progress. Operations that were in progress fail with WSAENETRESET.
Subsequent operations fail with WSAECONNRESET.
Now the nt[0]=517 is the error code TNS interpreted from the network protocol error 54.
This can be found using the Oracle UNIX error tool “oerr” or by looking up the error
code in the Oracle Error Messages manual (oerr is not available in Micorosft’s NT). To
use “oerr” to discover more information about Oracle Net return codes, enter:
oerr tns error_number
As it can be seen, a network protocol reset would result in the client having “Lost
contact” with the server.
The explanation of the NS errors, ns2=12569 and ns=12547 can also be found using the
“oerr” tool or in the Oracle Error Messages manual. These would be:
The “internal inconsistency” would be that there was no log off from the database before
the communication terminated.
The Trace Configuration Information contains the path and name of the trace file created
for this particular process and the trace level used.
The trace header records exactly what level of information is being collected. Trace
levels can be set in either the system or user SQLNET.ORA files on the client and server
or in the LISTENER.ORA file on the server. The SQLNET.ORA file turns on/off tracing
for the client or server, while the LISTENER.ORA turns on/off tracing for the Oracle
listener. Valid settings are:
The amount of information Oracle Net will place in the trace file increases as the values
of the trace level increase. These levels are set through the configuration editor, the
Oracle Net Manger, the Net8 Assistant, or by hand-editing the appropriate configuration
files. Performance degradation will be encountered when tracing is enabled, so it is
important to turn tracing off when debugging is completed as the trace files continue to
grow, and take up storage.
The file or files read and used by Oracle Net for configuration are listed under the
heading of Parameter Source Information. This information is relevant because more
then one configuration file can be used.
Two types of SQLNET.ORA configuration files can be located on the client or server:
system and user6. The system SQLNET.ORA file is located in either
ORACLE_HOME/network/admin or in the location specified by the environment
variable TNS_ADMIN. The system file is to be used by more then one client. This
provides the administrator with the ability to configure global parameters while allowing
the changing of parameters for a single client without interfering with all others. The
client is the sole user of the user SQLNET.ORA file and these parameters will override
the system file’s parameters. In Example 1, the system SQLNET.ORA file is loaded
successfully, but the user file is not loaded because it doesn’t exist on the client’s system.
6
NOTE: If it is not known where to locate the user file, examine the Parameter Source Information to find
out which directory the client or server is expecting it to reside in. Oracle Net Manager does not create this
file nor is its location modifiable.
The parameters read should coincide with the ones originally intended. If not, then the
either the wrong parameter files are being modified, or this may be the source of the
problem being diagnosed.
The Log Configuration Information contains the path and filename of the Oracle Net log
file. The log file contains the final error messages for the connection, but may not
contain the cause of the error message. With logging enabled, the error will first appear
there. If there is question as to why the error in the log file is being generated, enable
Oracle Net tracing and examine the resultant trace file for detailed information about the
error. Logging, when enabled, will only create a log file when Oracle Net detects an
error.
Before the connection request occurs, Oracle Net performs several procedures to set up
its running environment. This is the most work done by the Oracle Net features; so
illustrating the diagnosis of problems in this area will relay the needed knowledge of just
what is being done. A simple outline of the tasks needed to get to the database login is as
follows:
• Start Oracle Net, choose the naming method, and resolve service name to a
protocol address
• Authenticate the user through by an external means (authentication Authority).
• Initialize the Network Session, open the network protocol, and connect to the
database
• Negotiate Encryption and Integrity algorithms, Operating Systems, and Two Task
Common character set
• Log into the database
The Network Interface initializes and then queries NR to see if an Oracle Net router is to
be used. This router is Oracle Connection Manager, which is an OSI Session Layer
routing device. Connection Manager has the ability to gateway database communication
between two different protocols, like TCP and SPX. In earlier versions of Oracle Net,
this ability was configured at this point. However, being a gateway between two
protocols is not configured this way anymore, but the procedures are left for backward
compatibility reasons.
NN then takes over to search down the name space directory path for which kind of
naming directory service is to be used. The directory search path is set in the
SQLNET.ORA file by the parameter NAMES.DIRECTORY_PATH. Be careful about
setting the order of the entries in this parameter because NN will search the exact order
from left to right. For this trace file the parameter is set to:
Note from the start of the trace section to when it has been able to resolve the service
name and shut down NN. The trace starts at 15:22:37 and shuts down NN at 15:22:40.
Further down this trace, it is pointed out where the delay is. If the problem was too long
connection times, using time stamping will point out where any delay, like this one, can
be found.
From the start of this process, it took around three seconds to resolve the service name.
Listing TNSNAMES first in the NAMES.DIRECTORY_PATH configuration line would
speed up this service name resolution because it was where the service name was defined.
The proper setting of this parameter will depend on the Oracle Net environment.
At this point, Oracle Net selected the naming method and used it to resolve the service
name to a network address. It then opened the network protocol and connected the
transport layer to the destination Oracle Net Listener. Oracle Net then examined if the
user needed to be validated through Authentication Services, which was not needed this
time. It initialized the network session and several parameters needed.
The next section will explain how Oracle Net packets will traverse the network, showing
the process from connection through database login.
As pointed out earlier, each line in the trace file shows the procedures writing the
message to the trace file followed by the message:
shows the procedure nscon is performing a connection handshake to the database server.
Part of this connection handshake is to send the connection packet to the database server.
The following line indicates this type of packet:
The NSPTCN keyword is the packet type. The easiest way to scan a trace file for
specific packet types is to search for the prefix “NSPT.” The following is the list of all
the Oracle Net packet types available:
7
NOTE: The packet traversal portion is the only item in this document requires trace level 16 (SUPPORT)
tracing.
Sometimes the data that flows inside the packet is viewable to the right of the
hexadecimal format, as in the example below. This example shows the dump of a
connect packet (NSPTCN) with the data viewable to the side of the hexadecimal format.
In packets dumped in this manner, it is possible to view the SQL queries or even the row
data returned from the server. However, database data was not created for ASCII
viewing, so the information displayed can be limited. If the data is encrypted (through
Oracle Advanced Security) or EBCIDIC data is being transferred, the hexadecimal
format will not translate into a viewable form.
The only packets not a specific NSPT type are the security packets. To find the security
packets, look for a data packet (NSPTDA) containing a “magic number” in it. Look for a
NSPTDA packet containing the hexadecimal number “DE AD BE EF” displayed in the
packet. That packet will be a security negotiation packet.
Normally Oracle Net’s NS layer will oscillate between sending a packet and receiving
one. This is not always the case. Sometimes the server has a good deal of data to send
back to the client. In these cases, a series of packet receives may be displayed in the trace
file. This type of behavior is highly dependant on the network protocol’s capability to
stream data, such as TCP/IP’s windowing ability. Controlling this behavior in tuning the
network conversation between the client and server will be a subject of a later paper and
will not be covered in the scope of this document.
Oracle Net begins the security negotiation by setting up the encryption and integrity algorithms and
the system resources they will need. Oracle couples encryption, integrity, and authentication in its
security offerings so this code is bundled in the NA layer. The security packets created during the
process Oracle lovingly refers to as DE AD BE EF packets (as mentioned above).
NOTE: These packets, and all the rest in the trace, are NSPTDA packets. The special packets of
Oracle Netare finished being used as the network session connection between the client and the
server is complete.
Two-Task Common
Two-Task Common (TTC) is Oracle’s implementation of the OSI Presentation Layer.
TTC provides the negotiation of the version of Oracle Net, character set and data type
translation between the different operating systems on the client and server. This layer is
optimized on a per-connection basis to perform data conversion only when required.
TTC is responsible for evaluating the differences in internal data and character set
representations in Operating Systems and determine if a conversion is required for the
two computers to communicate. This is determined first by exchanging the type of
At this point the client is logged into the database. Many of the parameters sent to the
database server process as part of the logon process are visible from the database
V$SESSION view. What follows from here is all NSPTDA (data) packets further setting
up the client’s database session. For the purposes of this paper, the trace is ended here.
Oracle Net trace analysis can be a good source of information when having to resolve the
complex problems arising in computing environments using Oracle software. By
covering the initial connect and login to the Oracle database from the client perspective,
this paper presented several keys for understanding Oracle Net trace files.
This paper illustrated the layer and component structure of Oracle Net. It revealed how
to understand the cryptic lines in the trace file by pointing out the various layers and
procedures used. It pointed out the importance that analyzing the trace files in the proper
context can find both labeled and unlabeled errors (as shown in the example of the LDAP
server). Also explained was the order of the packet traversal for connections and the
various packets used.
Trace files provide an accurate view of Oracle Net activity and the activity between the
client and the database. With an understanding of Oracle Net components and a careful
reading of the messages in the trace files, the reader should be able to analyze Oracle Net
connections and find the source of many problems. It takes practice, but the more trace
files read, the easier they become to understand and use.