H

P
C
&L
St
ake
ho
ld
er
use
on
ly
.R
ep
ro
du
c tio
n
in
w
ho
le
or
i
BitSpyder - The Culture of Knowledge

n
pa
rt
w
ith
ou
tp
er
m
is
sio
n
is
pr
oh
ib
ite
d.
 BitSpyder - The Culture of Knowledge

Accelerated Interoperability and Troubleshooting HP Networks

Part 2: HP Networking Interoperability
Rev. 11.31 - Course #: 00314282
Part Number: 00314282S21104 – Book 2 of 2

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

Student guide
HP Partner Learning
 BitSpyder - The Culture of Knowledge

d.
ite
ib
oh
pr
is
n
ios
is
m
er
tp
 Copyright 2011 Hewlett-Packard Development Company, L.P.

ou
The information contained herein is subject to change without notice. The only warranties for

ith
HP products and services are set forth in the express warranty statements accompanying such

w
products and services. Nothing herein should be construed as constituting an additional

rt
warranty. HP shall not be liable for technical or editorial errors or omissions contained

pa
herein.
This is an HP copyrighted work that may not be reproduced without the written permission of
i n
HP. You may not use these materials to deliver training to any person outside of your
or
organization without the written permission of HP.
e

Printed in United States of America

l
ho

Accelerated Interoperability and Troubleshooting HP Networks – v11.31

w

Student guide – Book 2 of 2

in

April 2011
n

HP Restricted
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H
 BitSpyder - The Culture of Knowledge

Contents
Module 1: Troubleshooting Methodologies and Practices ................................. 1 - 1
Troubleshooting Methodology ................................................................ 1 - 2
Problem Solving Methodology ............................................................... 1 - 4
Identification and Analysis .................................................................... 1 - 6
Hypothesis and Validation..................................................................... 1 - 8
Implementation and Verification ............................................................ 1 - 10
Summary ........................................................................................... 1 - 11
Module 2: Layer 1 (Physical Layer) Troubleshooting and Problem Resolution ....... 2 - 1
“It’s the cable” ..................................................................................... 2 - 2
Physical Layer Symptoms ....................................................................... 2 - 3
Module 3: Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution ..... 3 - 1
Switching ............................................................................................ 3 - 2
VLANs ................................................................................................ 3 - 3
Switch VLAN port types ........................................................................ 3 - 4
Link Aggregation.................................................................................. 3 - 9
LACP – Link Aggregation Control Protocol .............................................. 3 - 14
Configurable LACP States ..................................................................... 3 - 14
Static vs. Dynamic Link Aggregation ...................................................... 3 - 15
Spanning Tree .................................................................................... 3 - 16
Basic IRF Concepts .............................................................................. 3 - 21
How IRF simplifies networks ..................................................................3 - 23
Lab 4: VLAN Switching ....................................................................... 3 - 29
Module 4: Layer 3 (Network Layer) Troubleshooting and Problem Resolution ...... 4 - 1
Forwarding between VLANs .................................................................. 4 - 2
VRRP Basics......................................................................................... 4 - 5
OSPF Basics ........................................................................................ 4 - 7
External and internal Border Gateway Protocol (BGP) .............................. 4 - 12
Network Address Translation (NAT) ....................................................... 4 - 14
Static and Dynamic NAT ...................................................................... 4 - 16
Lab 5: Layer 3 Practice and Tools .......................................................... 4 - 17
Lab 6: OSPF Routing Issues .................................................................. 4 - 18
Lab 7: Addressing Issues ...................................................................... 4 - 19
Lab 8: Inter-VLAN and Routing ............................................................. 4 - 20
Module 5: Layer 4 (Transport Layer) Troubleshooting and Problem Resolution ..... 5 - 1
Troubleshooting TCP/UDP ..................................................................... 5 - 2
Firewalls.............................................................................................. 5 - 7
Firewall types....................................................................................... 5 - 9
Network address translator (NAT) ......................................................... 5 - 11
Module 6: Layer 5 (Application Layer) Troubleshooting and Problem Resolution.. 6 - 1
QoS process flow ................................................................................. 6 - 2
802.1p traffic prioritization .................................................................... 6 - 8
Traffic marking by an end station .......................................................... 6 - 11
Retaining priority between VLANs ......................................................... 6 - 12
Normal priority data traffic ................................................................... 6 - 14
Lab 10: Quality of Service.................................................................... 6 - 15
Rev 10.41 i
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Module 7: Troubleshooting an End-to-End Complex, Integrated Multi-Protocol

Network .................................................................................................... 7 - 1
Lab 11: Final lab ................................................................................... 7 - 2

ii Rev 10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting Methodologies and Practices

Module 1

No network or networking technology operates smoothly all of the time. Every

network technician will be required at some time to troubleshoot issues in network
configuration and performance. This module introduces basic techniques for
network troubleshooting.
After completing this module, you will be able to:
 Describe a framework for basic network troubleshooting

Rev 10.41 1 –1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Troubleshooting Methodology
Network troubleshooting benefits from having:
 Methodology
 A discipline for evaluating, analyzing and investigating problem
conditions
 Includes determining the scope of the problem, developing a hypothesis,
testing it out, and if successful, implementing a resolution
 Skill sets
 Familiarity with network devices, how they operate and how they are
managed
 Technical tools that may be useful for investigating and verifying
problems; from CLI commands and protocol analyzers
 Good Q&A skills
 Experience
 Over time, applying a methodology and the technical tools helps develop
your own “library” of problem recognition capabilities and yields a more
efficient problem resolution process
The basics of troubleshooting any kind of networking trouble might be succinctly
stated as “keep eliminating obvious causes until the real cause presents itself.” But
understanding what this means requires a systematic approach and real discipline
when attempting to identify causes from symptoms and apply the right fixes or
workarounds.
Troubleshooting is a skill that all networking professionals learn by trial and error.
But skipping some of the more painful or obvious errors can make your learning
somewhat less trying than it might be otherwise. The most important
characteristic to cultivate when solving problems is calmness. If you can keep a
clear head when things fail or start degrading seriously, you’ll be better able to
assess your situation and better equipped to solve whatever problems you
discover.

Methodology
Development of problem solving techniques is often an on-the-job acquisition
process. Few of us can expect much along the lines of formal network
troubleshooting training in our job positions for a number of reasons. These
reasons may include:
 The relatively fast pace of the day-to-day job tasks and challenges yields little
time to pursue formal training on troubleshooting aspects such as technical
tools like a protocol analyzer.

1 –2 Rev 10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting Methodologies and Practices

 Few business environments provide the luxury of a “test lab” and the time to
hone your skills where a progression of test problems can be examined,
worked through, and resolutions tried out.
In the absence of a more ideal situation, a problem solving methodology can
increase the effectiveness of support staff by standardizing the approach used to
some extent. With a fairly modest amount of discipline, network technicians can
improve their problem resolution efficiency in terms of the effort needed and the
number of other people that must be directly involved.

Skill Sets
There are a variety of skill sets that can enhance a network technician’s success in
problem solving. Some of these skills are purely technical in nature. For instance, it
is important to understand the fundamentals of how network devices operate and
how they are managed. Having proficiency in reading logs or interpreting a
protocol analyzer display are examples of having familiarity with the potential tools
you may need to call upon from your “toolbox”.
Other skills are much less technical, but still very important. As part of the problem
investigation process, a network technician may need to talk with various levels of
staff. The staff may include non-technical end-users and business unit managers
to software and hardware vendor support people. Having sufficient interpersonal
skills coupled with good investigative reporter-like skills can expedite the isolation
of a problem and eliminate the “noise” that often conceals the real problem.
Proactive IT support groups tend to spend time on developing procedures and
tools to facilitate problem resolutions. Some examples of technical tools used by
the network technicians are:
 Device logs—Archived instances of the logs as well as the current one may
provide hints of where the problem may be. At the very least, familiarity with
a log file’s typical contents helps you differentiate normal from abnormal
situations.
 Device statistics and status information—Being able to determine the health of
a system or the network is important for gathering the “vital” signs. This type
of information can include anything from port statistics and CPU utilization to
network reachability results.
 Protocol analyzer—Although this may not be a frequently used tool, it can be
invaluable for examining what conversations are or are not occurring between
communicating devices.
A problem solving methodology that is refined over time can be very beneficial to
network technicians. Being methodical and learning from the macro and micro
levels of mistakes can help network technicians improve problem recognition
capabilities and yield a more efficient application of a problem resolution process.

Rev 10.41 1 –3
 BitSpyder - The Culture of Knowledge

Troublesh
hooting HP Networks

Prob
blem So
olving Methodologyy

Figure 1

A problem solving methodology

m y is a processs for managiing problem resolution.
Although h there is no one specificc model thatt may be use eful for all prroblem
situations, a general framework can c provide guidelines aand help enssure efficienccy
in the effforts made to solve a pro oblem. App plying a meth hodology ca an improve the
probabiliity of a succeessful resolution. This g raphic illustrrates the framework for a
general problem
p solvving method dology that h has many ap pplications, including in
today’s contemporarry network environmentss.
e six steps to
There are o the problemm solving m methodologyy outlined he ere. The step
ps
must be executed in order startin ng with iden ntification. T
The rules of tthe
methodo ology state that if a step fails, you m ust return to
o the precedding step abo
ove
or possib
bly return to the top leveel step.
The six stteps are:
 Iden
ntification—UUnderstand and
a documeent the prob blem from bo oth a user an
nd
technical perspeective. Some
etimes it is p
possible to lo
ose sight of w
what the
pote
ential problem is before searching foor a cause wh hen we don’’t consider
multtiple perspecctives.
 Anallysis—Evalua ate the situattion by invesstigating usiing problem
m resolution
toolss, product do
ocumentatio on and user input.

1 –4 Rev 1
10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting Methodologies and Practices

 Hypothesis—Develop possible resolutions based on the analysis and

document a possible resolution. This documentation may be fairly informal,
but it is important to be able to explain it in writing. Doing so can reveal a
hypothesis that is unclear and for which a possible resolution may not be
plausible.
 Validation—Run a validation process to prove or disprove the hypothesis. This
may not be particularly feasible, for example, if you have no test lab
equipment to try out your hypothesis. At the very least, performing a walk-
through of the hypothesis in an articulate manner with other team members
may help.
 Implementation—Develop an implementation plan along with a back-out
plan, just in case, and then implement the resolution. For example, have a
backup configuration and software image readily available.
 Verification—Verify the success or failure of the implementation. If it fails,
implement the back-out plan.

Rev 10.41 1 –5
 BitSpyder - The Culture of Knowledge

Troublesh
hooting HP Networks

Iden
ntification and
d Analyysis

Figure 2

The first step of the six-step

s methodology is identificatio
on, which is aan observatiion
process. Try to observe everything, not just the apparen nt problem, aand avoid
assuming g something g. Because network
n trou bleshooting
g primarily in
nvolves
evaluatin
ng and resolvving connecttivity issues, the generall procedure begins with an
o symptomss to determine the scop e of the issu
analysis of ue.
For exammple, it is impportant to de
etermine wh hether the prroblem is afffecting a single
host, a group of hostts, or the enttire network . If many ho osts are affeccted, determ
mine
what they have in common. For instance, if a host can communicate e with local
hosts, bu
ut not remote hosts, verify connectivvity with its d default gatewway. If all ho
osts
in the sam
me VLAN ca an communiccate with loccal hosts, buut not remote e hosts, the
issue may be a logica al problem with
w the defaault gatewayy or a physical problem
concerning connectivvity with the e default gateeway. Altho ough the deffault gatewayy
performss Layer 3 forw eir communication with the
warding on behalf of loccal hosts, the
default gateway
g is do
one using Laayer 2 addre ssing.
The identification pro
ocess consists of doing ttasks that caan include:
 Docu
umenting th
he physical se
ettings. Thee specifics wiill of course vary
depe
ending on th
he problem scenario,
s bu t some exam mples are the e following:

1 –6 Rev 1
10.41
 BitSpyder - The Culture of Knowledge

Troublesh
hooting HP Networks

Hypothesiss and Validati

V on

Figure 3

Step 3 is hypothesis. The hypothhesis step invvolves the evvaluation of the

information acquired nalysis step tto determine a number of probable
d from the an
causes.
Some thiings to keep in mind are
e:
 Wha
at is the tech
hnical reason
n for the bus iness proble
em?
 You need a validdation proced dure for the hypothesis to be usable e. Although
yourr intuition may prove to be correct aat times, in th
he business world, relyin
ng
on thhat primarilyy makes it diifficult for m anagement to feel confiident about the
proccess.
 Even
ntual resolution of the prroblem could
d create side me that are not
e effects, som
imm
mediately obvvious.
Validation, step 4, typ
pically involvves experimeentally deterrmining whe ether the
hypothessis is reasonaable. It incre problem will in
eases the co nfidence levvel that the p
fact be re
esolved afterr implementtation of a po otential solu
ution.

1 –8 Rev 1
10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting Methodologies and Practices

The validation step involves:

 Testing each hypothesis until you validate a probable cause with a high degree
of certainty. The objective is not necessarily to be 100% sure, but to balance
the time criticality of resolving a problem with the information you have
available.
 If validation fails for all probable causes you developed, then you may need to
return to the problem definition phase and start over. Despite what may
appear to be time wasted, you will likely have improved your awareness of the
problem situation and will have some additional facts to use when you
attempt to redefine the problem.

Rev 10.41 1 –9
 BitSpyder - The Culture of Knowledge

Troublesh
hooting HP Networks

Implementtation and
a Verificatio
on

Figure 4

Implemeentation, step
p 5, requires planning fo
or installation
n of some foorm of system
m
ork fix or modification alo
or netwo ong with preeparation for failure. If a
an
implementation fails,, you must be
b able to re store the sysstem to a prrevious stable
state.
The planning involve
es:
 Deve
elopment off a specific im
mplementati on plan.
 Deve elopment off a verificatio o prove the iimplementation was
on process to
succcessful.
 Deve
elopment off a back-out plan to ensu ure the imple ementation can be
oved, if it fails. It should
remo d also addresss how to haandle side efffects.
Verification, step 6, iss the processs of proving the implem
mentation wa as successful
and dete ermining thatt any side efffects are accceptable. If verification fails or side
effects arre unaccepta able, the bacck-out plan ddeveloped inn the implem mentation
phase is executed.
Upon succcessful com
mpletion, the
e user or custtomer must be informed d and the
problem resolution should
s be doocumented i n a trouble llog. Lack off
documen on for recurriing problem
ntation can lead to lengtthy resolutio ms.

1 –10 Rev 1
10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting Methodologies and Practices

Summary
 Network troubleshooting benefits from having a methodology, skill sets and
experience
 General problem solving methodology consists of six steps:
 Identification: Develop a problem statement
 Analysis: Narrow the scope
 Hypothesis: Define procedures to validate
 Validation: Test probable causes
 Implementation: Make changes with back-out plan ready
 Verification: Ensure that changes resolve problem without side effects

Rev 10.41 1 –11

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks
 BitSpyder - The Culture of Knowledge

Layer 1 (Physical Layer) Troubleshooting and

Problem Resolution
Module 2

In this module, various layer 1 problems will be discussed.

The technologies include:
 Cable / Link problems
 Link Errors

Rev 10.41 2 –1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

“It’s the cable”

Figure 2.1: Cables

Some of the most common Layer 1 problems can be isolated to the cable.
Common physical layer problems:
 Bad Cables can be terminated improperly or have physical breaks in one or
more conductors, etc.
 Mis-wired cables can be terminated in the wrong order. A common symptom
here is that a cable works with 10 or 100Mbps links but not 1 Gig links because
of the extra conductors required for Gigabit. It is also common to have fiber
links mis-wired so that transmit is connected to transmit and receive is connected
to receive.
 Interference is mostly a problem with unshielded copper cables. This can be
due to running data cable alongside power cable.
 Wrong cable types could be using a CAT3 cable with a Gigabit link or a
multimode fiber cable with transceivers that require single mode, etc.

2 –2 Rev 10.41
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Last 300 seconds output: 0 packets/sec 78 bytes/sec 0%

Input (total): 916 packets, 136158 bytes
186 unicasts, 79 broadcasts, 651 multicasts
Input (normal): 916 packets, - bytes
186 unicasts, 79 broadcasts, 651 multicasts
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 199 packets, 35587 bytes
146 unicasts, 10 broadcasts, 43 multicasts, 0 pauses
Output (normal): 199 packets, - bytes
146 unicasts, 10 broadcasts, 43 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier

[4800G]display brief interface

The brief information of interface(s) under route mode:
Interface Link Protocol-link Protocol type Main IP
NULL0 UP UP(spoofing) NULL --
Vlan1 UP UP ETHERNET 16.1.1.50

The brief information of interface(s) under bridge mode:

Interface Link Speed Duplex Link-type PVID
GE1/0/1 DOWN auto auto access 1
GE1/0/2 UP 1G(a) full(a) access 1
GE1/0/3 DOWN auto auto access 1
---- More ----

[4800G]display logbuffer reverse

Logging buffer configuration and contents:enabled
Allowed max buffer size : 1024
Actual buffer size : 512
Channel number : 4 , Channel name : logbuffer
Dropped messages : 0
Overwritten messages : 0
Current messages : 166

%Apr 26 13:54:59:803 2000 4800G LLDP/2/CREREM:Port

GigabitEthernet1/0/2 (IfIndex 9437185):Created new neighbor,
chassis ID: 001c-2e96-8900, port ID: 1.
%Apr 26 13:54:58:908 2000 4800G MSTP/2/PFWD:Instance 0's
GigabitEthernet1/0/2 has been set to forwarding state!
%Apr 26 13:54:58:907 2000 4800G IFNET/4/UPDOWN:
Line protocol on the interface Vlan-interface1 is UP
%Apr 26 13:54:58:907 2000 4800G IFNET/4/LINK UPDOWN:
Vlan-interface1: link status is UP
%Apr 26 13:54:58:873 2000 4800G IFNET/4/LINK UPDOWN:
GigabitEthernet1/0/2: link status is UP
%Apr 26 13:54:56:209 2000 4800G IFNET/4/UPDOWN:
Line protocol on the interface Vlan-interface1 is DOWN
---- More ----

2 –4 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 1 (Physical Layer) Troubleshooting and Problem Resolution

E3500yl# show interfaces 23

Status and Counters - Port Counters for port 23

Name :
MAC Address : 001c2e-968929
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 1,821,092 Bytes Tx :
304,614
Unicast Rx : 1626 Unicast Tx : 1938
Bcast/Mcast Rx : 10,253 Bcast/Mcast Tx : 503
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0
Others (Since boot or last clear) :
Discard Rx : 0 Out Queue Len : 0
Unknown Protos : 0
Rates (5 minute weighted average) :
Total Rx (bps) : 5,001,008 Total Tx (bps) : 3,010,520
Unicast Rx (Pkts/sec) : 0 Unicast Tx (Pkts/sec) : 0
B/Mcast Rx (Pkts/sec) : 0 B/Mcast Tx (Pkts/sec) : 0
Utilization Rx : 00.50 % Utilization Tx : 00.30 %

E3500yl# show interfaces brief

Status and Counters - Port Status

| Intrusion MDI Flow Bcast

Port Type | Alert Enabled Status Mode Mode Ctrl Limit
----- ----- + ------ ------- ------ ------- ----- ---- -----
1 1000 | No Yes Up 1000FDx MDI off 0
2 1000 | No Yes Down 1000FDx Auto off 0
3 1000 | No Yes Down 1000FDx Auto off 0
4 1000 | No Yes Down 1000FDx Auto off 0
5 1000 | No Yes Down 1000FDx Auto off 0
6 1000 | No Yes Down 1000FDx Auto off 0
-- MORE --, next page: Space, next line: Enter, quit: Control-C

E3500yl# log -r
Keys: W=Warning I=Information
M=Major D=Debug E=Error
---- Reverse event Log listing: Events Since Boot ----
I 10/22/10 17:52:38 00561 ports: port 1 Applying Power to PD.
I 10/22/10 17:52:38 00560 ports: port 1 PD Detected.
I 10/22/10 17:52:36 00076 ports: port 1 is now on-line
I 10/22/10 17:52:35 00565 ports: port 1 PD Removed.
I 10/22/10 17:52:34 00561 ports: port 1 Applying Power to PD.
I 10/22/10 17:52:34 00560 ports: port 1 PD Detected.
I 10/22/10 17:52:31 00565 ports: port 1 PD Removed.
I 10/22/10 17:52:30 00077 ports: port 1 is now off-line
-- MORE --, next page: Space, next line: Enter, quit: Control-C

Rev 10.41 2 –5
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Troubleshooting no link
Step 1: Determine if one or two fibers are in use. BX (bi-directional) transceivers use
only one fiber for both transmit and receive. There are two "flavors" of BX
transceiver. One is a "D" (downstream), the other is a "U" (upstream). You must
connect a "D" to a "U". You cannot connect a "D" to a "D", and you cannot
connect a "U" to a "U".
Is this a BX transceiver link?
Action: If BX, try using the other "flavor" (D or U). Or try a connection to a nearby
device, ensuring D connects to U.
Step 2: Roll (swap) transmit and receive fibers at only one place; for BX ensure "D"
connects to"U".
Does link come up?
Step 3: If no link after rolling the fibers, try connecting to a nearby device with
crossover fiber.
NOTE: Fiber must be "crossover", meaning transmit at one end connects to receive
at the far end. Many fiber patchcords are mis-labeled. Do not rely on color-coding
of strain relief, or "A" and "B" labels on the patchcord, to determine if patchcord is
crossover. (Those can be wrong.) Instead, use manufacturer's lettering on outside of
fiber to identify which strand is which. With connector nub facing up on each end,
and with each connector pointing the same direction, be sure lettering is on left at
one end, and on right at other end, as shown here:

With both connectors facing

same direction, crossover fiber
has lettering on left fiber at one
end, and lettering on right fiber
at other end.

Does link come up?

Action: If no link occurs using crossover fiber to nearby device with known-good
transceiver, then validate with physical inspection that this is a genuine HP
transceiver.

2 –6 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 1 (Physical Layer) Troubleshooting and Problem Resolution

Troubleshooting Errors on link

HP switches keep per-port statistics (counters) that help us diagnose problems on the
link or on the network. In addition to "normal" errors like an occasional bad packet
received (with incorrect FCS/CRC for example), HP switches alert users to abnormal
or "excessive" errors.
"Excessive" errors and FFI
FFI (Find, Fix, Inform) is a feature of HP switches that informs the user when the switch detects a
large number of errors in a short period of time, with specific parameters defined by the HP
Switch Lab. The feature was originally called "Fault-finder", and is a good indicator of
problems on the link or network. Here are the FFI messages and explanations from the "Help"
text in the menu-based event log. Description is what the switch detected. Possible causes are
documented, as are user Actions to resolve the problem.
Too many undersized/giant packets
Description: A device on this port is transmitting packets shorter than 64 bytes or
longer than 1518 bytes (longer than 1522 bytes if tagged), with valid CRCs.
Possible Causes: A misconfigured NIC or a malfunctioning NIC, NIC driver, or
transceiver.
Actions:
a. Check the NIC for a misconfiguration.
b. Update the NIC driver software.
c. Replace the malfunctioning NIC or transceiver.
d. Check for a short-circuit in the cable path connected to this port.
Excessive jabbering
Description: A device on this port is incessantly transmitting packets ("jabbering" is
detected as oversized packets with CRC errors).
Possible Causes: A misconfigured NIC, or a malfunctioning NIC or transceiver. It
could also be caused by a short-circuit in the network cable path.
Actions:
a. Check the NIC for a misconfiguration.
b. Update the NIC driver software.
c. Replace the NIC or transceiver.
d. Check for a short-circuit in the cable path connected to this port.
Excessive CRC/alignment errors
Description: A high percentage of data errors was detected on this port.
Possible Causes: Faulty cabling or topology, half/full duplex mismatch, a
misconfigured NIC, or a malfunctioning NIC, NIC driver, or transceiver.

Rev 10.41 2 –7
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Actions:
a. If this port is 100Base-T, make sure the cable, connectors, punch-down
blocks, and patch panels connecting to the port are Category 5 or better.
Verify the correctness of the installation using a Category 5 test device.
b. Check the directly-connected device for mismatches in half/full duplex
operation (half duplex on the switch and full duplex on the connected
device, or the reverse).
c. Update the NIC driver software.
d. Verify that the network topology conforms to IEEE 802.3 standards.
e. Replace or relocate the cable. Also check wiring closet components,
transceivers, and NICs for proper operation.
Excessive late collisions
Description: Late collisions (collisions detected after transmitting ~64 bytes) were
detected on this port.
Possible Causes: An overextended LAN topology, half/full duplex mismatch, or a
misconfigured or faulty device connected to the port.
Actions:

a. Verify that the network topology conforms to IEEE 802.3 standards. Insert
bridges or switches, if needed, to extend the network topology.
b. Check the directly-connected device for mismatches in half/full duplex
operation (half duplex on the switch and full duplex on the connected
device).
c. If this port is 100Base-T, make sure the cable connecting to that port is
Category 5 or better.
d. Check for faulty cabling, transceivers, and NICs.
High collision or drop rate
Description: A large number of collisions or packet drops have occurred on the
port.
Possible Causes: An extremely high level of traffic on this port, half/full duplex
mismatch, a misconfigured or malfunctioning NIC or transceiver on a device
connected to this port, or a topology loop in the network.
Actions:
a. Use a network monitoring device or application to determine the traffic
levels on the affected segment. If needed, consider subdividing that
segment with switches or bridges, or moving high-traffic devices to their
own switch ports.

2 –8 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 1 (Physical Layer) Troubleshooting and Problem Resolution

b. Check the directly-connected device for mismatches in half/full duplex

operation (half duplex on the switch and full duplex on the connected
device).
c. Check for a misconfigured NIC or transceiver (such as a transceiver
configured for "loopback test" or "SQE test").
d. Verify that there are no topology loops in your network. If not enabled,
you may also enable spanning.
Excessive broadcasts
Description: An excessively high rate of broadcast packets was received on the
port. This degrades the performance of all devices connected to this switch.
Possible Causes: This is usually caused by a network topology loop, but can also be
due to a malfunctioning device, NIC, NIC driver, or software application.
Actions:
a. Verify that there are no topology loops in your network.
b. Find and correct any malfunctioning devices or NICs on the segment.
c. Find and correct any malfunctioning applications on devices on the
segment.

Rev 10.41 2 –9
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks
 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and

Problem Resolution
Module 3

In this module, various layer 2 technologies will be reviewed and common problems
will be discussed.
The technologies include:
 Layer 2 switching
 VLANs
 Link Aggregation
 Spanning Tree
 IRF

Rev 10.41 3 –1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Switching

Figure 3.1: Switching

Today’s switches forward frames in two ways. They flood frame and they switch
frames. Frames are flooded if their destination is unknown. That is, the destination
doesn’t have an entry in the MAC address table. This is also the biggest difference
between hubs and switches. Hubs do not maintain a MAC address table.
When the destination address is known, then a frame is only forwarded towards that
destination. This has the effect of reducing traffic on a network because traffic is not
sent out on all links.

3 –2 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

[DeviceA-Gigabitethernet4/0/2] quit

4. Create aggregate interface Bridge-aggregation 2, configure the destination

MAC-based load sharing mode for the link aggregation group, and assign the
aggregate interface to VLAN 10.
[DeviceA] interface bridge-aggregation 2
[DeviceA-Bridge-Aggregation2] link-aggregation load-sharing mode
destination-mac
[DeviceA-Bridge-Aggregation2] port access vlan 10
[DeviceA-Bridge-Aggregation2] quit

5. Assign ports GE4/0/3 and GE4/0/4 to link aggregation group 2 and VLAN
10.
[DeviceA] interface gigabitethernet 4/0/3
[DeviceA-Gigabitethernet4/0/3] port link-aggregation group 2
[DeviceA-Gigabitethernet4/0/3] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/3] quit
[DeviceA] interface gigabitethernet 4/0/4
[DeviceA-Gigabitethernet4/0/4] port link-aggregation group 2
[DeviceA-Gigabitethernet4/0/4] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/4] quit

Rev 10.41 3 –13

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

LACP – Link Aggregation Control Protocol

Link Aggregation Control Protocol (LACP) is another option for creating ―port trunk
groups‖ on HP switches. LACP is defined by the IEEE standard 802.3ad. LACP was
standardized to allow a switch to automatically recognize coterminous, full duplex,
same-speed links between itself and another LACP-compliant switch.
Although LACP can automatically recognize links that are capable of aggregation,
the activation of an LACP trunk requires some configuration. You can’t simply connect
four links between the same two switches and expect them to act as a trunk.
When using dynamic LACP, you must define the trunk on one side, which is known
as the ―active‖ side. The active side sends Bridge Protocol Data Units (BPDUs) across
every link that has LACP defined statically.
Although a complete description of the fields in the BPDU is beyond the scope of this
course, a few BPDU fields relevant to dynamic operation are worth noting. They are:
 A system identifier, which is the switch’s MAC address.
 A priority value, which is a permutation of the MAC address.
 A port identifier, which contains a port number.
When a switch receives BPDUs through multiple ―passive‖ LACP ports that have the
same system identifier, it knows that those ports are linked to the same switch. If the
links are the same speed, the switch sends BPDUs to the ―active‖ partners on the
other side of the links, and the two switches agree to load share across the group of
links. Passive LACP ports only ―speak‖ when ―spoken‖ to; a passive LACP port sends
BPDUs only after it has received BPDUs from a connected switch.

Configurable LACP States

HP switches offer three possible options for LACP configuration:
 Passive
 Active
 Disabled - (default state)
LACP is configured on a per-port basis. When a port is configured for a passive
LACP state, it will be blocked for approximately five seconds when the switch is
initialized. This is appropriate for ports that are linked to active LACP partners
because it provides the ports with time to discover the LACP topology before
forwarding any traffic. However, this delay can be unacceptable for normal switch
operation.
Consequently, HP recommends that LACP remain in the default state of disabled for
all ports that will not participate in dynamic link aggregation.
If you define a trunk using the trunk command described earlier in this module, the
no lacp command is automatically executed and included in the configuration for

3 –14 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

the ports specified in the trunk command’s port list. Static and dynamic port trunking
cannot be simultaneously active on the same port.
Finally, is the case of 802.1X (Port-Based Access Control) being configured on a Port.
To maintain security, LACP is not allowed on ports configured for 802.1X

d.
authenticator operation. If you configure port security on a port on which LACP

ite
(active or passive) is configured, the switch removes the LACP configuration, displays

ib
a notice that LACP is disabled on the port(s), and enables 802.1X on that port.

oh
pr
Static vs. Dynamic Link Aggregation

is
n
io
One important advantage of dynamic link aggregation is its ability to recognize and

s
is
use trunk standby links. When two switches detect more than four coterminous, same

m
speed links, they aggregate the four links with the lowest port numbers. The

er
remaining links are used as standby links.

tp
ou
While dynamic LACP is the only way to set up standby links in a trunk, its

ith
disadvantage is that in certain circumstances it can give you less control.

w
The primary disadvantage of static link aggregation is its lack of support for standby

rt
pa
links. Switches configured for static link aggregation cannot automatically detect new

n
members of the trunk group and, therefore, cannot use standby links.
i
or
On the other hand, static aggregation enables administrators to retain more control
l e

of the operation of the trunk ports.

ho
w
in
n
ctio
du
ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 3 –15

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Spanning Tree

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
Figure 3.1: Spanning tree

ou
Multiple Spanning Tree Protocol (MSTP) enables the configuration of VLAN-aware

ith
Spanning Tree topologies. As described in IEEE 802.1S, multiple spanning trees

w
rt
allow frames assigned to different VLANs to follow different data routes within

pa
administratively established regions of the network.
i n
In this way, MSTP enables the configuration of Multiple Spanning Trees within a
or
physical topology, which provides significant improvement in the utilization of
l e
ho

redundant links. Furthermore, the standard notes that an MST configuration probably
w

will provide simple and full connectivity for frames even in the presence of
in

administrative errors in the allocation of VLANs to Spanning Trees.

n
tio

MSTP should not be confused with another VLAN-aware Spanning Tree protocol
c

known as Per VLAN Spanning Tree (PVST). In PVST configurations, a separate

du

Spanning Tree instance is created for each VLAN. BPDUs are transmitted with tags
ro
ep

that identify the STP instance and VLAN ID to which they belong. While this enables
.R

the use of redundant links if you apply priorities and costs intelligently, it can be a
ly

CPU-intensive process if there are many VLANs.

on

MSTP, on the other hand, enables the creation of multiple Spanning Tree instances
se

that are specifically mapped to VLANs. It is not necessary to literally have a one-to-
u

one correspondence between Spanning Trees and VLANs. In this way, MSTP
er
ld

combines the best of two extremes—the single Spanning Tree configurations of STP
ho

and RSTP and the Spanning Tree per VLAN configuration of PVST.
ke

MSTP Features
a
St
&L

 MSTP is the default protocol when Spanning Tree is enabled

C

MSTP allows for multiple instances of a redundant path for a set of VLANs within
P


H

the bridged network

 Each Spanning Tree instance has its own Root Bridge
 Traffic is distributed across redundant links

3 –16 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

 MSTP follows the same basic principles as STP

and RSTP
 Compatible and interoperable with STP and RSTP
 Emulates STP and RSTP behaviors when encountering switches that do NOT

d.
ite
support MSTP

ib
Because MSTP implements the same basic principles as the earlier Spanning Tree

oh
protocols, it is completely interoperable and compatible with STP and RSTP.

pr
Furthermore, MSTP will emulate STP and RSTP behaviors when encountering devices

is
that do not support MSTP.

n
sio
MSTP is the latest iteration of Spanning Tree, and is the default Spanning Tree

is
m
protocol on most switches. Check the release notes or manuals for a specific switch to

er
determine its default.

tp
ou
Comparing RSTP, PVST and MSTP

ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
use

Table 2.1: Comparing RSTP, PVST and MSTP

er
ld

Before the release of the MSTP standard, the only IEEE-standardized way to combine
ho

VLANs and Spanning Tree was to resolve loops within the topology without regard to
ke

VLAN configuration.
a
St

Cisco Systems Inc. developed PVST—and later PVST+—to enable the configuration of
&L

VLAN-aware Spanning Trees. PVST enables administrators to configure Bridge and

C

Port Priority settings and path costs so that any two paths between a pair of switches
P
H

can both be used. With PVST enabled, some Spanning Tree instances will take one
path while other instances take another path. However, each of the Spanning Tree
instances is separately configured, which results in more overhead than the simpler

Rev 10.41 3 –17

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

RSTP solution. Furthermore, the scalability of PVST is limited because of the increased
CPU utilization described earlier in this module.
MSTP, on the other hand, enables the configuration of fewer Spanning Tree
instances, typically between 1 and 16, with each VLAN mapped to the appropriate

d.
instance.

ite
ib
Spanning Tree for Instance 1

oh
pr
is
n
io
s
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in

Figure 3.3: Multiple spanning tree (1)

n
tio

With MSTP, Spanning Tree instances are associated with VLAN IDs, not with
c
du

individual links. Because a separate Root Bridge is elected for each MST instance,
ro

each instance uses a different set of links as the active path.

ep
.R

As with STP and RSTP, backup—or Blocking State—ports are not used in the primary
ly

active path, but they enable the quick restoration of connectivity in the event of link
on

failure.
se

In the graphic above, Edge_1 was elected as the Root Bridge for MST Instance 1,
u

which resulted in the topology shown. Instance 1 includes VLANs 2 to 10. The next
er
ld

slide illustrates the Spanning Tree topology for MST Instance 2.

ho
ake
St
&L
C
P
H

3 –18 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

Spanning Tree for Instance 2

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
n
Figure 3.4: Multiple spanning tree (2)
i
or
e

In the diagram above, Edge_2 has been elected as the Root Bridge for MST Instance
l
ho

2. Instance 1 includes VLANs 11 to 20.

w
in

Because of this election, the state of the physical links is different than in MST
n

Instance 1, shown on the previous slide.

ctio

MST Regions
du
ro

A group of switches that collectively define multiple Spanning Tree instances is

ep

known as an MST region

.R
ly

 Each switch can belong to only one region

on

 All switches in a region must have identical configuration attributes:

use

 Alphanumeric configuration name

er
ld

 Configuration revision number

ho

Associations between VLANs and Spanning-Tree instances

ake

A switch defines a region boundary if it receives BPDUs from:

St


&L

 A switch with different configuration attributes, or

C

 An STP or RSTP switch

P
H

MST Instances Within a Single Switch

 When MST is initially enabled, the default conditions are as follows:

Rev 10.41 3 –19

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

 Each switch defines its MAC address as its configuration name and ―0‖ as
its configuration revision number
 All of the VLANs defined on a switch belong to the Internal Spanning Tree
(IST) instance

d.
ite
 To cause the switch to interact correctly with other switches in the MST region,

ib
you must define common configuration attributes

oh
Any VLAN not explicitly mapped to a user-defined instance remains associated

pr


with the IST

is
n
 VLAN 1 is often associated with the IST

sio
is
Immediately after MSTP is enabled, all the VLANs configured on a switch are part of

m
the Internal Spanning Tree (IST), which is an RSTP instance that exists within the MST

er
tp
region. As you add new instances and associate them with VLANs, the VLANs are

ou
removed from the IST. However, the IST remains in place, even if no VLANs are

ith
explicitly mapped to it.

w
In most cases, user-defined VLANs are associated with user-defined instances

rt
pa
configured identically on all switches in the MST region. The default VLAN (VLAN ID
1) remains associated with the IST. This provides an important benefit: if the VLAN-to-
i n
or
instance mappings are misconfigured, you can still access the switch because the
IST’s association with VLAN 1 ensures that connectivity is not completely disrupted.
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

3 –20 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

Basic IRF Concepts

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
Figure 3.5: IRF concepts
i n
or
The devices that form an IRF virtual device are called IRF member devices. A member
l e

device assumes the role of master or slave. An IRF stack contains only one master,
ho

which manages the IRF virtual device. All other members operate as slaves and as
w
in

backups for the master. When the master fails, the IRF virtual device automatically
n

elects a new master from one of the slaves. Master and slaves are selected through
tio

the role election mechanism. The details of the role election mechanism will be
c
du

covered later in this module.

ro

A logical IRF port is a logical port dedicated to the internal connection of an IRF
ep

virtual device. These ports cannot act as access, trunk or hybrid ports. An IRF port is
.R

effective only when it is bound to a physical IRF port.

ly
on

Physical ports used for connecting members of an IRF virtual device are called
se

physical IRF ports. Typically, an Ethernet port or optical port forwards frames to the
u

network. When a physical port is bound to an IRF port, it acts as a physical IRF port
er

and forwards data traffic such as IRF-related negotiation frames and data traffic
ld
ho

among members.
ke

As shown in the figure above, an IRF stack can have a daisy chain topology or a
a
St

ring topology. A ring connection is more reliable than the daisy chain connection. In
&L

a daisy chain topology, the failure of one link can cause the IRF virtual device to
C

partition into two independent IRF virtual devices, which can disrupt connectivity as
P

well as IRF functioning. The failure of a link in a ring connection results in a daisy
H

chain connection, and does not affect IRF services.

Rev 10.41 3 –21

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

IRF application scenario: Increasing port density

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
Figure 3.6: IRF increases port density

ou
ith
IRF provides a simple, cost-effective solution to the issues that arise when use

w
population exceeds the available network ports. With IRF deployed, you can add

rt
new members to your virtual IRF device, adding port density with minimal

pa
configuration of the new switches.
i n
or
IRF application scenario: Expanding system processing
l e

capabilities
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L

Figure 3.7: IRF expands system processing capabilities

C
P
H

When the forwarding capability of the core switch cannot satisfy users’ needs, you
can add a switch to form an IRF stacking system with the original core switch. If the
forwarding capability of one switch is 64 Mpps, the forwarding capability of the
whole stack system is 128 Mbps after another switch is added. Note that this
increases the forwarding capability of the entire stacking system, not a single switch.
3 –22 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

IRF application scenario: Expanding bandwidth

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
Figure 3.8: IRF expands bandwidth

pa
n
You can increase the uplink bandwidth of an edge switch by adding another switch
i
or
to form a stacking system with the existing edge switch. You can configure multiple
e

physical links of the member devices as an aggregation group to increase the

l
ho

bandwidth of the link to the core switch. In the IRF configuration in the above Figure ,
w

four links (two from each switch) are aggregated to double the bandwidth from the
in

edge to the core. Adding a second edge switch without IRF would add more
n
tio

throughput to the core, but the bandwidth would be divided between the edge
c
du

switches and their corresponding clients. To the core switch, the number of edge
ro

switches does not change. The original edge switch will back up the current
ep

configurations to the newly added switch in batches, having minimal effect on

.R

network planning and configuration.

ly

How IRF simplifies networks

on
use
er
ld
ho
ake
St
&L
C
P
H

Figure 3.9: IRF simplifies networks (1)

Rev 10.41 3 –23

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

This network topology provides redundant links between the edge and the
distribution layer. MSTP is required to prevent loops introduced by these redundant
links.
VRRP is a protocol for providing router redundancy. For each of the two segments in

d.
the configuration shown, one router in the distribution layer acts as the master and

ite
does the actual routing and the other acts as a backup. If the master fails, the

ib
backup can take over the routing. In enterprise networks, VRRP is often combined to

oh
add Layer 3 redundancy to the Layer 2 redundancy provided by MSTP.

pr
is
n
io
s
is
m
er
tp
ou
ith
w
rt
pa
i n
or
e

Figure 3.10: IRF simplifies networks (2)

l
ho
w

In this solution, all four of the distribution layer switches are combined into one IRF
in

stack. All of the switches have the same routing table and can route packets received
n
tio

from the edge switches. The IRF master will run the routing protocol for the entire
c

virtual device.
du
ro

When configured as an IRF stack, the distribution layer switches now act as a single
ep

virtual switch. Loops can still occur, however between an edge switch and the IRF
.R

virtual switch. In order to retain the redundant links between the edge and
ly

distribution layers, the redundant links can be combined in a link aggregation,

on

creating a single logical link that spans two physical devices in the IRF virtual switch.
se

Advantages of this topology The IRF topology is simpler to configure and maintain
u
er

than the MSTP/VRRP solution. In the IRF implementation, the virtual switch is
ld

configured as if it were a single device. If the same switches were running MSTP and
ho

VRRP, each switch would need a distinctly different configuration to ensure the correct
ke

election of MSTP Root Bridge and VRRP Master. Furthermore, each switch would
a
St

need to be configured separately for all routing and switching functions.

&L

Architecture: Operational Planes

C
P
H

Plane Functions

3 –24 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

 Management interfaces (console, Telnet, SNMP, FTP,

TFTP, etc.)
Management  Internal/hardware monitoring: temperature, fan status,
module and power management, etc.

d.
 File system including: Configuration File

ite
ib
 Layer 2 protocols: LACP, RSTP, MSTP

oh
Control  Layer 3 Protocols: RIP, OSFP, BGP, ISIS, etc.

pr
 Routing Table

is
 ACLs and QoS Policies

n
sio
 FIB (Forwarding Information Base) and Local ACLs and

is
Forwarding QoS Policies

m
er
 Frame/packet forwarding and handling

tp
ou
ith
Modern Switches and Routers segregate their functions into different groups called

w
―operational planes‖ or simply ―planes‖.

rt
pa
Operational Planes in Standalone Switches
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R

Figure 3.11: Operational planes in standalone switches

ly
on

Modern Switches and Routers segregate their functions into different groups called
se

operational planes or simply planes.

u

The most common planes are:

er
ld

Control Plane: this group includes all internal monitoring and control functions
ho

related to power, temperature, and hardware state in general.

ake

Management plane: this functional group is where the user interface is located
St

and where and all protocols run, for example STP in Layer 2 and OSPF in layer
&L

3.
C
P

 It is in this plane that the routing table is built.

H

 Functions in this plane are software based to allow for upgrades.

 Forwarding Plane: this group of functions includes L2 and L3 forwarding, packet
filtering and QoS policies.

Rev 10.41 3 –25

 BitSpyder - The Culture of Knowledge

Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution

Lab 4: VLAN Switching

Lab 4 is design to ensure you can use a structured troubleshooting methodology to
resolve VLAN switching problems. There are three trouble tickets in this lab. Refer to

d.
your lab guide for instructions on how to do this lab.

ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
ctio
du
ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 3 –29

H
P
C
&L
St
ake
ho
ld
er
use
Troubleshooting HP Networks

on
ly
.R
ep
ro
du
c tio
n
in
w
ho
le
or
i
BitSpyder - The Culture of Knowledge

n
pa
rt
w
ith
ou
tp
er
m
is
sio
n
is
pr
oh
ib
ite
d.
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and

Problem Resolution
Module 4

d.
ite
ib
oh
In this module, various layer 3 technologies will be reviewed and common problems

pr
will be discussed.

is
The technologies include:

n
io
IPv4 Routing and Addressing

s
„

is
m
„ Inter-VLAN Routing

er
tp
„ VRRP

ou
„ OSPF

ith
iBGP/eBGP

w
„

rt
NAT

pa
„

i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 4 ²1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Forwarding between VLANs

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
Figure 4.1: forwarding between VLANs

ith
w
As is shown in the example above, IP address 10.1.2.1 with the 24-bit mask

rt
pa
(255.255.255.0) defines a range of local IP addresses between 10.1.2.0 and
10.1.2.255. When using this mask, the first 24 bits of the IP address are recognized
i n
as the "network" portion; the addresses of all the hosts in this range have the same
or
value in the network portion.
l e
ho

Layer 3 forwarding - host to router

w
in
n
ctio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

Figure 4.2: Layer 3 forwarding

The router has traditionally been a tool for interconnecting networks. As a layer 3
device, it uses layer 3 information to make forwarding decisions and requires that

4 ²2 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

each interface leads to a different network. The diagram above illustrates layer 3
forwarding.
When Host 1 wants to talk to Host 2, it first determines whether Host 2 is local to its
own network. Host 1 uses its own IP address and mask to determine the range of

d.
addresses that are local. In the example above, Host 2 is not in the same address

ite
range as Host 1. The local range of Host 1 is 10.1.2.0 – 10.1.2.255.

ib
oh
Since the intended destination is remote, Host 1 sends the traffic to the MAC address

pr
of its configured default gateway, which is a local router interface. All traffic

is
destined for address ranges other than the local network are directed toward the

n
default gateway. While Host 1 maintains an ARP cache that contains information

sio
about local hosts, including the default gateway, it has no knowledge of layer 2

is
m
addresses on the other side of the router.

er
tp
Layer 3 forwarding ² router to host

ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on

Figure 4.3: Forwarding router to host

use

A router is not transparent to end stations; IP hosts are configured with a local
er

router's address as a default gateway and they send to the router all traffic destined
ld
ho

for hosts on other networks or subnetworks.

ke

The router performs a lookup operation on the packet's destination IP address

a
St

against the entries in a routing table or cache. A successful lookup returns an

&L

outbound interface.
C

The router performs an ARP cache lookup operation to resolve the layer 2 address of
P
H

the destination IP host. In the slide below, the destination host is on a network that is
directly connected to the router. If the destination network is not directly attached to
the router, it sends the packet to another router that leads toward the destination
network.

Rev 10.41 4 ²3
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

VRRP Basics

d.
ite
ib
oh
pr
is
n
sio
is
m
Figure 4.5: VRRP basics

er
tp
Basic default gateway redundancy operation

ou
Common goals for default gateway redundancy methods:

ith
„

w
 Enable continuity for off-network communication despite the failure of the

rt
primary default gateway

pa
n
 Provide for automatic failover from primary to backup default gateway
i
or
within typical session timeout intervals
l e

Common technologies and implementation methods:

ho

„
w

 Routers use shared IP address (virtual address or interface on one router)

in

that is the default gateway address for hosts

n
tio

 Backup router takes over forwarding if Master router fails or is unavailable

c
du

VRRP: automatic failover for default gateway

ro
ep

„ Virtual Router Redundancy Protocol (VRRP) provides automatic failover for default
.R

gateways
ly


on

Specified in IETF RFC 3768

se

 Enables load sharing in designs that coordinate VRRP and MSTP

u
er

 Provides industry standard for default gateway provisioning

ld
ho

 Implemented on all HP Networking E-Series ProVision ASIC switches

ke

VRRP terminology review

a
St

„ A virtual router consists of a set of router interfaces on the same network that
&L

share:
C

 A virtual router identifier (VRID)

P
H

 A virtual IP address
„ One router in the group becomes the VRRP Master; other routers are VRRP
Backup(s)

Rev 10.41 4 ²5
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

 The VRRP Master router periodically sends advertisements to a reserved

multicast group address
 VRRP Backup routers listen for advertisements and assumes Master role if
necessary

d.
ite
„ A VRRP router can support many virtual router instances, each with a unique

ib
VRID/IP address combination

oh
Client interacts with virtual router

pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
Figure 4.6: Client interacts with virtual router
or
l e

Hosts on VRRP-protected networks learn the default gateway’s virtual MAC

ho

address from the Master’s via ARP request

w
in

„ Hosts send all off-network traffic to the local virtual MAC address without
n

knowing it is not a physical address

c tio
du

Automatic failover
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St

Figure 4.7: Automatic fallover

&L
C

„ If the Owner fails, the non-Owner (backup) begins forwarding traffic addressed
P
H

to the VRID 2 virtual MAC address (same as the Router 1 virtual MAC address)
„ Host does not require any configuration changes or session restarts
 Host is unaware that a different router is forwarding its off-network traffic

4 ²6 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

OSPF Basics
„ Benefits
 Offers faster convergence than RIP

d.
ite
 Scales to meet the needs of very large intranets

ib
oh
„ Characteristics

pr
 OSPF routers advertise the state of connected links

is
n
Š Flood advertisements to neighbors, who flood to other neighbors

sio
 Depends on router adjacency, formal relationship used to share routing

is
m
information

er
tp
 Intelligent path selection based on bandwidth-sensitive link costs

ou
 Divide large domain into smaller areas to enhance efficiency

ith
w
Š Careful design can avoid router overload

rt
pa
As described in OSPF is a sophisticated routing protocol
designed to scale to meet the needs of very large enterprise networks. OSPF offers
i n
or
several important advantages over the older Routing Information Protocol (RIP),
including faster convergence times as well as scalability.
l e
ho

OSPF uses hierarchical areas to enhance efficiency. By making sound decisions

w

when defining area borders, network designers can develop routing hierarchies that
in

scale readily without placing undue load on the routers.

n
tio

This module will describe the design, deployment, and configuration of OSPF
c
du

networking using the E-Series ProVision ASIC switches.

ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P

Figure 4.8: OSPF basics

H

„ OSPF provides a hierarchical routing structure based on multiple areas

 Backbone area (Area 0) required

Rev 10.41 4 ²7
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Š Other area types include stub and NSSA

„ Router roles:
 Area Border Router (ABR)

d.
 Autonomous System Boundary Router (ASBR)

ite
As described in OSPF provides a hierarchical routing

ib
oh
structure that can scale to meet enterprise needs. The graphic, adapted from IRF,

pr
illustrates some basic elements of the OSPF topology.

is
For more detail, consult IRF.

n
sio
Enabling OSPF
Enabling OSPF

is
m
er
tp
10.1.65.0/30 10.1.67.3024

ou
Server VLAN 10
E5406_A Student VLAN 30

ith
w
5406zl_A(config)# ip router-id 10.1.0.3 Define Router ID

rt
pa
5406zl_A(config)# router ospf
5406zl_A(ospf)# area 0 Enable OSPF and create Area 0

i n
or
5406zl_A(ospf)# vlan 10 Enable OSPF on each VLAN and
5406zl_A(vlan-10)# ip ospf [area 0] the loopback interface; area ID
e

5406zl_A(vlan-10)# ip ospf passive

defaults to Area 0
l
ho

5406zl_A(vlan-10)# vlan 30
5406zl_A(vlan-30)# ip ospf
Optionally, define stub networks as
w

5406zl_A(vlan-30)# ip ospf passive

―passive‖
in

5406zl_A(vlan-30)# vlan 65
5406zl_A(vlan-65)# ip ospf
n

5406zl_A(vlan-65)# vlan 67
tio

5406zl_A(vlan-67)# ip ospf
c

5406zl_A(vlan-67)# interface loopback 0

du

5406zl_A(lo-0)# ip ospf all

ro

7 Rev. 10.41 Figure 4.9: Enabling OSPF

ep
.R

Before enabling OSPF on an IP router, it is advisable to statically define a Router ID.

ly

If no Router ID is configured, the switch will assign one automatically. On the E-

on

Series ProVision ASIC switches, the choice of ID will depend on other configuration
se

items. Five possible cases are:

u
er

1. A single loopback interface and multiple VLANs with addresses

ld

The loopback interface will be used as ROUTER ID.

ho
ke

2. A single loopback interface with multiple IP addresses

a

The lowest loopback IP address will be used as Router ID.

St
&L

3. Multiple loopback interfaces with multiple IP addresses

C

The lowest loopback number and lowest loopback IP address will be used as
P

Router ID.
H

4. Multiple VLANs with a single IP Address in each VLAN

The IP address of the VLAN that becomes active first will be used as a Router ID.
Typically, on E-Series switches, the lowest number VLAN becomes active first.
Consequently, if an address is defined in VLAN 1, it will become the Router ID.
4 ²8 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

If VLAN 1 is down, the switch will use the next lowest number VLAN IP address
as the Router ID.
5. Multiple VLANs with multiple IP addresses in each VLAN
The lowest IP address of the first active VLAN will be used as a Router ID. In

d.
most cases, this will be a default VLAN IP address.

ite
ib
After the ID is defined, two separate commands are required to enable OSPF

oh
globally on the E-Series ProVision ASIC switches. In the first, you simply enable OSPF

pr
by issuing the router ospf command. In the second, you define at least one area.

is
To form adjacencies, which are fundamental to OSPF operation, two OSPF routers

n
must agree on an area ID, among other items.

sio
is
Note that the configuration for the loopback interface must include an argument

m
specifying which IP addresses will be included in OSPF advertisements. In the

er
tp
example on the previous page, ―all‖ indicates that all addresses will be included.

ou
Alternatively, the administrator could specify any address configured on the interface

ith
as this argument.

w
On the E-Series ProVision ASIC switches, configuration of OSPF at the global and

rt
pa
interface level is dynamic. Enabling OSPF on an interface may cause the router to:
1. i n
Begin sending Hello packets through this interface in an effort to establish
or
adjacencies.
l e
ho

2. Include the network address range associated with this interface in its Router
w

LSA.
in

To minimize OSPF processing overhead, interfaces with no neighboring routers, such

n
tio

as VLANs 10 and 30 in the example on the previous page, may be defined as

c
du

―passive.‖ The router does not send Hello messages over a passive interface, which
ro

means it can never form an adjacency and will never send Link State Updates over
ep

this type of interface.

.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 4 ²9
 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

Viewing OSPF neighbor

Viewing states
OSPF neighbor states
10.1.64.0/30
– E8212_A has full adjacency with one
neighbor on each of the following
networks:

d.
E8212_A E8212_B

ite
10.1.0.1 10.1.0.2
‡ 10.1.64.0/30

ib
‡ 10.1.65.0/30 10.1.65.0/30 10.1.66.0/30

oh
‡ 10.1.68.0/30

pr
10.1.67.0/30 10.1.68.0/30
– With equal interface priorities, the OSPF

is
router with the highest router ID becomes E5406_A E5406_B

n
10.1.0.3 10.1.0.4
the Designated Router

s io
is
E8212_A(config)# show ip ospf neighbor

m
OSPF Neighbor Information

er
Router ID Pri IP Address NbIfState State Rxmt QLen Events

tp
--------------- --- --------------- --------- -------- --------- ----------
10.1.0.2 1 10.1.64.2 DR FULL 0 6

ou
10.1.0.3 1 10.1.65.2 DR FULL 0 6

ith
10.1.0.4 1 10.1.68.2 DR FULL 0 7

w
9 Rev. 10.41 Figure 4.11: OSPF neighbor states

rt
pa
The figure on the previous page showed how information from the OSPF interface
in
and neighbor tables can be combined to learn the state the router interfaces on a
or
given network. In the figure above, the neighbor table from a different router,
l e

E8212_A, which has three neighbors. Because all of E8212_A’s neighbors have
ho

Router IDs that are higher than E8212_A’s Router ID, which is 10.1.0.1, all three
w
in

neighbors have assumed the role of Designated Router on their respective networks.
n

If you were to view the OSPF interface table, you would see that E8212_A has the
tio

Backup DR state for the three networks that support its full adjacencies.
c
du

As shown, the neighbor table identifies each adjacent router by its Router ID and the
ro

IP address on the interface where the adjacency has formed. The table also
ep

indicates each neighbor’s priority and state. Use the OSPF neighbor table to
.R

troubleshoot routing problems that may arise from the failure to form an adjacency.
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 4 ²11

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

External and internal Border Gateway Protocol

(BGP)
BGP uses the Transmission Control Protocol (TCP) as its transport protocol, using port

d.
ite
179 for establishing connections. Running over a reliable transport protocol

ib
eliminates the need for BGP to implement update fragmentation, retransmission,

oh
acknowledgment, and sequencing.

pr
The Internet is organized in a multitude of administratively independent networks

is
n
called domains or Autonomous Systems (AS). For example, an AS can be an Internet

io
Service Provider (ISP), a University campus or a corporate network.

s
is
m
The Border Gateway Protocol is an inter-Autonomous System routing protocol. The

er
primary function of a BGP speaking system is to exchange network reachability

tp
information with other BGP systems. This network reachability information includes

ou
information on the list of Autonomous Systems (ASs) that reachability information

ith
traverses. This information is sufficient to construct a graph of AS connectivity from

w
which routing loops may be pruned and some policy decisions at the AS level may

rt
pa
be enforced
i n
The route to each destination is called the AS path, and the additional route
or
information is included in path attributes. BGP uses the AS path and the path
l e

attributes to completely determine the network topology, detect and eliminate routing
ho
w

loops and it can enforce administrative preferences and routing policy decisions.
in

Contrasting eBGP and iBGP

n
c tio

BGP-4 provides a new set of mechanisms for supporting CIDR. These mechanisms
du

include support for advertising an IP prefix and they eliminate the concept of network
ro

"class" within BGP. BGP-4 also introduces mechanisms which allow aggregation of
ep

routes, including aggregation of AS paths.

.R
ly

Once BGP speakers are connected they exchange messages to start a BGP session
on

with a neighbor. This initial message identifies the sender’s AS number and BGP
se

identifier. Dependent upon whether the two speakers are in the same AS or different
u

will govern the session type. There are two basic session types for BGP, interior and
er

exterior.
ld
ho

While there are many similarities between exterior and interior BGP, the most
ke

important difference is that the BGP speakers in an interior BGP peer session are in
a
St

the same AS. Interior BGP is used within a transit AS, as is shown in the diagram
&L

below.
C
P
H

4 ²12 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
Figure 4-12: Contrasting eBGP and iBGP

rt
pa
Note that BGP routers at the "edge" of a domain will support both interior BGP

n
peers and exterior BGP peers.
i
or
BGP messages and route selection
l e
ho

Routers send open messages to each other to open or establish a BGP connection.
w

The two routers must first establish a TCP connection between them. After which the
in

sending of the Open Message is bidirectional.

n
tio

Routers send Open Messages out and wait until they receive an Open Message from
c
du

their peer before continuing. Once the BGP peer is established, routers can
ro

exchange routing information. This routing information is contained in Update

ep

Messages
.R
ly

Once the connection has been established, the routers send incremental updates that
on

include summarized address ranges and AS numbers. (Messages vary somewhat

se

dependent upon whether they are between interior or exterior BGP speakers.) They
u

also send ―keepalives‖ to maintain the session. The router builds a graph or table of
er

the destinations and the attributes. BGP uses the AS or Autonomous System number
ld
ho

to select the shortest path to route data and avoid routing loops.
ke

The two routers use UPDATE messages to add new routes, replace existing routes,
a
St

withdraw invalid routes, and communicate attributes.

&L

BGP Notification Messages are an error message. The router selects the error type,
C

and puts it into the Notification Message and sends it to the peer. It then tears down
P

the peer connection

H

Notification Messages consist of multiple pieces, including the BGP header, error
code, error sub-code, and data that describes the error. This is important as it helps
the Notification Message recipient router to troubleshoot BGP peering problems

Rev 10.41 4 ²13

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Lab 6: OSPF Routing Issues

Lab 6 is designed to ensure you can use a structured troubleshooting methodology to
resolve OSPF routing problems. There is one trouble ticket in this lab. Refer to your

d.
lab guide for instructions on how to do this lab.

ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

4 ²18 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 3 (Network Layer) Troubleshooting and Problem Resolution

Lab 7: Addressing Issues

Lab 7 is design to ensure you can use a structured troubleshooting methodology to
resolve IP addressing problems. There is one trouble ticket in this lab. Refer to your

d.
lab guide for instructions on how to do this lab.

ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 4 ²19

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Lab 8: Inter-VLAN and Routing

Lab 8 is design to ensure you can use a structured troubleshooting methodology to
resolve inter-VLAN routing issues. There is one trouble ticket in this lab. Refer to your

d.
lab guide for instructions on how to do this lab.

ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

4 ²20 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 4 (Transport Layer) Troubleshooting and

Problem Resolution
Module 5

d.
ite
ib
oh
This module focuses on troubleshooting at the transport layer 4. Upper layer

pr
protocols such as TCP, UDP, HTTP, FTP and Telnet run on top of the IP layer 3.

is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n

Figure 5.1: The 5 layer IETF model

ctio

In the course the five Layer IETF model is used to describe a layered approach to
du

networking. The TCP/IP model consists of four Layers. Even though there are some
ro

architectural differences, both models have interchangeable transport and network

ep

layers and their operation is based upon packet-switched technology.

.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Rev 10.41 5 –1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Troubleshooting TCP/UDP
The Host-to-Host (Transport) Layer contains two protocols; Transmission Control
Protocol (TCP) and User Datagram Protocol (UDP). TCP and UDP are used to

d.
transmit datagrams.

ite
ib
oh
pr
is
n
sio
is
m
er
tp
Figure 5.2: Contrasting TCP and UDP

ou
ith
Below is a description of major differences between TCP and UDP.

w
Reliable/Connection-Oriented

rt


pa
TCP is a connection-oriented protocol. When a file or message send it will get
n
delivered unless connections fails. If connection lost, the server will request the
i
or
lost part. There is no corruption while transferring a message.
l e
ho

 Unreliable/connectionless
w

UDP is connectionless protocol. When you a send a datagram or message, you

in

don't know if it'll get there, it could get lost on the way. There may be
n
tio

corruption while transferring a message

c
du

 Ordered
ro
ep

Each message is sent with a sequence number, so that even if they arrive out of
.R

order, they can be reassembled in the correct order.

ly

Not Ordered
on

If you send two messages out, and they arrive out of order, the application itself
se

would be responsible for reassembly in the proper order.

u
er

Heavyweight
ld


ho

When the low level parts of the TCP "stream" are lost, resend requests have to
ke

be sent, and all the out of sequence parts have to be put back together, so
a
St

requires a bit of work to piece together.

&L

 Lightweight
C

No ordering of messages, no tracking connections, etc. This means it's a lot

P
H

quicker, and the network card / OS have to do very little work to translate the
data back from the packets.
 Streaming

5 –2 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 4 (Transport Layer) Troubleshooting and Problem Resolution

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
w
rt
pa
i n
or
e

Figure 5.4: TCP packet capture

l
ho
w
in

UDP does not ensure that the data bytes sent will arrive at the other site. Thus, UDP
n

imposes less network overhead than TCP.

ctio

Source Port: The 16-bit port number of the process that originated the UDP
du

message on the source device. This will normally be an ephemeral (client) port
ro

number for a request sent by a client to a server, or a well-known/registered

ep

(server) port number for a reply sent by a server to a client.

.R
ly

 Destination Port: The port number of the process that is the ultimate intended
on

recipient of the message on the destination device. This will usually be a well-
se

known/registered (server) port number for a client request, or an ephemeral

u

(client) port number for a server reply.

er
ld

 Length: The length of the entire UDP datagram, including both header and Data
ho

fields.
ake

Checksum: An optional checksum computed over the entire UDP datagram plus
St

a special “pseudo header” of fields. See below for more information.

&L

Data: The encapsulated higher-layer message to be sent.

C


P
H

Rev 10.41 5 –5
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

d.
ite
ib
oh
Figure 5.5: UDP message segment format

pr
is
Below is a picture of a packet capture of the UDP section of the Ethernet frame. Note

n
io
that the UDP packet capture shows the Source port, Destination port, Length and

s
Checksum

is
m
er
tp
ou
ith
w
rt
pa
i n
or
l e
ho
w
in
n
tio

Figure 5.6: UDP packet capture

c
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

5 –6 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 4 (Transport Layer) Troubleshooting and Problem Resolution

Firewalls
Layer 4 protocols are subject to packet filters and firewalls. It is possible to have IP
connectivity between the network components but certain packets are unable to

d.
traverse between a source and destination address. These types of connectivity

ite
issues may cause by problems with:

ib
oh
 Firewalls

pr
 Packet filters

is
n
 Servers

sio
Authentication and authorization

is


m
Application software interoperability

er


tp
 Operating system interoperability

ou
ith
In this section we are going to look at troubleshooting firewall and packet filter

w
issues.

rt
pa
Firewall configurations
i n
You have many options when deciding where or how to implement your firewall. The
or
configuration typically includes a combination of routers, gateways, and servers on
l e
ho

the edge of a trusted network. Firewalls can be configured in (but are not limited to)
w

the following architectures shown in the picture below.

in
n
ctio
du
ro
ep
.R
ly
on
use
er
ld
ho
ake
St
&L
C
P
H

Figure 5.7: Firewall configurations

Rev 10.41 5 –7
 BitSpyder - The Culture of Knowledge

Layer 4 (Transport Layer) Troubleshooting and Problem Resolution

Firewall types
Firewalls fall into one or more of the following categories:
Packet-filtering firewall:

d.
ite
 Must establish a predefined table of rules against which a packet-filtering

ib
firewall compares the full association of the packets.

oh
Must specify which packets should be accepted and which denied.

pr


is
 Can create rules that will drop packets from specific untrusted servers, which you

n
io
identify by IP address.

s
is
Can also create rules that permit particular types of connections (such as FTP

m


connections) only if they are using the appropriate trusted servers (such as the

er
tp
FTP server).

ou
Circuit-level gateway

ith
w
 Acts as a proxy server to establish a circuit with the internal computers.

rt
pa
 All outgoing packets from the trusted clients appear to have the proxy server’s
source IP.
i n
or
 After a connection is established, the circuit-level gateway simply copies and
e

forwards packets back and forth without filtering them further.

l
ho
w

Application-level gateway
in

Acts as a proxy server between a trusted client and an untrusted host.

n


tio

Only accept packets generated by services they are designed to copy, forward,
c


du

and filter.
ro

For example, only a telnet proxy can copy, forward, and filter telnet traffic.
ep

Stateful-inspection firewall
.R
ly

 Combines all of the above.

on

Filtering all incoming and outgoing packets based on source and destination IP
se

addresses and port numbers.

u
er

Ensuring packets in a session are appropriateEvaluates the contents of each packet

ld
ho

up through the application layer and ensures that these contents match the rules
ke

in your company’s network security policy.

a
St

 Algorithms compare packets against known bit patterns of authorized packets.

&L
C
P
H

Rev 10.41 5 –9
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

d.
ite
ib
oh
pr
is
n
sio
is
m
er
tp
ou
ith
Table 5.1: Contrasting firewall types

w
Few firewalls belong in only one of these categories, and fewer still exactly match the

rt
pa
definition for any one category. These categories, however, do reflect the key

n
capabilities that differentiate one firewall from another.
i
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on

Figure 5.8: Stateful-inspection firewalls

se

In a specific firewall implementation, the various types can be combined to create

u

complex, sophisticated solutions. For example, a dual-homed host can be either a

er
ld

circuit-level gateway or an application-level gateway. A screened subnet includes at

ho

least two packet-filtering firewalls.

ake
St
&L
C
P
H

5 –10 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 4 (Transport Layer) Troubleshooting and Problem Resolution

Network address translator (NAT)

There are various types of NAT technology available. These include
 Single IP address translation

d.
ite
 Static NAT and dynamic NAT

ib
Port Address Translation (PAT)

oh


pr
 NAT Traversal (NAT T)

is
Network address translation (NAT) was discussed in an earlier module. This module

n
io
extends this discussion to include Port Address Translation (PAT).

s
is
m
Often, a company’s global address pool does not contain enough public IP

er
addresses to ensure all hosts in the trusted network can be mapped to an Internet

tp
address when they need to be. In this situation, the company should implement Port

ou
Address Translation (PAT). PAT maps each host in the trusted network to a global IP

ith
address and also to a unique TCP or UDP port number on the NAT-enabled router.

w
In this way, PAT can map the same global IP address to a number of private IP

rt
pa
addresses; it uses the unique port number to distinguish between them.
i n
or
l e
ho
w
in
n
ctio
du
ro
ep
.R

Figure 5.9: Port address translation (PAT)

ly
on

The router stores the original IP address and port against the new IP address and
se

port in the address translation table. When the destination computer on the
u

untrusted network sends a reply packet back through the router, the router identifies
er

the recipient on the trusted network using the address translation table and routes the
ld

packet appropriately.
ho
ke

Enabling PAT NAT

a
St

PAT must be enables when you configure NAT, that is:

&L
C

 Configure a basic or advanced ACL for each range of private addresses for
P

which you want to provide NAT.

H

 Configure a pool for each consecutive range of Internet addresses to which you
want NAT to be able to map the private addresses specified in the ACLs. Each
pool must contain a range with no gaps. If your Internet address space has

Rev 10.41 5 –11

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

gaps, configure separate pools for each consecutive range within the address
space.
 Associate a range of private addresses (specified in a basic or advanced ACL)
with a pool.

d.
Enable the Port Address Translation feature if you have more private addresses

ite


ib
that might need NAT than the Internet address pools contain.

oh
Enable outbound NAT on the interface connected to global addresses. The following

pr
commands configure a basic ACL for the private subnet 10.10.10.x/24, then enable

is
inside NAT for the subnet. This example has Port Address Translation Enabled.

n
sio
# acl number 2001

is
m
rule permit source 10.10.10.0 0.0.0.255

er
tp
#

ou
nat address-group 1 209.157.1.2 209.157.1.254

ith
w
)# interface Serial 5/0

rt
pa
nat outbound 2001 address-group 1
i n
or
l e
ho
w
in
n
c tio
du
ro
ep
.R
ly
on
u se
er
ld
ho
ake
St
&L
C
P
H

5 –12 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 5 (Application Layer) Troubleshooting and Problem Resolution

Traffic marking by an end station

 Many IP phones mark their traffic for high-priority handling. In this illustration:
1. Phone marks priority level in IEEE 802.1Q tag
2. The edge switch
a. Classifies traffic based on priority marker in tag
b. Schedules packet for delivery by placing it in queue associated with
traffic class

 Figure 6.5: IP phone illustration

Display the QoS policy applied to VLAN

Below is an example of how to display the parameters of VLAN 2.
# show qos vlan-policy vlan 2
VLAN priorities
VLAN ID Apply rule | DSCP Priority
------- ----------- + ------ -----------
1 No-override | No-override
500 No-override | No-override

Rev 10.41 6 –11

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Retaining priority between VLANs

Continuing the previous example:
3. The core switch classifies traffic based on priority marker in tag
4. The core switch
a. Marks SULRULW\LQSILHOGRIRXWERXQGSDFNHW·V4WDJ
b. Schedules packet for delivery by placing it in appropriate queue
c. Classifies and schedule delivery

Figure 6.6: Continuation of IP phone illustration

Configuring port priority

Below is an example of how port priorities can be set per VLAN or per interface.
# vlan 500
(vlan-500)# qos priority
(vlan-500)# qos
dscp Specify DSCP policy to use.
priority Specify priority to use.
(vlan-500)# qos priority
0
1
2
3
6 –12 Rev 10.41
 BitSpyder - The Culture of Knowledge

Layer 5 (Application Layer) Troubleshooting and Problem Resolution

4
5
6
7
# interface A1
(eth-A1)# qos priority
(eth-A1)# qos
dscp Specify DSCP policy to use.
priority Specify priority to use.
(eth-A1)# qos priority
0
1
2
3
4
5
6
7

Rev 10.41 6 –13

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Normal priority data traffic

 This last illustration show the normal priority data traffic. In this example the edge
switch uplink (port 50) is a tagged member of VLAN 10; 802.1p field in tag contains
´µYDOXH.
The steps are:
1. 7KHXVHU·VGDWDtraffic is sent untagged, with no priority marker
2. The edge switch
a. Classifies the WUDIILFDV¶QRUPDO·
b. Then marks ´µYDOXHLQSILHOGRIRXWERXQGSDFNHW·V4WDJ
c. Schedules packet for delivery, assigning it to the queue associated with
´QRUPDOµWUDIILF

Figure 6.7: Normal priority data traffic

6 –14 Rev 10.41

 BitSpyder - The Culture of Knowledge

Layer 5 (Application Layer) Troubleshooting and Problem Resolution

Lab 10: Quality of Service

Lab 10 is designed to ensure you can use a structured troubleshooting methodology
to resolve Quality of Service problems. There is one trouble ticket in this lab. Refer to
your lab guide for instructions on how to do this lab.

Rev 10.41 6 –15

 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks
 BitSpyder - The Culture of Knowledge

Troubleshooting an End-to-End Complex,

Integrated Multi-Protocol Network
Module 7

This module brings all the lessons from the previous modules and challenges you to
resolve a complex multi-protocol problem.
Stable network operations are critical to most enterprises. Failure of the network
results in productivity and revenue losses. Troubleshooting multiprotocol networks
can be complex and formidable, however following a structured approach diagnosis
and resolution can help resolve problems quickly and effectively.
In this lab you will solve a trouble ticket that has several problems. To do this lab,
you should use a structured approach to troubleshooting and document your steps.

Rev 10.41 7 –1
 BitSpyder - The Culture of Knowledge

Troubleshooting HP Networks

Lab 11: Final lab

Lab 11 is designed to ensure you can use a structured troubleshooting methodology
to resolve problems at multiple protocol layers. There is one trouble ticket in this lab
that contains several problems. Refer to your lab guide for instructions on how to do
this lab.

7 –2 Rev 10.41
 BitSpyder - The Culture of Knowledge

To learn more about HP Networking, visit

www.hp.com/networking
© 2010 Hewlett-Packard Development Company, L.P. The information contained herein is
subject to change without notice. The only warranties for HP products and services are set forth
in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical
or editorial errors or omissions contained herein.