Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer
Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer
Advanced Wireless Troubleshooting: Tim Smith, Technical Consulting Engineer
Troubleshooting
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Agenda
• Section 1 • Section 4
➢ Where do we start? ➢ Device Troubleshooting
• Section 2 ➢ AP Join Troubleshooting
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Advanced Wireless
Troubleshooting – Section 1
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Where do we Start?
Where is my problem?
EAP
IP RADIUS ISE
CAPWAP
EOIP
IP
802.11 IP
CAPWAP WLC
DHCP
802.11 Management
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Where is my problem? New!
ISE
CAPWP
In the APs?
EOIP
• Different client types impacted
802.11 • AP reload fixes the issue
Radio resets
• APWAP WLCreported
• SSID not heard DHCP
• One traffic QoS works and not
others
• Stops working after X days
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Where is my problem? New!
ISE
CAPWP
EOIP
In the Controllers?
802.11
APWAP WLC
• Multiple different clients impacted across
different Aps DHCP
• Same client type across different Aps
• Authentication issues
• Ping works, but TCP/UDP not
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Where is my problem? New!
ISE
CAPWP
EOIP
In the Clients?
802.11
Solution(s)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Reducing Scope New!
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Reducing Scope New!
Connected, no
No DNS traffic
Wifi does
not work
No connection
Can ping GW
AAA/Dot1x
Nothing Pending
Client State
No Connection
WebAuth
Required
DHCP
Required
Association
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
• Wrong Password/PSK
Reducing Scope •
•
Radius Config
Shared Secret New!
• EAP method
• Wrong Client Config
• Unsupported IE
AAA/Dot1x
Nothing Pending
• Exhausted pool
Permanent • DHCP server reachability
No • DHCP required and static IP
Connection
WebAuth • Queue stuck
• Certificate issues DHCP • VLAN config
Required
• No DNS Required
• No GW
• URL
• Preauth ACL
Association
• Rates
• Unsupported IE
• Queue Stuck
• AP Memory leaks
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Troubleshooting Basics
▪ Troubleshooting is an art with no right or wrong procedure, but best with a logical
methodology.
▪ Step 1: Define the problem
• Bad description: “Client slow to connect”
• Good description: “Windows 10 clients associations with card 8260 are rejected
with Status17 several times before they associate successfully.”
• Reduce Scope!
• Isolate multiple possible problems over same setup
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Troubleshooting Basics
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Troubleshooting Basics
▪ Step 4: Reproducibility
• Any problem that has a known procedure to reproduce (or frequently randomly
occurs) should be easier to diagnose
• Being able to easily validate or disprove a potential solution saves time by being
able to quickly move on to the next theory
• If the problem can be reproduced, it makes things much easier to work with
development, test the fix and deliver with lower impact to the end customer
• Tests will be conducted to isolate the root cause
▪ Step 5: Fix
• Validate Root Cause Analysis
• Local Reproduction
• Develop Fix
• Validate Fix
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Advanced Wireless
Troubleshooting – Section 2
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Client On-Boarding
What is Client On-Boarding?
The process for a Wireless device to find and connect to a Network,
including the following:
• Scanning the RF channels by sending Probe messages looking for the
target SSID
• Once the best AP servicing that SSID is found, send the initial
Authentication (typically this will be an Open Authentication message)
• Association request sent to the target AP
• Perform any optional 802.1x Authentication
• Encryption Key handshake
• IP addressing to be used (either static IP or DHCP)
• Perform any optional Layer 3 Authentication (Webauth)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
What State are We in? Unlocking the key…
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
What is the Client state?
START
Auth Failure
AUTHCHECK L2AUTHCOMPLETE
8021X_REQD
DHCP_REQD
DHCP
Failure
Auth Failure
WEBAUTH_REQD
FastPath
Failure
DHCP_NOL3SEC
Auth Failure
RUN
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Understanding the Client State
Name Description
8021X_REQD 802.1x (L2) Authentication Pending
DHCP_REQD IP Learning State
WEBAUTH_REQD Web (L3) Authentication Pending
RUN Client Traffic Forwarding
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
The Split MAC Architecture (Local Mode AP)
1. Transmitting Beacons
2. Probe Requests
State 1:
Unauthenticated, AP
Unassociated 3. Probe Responses
4. Authentication Request
5. Authentication Response
State 2:
Authenticated,
6. Association Request
Unassociated
7. Association Response WLC
State 3: 8. (Optional: EAPOL Authentication)
Authenticated,
Associated 9. (Optional: Encrypt Data)
10. Forward User Data
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Understanding Debug Commands
• There are many debug commands available for use, key will be to
know which to use for what issue.
• For most client troubleshooting use the “debug client <MAC>”
• Debug commands will remain active until either the session is
closed or “debug disable-all”
• Debug may be filtered via MAC address (up to 10)
• Be aware that running many debugs on busy systems may affect
WLC overall performance (Debug client has been tested on fully loaded
systems without any adverse affects)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Common Debugs (Local Mode APs)
Client Troubleshooting: Radius:
(wlc3504) >debug client 11:22:33:44:55:66 (wlc3504) >debug aaa ?
MAC Addr 1.................................. 11:22:33:44:55:66 all Configures debug of all AAA messages.
detail Configures debug of AAA detailed events.
SGT debugging .............................. Disabled events Configures debug of AAA events.
Flex-AP Client Debugging .............. Disabled ldap Configures debug of AAA LDAP events.
Flex-Group Client Debugging ......... Disabled local-auth Configures debug of AAA Local Authentication.
packet Configures debug of AAA packets.
Debug Flags Enabled: tacacs Configures debug of AAA TACACS+ events.
dhcp packet enabled.
Client Event enabled.
dot11 mobile enabled.
dot11 state enabled Web Authentication:
dot1x events enabled. (wlc3504) >debug web-auth redirect enable mac <MAC>
dot1x states enabled.
mobility client handoff enabled. Bonjour:
pem events enabled. (wlc3504) >debug mdns all enable
pem state enabled.
802.11r event debug enabled. AP Join:
802.11w event debug enabled. (wlc3504) >debug capwap errors enable
CCKM client debug enabled.
(wlc3504) >debug capwap events enable
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Open Authentication, No Encryption
START
Auth Failure
AUTHCHECK L2AUTHCOMPLETE
8021X_REQD
DHCP_REQD
DHCP
Failure
Auth Failure
WEBAUTH_REQD
FastPath
Failure
DHCP_NOL3SEC
Auth Failure
RUN
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Client Association
▪ Received management frame ASSOCIATION REQUEST on BSSID 70:70:8b:a3:2e:2f destination addr 70:70:8b:a3:2e:2f
▪ Station: 38:71:DE:4E:43:8B trying to join WLAN with RSSI -42. Checking for XOR roam conditions on AP:
70:70:8B:A3:2E:20 Slot: 1
▪ 0.0.0.0 START (0) Change state to AUTHCHECK (2) last state START (0)
▪ 0.0.0.0 AUTHCHECK (2) Change state to L2AUTHCOMPLETE (4) last state AUTHCHECK (2)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Client Association
▪ 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 70:70:8b:a3:2e:20 vapId 1 apVapId 1
▪ 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7) last state L2AUTHCOMPLETE (4)
▪ session timeout for station 38:71:de:4e:43:8b - Session Tout 1800, apfMsTimeOut '1800' and
sessionTimerRunning flag is 0
▪ Sending Assoc Response (status: '0') to station on AP CAP2802 on BSSID 70:70:8b:a3:2e:2f ApVapId 1 Slot 1,
mobility role 0
Since 11k was enabled
▪ Mobility query, PEM State: DHCP_REQD on WLAN, Client is
▪ Mobile Announce sent to 1 members of the local group. requesting Neighbor
Report
▪ Successful transmission of LWAPP Add-Mobile to AP 70:70:8b:a3:2e:20
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Client Association - Common Problems
There can be a number of reasons why a client association request fails, some examples:
• MAC Filtering – Access Reject received from Radius
Access-Reject received from RADIUS server 10.100.76.10 for mobile bc:9f:ef:1b:89:ef
Sending assoc-resp with status 1 station:bc:9f:ef:1b:89:ef AP:70:70:8b:a3:2e:20-01
Sending Assoc Response (status: 'unspecified failure') to station on AP CAP2802 on BSSID
70:70:8b:a3:2e:2f
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Client Association - Common Problems
Be mindful of enabling Cisco specific features that may cause Client Association Failures
• Aironet Extensions
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Client DHCP
Client State = “DHCP_REQD“
• Proxy Enabled:
Client DHCP Discover Client DHCP Discover Is
DHCP Relay/Proxy Unicast to DHCP Servers Bridged to DS
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Client DHCP (IP Learn)
▪ DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 1, encap 0xec03, xid 0x268b23e)
▪ DHCP dropping packet due to ongoing mobility handshake exchange, (siaddr 0.0.0.0, mobility state =
'apfMsMmQueryRequested'
▪ 0.0.0.0 DHCP_REQD (7) mobility role update request from Unassociated to Local Peer = 0.0.0.0, Old Anchor =
0.0.0.0, New Anchor = 192.168.158.80
▪ 0.0.0.0 DHCP_REQD (7) State Update from Mobility-Incomplete to Mobility-Complete, mobility role=Local, client
state=APF_MS_STATE_ASSOCIATED
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Client DHCP
DHCP processing DHCP DISCOVER (1)
DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0
DHCP xid: 0x6e43ffb3 (1849950131), secs: 0, flags: 0
DHCP chaddr: 10:4a:7d:b1:a8:e1
Discover
DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0
DHCP successfully bridged packet to DS
..
DHCP received op BOOTREPLY (2) (len 308,vlan 10, port 1, encap 0xec00, xid 0x6e43ffb3) Offer
DHCP processing DHCP OFFER (2)
..
DHCP received op BOOTREQUEST (1) (len 343,vlan 0, port 1, encap 0xec03, xid 0x6e43ffb3)
DHCP processing DHCP REQUEST (3)
.. Request
DHCP received op BOOTREPLY (2) (len 308,vlan 10, port 1, encap 0xec00, xid 0x6e43ffb3
DHCP processing DHCP ACK (5)
DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 1
DHCP xid: 0x6e43ffb3 (1849950131), secs: 0, flags: 0
DHCP chaddr: 10:4a:7d:b1:a8:e1
DHCP ciaddr: 0.0.0.0, yiaddr: 172.18.254.130 ACK
DHCP siaddr: 0.0.0.0, giaddr: 172.18.254.1
DHCP server id: 172.18.108.43 rcvd server id: 172.18.108.43
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Learning IP without DHCP
Orphan Packet from 10.99.76.147 on mobile
Installing Orphan Pkt IP address 10.99.76.147 for station10.99.76.147 DHCP_REQD
(7) Change state to RUN (20) last state RUN (20)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Client DHCP - Common Problems
• Internal DHCP -> DHCP proxy must be enabled!
• Internal DHCP -> point DHCP server to management
• Bridge mode: check for DHCP server or relay agent on vlan
• DHCP pool exhaustion
• Dirty Server -> Interface Groups.
• Static IP with DHCP required
• DHCP required with RFC compliant clients
Dirty Server
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Pre-Shared Key Encryption
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Client Encryption Key Exchange Packet Flow
AP WLC Radius
Probe Request
Probe Response
Auth Request
Auth Response
Association Request
Association Response
DATA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Client Encryption Key Exchange
• Change state to AUTHCHECK (2) last state START (0)
• AuthenticationRequired = 1
• 0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state AUTHCHECK (2)
• Sending EAPOL-Key Message to mobile bc:9f:ef:1b:89:ef state INITPMK (message 1), replay counter
00.00.00.00.00.00.00.00
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Client Encryption Key Exchange
• Encryption Policy: 4, PTK Key Length: 48
• Sending EAPOL-Key Message to mobile bc:9f:ef:1b:89:ef state PTKINITNEGOTIATING (message 3), replay
counter 00.00.00.00.00.00.00.01
• 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state 8021X_REQD (3)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Encryption Key Exchange - Common Problems
• Client taking a long time or not responding at all to respond to M1 key sent from
WLC
➢ “show advanced EAP” can be used to check your EAP timers, some clients need more time to
respond
Sending EAPOL-Key Message to mobile 00:44:19:6e:33:11
802.1x 'timeoutEvt' Timer expired for station 00:44:19:6e:33:11 and for message = M2
Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:44:19:6e:33:11
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
EAP Authentication (802.1x)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
802.1x EAP Authentication Packet Flow
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAP Start
EAP ID Request
EAP ID Response
EAP Method
Between 4 and
20+ frames EAP Success
EAPoL 4 way Exchange
DATA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
802.1x EAP Authentication
• Sending EAP-Request/Identity to mobile 70:d4:f2:c9:bb:ae (EAP Id 1)
• Received EAP Response from mobile 70:d4:f2:c9:bb:ae (EAP Id 2, EAP Type 25)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
802.1x EAP Authentication
• Received EAP Response from mobile 70:d4:f2:c9:bb:ae (EAP Id 3, EAP Type 25)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
802.1x EAP Authentication - Common Problems
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Local Web Authentication
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Local Web Authentication Packet Flow
Client Controller Radius
Association
1
Association Response
2
DHCP
3
HTTP Request
4
Redirect URL to Captive Portal
HTTP Request
5
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Local Web Authentication Packet Flow
Client Controller Radius
Network Access
9
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Local Web Authentication
• 172.18.254.134 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state DHCP_REQD (7)
• captive-bypass detection disabled, Not checking for wispr in HTTP GET, client mac=38:71:de:4e:43:8b
• unable to get the hostName for virtual IP, using virtual IP =1.1.1.1
• Web-auth type Internal, no further redirection needed. Presenting default login page to user
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Local Web Authentication
• parser host is captive.apple.com
• Access-Accept received from RADIUS server 172.18.123.43 (qid:11) with port:1812, pktId:4
• 72.18.254.134 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state WEBAUTH_NOL3SEC (14)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Local Web Authentication - Common Problems
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Local Web Authentication - Common Problems
Use the command “show custom-web sleep-client summary” to display the list of any clients in this state
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Central Web Authentication using ISE
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Central Web Authentication Packet Flow
Client Controller ISE Radius
Association
1
MAC Authentication
2
Access Accept with Redirect URL and ACL returned
Association Response
3 DHCP
HTTP Request
4
Redirect URL from ISE
HTTP Request
5
HTTP Response with ISE guest Portal page
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Central Web Authentication Packet Flow
Client Controller ISE Radius
8 MAC Authentication
Network Access
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Central Web Authentication
• Association received from mobile on BSSID 70:70:8b:a3:2e:2e AP CAP2802
• Applying Interface(vlan10) policy on Mobile, role Unassociated. Ms NAC State 0 Quarantine Vlan 0 Access Vlan 0
• WLAN CiscoLive-CWA has ISE-NAC security policy, using external RADIUS only for MacAuth-Request
• Redirect URL received for client from RADIUS. Client will be moved to WebAuth_Reqd state to facilitate
redirection. Skip web-auth Flag = 0
• AAA Override Url-Redirect-Acl 'CWA_redirect' mapped to ACL ID 0 and Flexconnect ACL ID 65535
• 172.18.254.81 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state DHCP_REQD (7)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Central Web Authentication
• Client configured with AAA overridden redirect URL
https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e50000000025adf905e&portal=f079c670-7159-
11e7-a355-005056aba4
• AAA redirect is NULL. Skipping Web-auth for Radius NAC enabled WLAN
• WLAN CiscoLive-CWA has ISE-NAC security policy, using external RADIUS only for MacAuth-Request
• CoA Request on MAC-Filter enabled security, initiate mac-auth for the client 38:71:de:4e:43:8b
• 38:71:de:4e:43:8b Send Radius Auth Request with pktId:8 into qid:11 of server at index:0
• 8:71:de:4e:43:8b Access-Accept received from RADIUS server 172.18.123.43 (qid:11) with port:1812, pktId:8
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Central Web Authentication
• 38:71:de:4e:43:8b Resetting web IPv4 acl from 0 to 255
• 38:71:de:4e:43:8b Applying Interface(vlan10) policy on Mobile, role Local. Ms NAC State 2 Quarantine Vlan 0
Access Vlan 10
• 172.18.254.158 WEBAUTH_REQD (8) Change state to START (0) last state WEBAUTH_REQD (8)
• 72.18.254.158 START (0) Change state to AUTHCHECK (2) last state START (0)
• 38:71:de:4e:43:8b AuthenticationRequired = 0
• AUTHCHECK (2) Change state to L2AUTHCOMPLETE (4) last state AUTHCHECK (2)
• L2AUTHCOMPLETE (4) Change state to STATICIP_NOL3SEC (12) last state L2AUTHCOMPLETE (4)
• STATICIP_NOL3SEC (12) Change state to RUN (20) last state STATICIP_NOL3SEC (12)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Central Web Authentication - Common Problems
• Verify the “Allow AAA override” is checked and “NAC State” set to “ISE NAC”
• AAA override passing down an ACL name that does not exist on the WLC (case matters)
AAA Override Url-Redirect-Acl 'CWA_redirect' mapped to ACL ID 0 and Flexconnect ACL ID 65535
• Verify the packet flow for the ACL used. If the ACL is wrong, you will begin to loop:
Preparing redirect URL according to configured Web-Auth type
Client configured with AAA overridden redirect URL
https://172.18.123.43:8443/portal/gateway?sessionId=c0a89e50000000215ad105a7&portal=f079c670-7159-
11e7-a355-005056aba4
[Looping…]
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Central Web Authentication - Common Problems
• Verify that you have “Support for CoA” set to “Enable” on your Radius Authentication Server
entry on the WLC
• Central Web Authentication on the WLC and ISE Configuration Example:
https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/115732-central-web-auth-
00.html
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Advanced Wireless
Troubleshooting – Section 3
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Client Run State
RUN State
• RUN means: client has completed all required policy states
• “NPU entry of Type 1” is the goal
172.18.254.81 DHCP_REQD (7) Change state to RUN (20) last state DHCP_REQD (7)
172.18.254.81 RUN (20) Reached PLUMBFASTPATH: from line 7656, null
RUN (20) Replacing Fast Path rule
type = Airespace AP Client
on AP 70:70:8b:a3:2e:20, slot 1, interface = 1, QOS = 0
IPv4 ACL ID = 255, IPv6 ACL ID
172.18.254.81 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID = 64206, IntfId = 10 Local
Bridging Vlan = 10, Local Bridging intf id = 10
172.18.254.81 RUN (20) Successfully plumbed mobile rule (IPv4 ACL ID 255, IPv6 ACL ID 255, L2 ACL ID
255,URL ACL ID 255,URL ACL Action 0)
172.18.254.81 RUN (20) No 11v BTM
172.18.254.81 RUN (20) NO release MSCB
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
RUN State - AP Radio Events
• Random Disconnections – Radio Reset
00:1a:70:35:84:d6 Cleaning up state for STA 00:1a:70:35:84:d6 due to event for AP
04:da:d2:4f:f0:50(0)
00:1a:70:35:84:d6 Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
RUN State - High RF Channel Utilization
%DOT11-3-NO_BEACONING: Error on Dot11Radio0 - Not Beaconing for too long - Current 2887074 Last
2887074
%LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
%LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
RUN State - High RF Channel Utilization
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Run State – High RF Duty Cycle
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
RUN State - RF Analysis WLCCA
WLCCA
• Tool for quick RF analysis
• RF Health - > simplified quick
view on RF, per Band, AP, AP
Group, Flex Group
Download:
https://developer.cisco.com/docs
/wireless-troubleshooting-
tools/#!wireless-config-analyzer
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
RUN State - RF Analysis WLCCA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Deauthenticated Client
Idle Timeout (300 sec)
38:71:de:4e:43:8b Received DELETE mobile, reasonCode MN_IDLE_TIMEOUT, deleteReason 4 from AP
38:71:de:4e:43:8b apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4,
reasonCode 4
38:71:de:4e:43:8b Sent Disassociate to mobile on AP 70:70:8b:a3:2e:20-1 on BSSID
70:70:8b:a3:2e:2d (reason 4)
38:71:de:4e:43:8b Sent 1x reauth initiate message to multi thread task for mobile
38:71:de:4e:43:8b
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Deauthenticated Client
WLAN Change
Modifying a WLAN in anyway Disables and Re-enables WLAN
38:71:de:4e:43:8b Successfully freed AID 1, slot 1 on AP 70:70:8b:a3:2e:20, #client on this
slot 0
Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Associated to
Disassociated
Sent Disassociate to mobile on AP 70:70:8b:a3:2e:20-1 on BSSID 70:70:8b:a3:2e:2f(reason 1)
Manual Deauthentication
Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
Changing state for mobile 38:71:de:4e:43:8b on AP 70:70:8b:a3:2e:20 from Associated to
Disassociated
Sent Disassociate to mobile on AP 70:70:8b:a3:2e:20-1 on BSSID 70:70:8b:a3:2e:2f(reason 1)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Deauthenticated Client
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Cloud tools - WLC Debug Analyzer
Tool which Automates debug analysis from debug client
https://cway.cisco.com/tools/WirelessDebugAnalyzer/
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Cloud tools – WLC Debug Analyzer
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Cloud tools - WCAE
https://cway.cisco.com/tools/WirelessAnalyzer/
https://developer.cisco.com/docs/wireless-troubleshooting-tools
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Cloud tools - WCAE
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Run State – Key Ideas
▪ Client can be removed for numerous reasons
✓ WLAN change, AP channel change, configured interval (session timeout)
▪ Further Troubleshooting
✓ Client debug should give some indication of what kind of deauth is
happening
✓ Packet capture or client logs may be required to see the exact reason
✓ Never forget Radio status and RF conditions
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Flexconnect Clients
The Split MAC Architecture
(FlexMode AP, Central Auth, Central Switching)
1. Transmitting Beacons
State 1: 2. Probe Requests
Unauthenticated,
Unassociated 3. Probe Responses
4. Authentication Request
AP
5. Authentication Response
State 2:
Authenticated,
6. Association Request
Unassociated
7. Association Response
State 3: 8. (Optional: EAPOL Authentication)
Authenticated,
Associated 9. (Optional: Encrypt Data) WLC
10. Forward User Data
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
The Split MAC Architecture
(FlexMode AP, Local Auth, Local Switching)
1. Transmitting Beacons
State 1: 2. Probe Requests
Unauthenticated,
Unassociated 3. Probe Responses
4. Authentication Request
State 2:
5. Authentication Response
AP
Authenticated,
6. Association Request
Unassociated
7. Association Response
State 3: 8. (Optional: EAPOL Authentication)
Authenticated,
Associated 9. (Optional: Encrypt Data)
WLC
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Show commands on COS Flex APs
Ap2802# show flexconnect ?
00:04:20:FA:DA:00 0 1 5 FWD AES_CCM128 none none none Local Central Local …..
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Show commands on COS Flex APs
Ap2802# show flexconnect wlan
Status of my
Flexconnect WLANs:
Flex wlans,
Radio Vap SSID State Auth Assoc Switching showing client
switching mode
0 0 DOWN Central Local Central
Client on-boarding
status per
radio/VAP
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Show commands on COS Flex APs
1 00:04:20:FA:DA:00 3 true 0 1
2 64:52:99:B0:46:5A 4 true 0 5
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Show commands on COS Flex APs
Ap2802# show dot11 clients
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Show commands on COS Flex APs
Ap2802# show dot11 clients deauthenticated
“wlan” here is
really the VAP
Ap2802# show dot11 clients onboarding dot11radio 1 wlan 0
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Show commands on COS Flex APs
Ap2802# show flexconnect wlan vlan
Native Vlan: 1
0 1 false
1 1 false
2 5 false
3 1 false
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
COS AP – Debug Client
Ap2802# debug client 96:d3:a5:83:85:c7
Client 96:d3:a5:83:85:c7 debugging enabled for critical, errors, events, info, arp, dhcp, eapol, access-lists
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
COS AP – Debug Client Walk Through
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [U:W] DOT11_AUTHENTICATION : (.)
CLSM[96:D3:A5:83:85:C7]: US Assoc Req(0) seq 2612 IF 29 slot 1 vap 0 len 237 state AUTH
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
COS AP – Debug Client Walk Through
CLSM[96:D3:A5:83:85:C7]: client moved from AUTH to ASSOC
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010b
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] EAPOL_KEY.M4 : DescType 0x02 KeyInfo 0x030b
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
COS AP – Debug Client Walk Through
CLSM[96:D3:A5:83:85:C7]: ADD_MOBILE AID 8
CLSM[96:D3:A5:83:85:C7]: Client ADD Encrypt Key success AID 8 Radio 1 Enc 4 Key Len 16 Key idx 0 Key 36 7d
CLSM[96:D3:A5:83:85:C7]: TLV_FLEX_CENTRAL_AUTH_STA_PAYLOAD
Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] ARP_QUERY : Sender 192.168.1.196 TargIp 192.168.1.1
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [U:E] ARP_QUERY : Sender 192.168.1.196 TargIp 192.168.1.1 **[U:E=APtoSw]
[Ap2802-Office] [96:d3:a5:83:85:c7] <wired0> [D:E] ARP_REPLY : Sender 192.168.1.1 HwAddr e0:cb:bc:29:e2:50 **Sw to AP
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
COS AP – Debug Client Walk Through
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] DHCP_REQUEST : TransId 0xb78335bc
[Ap2802-Office] [96:d3:a5:83:85:c7] < wifi1> [U:W] ICMPV6_RS : Src ff02:0:0:0:0:0:0:2 Dst ff02:0:0:0:0:0:0:2
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
COS AP – Debug Client – Client Disassociation
[Ap2802-Office] [96:d3:a5:83:85:c7] <apr1v0> [U:W] DOT11_DISASSOC : (.)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Client Roaming
Client Roaming - Definition
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Client Roaming
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Roaming Triggers
• Possible triggers
✓ RSSI
✓ Beacon loss
✓ Packet errors
✓ 802.11v BSS transition request
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Roaming Patterns
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
Roaming – Avoiding micro cells
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Client Roaming - Security
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
802.1x EAP Authentication Packet Flow
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAP Start
EAP ID Request
EAP ID Response
EAP Method
Between 4 and
20+ frames EAP Success
EAPoL 4 way Exchange
DATA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
CCKM Roaming Packet Flow
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Re Association Request
Re Association Response
DATA
Between 4 and
20+ frames
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
FT Roaming Packet Flow – Over Air
Old AP
Probe Request New AP WLC
Probe Response
Auth Request
Auth Response
Re Association Request
Re Association Response
DATA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
FT Roaming Packet Flow – Over DS
Old AP
New AP WLC
Probe Request
Probe Response
FT Request
FT Response
Auth Request
Auth Response
Re Association Request
Re Association Response
DATA
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
FT Roam – Over air
• *apfOpenDtlSocket: Received management frame AUTH on BSSID a8:0c:0d:db:d4:1d
destination addr a8:0c:0d:db:d4:1d
• *apfMsConnTask_6: Doing preauth for this client over the Air
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
FT Roam – Over air
• *Dot1x_NW_MsgTask_4: 11r roamed client, ft force auth with pskMode : 0
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Advanced Wireless
Troubleshooting – Section 4
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Device
Troubleshooting
WLC Serviceability
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
WLC Serviceability
(wlc3504) > show traplog
Number of Traps Since Last Reset ……............ 95
Number of Traps Since Log Last Displayed .... 1
1 Sat Apr 21 20:30:31 2018 RF Manager updated TxPower for AP CAP2802 Base Radio MAC:
70:70:8b:a3:2e:20 and Radio Type: 802.11a New Tx Power is: 8 ,
Reason: DTPC
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
WLC Serviceability
(wlc) > show logging last-reset
!!!Message and Trap Logs from Previous Reset
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
WLC Serviceability
WLC inter-process communication is handled via Queues
(wlc3504) > show queue-info
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
AP Serviceability
IOS APs have a “flight recorder”
CAP2702# dir
Directory of flash:/
2 -rwx 337 Jan 1 1970 00:05:17 +00:00 info
3 -rwx 64 Apr 18 2018 15:40:45 +00:00 sensord_CSPRNG0
25 drwx 2368 Apr 15 2018 18:07:36 +00:00 ap3g2-k9w8-mx.v153_3_jf.20
4 -rwx 368 Apr 18 2018 15:43:05 +00:00 capwap-saved-config
9 -rwx 965 Dec 20 2017 22:00:13 +00:00 lwapp_mm_mwar_hash.cfg
19 -rwx 326 Apr 18 2018 15:43:05 +00:00 env_vars
69 -rwx 75228 Apr 18 2018 15:40:49 +00:00 event.log
70 drwx 704 Mar 1 1993 00:00:42 +00:00 configs
7 -rwx 64 Apr 18 2018 15:40:45 +00:00 sensord_CSPRNG1
6 -rwx 60266 Apr 2 2018 17:56:17 +00:00 event.capwap
11 -rwx 280 Apr 15 2018 18:10:14 +00:00 lwapp_officeextend.cfg
14 -rwx 368 Apr 21 2018 18:08:15 +00:00 capwap-saved-config-bak
23 -rwx 95008 Apr 18 2018 15:40:34 +00:00 lwapp_reap.cfg
18 -rwx 0 Aug 4 2016 16:40:07 +00:00 config.txt
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
AP Serviceability
COS APs have various Flash Directories to view syslog, crash and core dumps
CAP2802# show flash syslogs
Directory of /storage/syslogs/
total 332
-rw-r--r-- 1 root root 855 Apr 12 16:50 12
-rw-r--r-- 1 root root 20404 Mar 27 23:21 12.0
-rw-r--r-- 1 root root 4809 Apr 12 16:50 12.last_write
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
AP Serviceability
Any of the Syslog Files on COS AP’s may be reviewed using the “more” command,
these syslog files are a rolling log so older information may also be viewed
CAP2802# more syslogs 16
Apr 21 18:28:32 kernel: [*04/21/2018 18:28:32.9447] Stopped Radio 1
Apr 21 18:28:32 kernel: [*04/21/2018 18:28:32.9619] DOT11_DRV[1]: set_channel Channel set to 56
Apr 21 18:28:33 kernel: [*04/21/2018 18:28:33.0671] DOT11_DRV[1]: set_channel Channel set to 56
Apr 21 18:28:33 kernel: [*04/21/2018 18:28:33.4794] DOT11_DRV[1]: set_channel Channel set to 56
Apr 21 18:28:34 kernel: [*04/21/2018 18:28:34.2099] 1:change to DFS channel 56, CAC for 60 seconds.
Apr 21 18:28:34 kernel: [*04/21/2018 18:28:34.2897] Started Radio 1
Apr 21 18:28:48 NCI: I1: openSensor(slot=1)
Apr 21 18:28:50 NCI: I1: SensorApp=1.15.4
Apr 21 18:28:50 NCI: I1: SensorHdw=1.2.3.0
Apr 21 18:28:50 NCI: I1: Hardware Radio Band = [4890, 5935] MHz, BW=150625
Apr 21 18:28:50 NCI: slot=1 mode=0 chanCnt=1 cw=1
Apr 21 18:28:50 NCI: chans: 56 0 0 0 0 0 0 0 0 0 0
Apr 21 18:28:50 NCI: I1: channel map channels: in=1 cloned=1
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
AP Serviceability
Methods of Accessing the AP
• Console
• Telnet / SSH
• No GUI support
• AP Remote Commands
Enabling Telnet/SSH
• WLC CLI: config ap [telnet/ssh] enable <ap name>
• WLC GUI: Wireless > All APs > Select AP > Advanced > Select [telnet/ssh] > Apply
• (No telnet on AP-COS)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
AP Serviceability
▪ Show controller Do[0/1] (or Show Tech)
Must have! Before/During/After event
AP# show capwap client rcb
▪ Show log AdminState : ADMIN_ENABLED
OperationState : UP
Name : CAP2802
▪ WLC: show ap eventlog <ap name> SwVer : 8.5.124.43
HwVer : 1.0.0.0
▪ Show capwap client <?> MwarApMgrIp : 192.168.158.80
MwarName : wlc3504
▪ CLI Tips MwarHwVer : 0.0.0.0
Location : default location
Debug capwap console cli (IOS AP only) ApMode : Local
ApSubMode : Not Configured
Debug capwap client no-reload CAPWAP Path MTU : 1485
CAPWAP UDP-Lite : Enabled
IP Prefer-mode : IPv4
AP Link DTLS Encryption : OFF
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
AP Wired Capture
(New for 8.10 MR1) New!
SS-I-1#Killed
######################################################################## 100.0%
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
AP Wired Capture New!
(New for 8.10 MR1)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
AP Wireless Dump
(New for 8.10MR1) New!
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
AP Wireless Dump
(New for 8.10MR1) New!
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
AP Wireless Remote Dump
(New for 8.10MR1) New!
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
AP Join
Troubleshooting
AP Join Process
▪ WLC Discovery
▪ DTLS/Join
▪ Image Download
▪ Configuration Check
▪ REG
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
L3 WLC Discovery
AP tries to send discover messages to all the WLC addresses that its hunting process
has turned up
Discover
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
AP Discover/Join
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
AP Join Debugging
Since AP’s are always sending packets to the WLC, be careful to make sure you have
the debug mac filter enabled before turning on the other debugs
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
AP Discover/Join – AP Side
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
AP Discover/Join – AP Side
%CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
%CAPWAP-3-ERRORLOG: Selected MWAR '5500-5'(index 0).
%CAPWAP-3-EVENTLOG: Selected MWAR '5500-5' (index 2).
%CAPWAP-3-EVENTLOG: Ap mgr count=1
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
AP Discover/Join – WLC Side
04:da:d2:4f:f0:50 Discovery Request from 192.168.5.156:7411
04:da:d2:4f:f0:50 ApModel: AIR-CAP2602I-E-K9
apModel: AIR-CAP2602I-E-K9
apType = 27 apModel: AIR-CAP2602I-E-K9
apType: Ox1b bundleApImageVer: 8.3.141.0
version:8 release:3 maint:141 build:0
44:03:a7:f1:cf:1c DTLS keys for Control Plane are plumbed successfully for AP
192.168.5.156. Index 7
44:03:a7:f1:cf:1c DTLS Session established server (192.168.5.55:5246), client
(192.168.5.156:7411)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
AP Join – Country Mismatch – WLC Side
*** The key point is to make sure you look at both sides of an AP Join issue.
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
Lightweight AP Joins – Key Ideas
➢ Make sure the AP is getting an address from DHCP (check the DHCP server leases
for the AP’s MAC address)
➢ If the AP’s address is statically set, ensure it is correctly configured. The AP will
switch back to DHCP if a number of join failures occur.
➢ Try pinging from AP to controller and vice versa
➢ If pings are successful, ensure the AP has at least one method to discover the WLC
➢ If you do not have access to APs, use “show cdp neighbors port <x/y> detail” on
connected switch to verify if the AP has an IP address
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
Advanced Wireless
Troubleshooting – Section 5
Tim Smith, Technical Consulting Engineer
DGTL-BRKEWN-3011
#CiscoLive
Understanding
Multicast
Multicast Transport
• Problem: how to replicate multicast traffic to all Aps
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
Multicast Transport
• Solution 1: Unicast replication
• Simple, but costly
AP to WLC: IP SRC: AP IP DST: WLC CAPWAP IP SRC: Client IP DST: Multicast Payload
WLC to Aps: IP SRC: WLC IP DST: AP1 CAPWAP IP SRC: Client IP DST: Multicast Payload
IP SRC: WLC IP DST: AP2 CAPWAP IP SRC: Client IP DST: Multicast Payload
IP SRC: WLC IP DST: APN CAPWAP IP SRC: Client IP DST: Multicast Payload
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Multicast Transport
• Solution 2: Multicast replication
AP to WLC: IP SRC: AP IP DST: WLC CAPWAP IP SRC: Client IP DST: Multicast Payload
WLC to Aps: IP SRC: WLC IP DST: MCAST CAPWAP IP SRC: Client IP DST: Multicast Payload
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Multicast Transport
• WLC will construct “MGID” to represent the multicast traffic relation between AP and
WLCs, against VLANs
• Common problems:
• Traffic is lost from WLC to AP due to multicast drop in network
• Using wrong multicast destination address
• Duplicate address between WLCs
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
Multicast Verification
• WLC L2 MGID
(3500-1) >show network multicast mgid summary
Layer2 MGID Mapping:
-------------------
Interface Name vlanId MGID
-------------------------------- ------ ----
management 15 0
vlan2 2 11
vlan50 50 10
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Multicast Verification
• WLC L3 MGID
(3500-1) >show network multicast mgid summary
..
Layer3 MGID Mapping:
-------------------
Number of Layer3 MGIDs........................... 5
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Multicast Verification
• AP L2 MGID
ap1700-sw1mgig-1-0-18#sh capwap mcast mgid all
L2 MGID Information:
L2 MGID = 1 WLAN bit map (all slots) = 0x0005 VLAN ID = 0
Slot map/tx-cnt: R0:0x0005/8 R1:0x0005/11 R2:0x0000/0
L3 MGID Information
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Multicast Verification
• AP L3 MGID
L3 MGID Information
L3 MGID = 12356 WLAN bitmap = 0x0001
Slot map/tx-cnt: R0:0x0000/0 R1:0x0001/0 R2:0x0000/0
Clients per Wlan
Wlan : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Clients R0:
Clients R1: 1
Clients R2:
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
Multicast Verification
• At Switch
r3-sw1#sh ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
Group Accounted
239.255.255.240 Vlan30 00:43:28 00:02:43 192.168.30.115
239.0.85.86 Vlan5 2w1d 00:02:35 192.168.5.186
234.5.6.7 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.9 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.8 Vlan50 00:00:23 00:02:36 192.168.50.15
234.5.6.11 Vlan50 00:00:23 00:02:36 192.168.50.15
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
Bonjour
Troubleshooting
mDNS – Multicast DNS
• mDNS is typically see used for both Apple and Google/Android devices
• Enabling the mDNS service (mDNS Global Snooping checkbox) on the WLC will enable the
caching of mDNS service advertisements.
• Verify that Multicast has been enabled on the WLC. If you are using the Multicast/Multicast AP
multicast mode verify that you have full multicast routing enabled on the wired infrastructure.
• If your mDNS services and clients are on the same VLAN, then the mDNS server is not needed,
but the WLC multicast must be enabled.
• Each query or advertisement is sent to the Bonjour multicast address for delivery to all clients on
the subnet. Apple’s Bonjour protocol relies on Multicast DNS (mDNS) operating at UDP port
5353 and sends to these reserved group addresses:
➢ IPv4 Group Address - 224.0.0.251 debug mdns error enable
➢ IPv6 Group Address - FF02::FB debug mdns message enable
debug mdns detail enable
debug mdns all enable
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
mDNS – Default Services
The mDNS default services on the WLC
(wlc)> show mdns service summary
Number of Services.............................. 10
Mobility learning status ........................ Enabled
Service-Name LSS Origin No SP Service-string
-----------------------------------------------------------------
AirTunes No All 0 _raop._tcp.local.
Airplay No All 0 _airplay._tcp.local.
HP_Photosmart_Printer_1 No All 0 _universal._sub._ipp._tcp.local.
HP_Photosmart_Printer_2 No All 0 _cups._sub._ipp._tcp.local.
HomeSharing No All 0 _home-sharing._tcp.local.
Printer-IPP No All 0 _ipp._tcp.local.
Printer-IPPS No All 0 _ipps._tcp.local.
Printer-LPD No All 0 _printer._tcp.local.
Printer-SOCKET No All 0 _pdl-datastream._tcp.local.
iTuneWirelessDeviceSharing 2 No All 0 _apple-mobdev2._tcp.local.
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
mDNS – Custom Service Strings
(wlc) > show mdns service detailed GoogleCast
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
mDNS – Services Learned
WLC> show mdns service detailed GoogleCast
Service Name..................................... GoogleCast
Service String..................................... _googlecast._tcp.local.
Service Id............................................ 4
Service query status........................... Enabled
Service LSS status.............................. Enabled
Service learn origin............................. Wireless
Number of Profiles.............................. 1
Profile.................................................. default-mdns-profile
ServiceProvider MAC Address AP Radio MAC Vlan Id Type TTL Time left
-----------------------------------------------------------------------------------------------
39b124._googlecast._tcp.local. 54:60:09:B5:A2:60 00:A6:CA:F1:4B:00 279 Wireless 4500 4361
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
mDNS - Query and Response - Success
38:71:de:4e:43:8b Query Service Name: _airplay._tcp.local., RR-Type: TYPE_DOMAIN_NAME_PTR , Class: 1
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
mDNS - Query and Response - Failure
Watch out for applications that use sub-types in the mDNS requests (examples are Netflix, HBO
GO, etc…), while the WLC has support for Apple sub-types (see CSCue05421), not all sub-types
may be supported with all service strings. (see CSCvc99976)
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 159
Mobility
Troubleshooting
What is Mobility
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Mobility Group vs. Mobility Domain
• Auto Anchoring
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
Layer 2 roaming
Client
▪ Same VLAN on both WLCs Entry
Data
▪ Only one Client entry Old WLC
remains
Association
▪ Data termination point is
moved to new WLC (local)
New
WLC
Mobile Announce Entry
Mobile Handoff
Data
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
Layer 3 roaming
Client
▪ VLAN does not match Entry
Data
▪ Two Client entries Old WLC
▪ L3 data termination point Association
remains at old WLC
(Anchor)
New
▪ New WLC is L2 termination WLC
(Foreign) Mobile Announce Entry
Mobile Handoff
EthoIP
#CiscoLive DGTL-BRKEWN-3011 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
Thank you
#CiscoLive