Troubleshooting LAN Protocols
Contents
1. and Layer 3 Connectivity Issues Define Problems Core Core Switch Switch Workstation A Access Access Workstation B Switch Distribution Switch Switch Performance latency jitter packet loss Connectivity link reachability O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Troubleshooting Methodology HSRP Active HSRP Active Even Vlans erf Odd Vlans LEED Workstation A Access Access Workstation B Switch Distribution Switch Switch Define issue between two specific stations Determine path of respective packets Begin systematic examination of path devices Troubleshooting Layer 1 Connectivity Do we have a link Traffic Are packets passing Distribution How many Switch Speed duplex Do both sides match Access Switch O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Link Comes up for 10 100 Mbs but Not for 1000Mbps Is one of the four pairs in a category 5 cable broken The Time Domain Reflectometry TDR test can be run without having to disconnect the cables to determine if there are any broken wires in them Helps network administrator to discriminate between cables that can support the upgrade to higher speed and the ones that cannot TDR support is available for copper ports at this time no support for optical as of today I TAA RW cS AA A PAPST SEE Cable Fault2. 14733424090 bytes 0 no buffer Received 224084097 broadcasts 201828280 multicast O runts O giants O throttles O input errors O CRC O fr O input packets with dribble condition de 35622 packets output 5452233 bytes 0 underruns O output errors 0 collisions 0 interface resets O babbles O0 late collision 0 deferred O lost carrier 0 no carrier O output buffer failures 0 output buffers swapped out Are Input Output Counters Incrementing 0 ignored Check for Any errors crc collisions Symptoms of Port Start Up Delay Dynamic Host Configuration Protocol DHCP address is not resolved 802 1x Client failing or delayed to get authenticated Category 5 Cable Is It a Physical Layer Issue O 2006 Cisco Systems Inc All rights reserved Presentation ID scr 12 Pos EEB ES Port Start up Delay Problem and Solution On linkup it takes up to 30 45 seconds for packets to flow Three things contribute to delay in packet forwarding on link up opanning Tree Trunk auto negotiation Channel auto negotiation BRKRST 3131 E 14513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public 25 Check Duplex Setting and Verify Topology Layer 1 Troubleshooting BRKRST 3131 14513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public O 2006 Cisco Systems Inc All rights reserved 13 Presentation ID scr 26 Troubleshooting Layer 2 Tru
3. 5 minute input rate 7890000 bits sec 4560 packets sec 5 minute output rate 7500 bits sec 10 packets sec 2006 Cisco Systems Inc All rights reserved 54 Presentation ID scr Basic Commands to Understand CPU Utilization 109 What Should Be Our Approach A local span session can be configured to capture the traffic for analysis Check log for any error messages which tell us about resource issues Make sure Spanning Tree is stable We can capture traffic going to CPU with help of TAC on Cat6500 Cat4500 BRKRST 313 O 2006 Cisco Systems Inc All rights reserved 55 Presentation ID scr CE o Storm Control Can Help to Protect CPU Configuring traffic storm control to avoid packets flood the LAN creating excessive traffic and degrading network performance Router config if storm control broadcast level levell level WS C3750 24TC L A config storm control broadcast level pps 1000 500 Router config if storm control multicast level level level WS C3750 24TC L A config if fstorm control multicast level bps 100000 1000 a Best Practices Following building codes results in solid well constructed homes and buildings Following LAN switching building code results in resilient well constructed and stable switched networks Practices for Cisco Catalyst 4500 4000 5500 5000 and 6500 6000 Series Switches Running Cisco CatOS IOS Configuration and Management http www cis
4. Bypasses listening and learning stages of STP Reduces failover time to 2 3 seconds from 30 seconds Auto populates upstream address tables dummy mcast Default in RSTP 2006 Cisco Systems Inc All rights reserved 31 Presentation ID scr BackboneFast Spanning Tree enhancement to reduce failover convergence time Targeted at indirect failures Enabled on all switches Bypasses max age Reduces failover time to 30 seconds from 50 seconds Default in RSTP 802 1s MST Overview TWO active topologies VLANs 1 50 VLANs 51 99 Instance 1 Instance 2 All VLANs mapped to one 5 of two topologies Lower BPDU counts Much less CPU utilization Very high scalability 802 1s 12 1 11 EX Reduces complexity of numerous topologies The Problem with Running a Single Instance of STP Is That Any Blocked Link Is Unable to Actively Participate in the Forwarding of Data thus It Becomes a Wasted Resource O 2006 Cisco Systems Inc All rights reserved 32 Presentation ID scr PESO ESSES Spanning Tree Issues 802 1D based Spanning Tree implementations don t converge fast 2 x Fwd Delay Max Age Traditional Spanning Tree is based on network wide timers Cisco s PortFast UplinkFast and BackboneFast help but standardization would be better IEEE work resulted in new standard Rapid Spanning Tree Protocol RSTP defined in 802 1w RSTP 802 1w Overvi
5. Interface Ios Vian Role Sts Cost Prio Nbr Type VLAN0001 Root FWD 3 128 833 P2p Router Port Mode Encapsulation Status Native vlan Gi3 1 802 1q 1 Port Vlans allowed on trunk Gi3 1 1 4094 Port Vlans allowed and active in management domain Gi3 1 1 58 60 899 902 998 1000 1001 Port Gi3 1 1 58 60 899 902 998 1000 1001 TT NR Pme A Gm ms dus SEE jJ Am I Seeing Mac Address on Correct Interface Layer 2 Bridging Ios Codes primary entry vlan mac address type learn qos ports 4 4 4 q4 4 wo d 0001 c912 7bff dynamic No Pol IOS address address keyword aging time aging time keyword count count keyword dynamic dynamic entry type interface interface keyword module display entries in DFCcard multicast multicast info for selected wildcard static static entry type vlan vlan keyword Output modifiers cr IOS Vlan Role Sts Cost Prio Nbr Type VLANOOO1 Root FWD 3 128 833 P2p 2006 Cisco Systems Inc All rights reserved Presentation ID scr Troubleshooting Layer 3 Route ARP Ios Routing entry for 162 123 74 0 24 Known via eigrp 1 distance 170 metric 130816 type external Redistributing via eigrp 1 Last update from 10 1 1 1 on Vlanl 00 01 13 ago Routing Descriptor Blocks 10 1 1 1 from 10 1 1 1 00 01 13 ago via Vlanl Route metric is 130816 traffic share count is 1 Total delay is 5010 microseconds minimum bandwi
6. Spanning Tree Protocol Troubleshooting Commands Track down source of changes TCN logs network management Protect against the changes UDLD PortFast network management O 2006 Cisco Systems Inc All rights reserved Presentation ID scr 37 Can Unidirectional Link Detection UDLD Help to Avoid Spanning Tree Loop What Is UDLD Detects one way logical connectivity Physical layer errors are detected by auto Faulty Gbic negotiation and FEFI E Ee 4 Detects faults at Layer 2 sl ia PAN TX RX 6500 1 gt sh int g2 1 GigabitEthernet2 1 is up line protocol is up FEFI Far End Fault Indication Root xmits BPDUs Neighbor doesnt receive them and thinks the root is dead now claims it s the new root Bottom switch opens up its blocked port loop in the network Network goes down troubleshooting very difficult O 2006 Cisco Systems Inc All rights reserved 38 Presentation ID scr Show UDLD BRKRST 3131 14513 04 2008 c2 isco Systems Inc All rights reserved Cisco Public 77 Spanning Tree Commands UDLD Enable Aggressive Native can have standard or aggressive configured globally and per port exceptions BRKRST 3131 14513_04_2008_c2 2008 Cisco Systems Inc All rights reserved Cisco Public 78 2006 Cisco Systems Inc All rights reserved 39 Presentation ID scr Spanning Tree Protocol STP Loop Recovery Do not power off switches pull shut redu
7. Time Domain Reflectometry TDR TDR determines cable faults Cat 5 cable has four cable pairs TDR detects faults in cable pairs such as opens or shorts TDR determines position of cable fault TDR test is invasive link will be down for the test duration TDR test shows the result for each of the four cable pairs O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Cable Fault TDR Router test cable diagnostics tdr interface GigabitEthernet3 1 Link state may be affected during TDR test TDR test started on interface Gi3 1 A TDR test can take a few seconds to run on an interface Use show cable diagnostics tdr to read the TDR results Router show cable diagnostics tdr int g3 1 TDR test last run on April 27 1 29 58 Interface Speed Pair Cable length Distance to fault Channel Pair status Gi3 1 100 1 2 N A N A Pair A Terminated 3 4 N A N A Pair B Terminated 5 6 N A 5 2 m Invalid Short 7 8 N A 5 2 m Invalid Short Digital Optical Monitoring DOM Digital Optical Monitoring DOM is an industry wide standard known as Digital Diagnostic Monitoring iere JOE Optical HE or SFF 8472 ftp ftp seagate com sfi SFF 8472 PDF intended to define a digital iinterace me access al time transceivers operating parameters such as Optical TX power Optical RX power Laser bias current Temperature Transceiver supply voltage With DOM the user has capability of performing in s
8. of DHCP traffic allows only designated DHCP ports or uplink ports trusted to relay DHCP messages Builds a DHCP binding table containing client IP address client MAC address port VLAN number Benefit Eliminates rogue devices from Client behaving as the DHCP server Rogue Server DHCP Snooping Switch config ip dhcp snooping Switch config ip dhcp snooping vlan 10 100 Switch config int f6 1 Switch config if ip dhcp snooping trust Switch config if ip dhcp snooping limit rate lt rate gt Switch show ip dhcp snooping Switch DHCP snooping is enabled DHCP snooping is configured on following VLANs 1 Insertion of option 82 is enabled Interface Trusted Rate limit pps FastEthernet2 1 yes 100 Switch show ip dhcp snooping binding MacAddress lpAddress Lease sec Type VLAN Interface 0000 0100 0201 10 0 0 1 1600 dynamic 100 Fa2 1 2006 Cisco Systems Inc All rights reserved Presentation ID scr What We Should Know Before We Start Troubleshooting Configured to rate limit the incoming DHCP packets Points to note DHCP request broadcasted to only trusted ports in that vlan DHCP responses unicasted to the client port only DHCP responses on untrusted port is dropped Option 82 enabled by default when dhcp snooping is enabled Option 82 DHCP pkt is dropped when rcvd on untrusted port When Upstream Switch Is a Relay Agent Interface Config ip dhcp relay information option tru
9. Cis soll vel Networkers n June 22 26 2008 Orlando FL The Power ot A Collaboratio THEFT CISCO bol i A CICCA 9 Ww Networkers Troubleshooting LAN Protocols BRKRST 3131 2006 Cisco Systems Inc All rights reserved Presentation ID scr Agenda Session Overview Troubleshooting Layer 1 Layer 2 and Layer 3 Connectivity Issues Spanning Tree Protocol Security Common Issues for High CPU Utilization Session Overview O 2006 Cisco Systems Inc All rights reserved Presentation ID scr e Related Sessions RST 3141 Troubleshooting Cisco Catalyst 3750 3550 and 2900 Series Switches by Michel Peters Tuesday 2 00 PM Wednesday 4 30 PM Thursday 4 30 PM RST 3142 Troubleshooting Cisco 4500 Series Switches by Wendy Hower Tuesday 4 30 PM Thursday 10 30 AM RST 3143 Troubleshooting Catalyst 6500 Series Switches by Barnaby Dianni Wednesday 2 00 PM Thursday 2 00 PM Thursday 4 30 PM Networking Concepts and Operations Be familiar with switching and routing concepts Understand the configurations on network devices Know what features are active and where Be familiar with Cisco s web sites Configuration guides Release notes Troubleshooting tips Software download page 2006 Cisco Systems Inc All rights reserved Presentation ID scr ee e E a a O EBO PO J EB Building Codes Reduce the Severity of Disasters Network Diagram C
10. Eat Yew Go Capture Anabyze Statistics Help B ote BB Bee eo Fe BW aaam wets x o piter Expression Clear Apply Source Destination Protocal Info u CBR WTR OTe Rage DLG Co perto ib Switch Port 15 ic e cepi EDP TREVO jJupLD coh Device Ib Switch Port 10 tigabitithernerd 21 CDP vTP OTP PAgP uDLD Coh Device Ib Switch Port 10 igabirtEzbernerd 21 EDP ROT PRADO ubLD Con Device Ib Switch Port db digabit tiermeta To PoIP OIT UICE menamir irunking Prarocn s Frame Li oo bytes on wire oo byter captured E IEEE 302 7 Ethernet X Dest nation OOF VTP OTe Rage uno Z01r00r crccrccczor surg Clabes Sate 00 11 801 60 241 44 Lengrh 25 Trailer OO j Leglcal Link Control Ear Saar Onaga IG Bit mndividual ar SNAP xaa ca Bir Command corro field u F rzsur OwOX rganizarion Code Cisco OwODODUr PIG OF Ora j mynamic rrunking provoca Wersdom Gwil Comins x Srarus Omit Br o5 cc cc CC IM 3 CaL 5 1 Tr a O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Trunk Solution Trunking Modes Forms Forms Trunk Trunk with No with On Negotiate BRKRST 3131 14513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public 31 Trunk Problem Trunk Fails to Form Take help of CDP to verify topology One side configured for non negotiate and other side desirable BRKRST 3131 14513 04 2008 c2 2008 Cisco Systems Inc All r
11. TP may lead to undesirable results Spanning Tree Standards and Features Spanning Tree Toolkit 802 1D 802 1s 802 1w It OT 802 1D 1998 legacy standard for bridging and Spanning Tree STP 802 1D 2004 updated bridging and STP standard includes 802 1s 802 1t and 802 1w 802 1s Multiple Spanning Tree Protocol MSTP maps multiple VLANs into the same Spanning Tree instance 802 1t MAC address reduction extended system ID moves some BPDU bits to high numbered VLANs from the priority field which constrains the possible values for bridge priority unique MAC per chassis not port 802 1w Rapid Spanning Tree Protocol RSTP improved convergence over 1998 STP by adding roles to ports and enhancing BPDU exchanges Cisco Features Per VLAN Spanning Tree PVST PVST Rapid PVST Rapid PVST UplinkFast BackboneFast BPDU Guard RootGuard LoopGuard UDLD O 2006 Cisco Systems Inc All rights reserved 29 Presentation ID scr Spanning Tree Features PortFast bypass listening learning Root Distribution phase for access port Switches UplinkFast three to five seconds F F i AI Nl convergence after link failure yi e BackboneFast cuts convergence time by Max Age for indirect failure LoopGuard prevents alternate or root port from becoming designated in absence of BPDUs RootGuard prevents external switches from becoming root BPDUGuard disable PortFast en
12. UNAUTHORIZED interface Vlan1 MaxReq 2 ip address 10 48 72 177 255 255 254 0 MultiHosts Disabled RADUIS server is behind this L2 port Port Control Auto interface gi2 1 QuietPeriod 60 Seconds switchport Re authentication Disabled switchport mode access ReAuthPeriod 3600 Seconds switchport access vlan 1 ServerTimeout 30 Seconds enable 802 1x on the interface SuppTimeout 30 Seconds interface gi2 16 TxPeriod 30 Seconds switchport switchport mode access dot1x port control auto Debugging commands end debug dot1x event debug radius DES SS A E ES IE Sl PC Is Authenticated in Correct Vlan but Have IP Address from DHCP in Guest Vlan Tx period Default is 30 sec switch expects response from client before retransmitting EAP Identity Request frame again Max reauth req Default is 2 Configuring the minimum values a switch port can be deployed into the guest VLAN in 5 seconds if our timers are very aggressive DHCP and the 802 1x processes are completely asynchronous Client 802 1x Process EAP Failure D 01 80 c2 00 00 03 Upon link up EAP Identity Request Q D 01 80 c2 00 00 03 Upon link up EAP Success D 01 80 c2 00 00 03 O 5 seconds O 2006 Cisco Systems Inc All rights reserved Presentation ID scr DHCP Snooping What it does DHCP Switch forwards only DHCP DHCP Snooping Enabled Server requests from untrusted access ports drops all other types
13. abled port if a BPDU is received BPDUFilter do not send or receive BPDUS on PortFast enabled ports Wiring B Closet Switch Also Supported with MST and Rapid PVST What Is Root Guard Root guard forces a Layer 2 LAN interface to be a designated port and if any device accessible through the interface becomes the root bridge root guard puts the interface into the root inconsistent blocked state Router config if switchport Router config if spanning tree guard root 9eSPANTREE 2 ROOTGUARDBLOCK Port 3 3 tried to become non designated in VLAN 800 Moved to root inconsistent state O 2006 Cisco Systems Inc All rights reserved 30 Presentation ID scr What Is BPDU Guard PortFast BPDU guard can prevent loops by moving PortFast configured interfaces that receive BPDUs to errdisable rather than running opanning Tree across that port This keeps ports configured with PortFast from being incorrectly connected to another switch Router config if spanning tree portfast Router config if spanning tree bpduguard enable 1w2d SPANTREE 2 BLOCK_BPDUGUARD Received BPDU on port FastEthernet3 1 with BPDU Guard enabled Disabling port 1w2d PM 4 ERR DISABLE bpduguard error detected on Fa3 1 putting Fa3 1 in err disable state UplinkFast Spanning Tree enhancement to reduce failover convergence time Used when recovery path is known and predictable Enabled on access switch
14. ances Max Number of ATM trunks addc idibus VLANS on trunk 2 3550 128 VLANs Number of nontrunking ports 3750 E 128 VLANs Number of active VLANs x 3560 128 VLANs number of trunks number of 4000 Sup or I 1 500 VLANs access ports 4500 Sup Il IV V 3 000 VLANs VTP pruning does not remove STP from trunks 6000 Sup 4000 VLANs 6500 Sup II 14 000 VLANs 6500 Sup 32 11 000 VLANs 6500 Sup 720 14 000 VLANs See Respective Platform Release Notes for More Details Spanning Tree Protocol Troubleshooting Commands IOS show proc cpu CPU utilization for five seconds 1 0 one minute 2 five minutes 2 PID Runtime ms Invoked uSecs 5Sec 1Min 5Min TTY Process 1 0 1 0 0 00 0 00 0 00 0 Chunk Manager lt some output removed 79 0 256 0 0 00 0 00 0 00 O mls msc Process 80 30508 461976 66 0 40 0 43 0 44 0 Spanning Tree 81 108 27024 3 0 00 0 00 0 00 0 Ethchnl lt some output removed gt 162 12 41 292 0 00 0 01 0 00 1 Virtual Exec IOS show spanning tree summary lt some output removed gt Name Blocking Listening Learning Forwarding STP Active VLANOOO1 T 0 0 1 2 lt some output removed gt VLAN1005 0 0 0 1 1 282 vlans al 0 0 282 283 Number of Spanning Tree Instances O 2006 Cisco Systems Inc All rights reserved 36 Presentation ID scr Spanning Tree Protocol Troubleshooting Topology Change Don t Forget PortFast BRKRST 3131 14513 04 2008 c2
15. co com warp customer 4 7 3 103 html http www cisco com warp customer 4 7 3 185 html O 2006 Cisco Systems Inc All rights reserved 56 Presentation ID scr Recommended Reading Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested COE Profession Deeg books Routing TCP IP iph erra Gaerne Eire JA catadod auxmc nation of interior ros rio prptocoia Available Onsite at the Cisco Company Store e ss M 1 Complete Your Online Session Evaluation Give us your feedback and you could win Don t forget to activate fabulous prizes Winners announced daily your virtual account for access to Receive 20 Passport points for each session all session material evaluation you complete on demand and return for our live virtual event Complete your session evaluation online now in October 2008 open a browser through our wireless network Go to the Collaboration to access our portal or visit one of the Internet Zone in World of stations throughout the Convention Center solutions or visit d ri an 4 gt LN I ec i AN IE O 2006 Cisco Systems Inc All rights reserved Presentation ID scr 57 ad nili CISCO Rex ee EE c i References http www cisco com Cata
16. dth is 1000000 Kbit Reliability 255 255 minimum MTU 1500 bytes Loading 1 255 Hops 1 IOS Protocol Address Age min Hardware Addr Type Interface Internet 10 1 1 1 4 0001 c912 7bfc ARPA Vlani Troubleshooting Useful Tools IOS Protocol ip Target IP address 10 1 1 1 Repeat count 5 1 Datagram size 100 Timeout in seconds 2 Extended commands n y Source address or interface 10 1 1 2 Type of service 0 Set DF bit in IP header no Validate reply data no Data pattern OxABCD Loose Strict Timestamp Verbose none Number of hops 9 3 Loose Strict Record Timestamp Verbose RV Sweep range of sizes n Type escape sequence to abort Sending 1 100 byte ICMP Echos to 10 1 1 1 timeout is 2 seconds Packet has IP options Total option bytes 15 padded length 16 Record route 0 0 0 0 0 0 0 0 Reply to request 0 1 ms Received packet has options Total option bytes 16 padded length 16 Record route 10 1 1 2 10 1 1 1 10 1 1 1 lt gt End of list Success rate is 100 percent 1 1 round trip min avg max 1 1 1 ms O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Path of Packet Troubleshooting Useful Tools Path of Packet Troubleshooting Summary Baseline applications Define endpoints Map expected path Know features in path Change control iei Apply methodical process cy Access Switch Workstation A 46 O 2006 Cisco Syst
17. e Network Gig 0 1 Gig 0 1 155 3 77 254 155 3 76 254 VLAN187 VLAN186 Root 1 3 5 7 9 187 1 1 Root 2 4 6 8 186 0001 c912 7800 0003 6b73 9700 DIST 01 F 1 2 3 4 5 6 7 8 9 f DIST 02 192 168 1 1 3 1 4 n 192 168 1 2 F 1 2 3 4 5 6 7 8 9 F 1 8 9 A aaas E 1 67 23 412 413 4 4 41 M2 4 8 4 4 F 1 2 3 F 1 4 5 CCZ F 1 4 5 F 1 6 7 F 1 8 9 id ET e TA B 1 7 v B 8 B 1 9 O lo ses TS me cmm e OO seems cem 0001 c967 7800 0001 c9dd 7800 0001 c932 9700 0001 c949 9700 Closet1 Closet2 Closet3 Closet4 192 168 1 11 192 168 1 12 192 168 1 13 192 168 1 14 O 2006 Cisco Systems Inc All rights reserved 28 Presentation ID scr Spanning Tree Best Practice How Can Have a Spanning Tree Loop Don t Have Spanning Tree Enabled Cisco recommends leaving STP enabled for the following reasons If there is a loop induced by mispatching bad cable and so on STP will prevent detrimental effects to the network caused by multicast and broadcast data Protection against an EtherChannel breaking down Most networks are configured with STP giving it maximum field exposure more exposure generally equates to stable code Protection against dual attached NICs misbehaving or bridging enabled on servers Bridging between wired and wireless The software for many protocols such as PAgP IGMP snooping and trunking is closely related to STP running without S
18. ems Inc All rights reserved 23 Presentation ID scr EEE EPE What Caused VLANs to Disappear from My Network What Is Virtual Trunking Protocol VTP Purpose create delete VLANs on a centralized switch server and have leaf client switches learn information Runs only on trunks Four modes Server updates clients servers stores VLAN info in NVRAM Client receive updates cannot make changes Transparent lets updates pass through Off VTP turned off What Is VTP Configuration Rev No VTP Configuration Revision Number Increments for Each VLAN Change O 2006 Cisco Systems Inc All rights reserved 24 Presentation ID scr Aha Now Know What Happened VTP Bomb Occurs when a VTP Server with a Higher Revision of the VTP Database Albeit Loaded with Potentially Incorrect Information Is Inserted into the Production VTP Domain Causing the Loss of VLAN Information on All Switches in that VTP Domain Rev X 1 VTP Commands Show VTP Status Cisco lOS Nativefshow vtp status VTP Version 12 Configuration Revision ob Maximum VLANs supported locally 1005 Number of existing VLANs 6 VTP Operating Mode Server VTP Domain Name mydomain VTP Pruning Mode Disabled VTP V2 Mode Disabled VTP Traps Generation Disabled MD5 digest OxE3 OxE9 0x3A 0x43 0x69 0x2A 0x59 Configuration last modified by 127 0 0 12 at 2 23 02 21 43 44 Local updater ID is 10 118 2 159 on interface V11 lowest nu
19. ervice transceiver monitoring and troubleshooting operations O 2006 Cisco Systems Inc All rights reserved Presentation ID scr DOM Support on Cisco Transceivers DOM capabilities is supported on selected GBIC SFP Xenpak X2 and XFP Refer to the DOM Compatibility Matrix for details The following conditions must be met for a particular transceiver type to qualify as supported Cisco engineering has successfully verified the DOM functions during the qualification process of the transceiver All the modules that Cisco has been shipping under a particular Product ID have DOM capable hardware Cisco manufacturing tests and verifies DOM support before each module is shipped to customers Sometimes not all three conditions are met and DOM commands may work on transceivers which are not DOM supported An example could be XENPAK 10GB ER Digital Read Backs Interpretation Of the five digital diagnostic read backs the most relevant ones are Optical TX and RX power as well as temperature The operating ranges of these three values is unique available on the data sheets across all modules of the same type e g all DWDM Xenpaks The supply voltage is specified in the data sheet of most transceivers Typical values are 5V for GBICs 3 3V for SFPs In 10 G transceivers there are three voltage supplies 1 8 3 3 and 5V Not always all three voltages are utilized hence this information is not called out
20. ew Takes advantage of today s topologies full duplex Root point to point links NE ER No more network wide timers ES when all switches run 802 1w Proposal Agreement Handshake mechanism 1 2 between bridges EP P t i Proposal agreement messaging I want to become designated Proposal Agresment do you agree 3 4 Can achieve subsecond gue convergence N 2006 Cisco Systems Inc All rights reserved 33 Presentation ID scr RSTP Overview Cont Incorporates mechanisms similar to UplinkFast Root BackboneFast extensions CP E a Decouples port status role Proposal Agreement i e forwarding designated O 2 No need to tune timers nna Backwards compatible BpoUS 802 1D with 802 1d PVST on a od per port basis f y T L N N Spanning Tree Protocol Troubleshooting Commands O 2006 Cisco Systems Inc All rights reserved 34 Presentation ID scr Spanning Tree Protocol Troubleshooting Commands BRKRST 3131 14513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public 69 Spanning Tree Protocol Troubleshooting Commands BRKRST 3131 4513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public 70 2006 Cisco Systems Inc All rights reserved 35 Presentation ID scr Spanning Tree Protocol Logical Ports and STP Instances Number of non ATM trunks Recommended number of VLANS on trunk Inst
21. ights reserved Cisco Public 32 O 2006 Cisco Systems Inc All rights reserved 16 Presentation ID scr Trunk Commands Show Interfaces Switchport Cisco IOS Show interfaces int switchport show interfaces trunk Router sh int g3 1 trunk Port Mode Encapsulation Status Native vlan Gi3 1 desirable 802 1q trunking 1 Port Vlans allowed on trunk Gi3 1 1 4094 Port Vlans allowed and active in management domain Gi3 1 1 58 60 899 902 998 1000 1001 Port Vlans in spanning tree forwarding state and not pruned Gi3 1 1 58 60 899 902 998 1000 1001 Channel Problems Channel Fails to Form channel is a method of grouping multiple physical links between two devices into a single logical link EtherChannel PAgP Cisco proprietary port channeling IEEE 802 3ad LACP standards based port channeling Incorrect configuration Port is err disabled 2006 Cisco Systems Inc All rights reserved Presentation ID scr DE C cc o NN Mix of Modes Allowing PAGP to Form Channel Switch B Mix of Modes Allowing LACP to Form Channel Which mix of modes allows LACP to form an Etherchannel O 2006 Cisco Systems Inc All rights reserved 18 Presentation ID scr Channel Solution Channel Modes Forms Forms Forms Channel Channel Channel with Off with Auto with Desirable Forms Channel with On Au
22. in the data sheet Note that the voltage supply read back monitors just one voltage supply this works on GBICs and SFPs which have one voltage supply but with 10G pluggables which have three separate voltages this parameter is not applicable O 2006 Cisco Systems Inc All rights reserved 10 Presentation ID scr Accessing DOM transceiver type all no monitoring This command turns on off the DOM monitoring process for all transceiver types in the system Router config transceiver type all Router config xcvr type monitoring Router config xcvr type end DOM is accessible also via CLI interface with the show interface transceiver command show interfaces transceiver high alarm high warning low warning low alarm N A not applicable Tx transmit Rx receive mA milliamperes dBm decibels milliwatts Optical Optical Temperature Voltage Current Tx Power Rx Power Port Celsius Volts mA dBm dBm show interface int transceiver detail show interfaces Te3 1 transceiver detail SKIP High Alarm High Warn Low Warn Low Alarm Temperature Threshold Threshold Threshold Threshold Port Celsius Celsius Celsius Celsius Celsius Te3 1 31 6 79 1 74 1 4 1 0 8 With 10 GE Interfaces the Voltage Threshold Threshold Threshold Threshold Value Is Usually 0 Because el Port Volts Volts Volts Volts Volts s There the Voltage Supply IS be 22 e c rcc mana ne srs nnee
23. lyst 4000 Troubleshooting TechNotes http www cisco com en US products hw switches ps663 prod tech notes list html High CPU Utilization on Cisco IOS Software Based Catalyst 4500 Switches Document ID 65591 Best Practices for Catalyst 6500 6000 Series and Catalyst 4500 4000 Series Switches Running Cisco IOS Software Document ID 24330 Catalyst 4500 System Message Guide DOM Compatibility Matrix http www cisco com en US docs interfaces modules transceiver modules compatibi lity matrix OL 8031 html http www cisco com en US docs interfaces modules transceiver modules compatibi lity matrix OL 6974 html http www cisco com en US docs interfaces modules transceiver modules compatibi lity matrix OL 6981 html Cisco Transceiver Data Sheets http www cisco com en US products hw modules ps5455 products data sheets list html O 2006 Cisco Systems Inc All rights reserved 58 Presentation ID scr
24. mbered VLAN interface found 2006 Cisco Systems Inc All rights reserved 25 Presentation ID scr VTP Problem How Can We Avoid This Reset the configuration revision using domain name Change the VTP domain of the new switch to a bogus and non existent VTP domain name and then change the VTP domain back to the original name Zero ize when Change Domain Name VTP Problem How Can Avoid This Reset the configuration revision using VTP mode Change the VTP type from server the default to transparent and then change the mode back to client or server Zero ize When change mode from server to transparent and back to server 2006 Cisco Systems Inc All rights reserved 26 Presentation ID scr Agenda Session Overview Troubleshooting Layer 1 Layer 2 and Layer 3 Connectivity Issues Troubleshooting Spanning Tree Protocol Security Common Issues for High CPU Utilization Troubleshooting Spanning Tree Protocol 2006 Cisco Systems Inc All rights reserved 27 Presentation ID scr Spanning Tree Protocol Troubleshooting Methodology Start now be proactive Divide and conquer Document Spanning Tree topology Implement Spanning Ee Mey Tree enhancement Links o E Links features Develop recovery plan to include data collection for root cause analysis EE 1 l Spanning Tree Protocol Documenting Spanning Tree Topology Corporat
25. ndant links f possible initially disable ports that should be blocking Check and physically remove the connections to the ports that should be blocking Set up remote access to your network and call TAC Spanning Tree Protocol Troubleshooting Summary Be proactive Use the diagram of the network Know where the root is Know where redundancy is Minimize the number of Equal Cost blocked ports Links Keep STP even if it is unnecessary Have modem access to key devices call TAC Gas ec O 2006 Cisco Systems Inc All rights reserved Presentation ID scr Equal Cost P Es snm 40 Agenda Session Overview Troubleshooting Layer 1 Layer 2 and Layer 3 Connectivity Issues Troubleshooting Spanning Tree Protocol Troubleshooting Security Troubleshooting High CPU Utilization Troubleshooting Security 2006 Cisco Systems Inc All rights reserved 41 Presentation ID scr Port Security What it does Limits the number of MAC addresses that are able to connect to a switch and ensures only approved MAC addresses are able to access the switch Benefit Ensures only approved users can log on to the network Valid MAC Address semi Invalid MAC Address Port Security Details Configuration options Interface FastEthernet1 1 switchport port security switchport port security maximum 3 switchport port security aging time 1 switchpo
26. nk Desirable ON Channel Desirable ON Distribution Bridge table Switch MAC address cy learned correctly remm Switch Spanning Tree Ports forwarding as expected Workstation A Trunk Problem Trunk Fails to Form A trunk is a link between two devices that carries multiple VLANs simultaneously ISL Inter Switch Link Cisco proprietary IEEE 802 1q standards based trunk encapsulation Endpoint mismatch Inconsistent DTP configuration 2006 Cisco Systems Inc All rights reserved 14 Presentation ID scr Dynamic Trunk Protocol DTP What is DTP Automates ISL 802 1Q trunk configuration operates between switches Does not operate on routers not supported on 2900XL or 3500XL DTP synchronizes the trunking mode on link ends i e native VLAN mismatch VLAN range mismatch encapsulation etc DTP state on ISL dot1Q trunking port can be set to auto on off desirable or non negotiate Runs over link layer assumes point to point link DTP destination mac address is 01 00 0C CC CC CC Port should be able to operate as an access port to fall back to access mode During negotiation do not participate in STP VLAN1 should be added to trunk in ISL DTP pkts send on VLAN1 and for access or 802 1Q on native vlan The HDLC protocol type for DTP is 0x2004 which is the SNAP format DTP Packet Capture WyseFTP dat Wireshark File
27. noin aono onnenn Not Unique Unlike in GBICs Ieg 0 00 000 000 000 000 7 andSFPs o o Current Threshold Threshold Threshold Threshold Port milliamperes mA mA mA mA Te34 99 2 130 0 130 0 20 0 10 0 Optical Transmit Power Threshold Threshold Threshold Threshold Port dBm dBm dBm dBm dBm Tesi 33 35 80 40 5 Optical Receive Power Threshold Threshold Threshold Threshold Port dBm dBm dBm dBm dBm Te3A4 28 5 6 5 7 0 24 1 24 5 2006 Cisco Systems Inc All rights reserved Presentation ID scr 11 Is Physical Interface Up Troubleshooting Layer 1 IOSk show interface GigabitEthernet 1 1 GigabitEthernetl 1 is up line protocol is up connected Hardware is Gigabit Ethernet Port address is 0009 435f 8300 bia 0009 435f 8 MTU 1500 bytes BW 1000000 Kbit DLY 10 usec reliability 255 255 txload 1 255 rxload 1 255 Encapsulation ARPA loopback not set Keepalive set 10 sec Full duplex 1000Mb s link type is auto media type is SX output flow control is off input flow control is on ARP type ARPA ARP Timeout 04 00 00 Last input 00 00 00 output never output hang never Last clearing of show interface counters never Input queue 0 2000 0 0 size max drops flushes Total output drops 0 Queuing strategy fifo Output queue 0 40 size max 5 minute input rate 89000 bits sec 141 packets sec 5 minute output rate 23000 bits sec 24 packets sec 226241448 packets input
28. options Expired TTL ACL configured with log keyword ACL failed to get programmed in hardware P routes failed to get programmed in hardware 2006 Cisco Systems Inc All rights reserved 53 Presentation ID scr NN A W O Issues Encountered with High CPU Utilization Degrade performance of network On router HSRP status may flap from active to standby Router will lose its routing neighbors May fail to access the switch via SSH or Telnet and many more E ee SST O E EE E PE EEN Basic Commands to Understand CPU Utilization Know the CPU Baseline CAT6K STATICZshow processes cpu sorted CPU utilization for five seconds 7196 7096 one minute 096 five minutes 096 PID Runtime ms Invoked uSecs 5Sec 1Min 5Min TTY Process 239 32 59 542 0 15 0 01 0 00 1 Virtual Exec 118 388712 1264243 307 0 07 0 01 0 00 0 QOS Stats Export 71 Is the average total utilization during the last 5 seconds interrupts processes 70 Is the average utilization due to interrupts during the last 5 seconds Use show proc cpu history cmd to view a more detailed history of CPU utilization history CAT6K STATIC show interface vian 1 Vian1 is up line protocol is up Hardware is EtherSVI address is 000c cf2b 9c00 bia 000c cf2b 9c00 MTU 1500 bytes BW 1000000 Kbit DLY 10 usec Input queue 0 2000 9986 890 size max drops flushes Total output drops 0 Snip
29. ork on switchports O 2006 Cisco Systems Inc All rights reserved Presentation ID scr ip dhcp snooping ip dhcp snooping vlan 2 10 ip arp inspection vlan 2 10 interface fa3 1 switchport port security switchport port security max 3 switchport port security violation restrict switchport port security aging time 2 switchport port security aging type inactivity ip arp inspection limit rate 100 ip dhcp snooping limit rate 100 no ip dhcp snooping trust ip verify source vlan dhcp snooping Interface gigabit1 1 ip dhcp snooping trust ip arp inspection trust 51 NAC Sessions SEC 2041 Deploying Cisco NAC Appliance for Diverse Access Methods SEC 3040 Troubleshooting NAC SEC 3041 Troubleshooting Cisco NAC Appliance SEC 2030 Deploying Network Based Intrusion Prevention Systems SEC 2031 Deploying Host Based Intrusion Prevention Technology SEC 3030 Troubleshooting Intrusion Detection Systems D ee ER Agenda Session Overview Troubleshooting Layer 1 Layer 2 and Layer 3 Connectivity Issues Troubleshooting Spanning Tree Protocol Security Common Issues for High CPU Utilization O 2006 Cisco Systems Inc All rights reserved 52 Presentation ID scr Troubleshooting High CPU Utilization eee E a cC Common Reasons for High CPU Utilization Packets are process switched f switch cannot forward packet in hardware because fragmentation issue Packets coming with IP
30. orporate Network Gig 0 1 Gig 0 1 155 3 77 254 155 3 76 254 HSRP Active of VLAN187 VLAN186 Odd Vlans 11 Root 1 3 5 7 9 187 11 Root 2 4 6 8 186 ES 0001 c912 7800 I 0003 6b73 970 HSRP Active for Even Vlans DIST 01 EE DO F 1 234 56 789 ie DIST 02 192 168 1 1 A S 192 168 1 2 F 1 2 3 4 5 6 7 8 9 4 4 F 1 8 9_ 4 1 4n E 1 67 L2 412 aa 4 4 F 1 2 3 lt 7 F 1 4 5 F 1 6 7 F 1 8 9 B 1 5 i 1 2 B 1 7 1 1 1 1 T cwm 1 0 rr ce DoD wem cs f oO m mh 0001 c967 7800 0001 c9dd 7800 0001 c932 9700 0001 c949 9700 Closet1 Closet2 Closet3 Closet4 192 168 1 11 192 168 1 12 192 168 1 13 192 168 1 14 O 2006 Cisco Systems Inc All rights reserved Presentation ID scr PE O gg O O I Have a Plan Don t assume anything Define the problem Understand what is working and what is not Is it intra VLAN or inter VLAN issue Perform basic troubleshooting Keep the network diagram handy Keep a protocol analyzer handy Keep modem access ready for TAC support D X M AAA AEAAA M AS Agenda Session Overview Troubleshooting Layer 1 Layer 2 and Layer 3 Connectivity Issues Troubleshooting Spanning Tree Protocol Security Common Issues for High CPU Utilization 2006 Cisco Systems Inc All rights reserved Presentation ID scr Troubleshooting Layer 1 Layer 2
31. rt port security violation restrict switchport port security aging type inactivity Default action shutdown 1w2d PM 4 ERR DISABLE psecure violation error detected on Fa3 1 putting Fa3 1 in err disable state 1w2d PORT SECURITY 2 PSECURE VIOLATION Security violation occurred caused by MAC address 0005 dccb c941 on port FastEthernet3 1 O 2006 Cisco Systems Inc All rights reserved 42 Presentation ID scr Port Security Details Understanding 802 1x How It Works Each person trying to enter the network must receive authorization based on their personal username and password Valid Username Valid Password ee000000000000000000000000000000000099 P m Yes No Invalid Username Invalid Password Client Accessing TACACS or RADIUS Switch Server O 2006 Cisco Systems Inc All rights reserved 43 Presentation ID scr Understanding 802 1x enable AAA Switch sh dotlx aaa new model Sysauthcontrol Enabled use AAA for 802 1x only optional Dotlx Protocol Version 1 aaa authentication login default none Dotlx Oper Controlled Directions aaa authentication dot1x default group radius Both set IP address of radius server Dotlx Admin Controlled Directions radius server host 10 48 66 102 Both radius server key radius server key Cisco Switch sh dotlx interface g2 16 enable 802 1x AuthSM State HELD dot1x system auth control BendSM State IDLE L3 interface for accessing RADIUS server PortStatus
32. sted DHCP Server DHCP Relay Agent e g Cat6k mum Trusted Port eal Untrusted Ports Cat4k as Edge Switch Inserts Option 82 2006 Cisco Systems Inc All rights reserved 46 Presentation ID scr Dynamic ARP Inspection Dynamic ARP Inspection Protects Against ARP Poisoning Gateway 10 1 1 1 M Uses the DHCP snooping binding table Tracks MAC to IP from DHCP transactions Rate limits ARP requests Gratuitous ARP from client ports stop 10 1 1 50 MAC B port scanning Drop BOGUS ARPs prevents ARP poisoning MIM attacks Gratuitous ARP i a 10 1 1 1 MAC_B Attacker 10 1 1 25 Victim 10 1 1 50 MAC B MAC C Dynamic ARP Inspection 2006 Cisco Systems Inc All rights reserved Presentation ID scr 47 Dynamic ARP Troubleshooting IP Source Guard Protection Against Spoofed IP Addresses IP Source Guard Protects Against Spoofed IP Gateway 10 1 1 1 s Addresses ry Uses the DHCP snooping binding table Tracks IP address to port associations Dynamically programs port ACL to drop traffic not originating from IP address assigned via DHCP Attacker 10 1 1 25 Victim 10 1 1 50 O 2006 Cisco Systems Inc All rights reserved 48 Presentation ID scr IP Source Guard mmc Res e AZ ass ER Troubleshooting Commands IPSG in IP Mode O 2006 Cisco Systems Inc All rights reserved 49 Presentation ID scr Access Control Lists What i
33. t does Allows or denies access based on the source or destination address Types of ACLs Restricts users to designated Router ACL RACL areas of the network blocking unauthorized access to all other VLAN ACL VACL applications and information Port based ACL PACL Benefit Prevents unauthorized access to servers and applications Allows designated users to access specified servers Applying a RACL PACL Da E O 2006 Cisco Systems Inc All rights reserved Presentation ID scr VLAN ACL Map VACL mac access list extended drop appletalk permit any any protocol family appletalk ip access list extended ip2 permit ip any any vlan access map vacl 100 15 action drop match mac address drop appletalk vlan access map vacl 100 20 action forward match ip address ip2 vlan filter vacl 100 vlan list 201 VACLs match all packets on the VLAN VACLs may have IP based and MAC based ACLs with implicit deny all at the end This example will permit IP and drop all AppleTalk frames on VLAN 201 Catalyst Integrated Security Features Summary Cisco IOS IP Source Guard P Dynamic ARP Inspection Port security prevents MAC flooding attacks DHCP snooping prevents client attack on the switch and server Dynamic ARP inspection adds security to ARP using DHCP snooping table P source guard adds security to IP source address using DHCP snooping table All features w
34. to Passive Desirable Active BRKRST 3131 14513 04 2008 008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public Channel Problems Channel Fails to Form Misconfiguration auto on one side and on on the other BRKRST 3131 14513 04 2008 c2 2008 Cisco Systems Inc All rights reserved Cisco Public O 2006 Cisco Systems Inc All rights reserved 19 Presentation ID scr Channel Commands Show Interfaces EtherChannel Cisco IOS Show interfaces port channel 1 269 etherchannel IOSi Age of the Port channel 00d 00h 03m 10s Logical slot port 14 1 Number of ports 2 GC 0x00010001 HotStandBy port null Passive port list Fa3 45 Fa3 46 Port state Port channel L3 Ag Ag Inuse Ports in the Port channel Index Load Port EC state 4 4 4 0 55 Fa3 45 desirable sl 1 AA Fa3 46 desirable sl Time since last port bundled 00d 00h 02m 49s Fa3 46 Troubleshooting Layer 2 EtherChannel IOS Source XOR Destination IP address Native IOS cat6ki Trying Switch Entering CONSOLE for Switch Type C C C to end this session IOS cat 6k spf Would select Gil 1 of Pol IOS cat 4k Map port for Ip 1 1 1 1 2 2 2 2 is Gil 1 Pol NOTE Software forwarded traffic will use Gil 1 Pol O 2006 Cisco Systems Inc All rights reserved Presentation ID scr rS e E CMM CC ES MM CM Am Making Sure Spanning Tree Is Forwarding Vlan on Right
Download Pdf Manuals
Related Search