Troubleshooting


Lab 6 Volume 4

by: CCIE Pilot

Ticket 1: EIGRP

The slower EIGRP link via the FR cloud been used. There is an optimal path via port-channel between SW3 and SW3 found to be down.

The fix of this issue is related to port-channel.

Per Etherchannel tunneling rules, you need to have a single separate VLAN for every pair or opposing channel links. Meaning, every VLAN are used : VLAN 100 and VLAN 101.

Make sure that each access-port is having unique vlan id towards the  port-channel interface.

interface FastEthernet0/17

switchport access vlan 100  (or 101 on the second link)

switchport mode dot1q-tunnel

l2protocol-tunnel cdp

l2protocol-tunnel point-to-point lacp

no cdp enable

spanning-tree bpdufilter enable

The effect is that EIGRP will prefer the faster link vial the etherchannel.

Show ether-channel summary

Ticket 2: Connectivity

Use bottom up approach, check Layer by layer.

Here you will discover some frame-relay map statement is misconfigured.

Easily correct the config.

For RIP running on an NBMA interface, make sure that split-horizon is disabled to encourage route propagation.

Show ip interface serial 0/0/0

Show frame map

Ticket 3:  BGP

In dealing with BGP make sure to clear out all lower layer issue.

In this case, the keepalive or essentially the LMI is turn-off effecting ckt to be brought down.

Watch also for IBGP route reflection issue. Make sure RR is enabled or used if you are not having a full mesh connection.

Show ip bgp neig

Sh run interface

Ticket 4: IPv6

This case is related to tunneling IPV6. This case uses 6to4 automatic tunneling.

Make sure the source IPv4 address is properly configured and reachable.

Check static route of 2002://16 towards the Tunnel interface.

Ping ipv6

Ticket 5: Multicast

Perform basic multicast topology analysis. PIM should be enabled on the path from R3 and R6. Check for tunnel and should run PIM also.

Watch out for RFP failures.

Static mroute command can be useful also.

Ticket 6: Core Dumps

Check reflexive access-list along the way. Passived FTP should be use under normal circumstances, else no data session will be established.

Active FTP will not bypass the packet filter.

Check correct configuration for the core dumps.

ip ftp username R6CORE

ip ftp password CISCO

exception core-file R6DUMP.txt

exception protocol ftp

exception dump 148.6.3.100

Ticket: 7: Time Synchronization

Make sure authentication key is configured properly and should be trusted.

Make sure ACL is correctly configured.

ntp authentication-key 1 md5 13263E212823 7

ntp authenticate

ntp trusted-key 1

ntp access-group peer 5

ntp master 5

ntp peer 148.6.57.7

ntp server 204.12.1.254 key 1 prefer

access-list 5 permit 127.127.7.1

access-list 5 permit 204.12.1.254

access-list 5 permit 148.6.57.7

show ntp ass

show ntp ?

Ticket 8: NAT

This case is about NAT as a load balancer.

The real servers at the back should be define as type rotary.

Secondly, the access-list specifying the traffic to the virtual server should be mirrored – it should match traffic from sources to the virtual server’s IP address.

ip nat pool POOL1 <start-ip> <end-ip> prefix-length> 24 type rotary

ip access-list ext SERVERS

permit tcp any host x.x.x.x eq www (or 8080 or 443)

Ticket 9: Server Access

For RIP make sure the distance is not set to 255.

Any underlying layer 2 filtering like vlan filter will effectively drop traffic also.

Make sure that RIP udp port is not filtered out.

Take away unnecessary servers if needed.

Show ip route rip

Show vlan filter

Ticket 10: Convergence

Make sure you don’t make unwanted configuration for dampening.

interface FastEthernet0/1

no dampening 30 1000 17956 125 restart 17956

Rack6R5(config-if)#dampening ?

<1-30>  Half-life time for the penalty

<cr>

Rack6R5(config-if)#dampening 30 ?

<1-20000>  Value to start reusing an interface

<cr>

Rack6R5(config-if)#dampening 30 1000 ?

<1-20000>  Value to start suppressing an interface

Rack6R5(config-if)#dampening 30 1000 17956 ?

<1-255>  Maximum duration to suppress an interface

Rack6R5(config-if)#dampening 30 1000 17956 125 ?

restart  Enable restart penalty

<cr>

Rack6R5(config-if)#dampening 30 1000 17956 125 restart ?

<1-20000>  Penalty applied at restart

<cr>

Rack6R5(config-if)#dampening 30 1000 17956 125 restart 17956

Lab 5 Volume 4

Ticket 1: OSPF

-Case related to dot1q tunneling between 2 indirectly connected devices.

-Make sure the dot1q tunnel port has and access vlan assignment and l2protocol-tunnel allowed.

-Do the usual OSPF routine checking.

-This case is not actually related to OSPF.

Interface f0/x

switchport access vlan 100

switchport trunk encapsulation dot1q

switchport mode dot1q-tunnel

l2protocol-tunnel cdp

show cdp neighbor.

Ticket 2: OSPF

-Optimal path issue.

-Make sure in OSPF network NBMA type, the Hub router is has higher priority with the spoke router.

-Check Virtual links location and if working properly.

no ip ospf priority 0 (at the hub of course)

show ip ospf virtual-links

show ip ospf neighbor

Ticket 3:  BGP

-A reason of BGP router not receiving or dropping route advertisement is that AS path length limitation.

-Make sure AS path length configured properly. No limit by default.

%BGP-6-ASPATH: Long AS path 100 300 100 300 54 received from 204.12.1.6: More than configured MAXAS-LIMIT

router bgp 200

bgp maxas-limit 3

show ip bgp sum

clear ip bgp *

logging buffered

Ticket 4: IPv6

-The IPv6 router advertisement neighbor discovery helps the client IPV6 host in setting up its ipv6 address and default route.

ipv6 unicast-routing

interface FastEthernet0/0

ipv6 nd ra suppress <<-watch out for this!

show ipv6 interface

Ticket 5: MPLS VPN

-MPLS VPN via interface tunnel. Make sure tunnel source and destination ip address are reachable via global routing table.

-In EIGRP VPN, do not forget the AS number of the VPN address-family configuration.

show ip route vrf VPN_A eigrp

show ip eigrp neig

Ticket 6: MPLS VPN

-OSPF Sham links

-Make sure MPLS protocol is consistent either using TDP or LDP.

-Create a new loopback ip address on both PEs.

-Do not advertise the new loopbacks into OSPF but should be advertise via BGP or other protocol on the primary path.

-Make the OSPF area towards the backdoor as a sham.

-Increase OSPF cost at backdoor link to a high value.

sh ip ospf sham-links

show ip route vrf xxx

Ticket: 7: Multicast

-Watch out for PIM Stub routing feature.

-Both should be using DENSE mode, but the other side is filtering announcement using ip pim neighbor-filter command to avoid PIM neighborship between routers.

-To test do not forget the client router to run also pim DM/SM and use ip igmp-join group command.

-PIM SM and DM can combine on a multicast network.

-Identify the first hop router and next hop routers.

-Beware of RPF failures when dealing with multicast scenario.

-ip mroute x y z

-show ip pim interface x/x detail

-show ip pim interface

-show ip pim neighbor

Multicast Killer command:

sh run | i igmp|pim|mroute|multicast

Ticket 8: QoS

-Watch out for “Mr. 768” (that 768Kbps bandwidth), it means fragmentation is necessary and mandatory.

-For VOIP the recommended fragmentation size is = 768000 / 8  * .01 sec = 960 bytes.

To check Qos Fragmentation use:

Show frame-relay fragment

Configuration:

Frame-relay fragment 960 end-to-end (MQC compatible)

Else use map-class.

Ticket 9: Secure Access

-SSH Configuration Troubleshooting.

-Remember the 7 basic steps to a successful SSH server setup.

-Make sure routing is stable and the remote ssh server is ip reachable.

1. username and password

2. hostname

3. domain-name

4. crypto key generate rsa modulus

5. ip ssh version

6. line vty, transport input ssh – enable ssh as transport input.

7. line vty, login local

debug ip ssh

show line vty

ssh –l username –v 2 x.x.x.x

Ticket 10: Security

Dynamic ACL troubleshooting

-Make sure that username for the DYNACL has the autocommand access-host enable feature.

-Check AAA configuration and check AAA authorization – like

aaa authorization exec default local

-Watch out for the direction of ACL applied and the ACL config too.

-show run | i aaa

-show access-list

-debug aaa authentication

-debug aaa authorization

10:24 PM 3/16-25/2010 HKT (Redo)

Trouble Ticket 1: RIPV 2

-Do not discount VLAN filter in trouble shooting IP connectivity.

-Use CDP, trace spanning tree for the VLAN in question.

-Logging should be on to console.

-Always verify after putting the configuration or fix.

-Use CDP

-Ping test using source.

-Test if related to any layer 2 issue. (VTP, STP, Trunking, Allowed VLAN,

Duplex/speed issue).

-Check direction of filter.

-Enable ICMP debugging.

-Enable logging buffered to flash.

Filtering can be performed by:

a. ACL

b. PBR

c. Policing

d. uRPF

e. Filters on VLAN

f. FPM – Flexible Packet Matching.

Trouble Ticket 2: BGP Prefixes

-For eBGP using loopbacks as source and destination IP address, do not forget the ebgp-multihop command to reconsider adjusting the TTL for BGP.

-For back to back frame relay using, no keepalive on each site will turn

off LMI and bring up the ckt. PVC will be on static state.

-In BGP make sure the next-hop ip address is reachable and good.

-BGP is on layer 4, means the layer below has to be up and running first.

show ip route <next hop ip>

show ip bgp regex ..

Trouble Ticket 3: OSPF

-OSPF database entries cannot be filtered out by distribute-list.

-An access-list of deny only needs to have explicit allow / permit entries at the bottom of the ACL.

-Check for route-maps used in redistribution. Make sure the prefix needed is allowed.

-Check also for ospf network types. Always check whos should be the DR and BDR, check if the state should not be all to DROTHER.

-For compatible network types of broadcast or non-broadcast, a DR must e xist on a segment.

-debug ip ospf packets

-debug ip ospf events

-debug ip ospf adj

-debug ip ospf hello

Trouble Ticket 4: IPv6

-IPv6 bgp neighbors, make sure it is activated.

-Ebgp does not change the next hop, if route entry for the p2p Ebgp link, you can use next-hop-self to change ebgp next hop.

-to verify: -ping destination using source lan interface local or a loopback local interface.

-show bgp ipv6 unicast summary

-show bgp ipv6

Trouble Ticket 5: OSPF

-OSPF – even the mtu is mismatched the OSPF neighbors can still go up if both priority for broadcast segment is set to 0. This is not a true OSPF neighborship, since it is DROTHER, they do not exchange DBD anyway.

-to fix this do one of the following options

1. changed system mtu for switch

2. ip mtu command on the interface

3. ip ospf mtu-ignore.

-A route can be on the OSPF database but cannot make it on the routing table due to filters. check for filters.

show ip ospf database

Trouble Ticket 6: BGP

-check bgp neighborship.

-dont forget to check the return path of the broken route.

-you may manually advertise via network statement, redistribution in bgp to the BB neighbor to have a return path back to the source.

-check using ping with an explicit source ip address.

-Do not forget Ebgp multihop or use TTL security hops.

show ip bgp summary

Trouble Ticket 7:  MPLS VPN

MPLS VPN troubleshooting:

-check MPLS speaking interfaces and relationships to its LDP neighbor.

show mpls interface

show mpls ldp neighbor

-check LFIB for labels generated by loopback, do this again later to verify.

show mpls forwarding-table x.x.x.x

where x.x.x.x. is the loopback ip address of the LDP neighbor.

-check local labels that every router generates for that loopback interfaces.

show mpls ldp binding local x.x.x.x NN

where NN = prefix length i.e. 24 for /24, 32 for /32 ,usually a /32.

-check mask and ip address.

show run int loopback

-check VPN routes and neighborship.

show bgp vpnv4 unicast all summary

-to check consistency of RT values.

show bgp vpnv4 unicast all x.x.x.x | i RT

show ip vrf detail | i RT

-do ping test using source ip address.

ping xxxx source yyyy.

-do debug VPNv4 updates

debug bgp vpnv4  unicast updates

clear bgp vpnv4 unica * soft out

show ip cef vrf VPN_A x.x.x.x

Trouble Ticket 8: BGP Peering

-L2 issue on the ethernet channel.

-Make sure mode are compatible.

-Then BGP in the L4 will follow.

show etherchannel summary

Trouble Ticket 9: EIGRP

-Authorised arp is enable

-added static arp entry on the router.

-alternative solution is to use DHCP for ip address assignment on the interface.

Trouble Ticket 10 – Netflow

-make sure to know the the direction of netflow.

-Netflow is enabled using MQC.

-Watch out for the access-list requirement and direction of flow.

-check running config.

-check doc cd for syntax.

11:01 AM 3/7/2010 HKT

LAB 3 VOLUME 4 Notes:

Ticket 1 – RIPv2

• Fix the issue preventing R6 from learning RIPv2 prefixes from BB1.
• Do not change the IP addressing on R6 to resolve this problem.

-Know the scope of the problem.
-Check RIP running config first. Check interfaces running RIP.
-Check interface status up/down
-Check Layer 2 parameters, do ping test.
-PPP can be stuck in LCP exchange, check any authetication issue.
-This ticket is not directly related RIP but rather related to L2 PPP issue on the Serial link running RIP.
-Enable logging console and logging buffered

debug ppp negotiation – Look for “Authentication Failure” for hints.
show ip protocols
show ip route rip
show interface x/x
show frame-relay pvc yyy
show interface virtual-access x | i PPP|packets

Ticket 2 – BGP Peering

• The BGP session between R6 and SW2 is not coming up for some
reason.
• Find what might be causing this and fix the problem applying minimal
changes.

-BGP check – Remember BGP is a layer 4
-Underlying issue maybe related to switch trunking or any layer2 issue.
-Make sure the layer 2 is intact.
-Mark all access and trunk links.
-If using DTP (no switchport noneg) make sure VTP names and password are consistent, else trunks using DTP will not come up.
-Checking of logs is very critical, make sure to familiarized log checking.
-This case is not directly related to any BGP process or layer 4 issue, but rather some layer 2 trunking issue.
-Check BGP decent running config.

show spanntree vlan xxx
show interface fx/x switchport | i Mode
show vtp status
show ip bgp summary.

Ticket 3 – Backup Link

• Users on VLAN42 report terrible performance when connecting the
servers on VLAN5.
• You suspect that the problem is that packets are taking suboptimal path across the backup ISDN link between the two sites.
• Restore optimal performance and ensure the primary path over Frame-Relay cloud is being used.

-This setup is a dual link via Serial p2p and Serial frame-relay ckt.
-FR is down. Check for FR configuration, proper DLCI assignment.
-Check EEK.
-Check OSPF neighbor
-EEK is applied via a map-class, make sure it is consisted on both ends.

debug interface xx
debug ip ospf adj
show ip protocols
show interface xx | i LMI
show frame pvc xxx | s UNUSED
show frame-relay pvc | i DELETED

Ticket 4 – Network Optimization

This ticket requires Tickets 1 and 2 resolved prior to starting.
• After a recent merge process, some networks have been joined together.
• In order to provide end-to-end connectivity, mutual redistribution has been configured between RIP and OSPF on R1, R2, R3, R4 and SW1.
• However, the straight-forward configuration approach resulted in network instabilities and sub-optimal routing.
• Fix the network configuration to ensure optimal routing, which means
using high-speed links (Ethernet and Frame-Relay) for primary data paths.
• The routers behind R5 should receive a single default route to reach the rest of the network.
• R4 may still use the Serial link to reach the networks behind R5.

-Redistribution issue.
A. layout 1

RIP-R2–OSPF—-R1—–OSPF—-R3—
|        |
|__RIP-SW2-RIP__|

Beware of layout 1 when doing mutual redistribution  on all router involved (R1, R2, R3,SW2). The ospf and RIP domain will by default will looped back.

N: At R2 make rip distance lesser than OSPF (e.g. distance 109 – router rip)

B. Redistribution troubleshooting technique.

-Shut down some interfaces to make the topology simple. Check whether you can get expected routes from the neighboring routers.
-Check for filtering in or out direction on the 1. interface, routing process.
-Alternative to bumping up AD of OSPF versus RIP, you can bump down RIP AD targeting only from a specific neighbor.

e.g.

router rip
distance 109 x.x.x.x 0.0.0.0
x.x.x.x is the neighbor.

-this makes routes from neighbor x.x.x.x to have a better distance learned from OSPF.

or it is usually like this:

router ospf 1
distance ospf external 121
! or
distance 110 x.x.x.x 0.0.0.0
! or
router ospf 1
redistribute rip metric 10000 subnets <<<<<<<makes sure the the redistributed routes to OSPF has a worse metric.

General Rule in Redistribution:

1. Native prefixes for one protocol should be reached via this protocol, can have the best metric against others.
2. External prefixes should have AD that is worse than the  AD of the native prefixes from different protocol.
3. When injecting external information into a protocol, assign it a high metric value to make is less preferred.
4. You can safely generate default route for stub domains but watch out and do filter of this default if needed.

Redistribution commandments notes:

1. ospf to eigrp -> 110 to 170 -> ok
2. eigrp to ospf->  90 to 110 -> ok
3. eigrp(external) to ospf -> 170 to 110 <<<– problem here, not ok, watch out!
4. lower AD wins

TECHNIQUES:

1. route tagging
2. route summarization
3. distance

To the following with redistributing:

1. Check first all native routes per router.
2. Check all external routes per protocol type, perrouter.
3. Think of the redistribution commandments.
4. Can make ospf acts like EIGRP on dealing with external prefixes. (distance ospf external 171)
5. Bump up or bump down certain prefixes. (external or internal).
6. The distance of external EIGRP cannot be changed on a per prefix basis.
7. ping the destination using the source IP address interface on different network
8. debug ip packet detail with an ACL (ACL is permit icmp from any to any or host)
9. Adjust default AD of IGP. RIP->109, OSPF= 121, OSPF External = 171, ospf = 89, depends on the situation and requirments.

Watchout on the following with redistributing.

1. route feedback.
2. manually tag the route with higher AD with put into the redistributed towards the lower AD.
3. lockdown pre-redistribution externals to equal to the native AD (e.g. distance 110 0.0.0.0 255.255.255.255  ACL_EXTERNAL_ALREADY
4. The rest of new routes coming is can be bump up (distance ospf external 171)
5. Item 3,4 are OSPF and EIGRP samples.

-debug ip routing
-show ip route xxx
-show ip rip data
-show ip ospf topology

Ticket 5 – Server Farm

• Two new servers have been recently connected to SW1 port 0/18 and SW2 port 0/19 respectively.
• For the lab purpose, the servers are emulated by L3 ports in SW3 and SW4 respectively.
• The following is the configuration policy for the server block:
o The new servers are assigned to the subnet 192.10.X.0/24
o To ensure security, the servers should not be able to communicate directly.
o The servers should be able to reach and communicate via R4.
• However, the configuration appears to be incomplete, as servers cannot reach to each other.
• Ensure the connectivity via R4 and verify it by using IP addresses assigned to the switchports of SW3 and SW4.

-This case is related to private vlan configurations.

show interface f0/x switchport | i private-vlan
show interface trunk
show vlan id
show ip interface f0/xx | i arp

Ticket 6 – BGP

• AS254 and AS54 cannot exchange routes across AS 100, AS 200 and AS 300.
• Applying changes to BGP configurations only, restore the end-to-end connectivity.
• Do not apply any changes to AS 300 routers to accomplish this.
• The ticked will be resolved when AS 54 and AS 254 could see each other’s routes.

-External BGP loop prevention mechanism uses AS-PATH information.
-The use local-as on the source AS will generate the AS path information.
-Use allowas-in on the far receiving end
-Make user no other L2/L3 issue exist. IGP should be stable at this time. No routing loops occured.
-Check for IBGP relationships and watch out for route-refrelector requirments.

show ip bgp regexp _
show ip bgp neig xxx advertise-routes
show ip bgp sum
show ip bgp neig | i state|neighbor|index|update-group|reflec
show ip bgp update-group

Ticket 7 – L2VPN

• You have tasked a junior system administrator to configure an L2VPN tunnel between VLAN64 and VLAN46 interfaces of R6 and R4.
• Some time after this, he reported that the configuration has been done, but there are some issues preventing it from working properly.
• Find and resolve the issues preventing the Layer 2 VPN tunnel from coming up.

-Watch out for inconsistend ckt id name. Ckt id should be the same on both ends.

show vpdn tunnel
show vpdn session
show l2tun session all
show sss circuits

Ticket 8 – L2VPN Issues

• The Frame-Relay connection between R1 and R2 has been recently replaced with L2 VPN. Soon after this, users on VLAN263 started complaining that they cannot download files from the servers on VLAN42.
• They still could use telnet and ssh to access the servers though.
• Suspecting that this may be caused by Path MTU discovery failures, you configured a workaround by clearing DF bit on all TCP packets received by R2 on its VLAN263 interface.
• However, this does not seem to help and the problem persists. Find the solution to this problem that works for all types of network traffic.

-Do ping test with oversized packet to check fragmentation issues.
-Fragments per packet is controlled by VFR – ip virtual-reassembly max-fragments x command.
-Using ip tcp adjust-mss will keep the router to modify and MSS field in TCP packet header.  MSS is 40 bytes less than the MTU. Only for TCP.
-IP virtual-reassembly, use to set max amount of fragmented packets

There are basically few ways to filter fragmented packets:
1) Using an access-group applied to an interface.
2) Using a policy-map that routes fragmented traffic to Null0.
3) Using a service-policy that drops fragments.
4) Configuring IP virtual-reassembly on interface.
5) Using IOS IPS to drop fragmented traffic.
6) Using CoPPr or Control-Plane policing to restrict packets going to the router.

show ip virtual-reassembly | i Fragments
show ip int xx | i list
show ip ips all
show ip policy

Ticket 9 – NTP

• You have recently re-loaded the configuration for R3 and R4 using the backup copies and soon after that discovered the logging timestamps went wrong.
• You know that NTP has been used for time synchronization on R3 and R4.
• R5 was supposed to be the NTP master for R3 and R4.
• You may assume that R5 is configured correctly and fix the issues in R3 and R4.

-NTP key should match on Server and client.
-Use key chain to decrypt md5 hash on same router.
-if the master NTP server is using ACLS on the peers and clients make sure the clients and peers are allowed on the ACL. Make sure NTP source is allowed.
-NTP authentications need to do “ntp authenticate” command
-NTP key should be manually trusted.

show ntp status
show ntp ass detail

Ticket 10 – Internet Access

• After the recent security policy changes users on VLAN5 started complaining they cannot reach YouTube videos anymore.
• Asking a few questions you found that uses cannot access any other WWW sites as well.
• After a short investigation you found that the uplink used to reach the YouTube website is via BB1.
• Talking to the users you found that only HTTP is being affected, FTP downloads work well.
• Applying minimal changes to the network configuration resolve this problem and restore WWW connectivity.

-Related to WWW access only. Described that FTP traffic is not affected.
-Not Path MTU issue.
-Watch out for TOS value is matched and on the route-map is set to drop it to null0 interface.
-The DF bit can be cleared using route-map. set ip df 0.

-Check MTU issue
-Check local policy-map
-Check ip ips
-Check route-maps applied to the interface, make sure it is not dropping it.
-enable http server on each router as a test to web traffic.

ip http server
ip http path flash:
copy http://x.x.x.x/test null:
sho ip policy
show ip ips all
show ip int virtual-access x | i access

Ticket 1 – WWW connectivity
-Default gateway is essential for a non routing end host.
-Remember by default IOS proxy arp feature is enable.
-Proxy arp – a neighbor router will proxy all destination if it has a route into it.
-PIng the destination using source on each end of the network.
-Enabled proxy arp.

Ticket 2 – Suboptimal Routing
-You are given 3 paralled links between R4 and R5 (1 Serial p2p, 1 Serial FR, 1 FE).
-All ckts are up/up state can reach each others p2p link.
-OSPF ADJ is not coming up. Stock in EXSTART state.
-Watch out for MTU mismatch on the interface.
-Debug ip ospf adj
-Use ip ospf mtu-ignore or change the ip mtu of the interface.
-Watch out for restrictions.

Ticket 3 – Connectivity Issue
-The presented case may not be directly related to the solution of the issue.
-Use basing traceroute and ping test to destination using an explicit source.
-Check IGP neighborship if applicable.
-Watch out for wrong ip address.
-Show ip ospf interface xx | i Inter|State|Time|Count to check for the ospf neighbor.
-Note that NBMA OSPF required unicasting its neighbor.

Ticket 4 – RIPv2
-Use debug interface to limit and set a condition only on that interface.
-Use it with deb ip rip, then it will tell something like authentication issue.
-Hidden debugging command, useful for taking clear text passwords- debug ip packet detail ACL dump.

Ticket 5 – Traceroute
-Watchout for local policy maps set to interface null0
-Make icmp catch basins to make bench marks on the network.
-Check interface ACLs.
-Debug ip icmp
-Debug ip packet detail 100 ( acl 100 permit udp any any )
-Check vlan filters (show vlan filter)
-Check Policy-map interface. (show policy-map interface).s
-Show ip inspect all
-Check ip local policy routing. (show ip local policy).

Ticket 6 – NAT
-IL Inside Local (source IP address)
-OG Outside Global (usually the final destination)
-IG Inside Global (From the outside network perspective, will reached this IP address locally. Same network with OG)
-OL Outside Local (From the source IP perspective, this the same network IP respresentative of the OG.)
-Check IP NAT translations.
-Check the beginning of the lab, maybe allowed to use static routing.
-Debug ip icmp to check ICMP status.

Ticket 7 – BGP
-BGP is in synchronization
-Make sure the route is in the IGP
-Make suer also that the source of the route is consistent. CHeck the router ID, For OSPF and BGP router ID should be the same.
-Check routing table, look for the source of the route match it agains the BGP table source enclose in ().

Ticket 8 – DHCP
-DHCP has 3 components, DHCP client, relay and server.
-Make sure you correctly point the relay to the server.
-Check for the IP connectivity issue between the DHCP server and client on both ways.
-Simulated a host, you can use any switch using its SVI interface and use ip address dchp.
-On server: debug ip dhcp server events, packets and class.
-On client: debug dhcp
-can use logging buffered.
-shut, no shut the client to simulated IP address request.
-Option 82 is a relay information option subsciber-id.
-This can be forcefully enable using “ip dhcp information option”
-Make sure that the relay agent information hex matches the output on the debug. THis is under the ip dhcp class.
-Edit the ip dhcp pool putting the “class” and put the address range for that particular class.
-show ip dhcp binding.

Ticket 9 – IPv6
-OSPFv3, make sure to have correct neighborship status link network types- p2p or broadcast or nbma.
-Note that ospf still brings up its adjacency as long as its timers matches regardless of network type.
-check also for islands of area 0.

Ticket 10 – Multicast
-Make sure tunnel intefaces have pim enabled
-Show ip pim interface
-Sheck pim interfaces, pim neighborship
-For sparse-mode only network using autorp, make sure you use the ip pim autorp listener for proper autorp operations.
-For NBMA network, go ahead enable the ip pim nbma-mode for proper operations.

IEWB-RS-VOL4.lab2.diagrams.v0.01

I was not able to get a seat for my 2nd attempt in all over ASIA (Sydney, Tokyo, Hong Kong) due to no more space available up to October 18, 2009 (Version 3.0). So I decided to go over the version 4.0 CCIE R&S by December 3, 2009 in Hong Kong.

I purchased the INE’s Volume 4.0 workbook. Below are my log on each lab tickets.  Sharing to you my thoughts.

Internetwork Expert’s R&S Lab Workbook Volume IV Logs – Lab 1

Ticket 1: VOIP Quality issue

-Think of a QOS policy configuration on the LLQ!
-Check for asymetric routing issue. Do traceroute.
-Do show frame-relay pvc xxx, check for  CIR, BC, Fragment size.
-Verify applied service policy and map-class frame-relay if any.
-Check for the queueing strategy on the interface. LLQ is “Conversation 40″, minimum bandwidth reservation is “Conversation 41″ on the  show policy-map interface command.
-Ping test with higher MTU size to check for fragmentation if working.

Ticket 2: OSPF load balancing.

-It is critical that the adjancency database to be sync, check individual database of the advertising router (show ip ospf data router adv <rid>)
-OSPF does not install routes on the routing table with different DB entries and interface types.
-Watch OSPF adjacencies can still be form even on a different network types, as long as timers are match. Here p2p and broadcast types can form adjacendy.
-Check basic connectivity.
-show ip route
-Check OSPF topology table on the LSA type.
-show ip ospf database router self-originate/advertising-router xxxx.
-debug ip bgp

Ticket 3: BGP Peering.

-BGP TTL Security HOPs. Nice TTL security feature for BGP. Sets outgoing TTL to 255 and the Incoming TTL equals to 255-x, where x is the TTL hops on the command “neighbor <iP> ttl-security hops x”.
-Without TTL security hops feature, OUT TTL is 1 by default and Incoming is 0. So it means ebgp sessions can be established from anywhere on the network.
-Check basic connectivity.
-Check for any filtering issue on tcp port 179.
-Debug ip packet detail ACL, ACL matches TCP source and destinations ports.

Ticket 4: Connectivity Issue

-Watch out for inconsistent RB for a vlan. Check spann tree guard root command. Check L2 diagram for the any spanning tree diagram (root down, down to up).
-Do basic ping test.
-Check for VACL’s applied on the switch.
-Verify CDP neighbors.
-Check interface ACL’s applied.
-show spanning tree vlan and verify proper tree distribution or path from the ROOT down.

Ticket 5. Old Backup
-check for frame relay mapping issue
-Check for EIGRP router-id should be unique throughout the entire EIGRP domain.
-RIP – Check out send and receive versions.
-Check for IN and OUT filters within the protocol by issuing the command show ip protocols.
-Can use temporary* static route to test connectivity issue.
-Basic ping test, -Verify CDP neighbors.
-Prefix not installed on the routing table – 1. check AD, 2. Distribute-list Filtering. 3. Same Router ID Filtering External Routes.

Ticket 6. BGP prefix

-check out for cluster id issue. If your router-id is taken by other IBGP neighbor, you can establish neighborship but does not exchange bgp routes. (this case the RR has same RID with the client. So the client does not receive any bgp routes from the RR due to CLUSTER-LIST loop.
-To fix this, issue a unique bgp cluster-id on the router bgp process.
-clear bgp neighborship using soft-in
-debug ip bgp xxxx updates

Ticket 7. DHCP

-Use debugging technique for DHCP to gather information.
-Use logging buffer.
-debug ip dhcp server packets
-debug ip dhcp server events
-show ip dhcp.
-check doc cd
-use all options ? command technique.


Ticket 8. Port Security Violation

-By default port security violation allows only 1 MAC address.
-If you use FHRP with port security be sure to allow at least 2 MAC address on the switch port.
-Else use standby use-bia on the Router port for it not to use any other MAC address.
-Be sure to shut/no shut switch port to re-enable interface with error-disable state.
-Show port-security interface f0/1
-logging console debugging.

Ticket 9. IPV6

-RIPng uses the link local address as next hop interface.
-Make sure to map the link local address on the frame map statement.
-Check IPV6 routing table
-Ping test.

Ticket 10. Multicast

-ping testing usually is automatically source on the loopback interface (sparse-dense), so make sure the loopback interface has ip pim mode on it.
-mtrace back from last hop router to the source.
-check for RP configs
-use process switch for troubleshooting and debug ip mpacket
-check for any RFP issue, use static mroute if needed only.
-mtrace from destination back to source.
-check rpf failure for the RP and source separately!
-check mroute table.
-show ip igmp groups
-show ip igmp rp mappings

I have a customer that is complaining a lot of CRC and input errors on the interface.
This has been diagnosed with the help of Cisco's Output Interpreter.

R1#sh int s0/0/0
Serial0/0/0 is up, line protocol is up
 Hardware is GT96K Serial
 Description: R1
 MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
 reliability 255/255, txload 39/255, rxload 38/255
 Encapsulation FRAME-RELAY IETF, loopback not set
 Keepalive set (10 sec)
 Carrier delay is 15 sec
 LMI enq sent  1146, LMI stat recvd 1146, LMI upd recvd 0, DTE LMI up
 LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
 LMI DLCI 1023  LMI type is CISCO  frame relay DTE
 FR SVC disabled, LAPF state down
 Broadcast queue 0/64, broadcasts sent/dropped 603/0, interface broadcasts 412
 Last input 00:00:00, output 00:00:00, output hang never
 Last clearing of "show interface" counters 03:10:56
 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 751448
 Queueing strategy: weighted fair
 Output queue: 63/1000/64/751448 (size/max total/threshold/drops)
 Conversations  1/3/256 (active/max active/max total)
 Reserved Conversations 0/0 (allocated/max allocated)
 Available Bandwidth 1158 kilobits/sec
 5 minute input rate 235000 bits/sec, 22 packets/sec
 5 minute output rate 237000 bits/sec, 21 packets/sec
 34006 packets input, 39923277 bytes, 0 no buffer
 Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
 4 input errors, 4 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
 38081 packets output, 42136153 bytes, 0 underruns
 0 output errors, 0 collisions, 0 interface resets
 0 output buffer failures, 0 output buffers swapped out
 0 carrier transitions
 DCD=up  DSR=up  DTR=up  RTS=up  CTS=up
R1#

If you paste this on the CISCO OUTPUT Interpreter you would get the following:

SHOW INTERFACE SERIAL NOTIFICATIONS (if any)

Interface Serial0/0/0 (up/up)
  WARNING: The counters for this interface have not been cleared for 3 hours 10
  minutes 56 seconds.
  TRY THIS: Use the 'clear counters Serial0/0/0' command to ensure current information
  is being displayed. This will assist when troubleshooting serial interface issues.

  WARNING: This interface has a high number of output drops.
  The input rate to this interface has exceeded the bandwidth available on the
  serial link.
  TRY THIS:
  1. Minimize periodic broadcast traffic like routing and Service Advertising
     Protocol (SAP) updates (if applicable) by using access lists or by other
     means.
  2. Turn off fast switching for heavily used protocols. For example, turn off
     IP fast switching by using the 'no ip route-cache' interface configuration
     command.
  3. Implement priority queuing on slower serial links.
  4. Submit the output from 'show buffers' to Output Interpreter to determine
     if buffers need to be tuned.
  REFERENCE: For more information see: Troubleshooting Output Drops

  WARNING: This interface has received a high number (0.01176% of input packets)
  of packets with incorrect CRCs (corrupted data).
  Problems that may cause this symptom include:
  a. Noisy serial line
  b. Serial cable is too long or cable from the CSU/DSU to the router is not
     shielded
  c. SCTE mode is not enabled on the DSU
  d. The CSU line clock is incorrectly configured
  e. A Ones density problem on the link (incorrect framing or coding
     specification), exists
  f. Verify the queuing strategies are the same on both ends of the link.
  TRY THIS:
  1. Ensure that the line is clean enough for transmission requirements. Shield
     the cable if necessary.
  2. Make sure the cable is within the recommended length (no more than 50 feet
     [15.24 meters], or 25 feet [7.62 meters] for the link).
  3. Ensure that all devices are properly configured for a common line clock.
     Set serial clock transmit external (SCTE) on the local and remote DSU. If
     you are attempting serial connections at speeds greater than 64 kbps with
     a CSU/DSU that does not support (SCTE), you might have to invert the
     transmit clock on the router. Inverting the transmit clock compensates
     for phase-shifts between the data and clock signals.
  4. Make certain that the local and remote CSU/DSU are configured for the
     same framing and coding scheme as that used by the leased-line or other
     carrier service (for example, ESF/B8ZS).
  5. Contact your leased-line or other carrier service and have them perform
     integrity tests on the line.

REFERENCE: For more information on Serial Lines, see:
  Troubleshooting Serial Line Problems
  Configuring Serial Interfaces
  Troubleshooting Serial Lines
  Loopback Tests for T1/56K Lines

REFERENCE: For more information on Frame-Relay, see:
  Frame Relay
  Configuring Frame Relay
  Configuring and Troubleshooting Frame Relay
  Configuring and Troubleshooting Frame Relay Broadcase Queue
  Troubleshooting Frame Relay Networks
SHOW INTERFACE SERIAL NOTIFICATIONS (if any) Interface Serial0/0/0 (up/up) WARNING: The counters for this interface have not been cleared for 3 hours 10 minutes 56 seconds. TRY THIS: Use the ‘clear counters Serial0/0/0′ command to ensure current information is being displayed. This will assist when troubleshooting serial interface issues. WARNING: This interface has a high number of output drops. The input rate to this interface has exceeded the bandwidth available on the serial link. TRY THIS: 1. Minimize periodic broadcast traffic like routing and Service Advertising Protocol (SAP) updates (if applicable) by using access lists or by other means. 2. Turn off fast switching for heavily used protocols. For example, turn off IP fast switching by using the ‘no ip route-cache’ interface configuration command. 3. Implement priority queuing on slower serial links. 4. Submit the output from ‘show buffers’ to Output Interpreter to determine if buffers need to be tuned. REFERENCE: For more information see: Troubleshooting Output Drops WARNING: This interface has received a high number (0.01176% of input packets) of packets with incorrect CRCs (corrupted data). Problems that may cause this symptom include: a. Noisy serial line b. Serial cable is too long or cable from the CSU/DSU to the router is not shielded c. SCTE mode is not enabled on the DSU d. The CSU line clock is incorrectly configured e. A Ones density problem on the link (incorrect framing or coding specification), exists f. Verify the queuing strategies are the same on both ends of the link. TRY THIS: 1. Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary. 2. Make sure the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for the link). 3. Ensure that all devices are properly configured for a common line clock. Set serial clock transmit external (SCTE) on the local and remote DSU. If you are attempting serial connections at speeds greater than 64 kbps with a CSU/DSU that does not support (SCTE), you might have to invert the transmit clock on the router. Inverting the transmit clock compensates for phase-shifts between the data and clock signals. 4. Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS). 5. Contact your leased-line or other carrier service and have them perform integrity tests on the line. REFERENCE: For more information on Serial Lines, see: Troubleshooting Serial Line Problems Configuring Serial Interfaces Troubleshooting Serial Lines Loopback Tests for T1/56K Lines REFERENCE: For more information on Frame-Relay, see: Frame Relay Configuring Frame Relay Configuring and Troubleshooting Frame Relay Configuring and Troubleshooting Frame Relay Broadcase Queue Troubleshooting Frame Relay Networks