Some Useful Commands to Troubleshoot IP Routing

To see some general routing information

#show ip route summary
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Replicates  Overhead    Memory (bytes)
connected       0           13          0           1408        3744
static          0           0           0           0           0
application     0           0           0           0           0
ospf 100        0           6           0           768         1752
  Intra-area: 6 Inter-area: 0 External-1: 0 External-2: 0
  NSSA External-1: 0 NSSA External-2: 0
bgp 14234       103         1006        0           106464      319392
  External: 629 Internal: 455 Local: 25
internal        63                                              111864
Total           166         1025        0           108640      436752

Shows how much memory is being consumed by each routing process.

To see information on OSPF Area Border Routers and Autonomous System Border Routers

#show ip ospf border-routers

            OSPF Router with ID (10.0.0.2) (Process ID 100)

                Base Topology (MTID 0)

Internal Router Routing Table
Codes: i - Intra-area route, I - Inter-area route

i 10.0.0.3 [10] via 10.0.23.2, Ethernet0/1, ABR/ASBR, Area 0, SPF 9
i 10.0.0.1 [10] via 10.0.12.1, Ethernet0/0, ASBR, Area 1, SPF 5
I 10.0.0.4 [20] via 10.0.23.2, Ethernet0/1, ASBR, Area 0, SPF 92

Basically shows how your router is intended to reach each of the ABRs and ASBRs it knows about.

To see which routing information is originated by your  OSPF-enabled router

#show ip ospf database self-originate

OSPF Router with ID (10.0.0.2) (Process ID 100)

Router Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.2        10.0.0.2        156         0x80000006 0x005D69 1

Summary Net Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.12.0       10.0.0.2        111         0x80000001 0x006EA5

Summary ASB Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.1        10.0.0.2        111         0x80000001 0x00EC2E

Router Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.2        10.0.0.2        156         0x80000003 0x00746B 1

Summary Net Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.23.0       10.0.0.2        111         0x80000001 0x00F414
10.0.34.0       10.0.0.2        111         0x80000001 0x00DF14

Summary ASB Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.3        10.0.0.2        111         0x80000001 0x00D840
10.0.0.4        10.0.0.2        111         0x80000001 0x0033DA

The output above is taken from Router that is ABR, so you can see Summary (Type 3) LSAs that this router generates and sends to Area 1 to tell about the routes belonging to Area 0 and vice versa. Also you can see Summary ASB LSAs that it sends to inform routers in Area 0 about an ASBR it sees in Area 1 and routers in Area 1 about ASBRs in Area 1.

MPLS L3 VPN. Part 1 – VRF

I have found for myself long ago that reading several articles and documents explaining the same thing, often using different approaches, helps me quite a lot to understand it. Trying to brush up my knowledge in Layer 3 MPLS VPN and doing some experiments in the lab environment I have decided to write a series of short posts on how it works with my way of explaining it. Hope it will help someone who also likes to read various materials on the same topic.

The solution, as many already know, consists of the following basic components:

  • VRF
  • Multiprotocol BGP
  • MPLS (label propagation and switching)
  • Some Interior Gateway Routing Protocol

And it all looks something like this (an example shows LDP as label propagation mechanism though it is not the only one possible):

MPLSL3VPN

The idea is to be able to pass Layer 3 private networks across some transport (MPLS-enabled) network leaving routing processes within these private networks independent of each other and of the ones that take place in the backbone network (unless you actually want to interchange some routing information).

The most difficult part is usually not to understand how each of the components works by itself, but go get the whole picture. I started by asking: “Why do we need all of these components to make the solution work?” Starting with the first one the question will be:

Why do we need VRF?

Remember the goal of L3 MPLS VPN – several independent Layer-3 networks should be able to send data over one transport network. This means we are going to have traffic of two or more private networks on the same router and we need to have separate routing instances for each of these networks. This is what VRF does.

How does it work?

VRF is often called Router Virtualization and is compared to Server/Desktop Virtualization. This analogy doesn’t seem very accurate to me. When you virtualize a server with let’s say VMWare ESXi you normally get all of it’s main components – CPU, RAM, disks, network cards, virtualized and managed separately from the same components of other virtual servers. In case of VRF what gets virtualized is just routing instance. You don’t manage how many hardware resources each virtual routing instance will get, router RAM and CPU resources remain shared.

What makes each VRF is its Layer-3 interface members and its proper routing table. To me an analogy with VLANs each having its Layer-2 port members and proper MAC-address table seems more correct.

A good example of network virtualization technology that gets really close to the server/desktop virtualization would be VDC in Cisco NX-OS, not VRF.

VRF instances are local to the router

This is an important thing to keep in mind. Two switches receiving Ethernet frames from each other across dot1q trunk understand which traffic belongs to which VLAN by looking at dot1q tag. There is nothing similar happening with VRF instances. Each VRF instance is local to the router where it is configured. There is no data in the packet that could indicate which VRF it comes from.

Configuration

VRF configuration is basically resumed to

Creating VRF instances

ip vrf Network-A
 rd 64600:100
!
ip vrf Network-B
 rd 64600:100

The meaning of the Route Distinguisher parameter configured with rd command will be explained in later post.

Assigning Layer-3 interfaces

interface FastEthernet1/0
ip vrf forwarding Network-A
ip address 192.168.1.1 255.255.255.0
!
interface FastEthernet2/0
ip vrf forwarding Network-B
ip address 172.16.1.1 255.255.255.0

Configuring routing

ip route vrf Network-A 192.168.2.0 255.255.255.0 192.168.1.254
ip route vrf Network-B 172.16.2.0 255.255.255.0 172.16.1.254

I have added just static routes, though it is also possible to use dynamic routing protocols capable to work with VRF.

As a result we have separate routing tables for each VRF instance:

R1#show ip route vrf Network-A

Routing Table: Network-A
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
S     192.168.2.0/24 [1/0] via 192.168.1.254

 

R1#show ip route vrf Network-B

Routing Table: Network-B
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/24 is directly connected, FastEthernet2/0
L        172.16.1.1/32 is directly connected, FastEthernet2/0
S        172.16.2.0/24 [1/0] via 172.16.1.254

The routes that don’t belong to any of the VRF instances created, stay in Global routing instance. Here I have FastEthernet0/0 interface that is not assigned to any VRF:

R1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next
hop override
Gateway of last resort is 10.0.0.254 to network 0.0.0.010.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/24 is directly connected, FastEthernet0/0
L        10.0.0.1/32 is directly connected, FastEthernet0/0

VRF as part of MPLS L3VPN

In MPLS L3VPN solution VRF is used on the transport network border routers (often called PE routers). Let’s say we have Network A and Network B across backbone network. Each of them will have its own VRF on every PE router that has Network A, B devices (CE-routers) connected.

VRF

 

Bidirectional Forwarding Detection Protocol

Bidirectional Forwarding Detection Protocol is a relatively new tool that allows to significantly lower dynamic routing protocol convergence times. Previously faster convergence in the LAN was often achieved by tuning protocol hello/keepalive timers. A good example is Fast Hello setting for OSPF. Lowering routing protocol timers, however, can lead to higher CPU resource utilization and somewhat unexpected behavior of the network. For example in a network where you have OSPF, BGP and LDP working simultaneously with two latter depending on the former, and you try to improve convergence times by tweaking the timers for all three protocols it is difficult to predict how the whole thing reacts on some link flapping in terms of both convergence time and hardware resource utilization during convergence period. Again, I am talking here about the complicated situations, such as when you lose communications over the link without interfaces going down, maybe just in one direction. Situations when the Layer 1 and Layer 2 go down normally trigger some protocol events and convergence occurs quite quickly.

BFD offers lightweight mechanism for detecting link communications failures and notifying routing protocol making them react quickly. Convergence times with BFD are lower than for routing protocols configuring with shorter timers (se my example for OSPF below). Of course, as detection times get shorter the sensitivity to link flapping grows which may be undesirable. Also BFD has dampening mechanism that allows to introduce exponential delay in communication failure detection mechanism.

For now BFD can work with the following protocols: 

  • Static Routing
  • BGP
  • EIGRP
  • ISIS
  • OSPF
  • HSRP
  • LDP
  • ATM Pseudowires

OSPF Example

Here is a simple topology I used to test BFD operations:

BFDTopology

One important thing to understand with BFD is that it works only in conjunction with the protocol it is supposed to notify of failures. If you just configure it on two adjacent interfaces like this:

R1 R2
interface Ethernet0/0
  bfd interval 50 min_rx 50 multiplier 5
interface Ethernet0/0
  bfd interval 50 min_rx 50 multiplier 5

It won’t do anything. No neighborship is going to be established a this stage, no packets will be sent. What you need is enable it for the routing protocol, in our case OSPF:

R1 R2
router ospf 100
bfd all-interfaces
router ospf 100
bfd all-interfaces

And here it is:

R1#show bfd neighbors

IPv4 Sessions
NeighAddr LD/RD RH/RS State Int
10.0.0.2 1/1 Up Up Et0/0

BFD neighborship is established:

Now, breaking the communication between OSPF neighbors (with debug bfd events enabled):

*Dec 21 17:14:32.811: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:ip access-group denyany in
*Dec 21 17:14:33.078: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD session ld:1 handle:1,is going Down Reason: ECHO FAILURE
*Dec 21 17:14:33.078: BFD-DEBUG Event: V1 FSM ld:1 handle:1 event:ECHO FAILURE state:UP (0)
*Dec 21 17:14:33.078: BFD-DEBUG EVENT: bfd_session_destroyed, proc:OSPF, handle:1 act
*Dec 21 17:14:33.078: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: bfd_session_destroyed, ld:1 neigh proc:OSPF, handle:1 act
*Dec 21 17:14:33.078: %OSPF-5-ADJCHG: Process 100, Nbr 10.0.0.2 on Ethernet0/0 from FULL to DOWN, Neighbor Down: BFD node down
*Dec 21 17:14:33.078: BFD-DEBUG Event: notify client(OSPF) IP:10.0.0.2, ld:1, handle:1, event:DOWN, cp independent failure (0)
*Dec 21 17:14:33.078: BFD-DEBUG Event: notify client(CEF) IP:10.0.0.2, ld:1, handle:1, event:DOWN, cp independent failure (0)

BFD reports communication failure in 267 ms and signals OSPF to shut down the neighborship which OSPF immediately does. This gives us convergence time a lot shorter than OSPF Fast Hello.

In real life you should, of course, check that you have enough hardware resources to run BFD with these parameters as well as understand whether you need such short convergence time (because of the possibility of false-positives).

What’s inside

Looking at how BFD packet capture you may expect to see some sort of keepalive exchange between neighbor IP addresses which in our case are 10.0.0.1 and 10.0.0.2. Here is what you actually see:

BFD Packet Capture

Both R1 and R2 are sending UDP messages to their own interface IP addresses. This is another important thing about BFD: it works on Layer 2. Here are these packets details:

BFD2Capt

R1 sends BFD control message to R2 MAC, R2 swaps source and destination MAC and sends it back out the interface it was received. R1 gets his own message back and sees that communication across the link is working. The same process happens in the opposite direction. So R2 doesn’t process BFD data coming from R1 and vice versa. Apart from this routers interchange control messages with link state information. These messages are sent on Layer 3:

BFDDiag

Here is what the link communication failure detection process looks like:

BFDFailure

NTP Synchronization Check with EEM Applet

A common NTP implementation in the enterprise network implies one or two border routers synchronizing with external NTP servers and peering between them. Other NTP-enabled devices and applications get time from these routers internal or loopback interface addresses which belong to private address space. I usually configure external NTP servers from the regional ntp.org pool:

ntp server 0.south-america.pool.ntp.org
ntp server 2.south-america.pool.ntp.org
ntp server 1.south-america.pool.ntp.org
ntp server 3.south-america.pool.ntp.org

Border router resolves the names and tries to syncronize to one of them. The experience has shown that these NTP servers are quite unstable (at least in the region where I currenty work). Both border routers can easily be left without a single server to synchronize, so the whole network stays without reliable time source. In order to be able to detect such situations I wrote a simple EEM Applet:

event manager applet ntp-sync-check
event tag 1 snmp oid 1.3.6.1.2.1.197.1.2.1 get-type next entry-op ne entry-val "6" entry-type value poll-interval 86400
event tag 5 none
trigger
correlate event 1 or event 5
action 0.10 info type routername
action 1.00 cli command "enable"
action 1.10 cli command "show ntp asso"
action 1.20 set ntpa "$_cli_result"
action 2.00 cli command "show ntp status"
action 2.20 set ntpst "$_cli_result"
action 3.00 syslog priority critical msg "NTP sync failed" facility "NTP"
action 3.10 mail server "exchange.ourdomain.com" to "alarmsandnotifications@ourdomain.com" from "$_info_routername@ourdomain.com" cc "sysadmin@ourdomain.com" subject "** NTP Sync Failure **" body "$ntpst

What it does is just polling SNMP OID that returns the router NTP synchronization status and if the synchronization is found to be failed sends a syslog message and e-mail notification containing show ntp associations and show ntp status commands outputs.

The values for this OID being polled are:

1 : notRunning
2 : notSynchronized
3 : noneConfigured
4 : syncToLocal
5 : syncToRefclock
6 : syncToRemoteServer
99 : unknown

The only value that satisfies us is 6 – NTP Synchronized to Remote Server. So if the OID value is not equal to 6 the applet gets launched (event 1 gets triggered).

A recommended practice for all EEM scripts is have event none added besides the main event and to configure the applet to run on any of the two events occurrence. In the example above it is added with tag 5. This allows you to test the applet by launching it manually without waiting for o provoking the main event (NTP synchronization loss in my case). An applet can be tested by executing the following command:

event manager run ntp-sync-check