MPLS L3 VPN. Part 2 – MP-BGP

This is my second post on MPLS VPN. The first one that described Virtual Routing and Forwarding can be found here. Now I want to move on to MP-BGP component.

Why do we need MP-BGP?

We know that VRF instances are local for each router. Two different routers that have VRF instances configured for the same VPN have no idea that these two VRF instances belong to the same VPN even though these instances names or Route Distinguishers may match. But they need to exchange routes belonging to this VPN between themselves, i.e. send and receive them and insert received routes into correct VRF instance. How can we manage this? Well, this is what MP-BGP is used for. It permits exchanging routing information from different VRF together with VPN identification information.

Route Distinguishers and Route Targets

To differentiate routes belonging to different VRF instances MP-BGP uses Route Distinguishers and Route Target Import/Export parameters. The difference between RD and RT is the following:

Route Distinguisher is appended to IPv4-prefixes belonging to different VPNs when you set up MP-BGP and it makes the prefixes unique. Two 10.0.0.0/8 prefixes belonging to different VPNs will exist as something like RD1:10.0.0.0/8 and RD2:10.0.0.0/8 so BGP process won’t mix them up within the same physical router. However, this will not have any influence on whether these routes will be installed into certain VRFs in the peer routers that they will be sent to.

This other part is manupulated by Route Target parameter. Source router decides whether it wants to export certain Route Targets and then RT is sent within BGP Update as extended BGP community. At the receiving PE router VRF configuration determines routes with which RT it wants to import. Here is a configuration example:

R4 R5
ip vrf network-a
rd 64650:10
route-target export 64650:1
route-target import 65600:1ip vrf network-b
rd 64700:10
route-target export 64700:1
route-target import 64750:1
ip vrf network-a
rd 65650:10
route-target export 65600:1
route-target import 64650:1ip vrf network-b
rd 64600:10
route-target export 64750:1
route-target import 64700:1

R4 will export routes from VRF network-a assigning them an RT of 64650:1 and R5 will install these routes into its network-a VRF as it has RT 64650:1 configured for import. R4 in turn will install routes coming from R5 with RT 65600:1 into network-a VRF. Similar exchange process will happen with network-b VRF.

MP-BGP Peering. Not to confuse with PE-CE VRF

PE router may have BGP peering with CE routers in order to exchange routes between Customer network and a VRF on PE router that is used to connect this Customer to Provider network. The terms Customer and Provider are applicable if we are talking of actually transporting some customer networks across some service provider network. MPLS VPN can be used in other circumstances where transport network and VPN networks belong to the same organization and the whole solution is used just to virtualize the networks. The concept, however, stays the same:

  • Between PE routers and MP-BGP (VPNv4) peering is used
  • Between PE and CE router classical IPv4 BGP peering is used but it is configured inside each customers (private network) VRF

BGP peering looks the following way (a lab example):

BGP-Peering
BGP Peering Diagram

Full BGP configuration examples:

R4

router bgp 64600
bgp router-id 10.4.4.4
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.5.5.5 remote-as 64600
neighbor 10.5.5.5 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 10.5.5.5 activate
neighbor 10.5.5.5 send-community extended
exit-address-family
!
address-family ipv4 vrf network-a
neighbor 10.0.47.2 remote-as 64650
neighbor 10.0.47.2 activate
exit-address-family
!
address-family ipv4 vrf network-b
neighbor 10.0.114.2 remote-as 64700
neighbor 10.0.114.2 activate
exit-address-family

R5

router bgp 64600
bgp router-id 10.5.5.5
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.4.4.4 remote-as 64600
neighbor 10.4.4.4 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 10.4.4.4 activate
neighbor 10.4.4.4 send-community extended
exit-address-family
!
address-family ipv4 vrf network-a
neighbor 10.0.58.2 remote-as 64651
neighbor 10.0.58.2 activate
exit-address-family
!
address-family ipv4 vrf network-b
neighbor 10.0.105.2 remote-as 64750
neighbor 10.0.105.2 activate
exit-address-family

Here is all neighbors status:

R4

#sh bgp vpnv4 all neighbors | inc BGP
BGP neighbor is 10.0.47.2, vrf network-a, remote AS 64650, external link
BGP version 4, remote router ID 172.16.1.1   <--- R7 (IPv4 VRF network-a)
BGP state = Established, up for 00:00:07
BGP table version 14, neighbor version 14/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.0.114.2, vrf network-b, remote AS 64700, external link
BGP version 4, remote router ID 10.0.114.2   <--- R11 (IPv4 VRF network-b)
BGP state = Established, up for 00:37:19
BGP table version 14, neighbor version 14/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.5.5.5, remote AS 64600, internal link
BGP version 4, remote router ID 10.5.5.5   <--- R5 (VPNv4)
BGP state = Established, up for 00:37:18
BGP table version 14, neighbor version 14/0
Last reset 00:37:25, due to BGP protocol initialization
BGP neighbor is 10.6.6.6, remote AS 64600, internal link
 

 

 

 

R5

#sh bgp vpnv4 unicast all neighbors | inc BGP
BGP neighbor is 10.0.58.2, vrf network-a, remote AS 64651, external link
BGP version 4, remote router ID 172.16.2.1   <--- R8 (IPv4 VRF network-a)
BGP state = Established, up for 04:00:16
BGP table version 20, neighbor version 20/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.0.105.2, vrf network-b, remote AS 64750, external link
BGP version 4, remote router ID 10.0.105.2   <--- R10 (IPv4 VRF network-b)
BGP state = Established, up for 00:38:08
BGP table version 20, neighbor version 20/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.4.4.4, remote AS 64600, internal link
BGP version 4, remote router ID 10.4.4.4   <--- R5 (VPNv4)
BGP state = Established, up for 00:31:54
BGP table version 20, neighbor version 20/0
Last reset 00:32:01, due to BGP Notification received of session 1, Administrative Reset

Here is an example of BGP VPNv4 Update message carrying both Route Distinguisher whithin NLRI and Route Target as Extended Community attribute for networks belonging to two configured VRF:

BGPUpd1

BGPUpd2

Route Distinguisher is carried together with network prefix information and Route Target – as an Extended BGP Community. AS-numbers are presented in asdot format so they look different from what is seen in the configuration.

Finally both routers have VPN routes from the other side present in correspondent VRF instances (local “customer” routes come from IPv4 session).

R4 VRF network-a

#sh ip route vrf network-a

Routing Table: network-a
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.47.0/30 is directly connected, Ethernet0/1
L 10.0.47.1/32 is directly connected, Ethernet0/1
172.16.0.0/24 is subnetted, 2 subnets
B 172.16.1.0 [20/0] via 10.0.47.2, 00:27:34
B 172.16.2.0 [200/0] via 10.5.5.5, 00:25:55

R4 VRF network-b

#sh ip route vrf network-b

Routing Table: network-b
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.114.0/30 is directly connected, Ethernet0/2
L 10.0.114.1/32 is directly connected, Ethernet0/2
B 192.168.1.0/24 [20/0] via 10.0.114.2, 00:27:36
B 192.168.2.0/24 [200/0] via 10.5.5.5, 00:25:57

R5 VRF network-a

#sh ip route vrf network-a

Routing Table: network-a
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.58.0/30 is directly connected, Ethernet0/1
L 10.0.58.1/32 is directly connected, Ethernet0/1
172.16.0.0/24 is subnetted, 2 subnets
B 172.16.1.0 [200/0] via 10.4.4.4, 00:00:06
B 172.16.2.0 [20/0] via 10.0.58.2, 00:28:30

R5 VRF network-b

#sh ip route vrf network-b

Routing Table: network-b
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.105.0/30 is directly connected, Ethernet0/2
L 10.0.105.1/32 is directly connected, Ethernet0/2
B 192.168.1.0/24 [200/0] via 10.4.4.4, 00:27:42
B 192.168.2.0/24 [20/0] via 10.0.105.2, 00:28:33

MPLS L3 VPN. Part 1 – VRF

I have found for myself long ago that reading several articles and documents explaining the same thing, often using different approaches, helps me quite a lot to understand it. Trying to brush up my knowledge in Layer 3 MPLS VPN and doing some experiments in the lab environment I have decided to write a series of short posts on how it works with my way of explaining it. Hope it will help someone who also likes to read various materials on the same topic.

The solution, as many already know, consists of the following basic components:

  • VRF
  • Multiprotocol BGP
  • MPLS (label propagation and switching)
  • Some Interior Gateway Routing Protocol

And it all looks something like this (an example shows LDP as label propagation mechanism though it is not the only one possible):

MPLSL3VPN

The idea is to be able to pass Layer 3 private networks across some transport (MPLS-enabled) network leaving routing processes within these private networks independent of each other and of the ones that take place in the backbone network (unless you actually want to interchange some routing information).

The most difficult part is usually not to understand how each of the components works by itself, but go get the whole picture. I started by asking: “Why do we need all of these components to make the solution work?” Starting with the first one the question will be:

Why do we need VRF?

Remember the goal of L3 MPLS VPN – several independent Layer-3 networks should be able to send data over one transport network. This means we are going to have traffic of two or more private networks on the same router and we need to have separate routing instances for each of these networks. This is what VRF does.

How does it work?

VRF is often called Router Virtualization and is compared to Server/Desktop Virtualization. This analogy doesn’t seem very accurate to me. When you virtualize a server with let’s say VMWare ESXi you normally get all of it’s main components – CPU, RAM, disks, network cards, virtualized and managed separately from the same components of other virtual servers. In case of VRF what gets virtualized is just routing instance. You don’t manage how many hardware resources each virtual routing instance will get, router RAM and CPU resources remain shared.

What makes each VRF is its Layer-3 interface members and its proper routing table. To me an analogy with VLANs each having its Layer-2 port members and proper MAC-address table seems more correct.

A good example of network virtualization technology that gets really close to the server/desktop virtualization would be VDC in Cisco NX-OS, not VRF.

VRF instances are local to the router

This is an important thing to keep in mind. Two switches receiving Ethernet frames from each other across dot1q trunk understand which traffic belongs to which VLAN by looking at dot1q tag. There is nothing similar happening with VRF instances. Each VRF instance is local to the router where it is configured. There is no data in the packet that could indicate which VRF it comes from.

Configuration

VRF configuration is basically resumed to

Creating VRF instances

ip vrf Network-A
 rd 64600:100
!
ip vrf Network-B
 rd 64600:100

The meaning of the Route Distinguisher parameter configured with rd command will be explained in later post.

Assigning Layer-3 interfaces

interface FastEthernet1/0
ip vrf forwarding Network-A
ip address 192.168.1.1 255.255.255.0
!
interface FastEthernet2/0
ip vrf forwarding Network-B
ip address 172.16.1.1 255.255.255.0

Configuring routing

ip route vrf Network-A 192.168.2.0 255.255.255.0 192.168.1.254
ip route vrf Network-B 172.16.2.0 255.255.255.0 172.16.1.254

I have added just static routes, though it is also possible to use dynamic routing protocols capable to work with VRF.

As a result we have separate routing tables for each VRF instance:

R1#show ip route vrf Network-A

Routing Table: Network-A
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
S     192.168.2.0/24 [1/0] via 192.168.1.254

 

R1#show ip route vrf Network-B

Routing Table: Network-B
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/24 is directly connected, FastEthernet2/0
L        172.16.1.1/32 is directly connected, FastEthernet2/0
S        172.16.2.0/24 [1/0] via 172.16.1.254

The routes that don’t belong to any of the VRF instances created, stay in Global routing instance. Here I have FastEthernet0/0 interface that is not assigned to any VRF:

R1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next
hop override
Gateway of last resort is 10.0.0.254 to network 0.0.0.010.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/24 is directly connected, FastEthernet0/0
L        10.0.0.1/32 is directly connected, FastEthernet0/0

VRF as part of MPLS L3VPN

In MPLS L3VPN solution VRF is used on the transport network border routers (often called PE routers). Let’s say we have Network A and Network B across backbone network. Each of them will have its own VRF on every PE router that has Network A, B devices (CE-routers) connected.

VRF