MPLS L3 VPN. Part 2 – MP-BGP

This is my second post on MPLS VPN. The first one that described Virtual Routing and Forwarding can be found here. Now I want to move on to MP-BGP component.

Why do we need MP-BGP?

We know that VRF instances are local for each router. Two different routers that have VRF instances configured for the same VPN have no idea that these two VRF instances belong to the same VPN even though these instances names or Route Distinguishers may match. But they need to exchange routes belonging to this VPN between themselves, i.e. send and receive them and insert received routes into correct VRF instance. How can we manage this? Well, this is what MP-BGP is used for. It permits exchanging routing information from different VRF together with VPN identification information.

Route Distinguishers and Route Targets

To differentiate routes belonging to different VRF instances MP-BGP uses Route Distinguishers and Route Target Import/Export parameters. The difference between RD and RT is the following:

Route Distinguisher is appended to IPv4-prefixes belonging to different VPNs when you set up MP-BGP and it makes the prefixes unique. Two 10.0.0.0/8 prefixes belonging to different VPNs will exist as something like RD1:10.0.0.0/8 and RD2:10.0.0.0/8 so BGP process won’t mix them up within the same physical router. However, this will not have any influence on whether these routes will be installed into certain VRFs in the peer routers that they will be sent to.

This other part is manupulated by Route Target parameter. Source router decides whether it wants to export certain Route Targets and then RT is sent within BGP Update as extended BGP community. At the receiving PE router VRF configuration determines routes with which RT it wants to import. Here is a configuration example:

R4 R5
ip vrf network-a
rd 64650:10
route-target export 64650:1
route-target import 65600:1ip vrf network-b
rd 64700:10
route-target export 64700:1
route-target import 64750:1
ip vrf network-a
rd 65650:10
route-target export 65600:1
route-target import 64650:1ip vrf network-b
rd 64600:10
route-target export 64750:1
route-target import 64700:1

R4 will export routes from VRF network-a assigning them an RT of 64650:1 and R5 will install these routes into its network-a VRF as it has RT 64650:1 configured for import. R4 in turn will install routes coming from R5 with RT 65600:1 into network-a VRF. Similar exchange process will happen with network-b VRF.

MP-BGP Peering. Not to confuse with PE-CE VRF

PE router may have BGP peering with CE routers in order to exchange routes between Customer network and a VRF on PE router that is used to connect this Customer to Provider network. The terms Customer and Provider are applicable if we are talking of actually transporting some customer networks across some service provider network. MPLS VPN can be used in other circumstances where transport network and VPN networks belong to the same organization and the whole solution is used just to virtualize the networks. The concept, however, stays the same:

  • Between PE routers and MP-BGP (VPNv4) peering is used
  • Between PE and CE router classical IPv4 BGP peering is used but it is configured inside each customers (private network) VRF

BGP peering looks the following way (a lab example):

BGP-Peering
BGP Peering Diagram

Full BGP configuration examples:

R4

router bgp 64600
bgp router-id 10.4.4.4
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.5.5.5 remote-as 64600
neighbor 10.5.5.5 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 10.5.5.5 activate
neighbor 10.5.5.5 send-community extended
exit-address-family
!
address-family ipv4 vrf network-a
neighbor 10.0.47.2 remote-as 64650
neighbor 10.0.47.2 activate
exit-address-family
!
address-family ipv4 vrf network-b
neighbor 10.0.114.2 remote-as 64700
neighbor 10.0.114.2 activate
exit-address-family

R5

router bgp 64600
bgp router-id 10.5.5.5
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.4.4.4 remote-as 64600
neighbor 10.4.4.4 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 10.4.4.4 activate
neighbor 10.4.4.4 send-community extended
exit-address-family
!
address-family ipv4 vrf network-a
neighbor 10.0.58.2 remote-as 64651
neighbor 10.0.58.2 activate
exit-address-family
!
address-family ipv4 vrf network-b
neighbor 10.0.105.2 remote-as 64750
neighbor 10.0.105.2 activate
exit-address-family

Here is all neighbors status:

R4

#sh bgp vpnv4 all neighbors | inc BGP
BGP neighbor is 10.0.47.2, vrf network-a, remote AS 64650, external link
BGP version 4, remote router ID 172.16.1.1   <--- R7 (IPv4 VRF network-a)
BGP state = Established, up for 00:00:07
BGP table version 14, neighbor version 14/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.0.114.2, vrf network-b, remote AS 64700, external link
BGP version 4, remote router ID 10.0.114.2   <--- R11 (IPv4 VRF network-b)
BGP state = Established, up for 00:37:19
BGP table version 14, neighbor version 14/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.5.5.5, remote AS 64600, internal link
BGP version 4, remote router ID 10.5.5.5   <--- R5 (VPNv4)
BGP state = Established, up for 00:37:18
BGP table version 14, neighbor version 14/0
Last reset 00:37:25, due to BGP protocol initialization
BGP neighbor is 10.6.6.6, remote AS 64600, internal link
 

 

 

 

R5

#sh bgp vpnv4 unicast all neighbors | inc BGP
BGP neighbor is 10.0.58.2, vrf network-a, remote AS 64651, external link
BGP version 4, remote router ID 172.16.2.1   <--- R8 (IPv4 VRF network-a)
BGP state = Established, up for 04:00:16
BGP table version 20, neighbor version 20/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.0.105.2, vrf network-b, remote AS 64750, external link
BGP version 4, remote router ID 10.0.105.2   <--- R10 (IPv4 VRF network-b)
BGP state = Established, up for 00:38:08
BGP table version 20, neighbor version 20/0
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
BGP neighbor is 10.4.4.4, remote AS 64600, internal link
BGP version 4, remote router ID 10.4.4.4   <--- R5 (VPNv4)
BGP state = Established, up for 00:31:54
BGP table version 20, neighbor version 20/0
Last reset 00:32:01, due to BGP Notification received of session 1, Administrative Reset

Here is an example of BGP VPNv4 Update message carrying both Route Distinguisher whithin NLRI and Route Target as Extended Community attribute for networks belonging to two configured VRF:

BGPUpd1

BGPUpd2

Route Distinguisher is carried together with network prefix information and Route Target – as an Extended BGP Community. AS-numbers are presented in asdot format so they look different from what is seen in the configuration.

Finally both routers have VPN routes from the other side present in correspondent VRF instances (local “customer” routes come from IPv4 session).

R4 VRF network-a

#sh ip route vrf network-a

Routing Table: network-a
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.47.0/30 is directly connected, Ethernet0/1
L 10.0.47.1/32 is directly connected, Ethernet0/1
172.16.0.0/24 is subnetted, 2 subnets
B 172.16.1.0 [20/0] via 10.0.47.2, 00:27:34
B 172.16.2.0 [200/0] via 10.5.5.5, 00:25:55

R4 VRF network-b

#sh ip route vrf network-b

Routing Table: network-b
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.114.0/30 is directly connected, Ethernet0/2
L 10.0.114.1/32 is directly connected, Ethernet0/2
B 192.168.1.0/24 [20/0] via 10.0.114.2, 00:27:36
B 192.168.2.0/24 [200/0] via 10.5.5.5, 00:25:57

R5 VRF network-a

#sh ip route vrf network-a

Routing Table: network-a
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.58.0/30 is directly connected, Ethernet0/1
L 10.0.58.1/32 is directly connected, Ethernet0/1
172.16.0.0/24 is subnetted, 2 subnets
B 172.16.1.0 [200/0] via 10.4.4.4, 00:00:06
B 172.16.2.0 [20/0] via 10.0.58.2, 00:28:30

R5 VRF network-b

#sh ip route vrf network-b

Routing Table: network-b
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.0.105.0/30 is directly connected, Ethernet0/2
L 10.0.105.1/32 is directly connected, Ethernet0/2
B 192.168.1.0/24 [200/0] via 10.4.4.4, 00:27:42
B 192.168.2.0/24 [20/0] via 10.0.105.2, 00:28:33

MPLS L3 VPN. Part 1 – VRF

I have found for myself long ago that reading several articles and documents explaining the same thing, often using different approaches, helps me quite a lot to understand it. Trying to brush up my knowledge in Layer 3 MPLS VPN and doing some experiments in the lab environment I have decided to write a series of short posts on how it works with my way of explaining it. Hope it will help someone who also likes to read various materials on the same topic.

The solution, as many already know, consists of the following basic components:

  • VRF
  • Multiprotocol BGP
  • MPLS (label propagation and switching)
  • Some Interior Gateway Routing Protocol

And it all looks something like this (an example shows LDP as label propagation mechanism though it is not the only one possible):

MPLSL3VPN

The idea is to be able to pass Layer 3 private networks across some transport (MPLS-enabled) network leaving routing processes within these private networks independent of each other and of the ones that take place in the backbone network (unless you actually want to interchange some routing information).

The most difficult part is usually not to understand how each of the components works by itself, but go get the whole picture. I started by asking: “Why do we need all of these components to make the solution work?” Starting with the first one the question will be:

Why do we need VRF?

Remember the goal of L3 MPLS VPN – several independent Layer-3 networks should be able to send data over one transport network. This means we are going to have traffic of two or more private networks on the same router and we need to have separate routing instances for each of these networks. This is what VRF does.

How does it work?

VRF is often called Router Virtualization and is compared to Server/Desktop Virtualization. This analogy doesn’t seem very accurate to me. When you virtualize a server with let’s say VMWare ESXi you normally get all of it’s main components – CPU, RAM, disks, network cards, virtualized and managed separately from the same components of other virtual servers. In case of VRF what gets virtualized is just routing instance. You don’t manage how many hardware resources each virtual routing instance will get, router RAM and CPU resources remain shared.

What makes each VRF is its Layer-3 interface members and its proper routing table. To me an analogy with VLANs each having its Layer-2 port members and proper MAC-address table seems more correct.

A good example of network virtualization technology that gets really close to the server/desktop virtualization would be VDC in Cisco NX-OS, not VRF.

VRF instances are local to the router

This is an important thing to keep in mind. Two switches receiving Ethernet frames from each other across dot1q trunk understand which traffic belongs to which VLAN by looking at dot1q tag. There is nothing similar happening with VRF instances. Each VRF instance is local to the router where it is configured. There is no data in the packet that could indicate which VRF it comes from.

Configuration

VRF configuration is basically resumed to

Creating VRF instances

ip vrf Network-A
 rd 64600:100
!
ip vrf Network-B
 rd 64600:100

The meaning of the Route Distinguisher parameter configured with rd command will be explained in later post.

Assigning Layer-3 interfaces

interface FastEthernet1/0
ip vrf forwarding Network-A
ip address 192.168.1.1 255.255.255.0
!
interface FastEthernet2/0
ip vrf forwarding Network-B
ip address 172.16.1.1 255.255.255.0

Configuring routing

ip route vrf Network-A 192.168.2.0 255.255.255.0 192.168.1.254
ip route vrf Network-B 172.16.2.0 255.255.255.0 172.16.1.254

I have added just static routes, though it is also possible to use dynamic routing protocols capable to work with VRF.

As a result we have separate routing tables for each VRF instance:

R1#show ip route vrf Network-A

Routing Table: Network-A
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
S     192.168.2.0/24 [1/0] via 192.168.1.254

 

R1#show ip route vrf Network-B

Routing Table: Network-B
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.1.0/24 is directly connected, FastEthernet2/0
L        172.16.1.1/32 is directly connected, FastEthernet2/0
S        172.16.2.0/24 [1/0] via 172.16.1.254

The routes that don’t belong to any of the VRF instances created, stay in Global routing instance. Here I have FastEthernet0/0 interface that is not assigned to any VRF:

R1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next
hop override
Gateway of last resort is 10.0.0.254 to network 0.0.0.010.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/24 is directly connected, FastEthernet0/0
L        10.0.0.1/32 is directly connected, FastEthernet0/0

VRF as part of MPLS L3VPN

In MPLS L3VPN solution VRF is used on the transport network border routers (often called PE routers). Let’s say we have Network A and Network B across backbone network. Each of them will have its own VRF on every PE router that has Network A, B devices (CE-routers) connected.

VRF

 

Layer 2 across Layer 3. Part 2 – EoMPLS

Continuing L2 across L3 topic started with post about L2TP.

Another easy-to-setup way to interconnect two Ethernet segments across Layer 3 network is EoMPLS, a particular case of AToM.

The diagram I used for my tests and the initial configuration are the same as in my previous post on L2TP.

Lab Diagram
Lab Diagram

 

R2 R3
interface FastEthernet0/0
  description -= WAN =-
  ip address 115.0.0.2 255.255.255.252
ip route 0.0.0.0 0.0.0.0 115.0.0.1
interface FastEthernet0/0
 description -= WAN =-
 ip address 116.0.0.2 255.255.255.252
ip route 0.0.0.0 0.0.0.0 116.0.0.1

To set up EoMPLS you need to have MPLS-enabled transport network in between the border routers. But if you don’t have this option, such as in case of interconnecting LAN segments across the internet you can just use GRE to make border routers look directly connected to each other across the tunnel. All the configurations are made for common Cisco IOS (not XE or XR).

Setting up GRE

GRE Tunnel uses outside physical interfaces IP addresses as source and destination:

GRE Tunnel
GRE Tunnel
R2 R3
interface Tunnel0
 ip address 1.1.1.1 255.255.255.252
 keepalive 1 3
 tunnel source 115.0.0.2
 tunnel destination 116.0.0.2
interface Tunnel0
 ip address 1.1.1.2 255.255.255.252
 keepalive 1 3
 tunnel source 116.0.0.2
 tunnel destination 115.0.0.2

Tunnel is up:

R3#sh ip int brief | inc Tunnel0
Tunnel0 1.1.1.2 YES manual up up

And we have R2 and R3 “directly connected”.

Setting up LDP

First, I have set up Loopback interfaces on both border routers:

R2

interface Loopback0
 ip address 2.2.2.2 255.255.255.255

R3

interface Loopback0
 ip address 3.3.3.3 255.255.255.255

To have connectivity between Loopback interfaces  I have set up OSPF (one might use static routing as well):

R2

interface Tunnel0
 ip ospf 100 area 0
interface Loopback0
 ip ospf 100 area 0

R3

interface Tunnel0
 ip ospf 100 area 0
interface Loopback0
 ip ospf 100 area 0

So now both Loopback interfaces see each other:

R2#sh ip route ospf

Gateway of last resort is 115.0.0.1 to network 0.0.0.0

3.0.0.0/32 is subnetted, 1 subnets
O 3.3.3.3 [110/1001] via 1.1.1.2, 00:01:48, Tunnel0
R3#sh ip route ospf

2.0.0.0/32 is subnetted, 1 subnets
O 2.2.2.2 [110/1001] via 1.1.1.1, 00:01:20, Tunnel0

R3#ping 2.2.2.2 source 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 3.3.3.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 212/222/228 ms

Enforce the LDP router-ID on both routers:

R2

mpls ldp router-id loop0 force

R3

mpls ldp router-id loop0 force

Enable label exchange over GRE Tunnel:

R2

int tunnel0
mpls ip

R3

int tunnel0
mpls ip

Now the LDP neighborship is established:

R2#sh mpls ldp neighbor
Peer LDP Ident: 3.3.3.3:0; Local LDP Ident 2.2.2.2:0
TCP connection: 3.3.3.3.64467 - 2.2.2.2.646
State: Oper; Msgs sent/rcvd: 8/8; Downstream
Up time: 00:00:12
LDP discovery sources:
Tunnel0, Src IP addr: 1.1.1.2
Addresses bound to peer LDP Ident:
116.0.0.2 3.3.3.3 1.1.1.2R2#show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
16 Pop Label 3.3.3.3/32 0 Tu0 point2point

 

R3#sh mpls ldp nei
Peer LDP Ident: 2.2.2.2:0; Local LDP Ident 3.3.3.3:0
TCP connection: 2.2.2.2.646 - 3.3.3.3.64467
State: Oper; Msgs sent/rcvd: 8/8; Downstream
Up time: 00:00:32
LDP discovery sources:
Tunnel0, Src IP addr: 1.1.1.1
Addresses bound to peer LDP Ident:
115.0.0.2 2.2.2.2 1.1.1.1R3#show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
16 Pop Label 2.2.2.2/32 0 Tu0 point2point

We see some label (16) in show mpls forwarding-table command output, but it’s not the the label that will be used for tunneling traffic.  There will be a few words later in this post about MPLS labels use with EoMPLS.

Now we have the following:

LDP Relationship
LDP Relationship

Setting up the EoMPLS Interconnection

Similarly to L2TPv3 we now configure the interconnection (Virtual Circuit) using xconnect.

R2

interface fastethernet2/0
xconnect 3.3.3.3 2222 encapsulation mpls

R3

interface fastethernet2/0
xconnect 2.2.2.2 2222 encapsulation mpls

Verifying

Trying ping between hosts:

R4#ping 192.168.1.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/143/224 ms

R5#ping 192.168.1.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 96/134/208 ms

And to the broadcast address:

R5#ping 255.255.255.255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 255.255.255.255, timeout is 2 seconds:

Reply to request 0 from 192.168.1.4, 152 ms
Reply to request 1 from 192.168.1.4, 188 ms
Reply to request 2 from 192.168.1.4, 180 ms
Reply to request 3 from 192.168.1.4, 172 ms

Checking that the Virtual Circuit is established:

R2#show mpls l2transport vc detail
Local interface: Fa2/0 up, line protocol up, Ethernet up
Destination address: 3.3.3.3, VC ID: 2222, VC status: up
Output interface: Tu0, imposed label stack {17}
Preferred path: not configured
Default path: active
Next hop: point2point
Create time: 03:38:36, last status change time: 01:26:33
Signaling protocol: LDP, peer 3.3.3.3:0 up
Targeted Hello: 2.2.2.2(LDP Id) -> 3.3.3.3
Status TLV support (local/remote)   : enabled/supported
Label/status state machine        : established, LruRru
Last local dataplane   status rcvd: no fault
Last local SSS circuit status rcvd: no fault
Last local SSS circuit status sent: no fault
Last local  LDP TLV    status sent: no fault
Last remote LDP TLV    status rcvd: no fault
MPLS VC labels: local 17, remote 17
Group ID: local 0, remote 0
MTU: local 1500, remote 1500
Remote interface description: -= LAN =-
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1382, send 1394
byte totals:   receive 140659, send 185950
packet drops:  receive 0, seq error 0, send 3
R3#show mpls l2transport vc detail

Local interface: Fa2/0 up, line protocol up, Ethernet up
  Destination address: 2.2.2.2, VC ID: 2222, VC status: up
    Output interface: Tu0, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: point2point
  Create time: 03:36:10, last status change time: 01:24:29
  Signaling protocol: LDP, peer 2.2.2.2:0 up
    Targeted Hello: 3.3.3.3(LDP Id) -> 2.2.2.2
    Status TLV support (local/remote)   : enabled/supported
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: no fault
      Last local SSS circuit status rcvd: no fault
      Last local SSS circuit status sent: no fault
      Last local  LDP TLV    status sent: no fault
      Last remote LDP TLV    status rcvd: no fault
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 1500, remote 1500
    Remote interface description: -= LAN =-
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1374, send 658
    byte totals:   receive 139272, send 88278
    packet drops:  receive 0, seq error 0, send 1

In both commands output above we see Virtual Circuit ID that we have configured (2222), it’s status (Up), and the tunnel label (17). This is an actual label used in traffic “encapsulation/decapsulation”. The encapsulation itself consists in appending the label to the packet sent across the tunnel.

Looking at the ping capture between R4 and R5 we can see how the packet is formed:

ICMP Echo Request transferred over Layer 2 MPLS Tunnel
ICMP Echo Request transferred over Layer 2 MPLS Tunnel
ICMP Echo Reply transferred over Layer 2 MPLS Tunnel
ICMP Echo Reply transferred over Layer 2 MPLS Tunnel

Both captures show ICMP packet inside the Ethernet frame. Above that comes the Label 17 which we saw in show mpls l2transport vc detail command output. This is so-called VC Label that actually identifies Layer 2 interconnection. Above that comes GRE header and then IPv4 header used to deliver packet from one border router to another. This configuration theoretically can be used to establish EoMPLS interconnection across any sort of Layer 3 WAN, i.e. to perform the same task as L2TPv3 usually does.

Here are the final configurations for both routers:

R2

mpls ldp router-id Loopback0 force
!
interface FastEthernet0/0
  description -= WAN =-
  ip address 115.0.0.2 255.255.255.252
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.25
!
interface Tunnel0
 ip address 1.1.1.1 255.255.255.252
 keepalive 1 3
 tunnel source 115.0.0.2
 tunnel destination 116.0.0.2
 ip ospf 100 area 0
 mpls ip
!
interface Loopback0
 ip ospf 100 area 0
!
interface fastethernet2/0
 xconnect 3.3.3.3 2222 encapsulation mpls
!
ip route 0.0.0.0 0.0.0.0 115.0.0.1

R3

mpls ldp router-id Loopback0 force
!
interface FastEthernet0/0
 description -= WAN =-
 ip address 116.0.0.2 255.255.255.252
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
!
interface Tunnel0
 ip address 1.1.1.2 255.255.255.252
 keepalive 1 3
 tunnel source 116.0.0.2
 tunnel destination 115.0.0.2
!
interface fastethernet2/0
 xconnect 2.2.2.2 2222 encapsulation mpls
 ip ospf 100 area 0
 mpls ip
!
interface Loopback0
 ip ospf 100 area 0
!
ip route 0.0.0.0 0.0.0.0 116.0.0.1

The general case

The more general use case for EoMPLS is when a service provider uses it to give Layer 2 interconnectivity to its customer. The EoMPLS VC starts at one PE router, crosses the MPLS-enabled service provider network (a number of P routers) and gets terminated at another PE router. In this case instead of GRE the MPLS transport is used. The series of labels gets attached and popped from the packet above the VC label. The network diagram I used to emulate this looks the following:

mplsbblab2

This is how ICMP echo request from R5 to R6 (192.168.1.5 to 192.168.1.6) travels across the MPLS WAN:

R1 → R2

R1R2

R2 → R3

R2R3

R3 → R4

R3R4

The packet carries two labels: the inner one identifies the VC as in previous case and the outer one is used to transport packet across the MPLS network. The outer label gets discarded on R3, because of penultimate-hop-popping so we don’t see it on R3 → R4 segment.

The outer label can easily change on the way, because it is local for each MPLS router interface. The captures above just show the case where both R1 and R2 got the same label (23) assigned for packets that are being forwarded towards 4.4.4.4/32 prefix. However it’s not always the case. Here is an example with echo reply packet going back from R6 to R5:

R4 → R3

R4R3

R3 → R2

R3R2

R2 → R1

R2R1

Outer label gets switched from 18 to 16 and then discarded on R2. The inner label stays the same.

Here is a diagram showing the packet flow for echo-request and echo-reply:

mplsbblab3flow
Packet flow with EoMPLS across MPLS WAN