Traffic Storm-Control: a tiny detail to keep in mind

When you configure Traffic Storm Control on Cisco switches ports using bandwidth percentage as threshold level, be aware that this percentage will be calculated on the basis of negotiated bandwidth not the hardware port type. In other words, you may have 1000BaseTX interface configured something like this:

interface GigabitEthernet1/0/6
 storm-control broadcast level 1.00
 storm-control action trap

And you think that port will start dropping broadcast traffic when it reaches 10Mbps rate (1% of 1000Mbps). That is true if it gets negotiated at actual 1000Mbps. However, if for some reason the speed gets negotiated to let’s say 100Mbps drops will start at 1Mbps rate which is 1% of 100Mbps.

The documentation says “The traffic storm control level is a percentage of the total available bandwidth of the port” This available bandwidth can be seen in show interface command output.

When the port is at 1000Mbps:

#sh int gi1/0/27 | inc Giga|BW
GigabitEthernet1/0/27 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6cfa.8953.fd1b (bia 6cfa.8953.fd1b)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

When it is at 100Mbps:

#sh int gi1/0/29 | inc Giga|BW
GigabitEthernet1/0/29 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6cfa.8953.fd1d (bia 6cfa.8953.fd1d)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

PIM Sparse-Mode

My short study of how PIM Sparse-Mode multicast works:

Lab Topology
Lab Topology

The Sender

Here is what happens when sender originates streaming to a multicast group.

R4#ping 224.88.88.88 repeat 100

The router that is a PIM DR for this network segment (R6) sends Register message to the rendezvous-point:

R6#show ip interface brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       10.0.46.6       YES manual up                    up
GigabitEthernet2       10.0.56.6       YES manual up                    up
GigabitEthernet3       unassigned      YES unset  administratively down down
GigabitEthernet4       10.0.67.6       YES manual up                    up
Loopback0              6.6.6.6         YES manual up                    up
Tunnel0                10.0.56.6       YES unset  up                    up
R6#show ip pim interfaceAddress          Interface                Ver/   Nbr    Query  DR         DR
Mode   Count  Intvl  Prior
10.0.56.6        GigabitEthernet2         v2/S   1      30     1          10.0.56.6
10.0.67.6        GigabitEthernet4         v2/S   1      30     1          10.0.67.7
10.0.46.6        GigabitEthernet1         v2/S   0      30     1          10.0.46.6(R6 Gi1 IP)

R6#debug ip pim

PIM(0): Send v2 Data-header Register to 3.3.3.3 for 10.0.46.4, group 224.88.88.88

The router that has one of its interfaces configured as a rendezvous-point checks the reverse path, creates the PIM tunnel and responds with Register-Stop message:

R3#debug ip pim

PIM(0): Check RP 3.3.3.3 into the (*, 224.88.88.88) entry
PIM(0): Adding register decap tunnel (Tunnel1) as accepting interface of (*, 224.88.88.88).
PIM(0): Adding register decap tunnel (Tunnel1) as accepting interface of (10.0.46.4, 224.88.88.88).
PIM(0): Send v2 Register-Stop to 10.0.56.6 for 10.0.46.4, group 224.88.88.88

Here is how these messages look:

PIM-Register Message
PIM-Register Message
PIM-Register-Stop Message
PIM-Register-Stop Message

Notice the structure of PIM Register – it is a unicast IP packet that carries PIM datagram together with original ICMP packet sent by R4 to the multicast group.

R3 now has (*,G) and (S, G) entries created.

R3#show ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:01:30/stopped, RP 3.3.3.3, flags: SP
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list: Null

(10.0.46.4, 224.88.88.88), 00:01:30/00:01:29, flags: P
  Incoming interface: Ethernet0/1, RPF nbr 10.0.235.5
  Outgoing interface list: Null

Both entries have P-flag. This means that the traffic for this multicast group is pruned, because there are no receivers. For the same reason (S,G) entry has Null in Outgoing interface. Incoming interface is already known at this stage.

 

The receiver

Make a receiver join the group:

R1#debug ip igmp
IGMP debugging is on
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int e0/0
R1(config-if)#ip igmp join-group 224.88.88.88

IGMP(0): WAVL Insert group: 224.88.88.88 interface: Ethernet0/0 Successful
IGMP(0): Send v2 Report for 224.88.88.88 on Ethernet0/0

And it sends the IGMP membership report frame structure:

IGMP Membership Report
IGMP Membership Report

This basicaly says “I want to listen to this group”.

The router that receives IGMP Report messages sends PIM-Join Message to the RP:

PIM Join Message
PIM Join Message

On RP we now can see outgoing interface in (*,G) entry. This is where it sees the receiver.

R3#sho ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:18:09/00:03:00, RP 3.3.3.3, flags: S
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
    Ethernet0/1, Forward/Sparse, 00:18:09/00:03:00

(10.0.46.4, 224.88.88.88), 00:01:59/00:01:11, flags: PT
Incoming interface: Ethernet0/1, RPF nbr 10.0.235.5
Outgoing interface list: Null

(*, 224.0.1.40), 01:00:34/00:03:26, RP 3.3.3.3, flags: SJCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse, 01:00:33/00:03:26

Now, if we start sending traffic to the group again we see get replies from the receiver:

R4#ping 224.88.88.88 repeat 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 224.88.88.88, timeout is 2 seconds:Reply to request 0 from 10.0.12.1, 62 ms
Reply to request 1 from 10.0.12.1, 8 ms

But on the RP we still have the trafic for 224.88.88.88 pruned and no record that has both incoming and outgoing interface.

R3#sh ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:49:35/00:03:03, RP 3.3.3.3, flags: S
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse, 00:03:44/00:03:03

(10.0.46.4, 224.88.88.88), 00:01:01/00:02:09, flags: PT
Incoming interface: Ethernet0/1, RPF nbr 10.0.235.5
Outgoing interface list: Null

(*, 224.0.1.40), 01:32:00/00:03:19, RP 3.3.3.3, flags: SJCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse, 01:32:00/00:03:19

This is because the traffic has switched to Shortest-Path Tree which doesn’t include R3. R6 forwards packets coming from 10.0.46.4 out interface GigabitEthernet2 (Towards R5):

R6#sh ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:02:19/stopped, RP 3.3.3.3, flags: SPF
Incoming interface: GigabitEthernet2, RPF nbr 10.0.56.5
Outgoing interface list: Null

(10.0.46.4, 224.88.88.88), 00:02:19/00:01:21, flags: FT
  Incoming interface: GigabitEthernet1, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet2, Forward/Sparse, 00:02:19/00:03:08

(*, 224.0.1.40), 01:31:36/00:03:09, RP 3.3.3.3, flags: SJCL
Incoming interface: GigabitEthernet2, RPF nbr 10.0.56.5
Outgoing interface list:
GigabitEthernet4, Forward/Sparse, 01:29:43/00:03:09

Then R5 forwards them out GigabitEthernet1 (Towards R2):

R5#show ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:04:29/stopped, RP 3.3.3.3, flags: SP
Incoming interface: GigabitEthernet1, RPF nbr 10.0.235.3
Outgoing interface list: Null

(10.0.46.4, 224.88.88.88), 00:04:29/00:02:52, flags: T
  Incoming interface: GigabitEthernet2, RPF nbr 10.0.56.6
  Outgoing interface list:
    GigabitEthernet1, Forward/Sparse, 00:04:29/00:03:02

(*, 224.0.1.40), 01:34:39/00:03:16, RP 3.3.3.3, flags: SJCL
Incoming interface: GigabitEthernet1, RPF nbr 10.0.235.3
Outgoing interface list:
GigabitEthernet2, Forward/Sparse, 01:33:47/00:03:16

Which forwards them to the receiver (R1) out Ethernet0/0:

R2#show ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route,
x - VxLAN group
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode(*, 224.88.88.88), 00:55:17/stopped, RP 3.3.3.3, flags: SJC
Incoming interface: Ethernet0/1, RPF nbr 10.0.235.3
Outgoing interface list:
Ethernet0/0, Forward/Sparse, 00:09:26/00:02:05

(10.0.46.4, 224.88.88.88), 00:06:43/00:02:13, flags: JT
  Incoming interface: Ethernet0/1, RPF nbr 10.0.235.5
  Outgoing interface list:
    Ethernet0/0, Forward/Sparse, 00:06:43/00:02:05

(*, 224.0.1.40), 01:39:03/00:02:59, RP 3.3.3.3, flags: SJCL
Incoming interface: Ethernet0/1, RPF nbr 10.0.235.3
Outgoing interface list:
Ethernet0/0, Forward/Sparse, 01:39:01/00:02:59

All of the multicast routes the packets are following have SPT-bit set (T-flag in multicast routing table) which means they are part of Shortest Part Tree.

GRE and IPIP Encapsulation – What Is The Difference?

We know that both GRE and IPIP encapsulate an IP packet into another IP packet, we know neither GRE nor IPIP provide encryption. But what is the difference between them?

Here it is:

IPIP
IPIP packet

 

GRE
GRE Packet

IPIP is really just another IP header on top of the original IP header, whilst GRE apart from traditional IP header data inserts GRE header that may contain checksum, authentication key, control the number of encapsulations, packet sequence, etc. The overhead for IPIP header is 20 bytes versus 24-36 bytes for GRE (the actual overhead depends on which GRE header data is being carried).

Some Useful Commands to Troubleshoot IP Routing

To see some general routing information

#show ip route summary
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Replicates  Overhead    Memory (bytes)
connected       0           13          0           1408        3744
static          0           0           0           0           0
application     0           0           0           0           0
ospf 100        0           6           0           768         1752
  Intra-area: 6 Inter-area: 0 External-1: 0 External-2: 0
  NSSA External-1: 0 NSSA External-2: 0
bgp 14234       103         1006        0           106464      319392
  External: 629 Internal: 455 Local: 25
internal        63                                              111864
Total           166         1025        0           108640      436752

Shows how much memory is being consumed by each routing process.

To see information on OSPF Area Border Routers and Autonomous System Border Routers

#show ip ospf border-routers

            OSPF Router with ID (10.0.0.2) (Process ID 100)

                Base Topology (MTID 0)

Internal Router Routing Table
Codes: i - Intra-area route, I - Inter-area route

i 10.0.0.3 [10] via 10.0.23.2, Ethernet0/1, ABR/ASBR, Area 0, SPF 9
i 10.0.0.1 [10] via 10.0.12.1, Ethernet0/0, ASBR, Area 1, SPF 5
I 10.0.0.4 [20] via 10.0.23.2, Ethernet0/1, ASBR, Area 0, SPF 92

Basically shows how your router is intended to reach each of the ABRs and ASBRs it knows about.

To see which routing information is originated by your  OSPF-enabled router

#show ip ospf database self-originate

OSPF Router with ID (10.0.0.2) (Process ID 100)

Router Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.2        10.0.0.2        156         0x80000006 0x005D69 1

Summary Net Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.12.0       10.0.0.2        111         0x80000001 0x006EA5

Summary ASB Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.1        10.0.0.2        111         0x80000001 0x00EC2E

Router Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum Link count
10.0.0.2        10.0.0.2        156         0x80000003 0x00746B 1

Summary Net Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.23.0       10.0.0.2        111         0x80000001 0x00F414
10.0.34.0       10.0.0.2        111         0x80000001 0x00DF14

Summary ASB Link States (Area 1)

Link ID         ADV Router      Age         Seq#       Checksum
10.0.0.3        10.0.0.2        111         0x80000001 0x00D840
10.0.0.4        10.0.0.2        111         0x80000001 0x0033DA

The output above is taken from Router that is ABR, so you can see Summary (Type 3) LSAs that this router generates and sends to Area 1 to tell about the routes belonging to Area 0 and vice versa. Also you can see Summary ASB LSAs that it sends to inform routers in Area 0 about an ASBR it sees in Area 1 and routers in Area 1 about ASBRs in Area 1.

Layer 2 across Layer 3. Part 1 – L2TP

Several tasks I’ve been up to recently have made me study solutions to interconnect Layer 2 network segments across Layer 3 infrastructure. I decided to do a series of quick review of the technologies that serve this purpose. The most detailed information could be found in standards and vendors documents, there is no point to rewrite all of it here. What I was intended to is just produce a sort of quick reference/cheatsheet for what I have studied.

The first is Layer 2 Tunneling Protocol.

Lab 

Here is a network diagram I used:

Lab Diagram
Lab Diagram
  • R4 and R5 represent two hosts in the same VLAN. Those are the ones that need to be interconnected across the WAN.
  • R2 and R3 represent border routers in each location.
  • R1 is just a transport WAN emulation.

Starting configuration for border routers:

R2 R3
interface FastEthernet0/0
  description -= WAN =-
  ip address 115.0.0.2 255.255.255.252
ip route 0.0.0.0 0.0.0.0 115.0.0.1
interface FastEthernet0/0
 description -= WAN =-
 ip address 116.0.0.2 255.255.255.252
ip route 0.0.0.0 0.0.0.0 116.0.0.1

The starting configuration for R1, R4 and R5 is just IP addresses assigned to interfaces according to the diagram.

The basic L2TPv3 configuration

L2TP is probably the most straightforward method to implement Layer-2 interconnection across Layer-3 WAN. Basic config looks like this:

R2

pseudowire-class test-pw-class
encapsulation l2tpv3
ip local interface FastEthernet0/0
interface FastEthernet2/0
description -= LAN =-
no ip address
xconnect 116.0.0.2 123 encapsulation l2tpv3 pw-class test-pw-class

R3

pseudowire-class test-pw-class
encapsulation l2tpv3
ip local interface FastEthernet0/0
interface FastEthernet2/0
description -= LAN =-
no ip address
xconnect 116.0.0.2 123 encapsulation l2tpv3 pw-class test-pw-class

Here is the result:

L2TPv3 Tunnel Established
L2TP Tunnel Established

All the traffic coming to FastEthernet 2/0 port of any of the border routers gets encapsulated and sent to the opposite site where it gets decapsulated while going out of another FastEthernet 2/0 interface. As a result we can ping R5 from R4 and vice versa as if they were connected directly by Layer-2 link:

R5#ping 192.168.1.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.4, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 88/98/120 ms
R4#ping 192.168.1.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 88/116/156 ms

The broadcast traffic also gets propagated successfully:

R4#ping 255.255.255.255
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 255.255.255.255, timeout is 2 seconds:
Reply to request 0 from 192.168.1.5, 152 ms
Reply to request 1 from 192.168.1.5, 108 ms
Reply to request 2 from 192.168.1.5, 152 ms
Reply to request 3 from 192.168.1.5, 104 ms

Some theory

The obvious advantage of L2TP is that in order to implement it you don’t need anything but Layer 3 connectivity between devices that will terminate both ends of the tunnel.

Here is what happens to the Layer 2 Payload when it gets transferred over the tunnel:

Encapsulation Scheme
Encapsulation Scheme

First, L2TPv3 Header gets added to the original Layer 2 Frame. It contains all the information needed to identify the Tunnel and the Session.

Second IP Delivery Header containing source and destination IP addresses of the tunnel peers (in our case R2 and R3 routers) and UDP Header are added to deliver the packet to the other end device that will decapsulate it. On transport layer L2TPv3 uses port 1701.

The packet flow to establish the connection is the following:

L2TP Control Packet Flow
L2TP Control Packet Flow

Verifying 

Checking tunnel and session states:

R2#show l2tun

L2TP Tunnel and Session Information Total tunnels 1 sessions 1

LocTunID   RemTunID   Remote Name   State  Remote Address  Sessn L2TP Class/
                                                           Count VPDN Group
1401476767 197244710  R3            est    116.0.0.2       1     l2tp_default_cl

LocID      RemID      TunID      Username, Intf/      State  Last Chg Uniq ID
                                 Vcid, Circuit
4141636091 3431548551 1401476767 123, Fa2/0           est    00:02:50 1

You can see the tunnel Virtual Circuit ID 123 configured and it’s state as established.

Here is a debug command showing tunnel establishment process where you can see packets related to tunnel setup marked bold. R2 plays the role of the server in this case, so the tunnel establishment process begins with it receiving SCCRQ, responds SCCRP and so on. Notice also the states the system passes through (Idle, Proc-SCCRQ, Wt-SCCN, etc.) till it gets to established.

R2#debug l2tp event
*Dec  8 23:52:23.159: L2X  tnl   08318:________: Create logical tunnel
*Dec  8 23:52:23.159: L2TP tnl   08318:________: Create tunnel
*Dec  8 23:52:23.163: L2TP tnl   08318:________:     version set to Cisco-V3
*Dec  8 23:52:23.163: L2TP tnl   08318:________:     remote ip set to 116.0.0.2
*Dec  8 23:52:23.167: L2TP tnl   08318:________:     local ip set to 115.0.0.2
*Dec  8 23:52:23.175: L2TP tnl   08318:70A47756: FSM-CC ev Rx-SCCRQ
*Dec  8 23:52:23.175: L2TP tnl   08318:70A47756: FSM-CC    Idle->Proc-SCCRQ
*Dec  8 23:52:23.179: L2TP tnl   08318:70A47756: FSM-CC do Rx-SCCRQ
*Dec  8 23:52:23.183: L2TP tnl   08318:70A47756:     class name l2tp_default_class
*Dec  8 23:52:23.191: L2TP       _____:________: Check for existing cc
*Dec  8 23:52:23.191: L2TP       _____:________:   115.0.0.2<->116.0.0.2
*Dec  8 23:52:23.195: L2TP       _____:________:   and peer router id: 116.0.0.2
*Dec  8 23:52:23.195: L2TP       _____:________:   and peer hostname: R3
*Dec  8 23:52:23.199: L2TP       _____:________:   and version: V3
*Dec  8 23:52:23.203: L2TP       _____:________:   No other cc exists to tie with
*Dec  8 23:52:23.203: L2TP tnl   08318:70A47756: FSM-CC ev SCCRQ-OK
*Dec  8 23:52:23.207: L2TP tnl   08318:70A47756: FSM-CC    Proc-SCCRQ->Wt-SCCCN
*Dec  8 23:52:23.207: L2TP tnl   08318:70A47756: FSM-CC do Tx-SCCRP
*Dec  8 23:52:23.211: L2X        _____:________: l2x_open_socket: is called
*Dec  8 23:52:23.211: L2TP tnl   08318:70A47756: Open sock 115.0.0.2:0->116.0.0.2:0
*Dec  8 23:52:23.215: L2TP tnl   08318:70A47756: FSM-CC ev Sock-Ready
*Dec  8 23:52:23.215: L2TP tnl   08318:70A47756: FSM-CC    in Wt-SCCCN
*Dec  8 23:52:23.219: L2TP tnl   08318:70A47756: FSM-CC do Ignore-Sock-Up
*Dec  8 23:52:23.355: L2TP tnl   08318:70A47756: FSM-CC ev Rx-SCCCN
*Dec  8 23:52:23.359: L2TP tnl   08318:70A47756: FSM-CC    Wt-SCCCN->Proc-SCCCN
*Dec  8 23:52:23.359: L2TP tnl   08318:70A47756: FSM-CC do Rx-SCCCN
*Dec  8 23:52:23.379: L2TP tnl   08318:70A47756: FSM-CC ev SCCCN-OK
*Dec  8 23:52:23.379: L2TP tnl   08318:70A47756: FSM-CC    Proc-SCCCN->established
*Dec  8 23:52:23.383: L2TP tnl   08318:70A47756: FSM-CC do Established
*Dec  8 23:52:23.383: L2TP tnl   08318:70A47756: Control channel up
*Dec  8 23:52:23.387: L2TP tnl   08318:70A47756:   115.0.0.2<->116.0.0.2
*Dec  8 23:52:23.431: L2X  _____:_____:________: Create logical session
*Dec  8 23:52:23.435: L2TP _____:_____:________: Create session
*Dec  8 23:52:23.435: L2TP _____:_____:________:   Using ICRQ FSM
*Dec  8 23:52:23.439: L2TP _____:_____:________: FSM-Sn ev created
*Dec  8 23:52:23.439: L2TP _____:_____:________: FSM-Sn    Init->Idle
*Dec  8 23:52:23.443: L2TP _____:_____:________: FSM-Sn do none
*Dec  8 23:52:23.443: L2TP _____:_____:________:     remote ip set to 116.0.0.2
*Dec  8 23:52:23.447: L2TP _____:_____:________:     local ip set to 115.0.0.2
*Dec  8 23:52:23.451: L2TP tnl   08318:70A47756: FSM-CC ev Session-Conn
*Dec  8 23:52:23.451: L2TP tnl   08318:70A47756: FSM-CC    in established
*Dec  8 23:52:23.455: L2TP tnl   08318:70A47756: FSM-CC do Session-Conn-Est
*Dec  8 23:52:23.455: L2TP tnl   08318:70A47756:   Session count now 1
*Dec  8 23:52:23.459: L2TP _____:08318:ECEDBDCE: FSM-Sn ev CC-Up
*Dec  8 23:52:23.459: L2TP _____:08318:ECEDBDCE: FSM-Sn    in Idle
*Dec  8 23:52:23.463: L2TP _____:08318:ECEDBDCE: FSM-Sn do CC-Up-Ignore0-1
*Dec  8 23:52:23.463: L2TP _____:08318:ECEDBDCE: Session attached
*Dec  8 23:52:23.467: L2TP _____:08318:ECEDBDCE: FSM-Sn ev Rx-ICRQ
*Dec  8 23:52:23.467: L2TP _____:08318:ECEDBDCE: FSM-Sn    Idle->Proc-ICRQ
*Dec  8 23:52:23.467: L2TP _____:08318:ECEDBDCE: FSM-Sn do Rx-ICRQ
*Dec  8 23:52:23.467: L2TP _____:08318:ECEDBDCE:   Chose application XCONNECT
*Dec  8 23:52:23.471: L2TP _____:08318:ECEDBDCE:   App type set to XCONNECT
*Dec  8 23:52:23.471: L2TP tnl   08318:70A47756:   XCONNECT Session count now 1
*Dec  8 23:52:23.471: L2TP _____:08318:ECEDBDCE: Remote AC is now UP
*Dec  8 23:52:23.471: L2TP _____:08318:ECEDBDCE: XCONNECT: process AVPs
*Dec  8 23:52:23.471: L2TP _____:08318:ECEDBDCE: Set HA epoch to 0
*Dec  8 23:52:23.475: L2TP _____:08318:ECEDBDCE:
*Dec  8 23:52:23.495: L2X  _____:_____:________: Destroying logical session
*Dec  8 23:52:23.495: L2TP 00001:08318:ECEDBDCE:
*Dec  8 23:52:23.499: L2TP 00001:08318:ECEDBDCE:   App type set to XCONNECT
*Dec  8 23:52:23.499: L2TP 00001:08318:ECEDBDCE:   Need cc version: Cisco-V3
*Dec  8 23:52:23.503: L2TP 00001:08318:ECEDBDCE:   Sequencing default tx disabled
*Dec  8 23:52:23.507: L2TP 00001:08318:ECEDBDCE:   Sequencing default rx disabled
*Dec  8 23:52:23.507: L2TP 00001:08318:ECEDBDCE: no cookies enabled
*Dec  8 23:52:23.511: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev ICRQ-OK
*Dec  8 23:52:23.511: L2TP 00001:08318:ECEDBDCE: FSM-Sn Proc-ICRQ->Wt-Tx-ICRP
*Dec  8 23:52:23.515: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Tx-ICRP-Local-Check
*Dec  8 23:52:23.515: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev Local-Cont
*Dec  8 23:52:23.519: L2TP 00001:08318:ECEDBDCE: FSM-Sn    Wt-Tx-ICRP->Wt-Rx-ICCN
*Dec  8 23:52:23.519: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Tx-ICRP
*Dec  8 23:52:23.523: L2X        _____:________: l2x_open_socket: is called
*Dec  8 23:52:23.527: L2TP 00001:08318:ECEDBDCE: Open sock 115.0.0.2:0->116.0.0.2:0
*Dec  8 23:52:23.527: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev Sock-Ready
*Dec  8 23:52:23.531: L2TP 00001:08318:ECEDBDCE: FSM-Sn    in Wt-Rx-ICCN
*Dec  8 23:52:23.535: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Ignore-Sock-Up
*Dec  8 23:52:23.555: L2TP 00001:08318:ECEDBDCE:
*Dec  8 23:52:23.559: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev DP-Setup
*Dec  8 23:52:23.559: L2TP 00001:08318:ECEDBDCE: FSM-Sn    in Wt-Rx-ICCN
*Dec  8 23:52:23.563: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Ignore-DP-Setup
*Dec  8 23:52:23.799: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev Rx-ICCN
*Dec  8 23:52:23.803: L2TP 00001:08318:ECEDBDCE: FSM-Sn    Wt-Rx-ICCN->Proc-ICCN
*Dec  8 23:52:23.807: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Rx-ICCN
*Dec  8 23:52:23.807: L2TP 00001:08318:ECEDBDCE:   MTU is 65535
*Dec  8 23:52:23.811: L2TP 00001:08318:ECEDBDCE: Session data plane UP
*Dec  8 23:52:23.815: L2TP 00001:08318:ECEDBDCE: XCONNECT: process AVPs
*Dec  8 23:52:23.819: L2TP 00001:08318:ECEDBDCE:
*Dec  8 23:52:23.839: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev ICCN-OK
*Dec  8 23:52:23.843: L2TP 00001:08318:ECEDBDCE: FSM-Sn    Proc-ICCN->established
*Dec  8 23:52:23.843: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Established
*Dec  8 23:52:23.847: L2TP 00001:08318:ECEDBDCE: Session up
*Dec  8 23:52:23.851: L2TP 00001:08318:ECEDBDCE:   115.0.0.2<->116.0.0.2
*Dec  8 23:52:23.895: L2TP 00001:08318:ECEDBDCE: XCONNECT: process AVPs
*Dec  8 23:52:23.899: L2TP 00001:08318:ECEDBDCE:
*Dec  8 23:52:23.903: L2TP 00001:08318:ECEDBDCE: FSM-Sn ev DP-Up
*Dec  8 23:52:23.907: L2TP 00001:08318:ECEDBDCE: FSM-Sn    in established
*Dec  8 23:52:23.907: L2TP 00001:08318:ECEDBDCE: FSM-Sn do Ignore-DP-UP

Similar packet exchange information can be seen with debug vpdn command.

In case you need to reset L2TPv3-tunnel, the following command can be used:

R2#clear l2tun remote ip 116.0.0.2