SD-WAN

Wanted to share some of my thoughts on the topic in simple words and being a bit more technical than the average on the internet.

What it is

Basically SD-WAN is a new way to build enterprise WAN infrastructure by employing some of the principles coming from SDN. Each SD-WAN vendor has its own view but what all the solutions have in common is

centralized control and management and
some form of Zero-Touch Provisioning for branch devices.

Besides branch devices (appliances or virtual machines) that plays the role similar to that of a customer-edge router typical SD-WAN solution has a Controller that ties together all the branches, some peace of software that manages configurations for both controller and branches and some monitoring/analytics software.

Central components may be composed in a different manner, for example Controller may also do the configuration management, or configuration management software do the monitoring as well but the functional blocks remain the same.

How it works

Branches interconnection

Controller and branch devices form an overlay network and provide at least hub-and-spoke connectivity for the whole enterprise. In most cases direct spoke-spoke tunneling is also possible which is important if you have significant VoIP and Videoconferencing traffic between branches. This is something similar to Cisco DMVPN. What most SD-WAN vendors try to add to this is multipath over redundant underlay WAN-circuits and greater control over traffic paths.

Traffic path control

In a traditional WAN architecture it was common to use some expensive WAN-circuit with guaranteed SLA as primary and then have some backup possibly over the Internet. SD-WAN solutions give you an opportunity to use primary and backup circuits in an active-active manner. They normally have some built-in mechanisms that monitor quality of each link and are able to distribute traffic between links dynamically as these links degrade and recover. In some solutions you can define SLA levels for each type of traffic you have, defining it by application signature or just L4/L3 information and the infrastructure will make sure it is always sent via circuits that satisfy these SLA at any given moment at the same time make sure not to use expensive bandwidth for non-sensitive traffic. In some cases this may mean that customer may cease to use MPLS circuits and move to 2 or 3 good internet connections from different service providers. The latter won’t have guaranteed SLA but branch and SD-WAN controller will be able to figure out where to send sensitive traffic to provide the best possible experience for end-users.

All this is typically implemented with some extensions to open-standard routing and monitoring protocols and functions. An interesting thing to note here is that some vendors completely hide this from system administrator providing him or her with a simple interface with sliders and boxes while others let you go deep and highly customize the operation.

Zero-Touch Provisioning and Configuration Management

All the SD-WAN solutions have some way to simplify new branches deployment as these has always been a huge burden on people who support WAN-infrastructures. The best analogy here would probably be a centralized Wi-Fi architecture. Just as in Wi-Fi you have a controller where you configure everything you need, plug your AP into switch port pointing it somehow to the controller with SD-WAN you do the same with the box you install in your branch. Again every SD-WAN vendor is different but normally you create the configuration in your Management Software, deliver the box on site where someone connects it to the WAN, the box somehow finds the Controller and Management Software, gets the configuration and starts doing its job. “Somehow” here may mean for example querying some predefined DNS-record or running a script with some parameters.

Same Wi-Fi analogy works with Configuration/Policy Management. One of the main benefits of SD-WAN is that an administrator no longer has the need to implement changes on each device when implementing a new application across the enterprise. All the changes are made centrally in the Management Software, typically based on predefined templates and pushed to all or selected the branch devices.

Direct Internet Access

Even though branch devices are centrally managed and the WAN traffic is typically tunnelled to a central site or some other branch location they are still able to do at least basic NAT to provide internet access locally. This is becoming essential as there are more and more cloud services being used by the enterprises. Some SD-WAN vendors provide Firewall and UTM functionality at the branch, so there is no need to backhaul all the traffic to your Datacenter for inspection and policy enforcement.

Other features

LTE support – several vendors already have it, some boxes may even support more than one SIM-card. Some vendors have in roadmap. Having LTE support makes it easier to acquire backup channel and allows to use boxes in vehicles.

Whiteboxing – yes, there are vendors that allow you to make CPE by installing their software on an x86 hardware from third-party vendors. Normally this hardware still has to be tested and approved by SD-WAN vendor but you have a better choice and lower price because hardware vendors competing with each other.

NFV – some vendors allow you to run/integrate 3rd party virtualized network services into their solutions.

As for CPE hardware an interesting moment might be that some vendors offer rugged appliances.

Conclusion

SD-WAN solution becomes the more attractive the more sites an enterprise have. There is already a bunch of successful implementation stories in retail, banking and insurance sector. More and more service providers sell SD-WAN as managed services. On the other hand there is a lot of diversity in understanding of what SD-WAN is among vendors, so you need to study particular solutions very well from the technical point of view in order to make sure it suits your needs.

Using Python, YAML and Jinja2 to Generate Config Files

You will need three things:

Jinja2 template with your configuration “skeleton”
YAML file with data you want to insert into your configuration
A Python script that feeds data taken from YAML file into Jinja2 template

Now some details. Jinja2 is a template engine designed to be used with Python. It’s similar to Django but is able to employ Python-like expressions. Here is an example for Cisco IOS-style config:

hostname {{ name }}

interface Loopback0
ip address 10.0.0.{{ id }} 255.255.255.255

{% for vlan, name in vlans.items() %}
vlan {{ vlan }}
name {{ name }}
{% endfor %}

router ospf 1
router-id 10.0.0.{{ id }}
auto-cost reference-bandwidth 10000
{% for networks in ospf %}
network {{ networks.network }} area {{ networks.area }}
{% endfor %}

The parts marked with {{ }} are variables and expressions. Here Hostname and the last octet of Loopback0 IP-address are to be substituted with variables but VLANs and OSPF network statements part contain Python-like For loops. These make you able to insert various VLAN and network statements getting data from YAML data file. Here is an example of such data file (I have taken a screenshot from Notepad++ to preserve all the text formatting):

Now a Python script that reads YAML file into dictionary and feeds it’s data into Jinja2 template:

#Import necessary functions from Jinja2 module
from jinja2 import Environment, FileSystemLoader

#Import YAML module
import yaml

#Load data from YAML into Python dictionary
config_data = yaml.load(open('./data_files/data_to_feed.yml'))

#Load Jinja2 template
env = Environment(loader = FileSystemLoader('./templates'), trim_blocks=True, lstrip_blocks=True)
template = env.get_template('template.txt')

#Render the template with data and print the output
print(template.render(config_data))

This script just prints the result, but it pretty straitforward to write the result into text file that you may load to your device. In my case the result looks like this:

hostname R1

interface Loopback0
ip address 10.0.0.1 255.255.255.255

vlan 10
name Users

vlan 20
name Voice

vlan 30
name Management

router ospf 1
router-id 10.0.0.1
auto-cost reference-bandwidth 10000
network 10.0.1.0 0.0.0.255 area 0
network 10.0.2.0 0.0.0.255 area 2
network 10.1.1.0 0.0.0.255 area 0

You may use loops in your script, feed it with YAML-serialized data for multiple devices and make it generate multiple config files at a time all based on the same Jinja2-formatted template. All data is nicely stored and there is no need for hard-to-read string concatenation expressions!

Anyconnect VPN on ASA Сheatsheet

A small example of Anyconnect VPN configuration on ASA with 9.6 software. Split-tunneling included.

ip local pool vpn_user 192.168.255.1-192.168.255.10 mask 255.255.255.224
!
access-list Split-Tunnelling-ACL standard permit 192.168.1.0 255.255.255.0
access-list Split-Tunnelling-ACL standard permit 192.168.2.0 255.255.255.0
access-list Split-Tunnelling-ACL standard permit 192.168.3.0 255.255.255.0
access-list Split-Tunnelling-ACL standard permit 192.168.4.0 255.255.255.0
!
webvpn
enable Outside
anyconnect image disk0:/anyconnect-win-4.4.01054-webdeploy-k9.pkg 1
anyconnect image disk0:/anyconnect-macos-4.4.01054-webdeploy-k9.pkg 2
anyconnect image disk0:/anyconnect-linux64-4.4.01054-webdeploy-k9.pkg 3
anyconnect profiles VPN-Profile disk0:/vpn-profile.xml
anyconnect enable
!
group-policy VPNPolicy internal
group-policy VPNPolicy attributes
dns-server none
vpn-tunnel-protocol ssl-client ssl-clientless
password-storage disable split-tunnel-policy tunnelspecified
split-tunnel-network-list value Split-Tunnelling-ACL
address-pools value vpn_user
webvpn
anyconnect profiles value VPN-Profile type user
!
username testvpnuser password testvpnpassword
username testvpnuser attributes
vpn-group-policy VPNPolicy
service-type remote-access
!
tunnel-group VPN type remote-access
tunnel-group VPN general-attributes
address-pool vpn_user
default-group-policy VPNPolicy

CCIE Certified

There was some silence in this blog as I was on a final stage of my CCIE lab exam preparation. Okay, now I am CCIE Routing & Switching!

It took me about 8 months to prepare. Hours and days of reading, watching and labbing.

Want to share some of my thoughts on all this.

You don’t know what you’ll face on the exam. As the routing and switching lab went virtual Cisco is constantly changing things, adding new twists to it, so the best choice is to master what you certainly will need.

So before going to the lab I tried to make sure that I

know the “core” technologies (L2/L3, MPLS/VRF, DMVPN etc.) really well. Just as Einstein said “be able to explain it to your grandmother”
can do the “core” level configurations really quickly without using any terminal hints, just by memory
know typical problems you may face with core technologies, which commands to use to find and verify them without spending much time guessing. Know the common reasons why OSPF neighborship fails, why BGP routes are not present in routing table etc.
know and master shortened versions of all the commands: no to show bgp vpnv4 unicast all summary, yes to sh bg vpnv4 uni all sum
know the terminal keyboard shortcuts/hotkeys. An official Cisco doc on it can be found here. Also there is a cheatsheet version published in Etherealmind
know all master all the CLI filtering keys (include, exclude, section). Of with their shortened versions (i, e, s). Forget about browsing through unfiltered command outputs
mastered fast typing

My idea was that this will make bigger part of configuration and troubleshooting I’ll have to do on my lab exam more like routine and liberate some of my time and brain resources to thoroughly think on issues I’ll certainly have and correct mistakes I’ll inevitably make.

Of course there is always a good and bad luck on every exam, but by proper training you are diminishing space for bad luck. Relaxing and sleeping well before exam also helps a lot.

And at last my answers to two common philosophical questions on obtaining CCIE status:

Q: How much does it have to do with a real life?

A: Not that much for me. I would say it has about 20% overlap with my job.

Q: Does studying for CCIE make you a better engineer?

A: Certainly yes. Just make sure you look as deep as you can into things while studying.

Notes on Routing

I will be posting some notes that I take while studying. These may be useful to someone who is preparing for his or her CCNP/CCIE exam. Some of these are not put explicitly in the documentation. This time it’s about RIP.

Managing RIP Versions

When you issue router rip and network x.x.x.x commands on a Cisco Router it brings up version 1. What it actually means is that it starts to send v1 updates but listens to both v1 and v2. You can verify it show ip protocols output:

When you issue version 2 global command it starts sending and receiving v2 updates only:

However you can turn on and off v1 and v2 updates on a per-interface basis and set whatever combination you want with two interface-level commands:

ip rip send version
ip rip receive version

Selectively Modifying Administrative Distance

You can modify administrative distance for routes based on:

source which the route comes from
the network part of the route

This is done with distance command under rip process. You specify the AD value you want to set, then sources range and wildcard mask and then the access-list that defines the network part of the routes (unfortunately there is no way to use prefix-lists here, so you can’t specify prefix length). Here is an example:

ip access-list standard ROUTES
 permit 1.1.1.1
 permit 2.2.2.2

router rip
 distance 125 10.0.0.2 0.0.0.0 ROUTES

Before:

After:

Routes to 1.1.1.1 and 2.2.2.2 changed their AD to 125, route to 3.3.3.3 stays intact. Same can be done under EIGRP or OSPF.

Checking RIP Route Information Sources

RIP has no “formal” neighborship concept it justs sends and receives updates in and out of interfaces where it is enabled so there is no analog of show ip ospf neighbors or show ip eigrp neighbors command for RIP. However show ip protocols command provides you with some useful data on routers neighbors:

Basic IKEv2 Site-to-Site VPN

Here is probably the most basic example of IKEv2 Site-to-Site (LAN-to-LAN) VPN between two routers called BRANCH-A and R5:

BRANCH-A R5

ip access-list extended VPN
permit ip host 1.1.1.1 host 2.2.2.2crypto ikev2 profile VPN-TO-R5
match identity remote address 172.16.198.35 255.255.255.255
identity local address 192.168.231.6
authentication remote pre-share key R5-PSK
authentication local pre-share key Branch-A-PSKcrypto map VPN-CM 10 ipsec-isakmp
set peer 172.16.198.35
set ikev2-profile VPN-TO-R5
match address VPN

interface Loopback0
ip address 1.1.1.1 255.255.255.255

interface Ethernet0/1
ip address 192.168.231.6 255.255.255.0
crypto map VPN-CM

interface Loopback0 ip address 1.1.1.1 255.255.255.255

ip access-list extended VPN
permit ip host 2.2.2.2 host 1.1.1.1crypto ikev2 profile VPN-TO-BRANCH-A
match identity remote address 192.168.231.6 255.255.255.255
identity local address 172.16.198.35
authentication remote pre-share key Branch-A-PSK
authentication local pre-share key R5-PSKcrypto map VPN-CM 10 ipsec-isakmp
set peer 192.168.231.6
set ikev2-profile VPN-TO-BRANCH-A
match address VPN

interface Loopback0
ip address 2.2.2.2 255.255.255.255

interface Ethernet0/0
ip address 172.16.198.35 255.255.255.248
crypto map VPN-CM

interface Loopback0 ip address 2.2.2.2 255.255.255.255

The routing part is omitted. 192.168.231.6 and 172.16.198.35 should be mutually reachable.

IKEv2 Profile

Here you have to specify three things:

The remote peer identity to match the profile, so that when you get message from the peer that presents this identity this particular IKEv2 profile gets applied. For this case I use the peer IP address as an identity:

crypto ikev2 profile VPN-TO-R5 match identity remote address 172.16.198.35 255.255.255.255

Your own identity (how you present yourself to remote peer):

identity local address 192.168.231.6

Authentication methods (in my case along with authentication data): these are methods that you are able to perform and methods you expect the remote peer to perform. Note that authentication in IKEv2 is asymmetric by default. You may authenticate your remote peer using one method while the remote peer authenticas you with another one. For this case I used pre-shared keys both ways and specified the keys:

authentication remote pre-share key R5-PSK authentication local pre-share key Branch-A-PSK

Crypto Map

Here you have to set up the remote peer IP Address:

crypto map VPN-CM 10 ipsec-isakmp set peer 172.16.198.35

The IKEv2 profile:
set ikev2-profile VPN-TO-R5

and an ACL to select interesting traffic:
match address VPN

Verifying

With the following debugs enabled on both routers:

debug crypto ikev2

debug crypto ipsec

Trying to generate interesting traffic:

BRANCH-A#ping 2.2.2.2 source 1.1.1.1

And here it goes:

BRANCH-A:

*May 5 19:00:36.318: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 192.168.231.6:500, remote= 172.16.198.35:500,
local_proxy= 1.1.1.1/255.255.255.255/256/0,
remote_proxy= 2.2.2.2/255.255.255.255/256/0,
protocol= ESP, transform= esp-aes esp-sha-hmac (Tunnel),
lifedur= 3600s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0
*May 5 19:00:36.318: IKEv2:Searching Policy with fvrf 0, local address 192.168.231.6
*May 5 19:00:36.318: IKEv2:Using the Default Policy for Proposal
*May 5 19:00:36.318: IKEv2:Found Policy 'default'
*May 5 19:00:36.318: IKEv2:(SESSION ID = 2,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 5
*May 5 19:00:36.318: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
*May 5 19:00:36.318: IKEv2:(SESSION ID = 2,SA ID = 1):Request queued for computation of DH key
*May 5 19:00:36.318: IKEv2:IKEv2 initiator - no config data to send in IKE_SA_INIT exch
*May 5 19:00:36.318: IKEv2:(SESSION ID = 2,SA ID = 1):Generating IKE_SA_INIT message
*May 5 19:00:36.318: IKEv2:(SESSION ID = 2,SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 15
AES-CBC AES-CBC AES-CBC SHA512 SHA384 SHA256 SHA1 MD5 SHA512 SHA384 SHA256 SHA96 MD596 DH_GROUP_1536_MODP/Group 5 DH_GROUP_1024_MODP/Group 2

*May 5 19:00:36.318: IKEv2:(SESSION ID = 2,SA ID = 1):Sending Packet [To 172.16.198.35:500/From 192.168.231.6:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

*May 5 19:00:36.319: IKEv2:(SESSION ID = 2,SA ID = 1):Insert SA

An IKE_SA_INIT message is sent to R5:

R5:

*May 5 19:00:36.320: IKEv2:Received Packet [From 192.168.231.6:500/To 172.16.198.35:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):Verify SA init message
*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):Insert SA
*May 5 19:00:36.320: IKEv2:Searching Policy with fvrf 0, local address 172.16.198.35
*May 5 19:00:36.320: IKEv2:Using the Default Policy for Proposal
*May 5 19:00:36.320: IKEv2:Found Policy 'default'
*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):Processing IKE_SA_INIT message
*May 5 19:00:36.320: IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieve configured trustpoint(s)
*May 5 19:00:36.320: IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
*May 5 19:00:36.320: IKEv2:Failed to retrieve Certificate Issuer list
*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 5
*May 5 19:00:36.320: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):Request queued for computation of DH key
*May 5 19:00:36.320: IKEv2:(SESSION ID = 16,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH secret key, DH Group 5
*May 5 19:00:36.331: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
*May 5 19:00:36.331: IKEv2:(SESSION ID = 16,SA ID = 1):Request queued for computation of DH secret
*May 5 19:00:36.331: IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Calculate SKEYSEED and create rekeyed IKEv2 SA
*May 5 19:00:36.331: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] SKEYSEED calculation and creation of rekeyed IKEv2 SA PASSED
*May 5 19:00:36.331: IKEv2:IKEv2 responder - no config data to send in IKE_SA_INIT exch
*May 5 19:00:36.331: IKEv2:(SESSION ID = 16,SA ID = 1):Generating IKE_SA_INIT message
*May 5 19:00:36.331: IKEv2:(SESSION ID = 16,SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 4
AES-CBC SHA512 SHA512 DH_GROUP_1536_MODP/Group 5
*May 5 19:00:36.331: IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieve configured trustpoint(s)
*May 5 19:00:36.331: IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
*May 5 19:00:36.331: IKEv2:Failed to retrieve Certificate Issuer list

*May 5 19:00:36.331: IKEv2:(SESSION ID = 16,SA ID = 1):Sending Packet [To 192.168.231.6:500/From 172.16.198.35:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 0
IKEv2 IKE_SA_INIT Exchange RESPONSE
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

*May 5 19:00:36.332: IKEv2:(SESSION ID = 16,SA ID = 1):Completed SA init exchange
*May 5 19:00:36.332: IKEv2:(SESSION ID = 16,SA ID = 1):Starting timer (30 sec) to wait for auth message

R5 responds with IKE_SA_INIT RESPONSE:

BRANCH-A:

*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):Received Packet [From 172.16.198.35:500/To 192.168.231.6:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 0
IKEv2 IKE_SA_INIT Exchange RESPONSE
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):Processing IKE_SA_INIT message
*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):Verify SA init message
*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):Processing IKE_SA_INIT message
*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):Checking NAT discovery
*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):NAT not found
*May 5 19:00:36.333: IKEv2:(SESSION ID = 2,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH secret key, DH Group 5
*May 5 19:00:36.343: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
*May 5 19:00:36.343: IKEv2:(SESSION ID = 2,SA ID = 1):Request queued for computation of DH secret
*May 5 19:00:36.344: IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Calculate SKEYSEED and create rekeyed IKEv2 SA
*May 5 19:00:36.344: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] SKEYSEED calculation and creation of rekeyed IKEv2 SA PASSED
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Completed SA init exchange
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Check for EAP exchange
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Generate my authentication data
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Use preshared key for id 192.168.231.6, key len 12
*May 5 19:00:36.344: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
*May 5 19:00:36.344: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Get my authentication method
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):My authentication method is 'PSK'
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Check for EAP exchange
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Generating IKE_AUTH message
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Constructing IDi payload: '192.168.231.6' of type 'IPv4 address'
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):ESP Proposal: 1, SPI size: 4 (IPSec negotiation),
Num. transforms: 3
AES-CBC SHA96 Don't use ESN
*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)

*May 5 19:00:36.344: IKEv2:(SESSION ID = 2,SA ID = 1):Sending Packet [To 172.16.198.35:500/From 192.168.231.6:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR

BRANCH-A sends IKEv2 AUTH REQUEST:

R5:

*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Received Packet [From 192.168.231.6:500/To 172.16.198.35:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)

*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Stopping timer to wait for auth message
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Checking NAT discovery
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):NAT not found
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Searching policy based on peer's identity '192.168.231.6' of type 'IPv4 address'
*May 5 19:00:36.346: IKEv2:found matching IKEv2 profile 'VPN-TO-BRANCH-A'
*May 5 19:00:36.346: IKEv2:Searching Policy with fvrf 0, local address 172.16.198.35
*May 5 19:00:36.346: IKEv2:Using the Default Policy for Proposal
*May 5 19:00:36.346: IKEv2:Found Policy 'default'
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Verify peer's policy
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Peer's policy verified
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Get peer's authentication method
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Peer's authentication method is 'PSK'
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Get peer's preshared key for 192.168.231.6
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Verify peer's authentication data
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Use preshared key for id 192.168.231.6, key len 12
*May 5 19:00:36.346: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
*May 5 19:00:36.346: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Verification of peer's authenctication data PASSED
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Processing INITIAL_CONTACT
*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):Processing IKE_AUTH message
*May 5 19:00:36.346: IKEv2:IPSec policy validate request sent for profile VPN-TO-BRANCH-A with psh index 1.

*May 5 19:00:36.346: IKEv2:(SESSION ID = 16,SA ID = 1):
*May 5 19:00:36.346: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*May 5 19:00:36.346: IPSEC(validate_proposal_request): proposal part #1
*May 5 19:00:36.346: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 172.16.198.35:0, remote= 192.168.231.6:0,
local_proxy= 2.2.2.2/255.255.255.255/256/0,
remote_proxy= 1.1.1.1/255.255.255.255/256/0,
protocol= ESP, transform= esp-aes esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0
*May 5 19:00:36.346: Crypto mapdb : proxy_match
src addr : 2.2.2.2
dst addr : 1.1.1.1
protocol : 0
src port : 0
dst port : 0
*May 5 19:00:36.347: (ipsec_process_proposal)Map Accepted: VPN-CM, 10
*May 5 19:00:36.347: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal - PASSED.

*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Get my authentication method
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):My authentication method is 'PSK'
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Get peer's preshared key for 192.168.231.6
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Generate my authentication data
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Use preshared key for id 172.16.198.35, key len 6
*May 5 19:00:36.351: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
*May 5 19:00:36.351: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Get my authentication method
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):My authentication method is 'PSK'
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Generating IKE_AUTH message
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Constructing IDr payload: '172.16.198.35' of type 'IPv4 address'
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):ESP Proposal: 1, SPI size: 4 (IPSec negotiation),
Num. transforms: 3
AES-CBC SHA96 Don't use ESN
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDr AUTH SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)

*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Sending Packet [To 192.168.231.6:500/From 172.16.198.35:500/VRF i0:f0]
Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
ENCR

R5 responds:

And creates Security Association:

*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):IKEV2 SA created; inserting SA into database. SA lifetime timer (86400 sec) started
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Session with IKE ID PAIR (192.168.231.6, 172.16.198.35) is UP
*May 5 19:00:36.351: IKEv2:IKEv2 MIB tunnel started, tunnel index 1
*May 5 19:00:36.351: IKEv2:(SESSION ID = 16,SA ID = 1):Load IPSEC key material
*May 5 19:00:36.351: IKEv2:(SA ID = 1):[IKEv2 -> IPsec] Create IPsec SA into IPsec database
*May 5 19:00:36.351: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*May 5 19:00:36.351: Crypto mapdb : proxy_match
src addr : 2.2.2.2
dst addr : 1.1.1.1
protocol : 256
src port : 0
dst port : 0
*May 5 19:00:36.351: IPSEC:(SESSION ID = 16) (crypto_ipsec_create_ipsec_sas) Map found VPN-CM, 10
*May 5 19:00:36.352: IPSEC:(SESSION ID = 16) (create_sa) sa created,
(sa) sa_dest= 172.16.198.35, sa_proto= 50,
sa_spi= 0xB0DF64D5(2967430357),
sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 9
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 172.16.198.35:0, remote= 192.168.231.6:0,
local_proxy= 2.2.2.2/255.255.255.255/256/0,
remote_proxy= 1.1.1.1/255.255.255.255/256/0
*May 5 19:00:36.352: IPSEC:(SESSION ID = 16) (create_sa) sa created,
(sa) sa_dest= 192.168.231.6, sa_proto= 50,
sa_spi= 0x4EDBBD5D(1323023709),
sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 10
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 172.16.198.35:0, remote= 192.168.231.6:0,
local_proxy= 2.2.2.2/255.255.255.255/256/0,
remote_proxy= 1.1.1.1/255.255.255.255/256/0
*May 5 19:00:36.352: IPSEC: Expand action denied, notify RP
*May 5 19:00:36.352: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Creation of IPsec SA into IPsec database PASSED
*May 5 19:00:36.352: IKEv2:(SESSION ID = 16,SA ID = 1):Checking for duplicate IKEv2 SA
R5#
*May 5 19:00:36.352: IKEv2:(SESSION ID = 16,SA ID = 1):No duplicate IKEv2 SA found
*May 5 19:00:36.352: IKEv2:(SESSION ID = 16,SA ID = 1):Starting timer (8 sec) to delete negotiation context

BRANCH-A processes IKEv2 AUTH RESPONSE from R5 and creates SA as well

`*May 5 19:00:36.352: IKEv2:(SESSION ID = 2,SA ID = 1):Received Packet [From 172.16.198.35:500/To 192.168.231.6:500/VRF i0:f0]`
`Initiator SPI : F178851A3C6C4300 - Responder SPI : 74741B9AAD85D582 Message id: 1`
`IKEv2 IKE_AUTH Exchange RESPONSE`
`Payload contents:`
`VID IDr AUTH SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)`

*May 5 19:00:36.352: IKEv2:(SESSION ID = 2,SA ID = 1):Process auth response notify
*May 5 19:00:36.352: IKEv2:(SESSION ID = 2,SA ID = 1):Searching policy based on peer's identity '172.16.198.35' of type 'IPv4 address'
*May 5 19:00:36.352: IKEv2:Searching Policy with fvrf 0, local address 192.168.231.6
*May 5 19:00:36.352: IKEv2:Using the Default Policy for Proposal
*May 5 19:00:36.352: IKEv2:Found Policy 'default'
*May 5 19:00:36.352: IKEv2:(SESSION ID = 2,SA ID = 1):Verify peer's policy
*May 5 19:00:36.352: IKEv2:(SESSION ID = 2,SA ID = 1):Peer's policy verified
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Get peer's authentication method
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Peer's authentication method is 'PSK'
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Get peer's preshared key for 172.16.198.35
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Verify peer's authentication data
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Use preshared key for id 172.16.198.35, key len 6
*May 5 19:00:36.353: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
*May 5 19:00:36.353: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Verification of peer's authenctication data PASSED
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Check for EAP exchange
*May 5 19:00:36.353: IKEv2:(SESSION ID = 2,SA ID = 1):Processing IKE_AUTH message
*May 5 19:00:36.353: IKEv2:IPSec policy validate request sent for profile VPN-TO-R5 with psh index 1.

*May 5 19:00:36.353: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*May 5 19:00:36.353: IPSEC(validate_proposal_request): proposal part #1
*May 5 19:00:36.353: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 192.168.231.6:0, remote= 172.16.198.35:0,
local_proxy= 1.1.1.1/255.255.255.255/256/0,
remote_proxy= 2.2.2.2/255.255.255.255/256/0,
protocol= ESP, transform= esp-aes esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0
*May 5 19:00:36.353: Crypto mapdb : proxy_match
src addr : 1.1.1.1
dst addr : 2.2.2.2
protocol : 0
src port : 0
dst port : 0
*May 5 19:00:36.353: (ipsec_process_proposal)Map Accepted: VPN-CM, 10
*May 5 19:00:36.353: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal - PASSED.

*May 5 19:00:36.357: IKEv2:(SESSION ID = 2,SA ID = 1):IKEV2 SA created; inserting SA into database. SA lifetime timer (86400 sec) started
*May 5 19:00:36.357: IKEv2:(SESSION ID = 2,SA ID = 1):Session with IKE ID PAIR (172.16.198.35, 192.168.231.6) is UP
*May 5 19:00:36.357: IKEv2:IKEv2 MIB tunnel started, tunnel index 1
*May 5 19:00:36.357: IKEv2:(SESSION ID = 2,SA ID = 1):Load IPSEC key material
*May 5 19:00:36.357: IKEv2:(SA ID = 1):[IKEv2 -> IPsec] Create IPsec SA into IPsec database
*May 5 19:00:36.357: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*May 5 19:00:36.357: Crypto mapdb : proxy_match
src addr : 1.1.1.1
dst addr : 2.2.2.2
protocol : 256
src port : 0
dst port : 0
*May 5 19:00:36.357: IPSEC:(SESSION ID = 2) (crypto_ipsec_create_ipsec_sas) Map found VPN-CM, 10
*May 5 19:00:36.357: IPSEC:(SESSION ID = 2) (create_sa) sa created,
(sa) sa_dest= 192.168.231.6, sa_proto= 50,
sa_spi= 0x4EDBBD5D(1323023709),
sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 10
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 192.168.231.6:0, remote= 172.16.198.35:0,
local_proxy= 1.1.1.1/255.255.255.255/256/0,
remote_proxy= 2.2.2.2/255.255.255.255/256/0
*May 5 19:00:36.357: IPSEC:(SESSION ID = 2) (create_sa) sa created,
(sa) sa_dest= 172.16.198.35, sa_proto= 50,
sa_spi= 0xB0DF64D5(2967430357),
sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 9
sa_lifetime(k/sec)= (4608000/3600),
(identity) local= 192.168.231.6:0, remote= 172.16.198.35:0,
local_proxy= 1.1.1.1/255.255.255.255/256/0,
remote_proxy= 2.2.2.2/255.255.255.255/256/0
*May 5 19:00:36.358: IPSEC: Expand action denied, notify RP
*May 5 19:00:36.358: IKEv2:(SA ID = 1):[IPsec -> IKEv2] Creation of IPsec SA into IPsec database PASSED
*May 5 19:00:36.358: IKEv2:(SESSION ID = 2,SA ID = 1):Checking for duplicate IKEv2 SA
*May 5 19:00:36.358: IKEv2:(SESSION ID = 2,SA ID = 1):No duplicate IKEv2 SA found

The whole message exchange capture:

Security Associacions:

BRANCH-A#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
1 192.168.231.6/500 172.16.198.35/500 none/none READY ! Peers addresses
Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK ! SA Parameters
Life/Active Time: 86400/1141 sec

IPv6 Crypto IKEv2 SA

BRANCH-A#sh crypto ipsec sa

interface: Ethernet0/1
Crypto map tag: VPN-CM, local addr 192.168.231.6

protected vrf: (none)
local ident (addr/mask/prot/port): (1.1.1.1/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0) ! Proxy ID
current_peer 172.16.198.35 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
#pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 192.168.231.6, remote crypto endpt.: 172.16.198.35 ! Peers addresses
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb Ethernet0/1
current outbound spi: 0xB0DF64D5(2967430357)
PFS (Y/N): N, DH group: none

inbound esp sas:
spi: 0x4EDBBD5D(1323023709)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 10, flow_id: SW:10, sibling_flags 80000040, crypto map: VPN-CM
sa timing: remaining key lifetime (k/sec): (4252494/2435)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xB0DF64D5(2967430357)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 9, flow_id: SW:9, sibling_flags 80000040, crypto map: VPN-CM
sa timing: remaining key lifetime (k/sec): (4252494/2435)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

Summary

IKEv2 has much clearer message exchange (and thus debugs output) than IKEv1.

Also, notice that there is no need to carefully configure all the IKE and IPSec parameters as we used to do in IKEv1. This is due to Cisco implementation of IKEv2 that implies the use of some default parameters, for example debug says:

*May 5 19:00:36.346: IKEv2:Using the Default Policy for Proposal
*May 5 19:00:36.346: IKEv2:Found Policy 'default'

And you can actually see this “default” IKEv2 policy and its parameters:

BRANCH-A#show crypto ikev2 policy

IKEv2 policy : default
Match fvrf : any
Match address local : any
Proposal : default

BRANCH-A#show crypto ikev2 proposal
IKEv2 proposal: default
Encryption : AES-CBC-256 AES-CBC-192 AES-CBC-128
Integrity : SHA512 SHA384 SHA256 SHA96 MD596
PRF : SHA512 SHA384 SHA256 SHA1 MD5
DH Group : DH_GROUP_1536_MODP/Group 5 DH_GROUP_1024_MODP/Group 2

Same thing with IPSec transform set:

BRANCH-A#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Transport, }

This simplifies configuration significantly.

Got my CWNA

Passed Cisco Wireless Network Administrator exam, a vendor-independent certification promoted by CWNP Foundation.

My past Wi-Fi experience was 99.9% Cisco and in my opinion Cisco has excellent matherials on how to design, deploy and troubleshoot its solutions, so does it pay great attention to theoretical basics. Cisco solution also provides you with very good set of troubleshooting tools. Whether in the future I still will be deploying and troubleshooting Cisco solutions or not, studying for a vendor independent certifications seemed to be very useful to better understand what I am doing and what is actually going on “in the air”.

May seem trivial but worth repeating: the deeper you go into theory the more developed and solid will be your ideas on how to design, configure or troubleshoot something. CWNA is definitely a good step towards this direction.

Now moving on to reading Certified Wireless Analysis Professional matherials.

Traffic Storm-Control: a tiny detail to keep in mind

When you configure Traffic Storm Control on Cisco switches ports using bandwidth percentage as threshold level, be aware that this percentage will be calculated on the basis of negotiated bandwidth not the hardware port type. In other words, you may have 1000BaseTX interface configured something like this:

interface GigabitEthernet1/0/6
storm-control broadcast level 1.00
storm-control action trap

And you think that port will start dropping broadcast traffic when it reaches 10Mbps rate (1% of 1000Mbps). That is true if it gets negotiated at actual 1000Mbps. However, if for some reason the speed gets negotiated to let’s say 100Mbps drops will start at 1Mbps rate which is 1% of 100Mbps.

The documentation says “The traffic storm control level is a percentage of the total available bandwidth of the port” This available bandwidth can be seen in show interface command output.

When the port is at 1000Mbps:

#sh int gi1/0/27 | inc Giga|BW
GigabitEthernet1/0/27 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6cfa.8953.fd1b (bia 6cfa.8953.fd1b)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

When it is at 100Mbps:

#sh int gi1/0/29 | inc Giga|BW
GigabitEthernet1/0/29 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6cfa.8953.fd1d (bia 6cfa.8953.fd1d)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

VRRP troubleshooting case

Got an interesting troubleshooting case. Two Layer-3 switches with VRRP configured on their downlinks:

The diagram shows only three downlinks with VRRP, the actual number is around 100.

SW2 being VRRP Backup in normal conditions was reporting VRRP flapping from time to time becoming Master and going back to Backup state again:

%VRRP-6-STATECHANGE: Vl366 Grp 166 state Backup -> Master
%VRRP-6-STATECHANGE: Vl60 Grp 60 state Backup -> Master
%VRRP-6-STATECHANGE: Vl673 Grp 73 state Backup -> Master
%VRRP-6-STATECHANGE: Vl479 Grp 79 state Backup -> Master

%VRRP-6-STATECHANGE: Vl366 Grp 166 state Master -> Backup
%VRRP-6-STATECHANGE: Vl60 Grp 60 state Master -> Backup
%VRRP-6-STATECHANGE: Vl673 Grp 73 state Master -> Backup
%VRRP-6-STATECHANGE: Vl479 Grp 79 state Master -> Backup

The clue that drove me to solve the case was that almost all the flappings were occurring in between 9:00 and 18:00. The Layer-3 interfaces had traffic shaping configured:

interface Vlan366
ip address x.x.x.73 255.255.255.248
vrrp 166 ip x.x.x.73
vrrp 166 preempt delay minimum 60
vrrp 166 priority 101
service-policy input Limitto2mbps
service-policy output Limitto2mbps

Looking at SNMP monitoring system I found that %VRRP-6-STATECHANGE Syslog messages timestamps match with time when the traffic on a given interface reaches the shaping limit. What was actually happening was at this moment policy-map was starting to drop traffic and VRRP messages that SW1 was sending to SW2 were occasionally being dropped as well. SW2 missing a subsequent VRRP message declared itself a Master, then got next VRRP message from SW1 and switched to Backup again.

So excluding VRRP (IP Protocol 112 to and from its multicast address 224.0.0.18) from traffic shaping by adding deny statement to traffic shaping ACLs

SW1#sh ip access-lists TS-ACL

Extended IP access list TS-ACL
4 deny 112 x.x.x.72 0.0.0.7 host 224.0.0.18
10 permit ip any any (4131183 matches)

solved the problem.

Nice thing to keep in mind: next time you do traffic shaping, make sure you don’t cause problems for your control plane traffic.

IPSec Proxy-ID

Proxy-IDs are entities that are used in IPSec tunnel negotiations (Phase-2 in case of IKEv1) to select which traffic actually goes to the tunnel. The always come in pairs (a sort of tuples) as Local+Remote. So in case of Cisco ASA or IOS-based router, when you make an ACL something like:

permit ip 192.168.1.0 255.255.255.0 10.1.1.0 255.255.255.0

You define 192.168.1.0/24 as Local Proxy ID and 10.1.1.0/24 as remote Proxy ID. I used subnet mask as in ASA, IOS-based routers use wilcard masks for ACL statements, but the idea is the same.

When the other end device receives this data it swaps Local and Remote Proxy IDs and starts looking for a match in its configuration. I.e. for IPSec security associacions to get established successfully the following definition needs to be found:

Local Proxy ID	Remote Proxy ID
10.1.1.0/24	192.168.1.0/24

and it has to match exactly.

A common doubt is whether it is possible to configure IPSec something like this:

Peer A:

permit ip 10.1.1.0 255.255.255.128 192.168.1.0 255.255.255.0

permit ip 10.1.1.128 255.255.255.128 192.168.1.0 255.255.255.0

Peer B

permit ip 192.168.1.0 255.255.255.0 10.1.1.0 255.255.255.0

And have the tunnel successfully established between 10.1.1.0/24 and 192.168.1.0/24? Two statements configured on Peer A match the same set of IP-addresses as statement configured on Peer B. So everything seems to be ok.

Well, it is not. As per configuration Peer A will generate two pairs of proxy ID (10.1.1.0/25, 192.168.1.0/24) and (10.1.1.128/25, 192.168.1.0/24) and send it to Peer B. Peer B will be waiting for (10.1.1.0/24, 192.168.1.0/24) and will not accept neither of pairs received from Peer A. The opposite way same thing will happen. Peer B will be sending (192.168.1.0/24, 10.1.1.0/24) and Peer A will reject it waiting for two Proxy-IDs pairs which are exactly inverse to the ones it has configured.

You will see a message similar to this in debug output:

[IKEv1]Group = 172.16.66.46, IP = 172.16.66.46, Rejecting IPSec tunnel: no matching crypto map entry for remote proxy 192.168.1.0/255.255.255.0/0/0 local proxy 10.1.1.0/255.255.255.0/0/0 on interface outside

To see this error you have to check debug on peer who receives the Proxy-ID pair. The one that sends it will just keep sending it and failing to establish the tunnel. This is generally a good hint for IPSec troubleshooting – always try to debug at the receiver side, because this is the one that compalins and reports errors.

So, whatever equipment you use from whatever vendor. The configuration that defines proxy IDs (in case of Cisco it is done with ACL statements) on one peer always has to be the mirror of the one from another peer.

Don't Fragment

Network Engineer experience

Author: asigachev

SD-WAN

What it is

How it works

Branches interconnection

Traffic path control

Zero-Touch Provisioning and Configuration Management

Direct Internet Access

Other features

Conclusion

Using Python, YAML and Jinja2 to Generate Config Files

Anyconnect VPN on ASA Сheatsheet

CCIE Certified

Notes on Routing

Managing RIP Versions

Selectively Modifying Administrative Distance

Checking RIP Route Information Sources

Basic IKEv2 Site-to-Site VPN

IKEv2 Profile

Crypto Map

Verifying

BRANCH-A:

R5:

BRANCH-A:

R5:

BRANCH-A processes IKEv2 AUTH RESPONSE from R5 and creates SA as well

The whole message exchange capture:

Security Associacions:

Summary

Got my CWNA

Traffic Storm-Control: a tiny detail to keep in mind

VRRP troubleshooting case

IPSec Proxy-ID