Got an interesting troubleshooting case. Two Layer-3 switches with VRRP configured on their downlinks:
The diagram shows only three downlinks with VRRP, the actual number is around 100.
SW2 being VRRP Backup in normal conditions was reporting VRRP flapping from time to time becoming Master and going back to Backup state again:
%VRRP-6-STATECHANGE: Vl366 Grp 166 state Backup -> Master%VRRP-6-STATECHANGE: Vl60 Grp 60 state Backup -> Master%VRRP-6-STATECHANGE: Vl673 Grp 73 state Backup -> Master%VRRP-6-STATECHANGE: Vl479 Grp 79 state Backup -> Master
|
The clue that drove me to solve the case was that almost all the flappings were occurring in between 9:00 and 18:00. The Layer-3 interfaces had traffic shaping configured:
interface Vlan366 ip address x.x.x.73 255.255.255.248 vrrp 166 ip x.x.x.73 vrrp 166 preempt delay minimum 60 vrrp 166 priority 101 service-policy input Limitto2mbps service-policy output Limitto2mbps |
Looking at SNMP monitoring system I found that %VRRP-6-STATECHANGE Syslog messages timestamps match with time when the traffic on a given interface reaches the shaping limit. What was actually happening was at this moment policy-map was starting to drop traffic and VRRP messages that SW1 was sending to SW2 were occasionally being dropped as well. SW2 missing a subsequent VRRP message declared itself a Master, then got next VRRP message from SW1 and switched to Backup again.
So excluding VRRP (IP Protocol 112 to and from its multicast address 224.0.0.18) from traffic shaping by adding deny statement to traffic shaping ACLs
SW1#sh ip access-lists TS-ACL
|
solved the problem.
Nice thing to keep in mind: next time you do traffic shaping, make sure you don’t cause problems for your control plane traffic.