VRRP troubleshooting case

Got an interesting troubleshooting case. Two Layer-3 switches with VRRP configured on their downlinks:

vrrp

The diagram shows only three downlinks with VRRP, the actual number is around 100.

SW2 being VRRP Backup in normal conditions was reporting VRRP flapping from time to time becoming Master and going back to Backup state again:

%VRRP-6-STATECHANGE: Vl366 Grp 166 state Backup -> Master
%VRRP-6-STATECHANGE: Vl60 Grp 60 state Backup -> Master
%VRRP-6-STATECHANGE: Vl673 Grp 73 state Backup -> Master
%VRRP-6-STATECHANGE: Vl479 Grp 79 state Backup -> Master

%VRRP-6-STATECHANGE: Vl366 Grp 166 state Master -> Backup
%VRRP-6-STATECHANGE: Vl60 Grp 60 state Master -> Backup
%VRRP-6-STATECHANGE: Vl673 Grp 73 state Master -> Backup
%VRRP-6-STATECHANGE: Vl479 Grp 79 state Master -> Backup

The clue that drove me to solve the case was that almost all the flappings were occurring in between 9:00 and 18:00. The Layer-3 interfaces had traffic shaping configured:

interface Vlan366
  ip address x.x.x.73 255.255.255.248
  vrrp 166 ip x.x.x.73
  vrrp 166 preempt delay minimum 60
  vrrp 166 priority 101
  service-policy input Limitto2mbps
  service-policy output Limitto2mbps

Looking at SNMP monitoring system I found that %VRRP-6-STATECHANGE Syslog messages timestamps match with time when the traffic on a given interface reaches the shaping limit. What was actually happening was at this moment policy-map was starting to drop traffic and VRRP messages that SW1 was sending to SW2 were occasionally being dropped as well. SW2 missing a subsequent VRRP message declared itself a Master, then got next VRRP message from SW1 and switched to Backup again.

So excluding VRRP (IP Protocol 112 to and from its multicast address 224.0.0.18) from traffic shaping by adding deny statement to traffic shaping ACLs

SW1#sh ip access-lists TS-ACL

Extended IP access list TS-ACL
4 deny 112 x.x.x.72 0.0.0.7 host 224.0.0.18
10 permit ip any any (4131183 matches)

solved the problem.

Nice thing to keep in mind: next time you do traffic shaping, make sure you don’t cause problems for your control plane traffic.

One thought on “VRRP troubleshooting case”

Leave a Reply

Your email address will not be published. Required fields are marked *