A common NTP implementation in the enterprise network implies one or two border routers synchronizing with external NTP servers and peering between them. Other NTP-enabled devices and applications get time from these routers internal or loopback interface addresses which belong to private address space. I usually configure external NTP servers from the regional ntp.org pool:

ntp server 0.south-america.pool.ntp.org ntp server 2.south-america.pool.ntp.org ntp server 1.south-america.pool.ntp.org ntp server 3.south-america.pool.ntp.org

Border router resolves the names and tries to syncronize to one of them. The experience has shown that these NTP servers are quite unstable (at least in the region where I currenty work). Both border routers can easily be left without a single server to synchronize, so the whole network stays without reliable time source. In order to be able to detect such situations I wrote a simple EEM Applet:

event manager applet ntp-sync-check event tag 1 snmp oid 1.3.6.1.2.1.197.1.2.1 get-type next entry-op ne entry-val "6" entry-type value poll-interval 86400 event tag 5 none trigger correlate event 1 or event 5 action 0.10 info type routername action 1.00 cli command "enable" action 1.10 cli command "show ntp asso" action 1.20 set ntpa "$_cli_result" action 2.00 cli command "show ntp status" action 2.20 set ntpst "$_cli_result" action 3.00 syslog priority critical msg "NTP sync failed" facility "NTP" action 3.10 mail server "exchange.ourdomain.com" to "alarmsandnotifications@ourdomain.com" from "$_info_routername@ourdomain.com" cc "sysadmin@ourdomain.com" subject "** NTP Sync Failure **" body "$ntpst

What it does is just polling SNMP OID that returns the router NTP synchronization status and if the synchronization is found to be failed sends a syslog message and e-mail notification containing show ntp associations and show ntp status commands outputs.

The values for this OID being polled are:

1 : notRunning
2 : notSynchronized
3 : noneConfigured
4 : syncToLocal
5 : syncToRefclock
6 : syncToRemoteServer
99 : unknown

The only value that satisfies us is 6 – NTP Synchronized to Remote Server. So if the OID value is not equal to 6 the applet gets launched (event 1 gets triggered).

A recommended practice for all EEM scripts is have event none added besides the main event and to configure the applet to run on any of the two events occurrence. In the example above it is added with tag 5. This allows you to test the applet by launching it manually without waiting for o provoking the main event (NTP synchronization loss in my case). An applet can be tested by executing the following command:

event manager run ntp-sync-check