Finding out-of-sync TLD nameservers (2024-05)

With recent news about c.root-servers.net not getting updated for near ten days, we realized our own existing tests which check for this didn't adequately check the . (dot) root zone.

This weekend, we did an analysis of 1448 TLDs and it reported thousands of anomalies and real DNS failures. Out of the DNS Institute DNS Analyzer's 160+ test cases, 67 unique test types reported problems. These are based on IETF/RFC standards, registry best practices, and government mandates, including for DNSSEC and IPv6. See the following examples of possible TLD zones that are out-of-date for some nameservers:

TD. — 204.61.216.129 served SOA SERIAL 692038076 while 185.28.194.194 served SOA SERIAL 2024042408. Two different serial number styles.
YE. — 195.94.13.22 SOA SERIAL 2024052592 vs. 195.94.0.35 SOA SERIAL 2024051919 and 195.94.0.34 SOA SERIAL 2024051919. If these are date stamps with counters, maybe six days behind?
RW. — 192.96.24.69 SOA SERIAL 2024052515 vs. 196.49.7.188 SOA SERIAL 2024052321. Two days behind?
KN. — 204.61.216.109 SOA SERIAL 2024052514 vs. 185.28.194.194 SOA SERIAL 2024050614. Eleven days?
KH. — 202.12.31.53 SOA SERIAL 2024041777 vs. 203.223.32.3 SOA SERIAL 2024051681. If date stamp, a month behind?

If the SOA serial numbers are counters, it detected 75 others that had a difference of at least 30. For example:

XN--NODE — 185.21.168.36 SOA SERIAL 2267371742 vs. 204.61.216.88 SOA SERIAL 2316459181. 2001:500:14:6088:ad::1 SOA SERIAL 2316459181 vs. 2a04:1b00:8::4 SOA SERIAL 2267371742. (Other nameservers too.) Difference of 49087439.

The following examples are Unix epoch times:

MK — 2001:628:453:bb::4 SOA SERIAL 1716661310 vs. 2001:148f:fffd::12 and 2001:1470:8000:53::44 SOA SERIAL 1716633410. Difference of 7 hours and 45 minutes. (Others showed other hours of differences too.)
XN--D1ALF — 78.104.145.4 SOA SERIAL 1716670315 vs. 153.5.81.140 SOA SERIAL 1716646015. Difference of 6 hours and 45 minutes. (Others showed other hours of differences too.)
BA. — 195.130.35.5 SOA SERIAL 1716649201 vs. 204.61.216.117 and 95.130.35.3 SOA SERIAL 1716638401. Difference of four hours.

In addition to SOA SERIAL comparisons, we also did experiments looking at DNSSEC signature expiration and inception times using same key tags. We saw 73 TLDs that had over 30 second differences in the SOA EXPIRATION times or for SOA INCEPTION times. While we saw anomalies, we did not yet decide if some nameservers are really behind or if there is any significance. Here are examples:

BT. — signature times were 339 days behind and expired. But it also has current RRSIGs for the same key tags and SOA SERIALs currently updated.
BE. — had RRSIG EXPIRATIONS that were slightly over 25 hours different (20240615184338 vs. 20240616195619). But the RRSIG INCEPTION times were only almost 11 minutes different. Maybe one of the signers had slighly different signature life expectancy?
BOND. — had at least four anomalies ranging from 305 second to 841 second differences for RRSIG INCEPTIONs and 17643 to 77787 second differences for RRSIG EXPIRATION times. It appears each nameserver is independent as the SOA SERIAL numbers use Unix epoch times which are just five seconds before the RRSIG (covering same SOA) INCEPTION times.
ARPA. — Some of the ARPA servers were 12 hours behind for expiration and inception times, but the SOA only incremented by 1. This wasn't reproduced nor seen in previous runs, so likely our analyzer saw it right on some 12 hour schedule. (Note the RRSIG EXPIRATION and INCEPTION times for ARPA are all at zero minute and zero second but our tests were at minute 27 from seconds 12 to 14.)

What techniques are you using to identify zones that are not getting updated?

Our DNS Institute DNS Analyzer also identified many other errors with TLDs, such as ns1.teleinfo.cn. (2402:7d80::1) for ANQUAN., SHOUJI., XIHUAN., XN--3DS443G., XN--KPUT3I., XN--RHQV96G., XN--VUQ861B.. and YUN. and b.dns-servers.vn. (203.119.73.105) for VN. nameservers responding with a source address that was different from the queried destination address. We also saw expired DNSSEC signatures for XN--FZC2C9E2C., XN--MGBAYH7GPA., and XN--XKC2AL3HYE2A. Again we saw 67 unique failure types.