Flag Day, reloaded
Further attempts to compel laggards to update obsolete Domain Name Server (DNS) infrastructure by dropping support for cumbersome workarounds are set to return next year.
Results of the first DNS Flag Day, which took place in February, were discussed at last week’s RIPE 78 security conference in Reykjavik, Iceland.
During the exercise, DNS vendors including Bind 9 and PowerDNS, along with large DNS resolver operators (Google, Cloudflare, et al), disabled workarounds for broken EDNS implementations.
In the weeks running up to Flag Day, there was an estimated breakage of 5.7% of all DNS servers.
Two large DNS operators were responsible for two-thirds of this breakage, it was found. This might sound bad, but no other significant breakage was reported on Flag Day itself, when core vendors behind the ‘internet’s address book’ turned off support for the workarounds.
Overall, the exercise was hailed success since the “pressure generated compelled the operators to fix their systems”, according to specialist DNS vendor Men & Mice.
Outdated DNS servers were updated, and incorrectly configured firewalls were fixed, resulting in the establishment of a more robust infrastructure.
Remaining problematic domains were largely unused (or parked), so that support lines set up in the run-up to DNS Flag Day stayed silent.
A amp-video recorded presentation at RIPE 78 by Petr Špaček from CZ.NIC and Ondřej Surý from ISC offered a summary of the outcome – and lessons learned – from the first edition of DNS Flag Day, along with draft plans for a 2020 edition.
Slides from the presentation can be found here (PDF).
Seconds out, round two
Because of the widely (but not universally) acknowledged success of the first DNS Flag Day, a second event is being planned.
Although no firm date has been set as yet, the next installment may well take place next February, this time with a focus on pushing towards tighter compliance to established standards for supporting DNS transport over the Transmission Control Protocol (DNS-over-TCP).
Some DNS responses are too big to be handled by the User Datagram Protocol (UDP), resulting in fragmentation of IP packets. This is a known problem in some networks that can present a security risk.
UDP works well for small responses, so what needs to happen is that resolvers must support fallback from UDP to TCP – something already achieved through standards-compliant software, but not universally applied.
“The [next] round of DNS Flag Day is being planned right now, with focus on operational and security problems in DNS caused by Internet Protocol fragmentation,” according to a statement on the official DNS Flag Day website.
In order for everything to work even more smoothly, the industry needs to move decisively towards established best practice. To that end, authoritative servers and DNS resolvers must be able to operate over TCP, in addition to UDP.
Most operators already adhere to best practice, but a small number of rogue parties are still causing headaches – hence the plan to oblige them to get in line or face negative consequences themselves.
Czech-based DDoS mitigation firm Qrator warns that, starting perhaps as early as February 2020, DNS servers that don’t support the processing of both UDP-based and TCP-based DNS queries may stop working.
Analysis of 34 million domains out of 59 TLDs “makes it evident that the requirement to use TCP leads to problems for approximately 7% of domains”, according to Qrator.
The company said that this is comparable to the situation in November 2018 – three months before the February 2019 DNS Flag Day – where 5.68% of tested sites still had EDNS issues.
On the client (stub resolver) side, DNS-over-TCP has been supported almost everywhere, including Windows, for some time, but there’s plenty of work to be done on the server side.
Authoritative servers of 10 companies account for the vast majority of problems in this area, with the Chinese ISP Hichina alone accounting for two-thirds of all known issues, according to Qrator.
Other Chinese providers AliDNS and Xinnet also present a problem. “Half of the names on the list also had issues with EDNS in November 2018, though resolved them successfully just about the time,” Qrator reports.
The idea for the second DNS Flag Day – much like the first – is to disable cumbersome workarounds and so force these operators and others into applying upgrades, rather than letting them rely on others for a helping hand.
Fragging hell
DNS-over-TCP offers security benefits to DNS-over-UDP, such as improved DDoS mitigation and DNSSEC support, as well as dealing with the IP fragmentation issue.
Cricket Liu, chief DNS architect at Infoblox, told The Daily Swig: “The point isn’t really that resolvers must move away from UDP; it’s that DNS servers MUST support TCP.
“This has basically been required for some time, since… some parts of DNS (e.g., DNSSEC, Response Rate Limiting) just don’t work with DNS servers that don’t also support TCP.”
Dan Kaminsky, chief scientist at whiteops.com and a security researcher who has done landmark work on DNS security, backed up the assessment that DNS servers needed to support TCP for multiple reasons.
“DNSSEC responses can be large, and IP fragmentation does not work very well,” Kaminsky told The Daily Swig.
“Also, there are DNS defences that depend on the protocol being run over TCP, which does slow things down some.”
“By some perspectives, this does end up pushing us away from DNS over UDP,” he added. “But there’s quite a bit going on in DNS-space, as people explore other protocols for synchronizing client nodes with directory services on the cloud edge.”
The philosophy behind Flag Day is that the organizers, ISPs, and DNS operators that make up the DNS community should no longer suffer the inconvenience and cost for workarounds to benefit a couple dozen companies that are not updating their servers.
Infobox’s Liu concluded: “I like the idea of having a sequence of DNS flag days to help ensure different aspects of DNS’s interoperability.”
RELATED Interview: ‘Middle-aged’ DNS tech still has legs to kick on