Add support for QUIC and DNS over QUIC/DoQ (RFC 9250)
This section is very likely to be changed/updated/expanded in the future.
Overview
One of the relatively recent additions of transport protocols for DNS is QUIC (DoQ, covered by RFC 9250) and HTTP/3 (DoH3), which also works on top of QUIC.
We need a generic implementation of the QUIC protocol in BIND's codebase to proceed with these new transports.
QUIC is a sophisticated transport that works on top of UDP and uses encryption on top of TLSv1.3. It is covered by multiple RFCs and is being actively worked on. The list of RFCs includes the following:
- https://www.rfc-editor.org/rfc/rfc9250.html
- https://www.rfc-editor.org/rfc/rfc8999.html
- https://www.rfc-editor.org/rfc/rfc9000.html
- https://www.rfc-editor.org/rfc/rfc9001.html
- https://www.rfc-editor.org/rfc/rfc9002.html
- https://www.rfc-editor.org/rfc/rfc9369.html
The protocol includes a lot of functionality, resembling the one from HTTP/2. Most notably, it is multiple uni- or bi-directional streams per connection. This aspect might have been influenced by the need to carry a new version of HTTP protocol (HTTP/3), which uses the multi-stream nature of QUIC instead of protocol-specific multiplexing found in HTTP/2.
Each bi-directional stream (in which we are the most interested, as these are used for DoQ) from the point of view of the higher-level code acts similarly to a TCP connection, though in the case of DNS, no more than one request/query per stream is allowed. That means that DNS pipelining is achieved by relying on the multistream nature of QUIC (again, similarly to HTTP/2). The thing is that each stream (being, effectively, a separate "connection") shares TLS parameters with others.
Another important aspect of QUIC is client connection end-point migration. That is, a client may change its location (= IP address and UDP port) while keeping the connection active. That functionality requires complete virtualisation of networking connections, which are now being identified not by IP address and port combination but by abstract connection identifiers (connection IDs). That brings a notion of a "connection path" into the picture as well as a procedure of "path migration" for a client.
Though that is my personal opinion, QUIC seems to be at least partially a result of large companies' experience of running user-space TCP/IP stacks on scale - when TCP/IP stack is implemented as a user-space library in an application which "directly" uses a dedicated network card. Indeed, in the case of QUIC, many things that we expect in-kernel TCP/IP implementation to do are brought under the control of a user-space application, including, but not limited to, such intricacies as congestion control. In this case, UDP can be seen as a portable kernel interface to the network card with the additional advantage of making it possible to run multiple applications simultaneously using a network card (which is not the case when using user-space TCP/IP stacks).
QUIC is often described as a replacement for the TCP protocol. One of the authors of "Computer Networks: A Systems Approach" argues that it is, in fact, an addition to the internet protocols suite, which is meant to implement a missing paradigm - a basis for Remote Procedure Calls (RPC) protocols. That is, it might be considered a replacement of the TCP only in the cases when TCP was used due to a lack of a better alternative.
It should be noted that while QUIC is meant to be used as a universal transport for DNS, it, unlike HTTP/2 (DoH), can be used for zone transfers as well. It is not always guaranteed that it will provide a significant performance boost. In fact, it might require more traffic in some cases, as even the initial QUIC message should be no less than 1200 bytes, which is a lot by common DNS measurements. However, it might compensate for that by lower latency due to 0-RTT support. Also, it seems to be more like a client-oriented protocol due to the ability to migrate client connections to new addresses (which is great for portable mobile devices). That being said, it is not clear how beneficial it would be to use it for server-to-server communications for things like zone transfers: servers do not change addresses often, and the protocol itself is more verbose than, e.g. DoT, so I fail to see the immediate benefits for this case (although it is standardised).
The State of Open Source QUIC Implementations and the Great OpenSSL Schism
As it was stated before, QUIC includes TLSv1.3. Thus, most implementations decided to dedicate TLS-specific functionality to base the crypto-related bits to OpenSSL and its derivatives, which makes sense as these libraries are widely deployed and used. However, initially, these libraries lacked QUIC-specific parts in their TLS implementations.
As the early QUIC adopters were also QUIC implementors, they forked OpenSSL and added the missing bits. The related changes to the API ended up in multiple OpenSSL forks, namely QuicTLS (maintained by Akamai and Microsoft), LibreSSL (maintained by OpenBSD), and BoringSSL (maintained by Google). These libraries only implement the low-level TLS 1.3 bits related to QUIC but do not have any internal QUIC implementations, as their early adopters developed their own.
For quite some time, the original OpenSSL had a merge request opened to implement the same API and remain mostly compatible with its forks, as was the case for a long time. However, OpenSSL authors decided to provide a high-level implementation of QUIC of their own making and eventually closed the MR. That caused a lot of drama, about which you can read here or here as well as in other places. Regarding OpenSSL, it is worth keeping multiple things in mind.
Firstly, OpenSSL's implementation is not ready yet, as it includes only minimal client-side implementation starting from the version of OpenSSL v3.2. The server-side support was planned for 3.3 (April 2024) according to the project's roadmap, but eventually, it was moved to 3.4 (October 2024), as there are many not completed tasks, some of them are marked as "Epic". Even after that, we most likely will have only the most basic (Minimal Working Product) implementation that is not as battle-tested as some others and with missing features.
Secondly, OpenSSL does not allow the use of third-party implementations of QUIC: with OpenSSL, the only option is to use the internal QUIC implementation. That was noted by Tatsuhiro Tsujikawa, the principal author of nghttp2/ngtcp2/nghttp3.
As a result of these decisions, most QUIC implementations chose to depend on QuicTLS and other forks that provide similar API. That list includes MS-QUIC, Quiche, Chromium QUIC, as well as internal (=not exposed as a redistributable library as it is very specific) implementation in HAProxy. Probably, most other libraries are likely doing the same.
There are notable exceptions to this, though.
Firstly, NGINX does not depend on a fork-related functionality to implement QUIC support, nor does it depend on the OpenSSL implementation of QUIC. One could get this impression after reading the announcement of this functionality in NGINX, which discusses that they implement only OpenSSL compatibility layer in order to remain compatible with both OpenSSL and its now numerous forks. It should be noted, though, that in this particular case, whoever wrote the announcement was modest, as, in fact, NGINX includes their own in-house QUIC implementation. And yes, it seems that they managed to do it without relying on any QUIC-related functionality in OpenSSL or its forks (like QuicTLS, LibreSSL, and BoringTLS). Their code seems to work with basically any OpenSSL-like library with TLSv1.3 support.
Secondly, there is ngtcp2, which itself does not depend on any cryptographic library per se. The library itself may use one of the provided backends implemented on top of QuicTLS, GnuTLS, PicoTLS or BoringSSL and implemented as separate libraries, but it does not have to, as an application can provide a custom implementation of the ngtcp2 crypto API as described in the programmer's guide. The library itself is very complete and seems to implement all the intricacies of the QUIC protocol, unlike the OpenSSL's implementation at the time of writing (and likely, for quiet some time after that).
At this point, it is clear that as far as QUIC support goes, the OpenSSL and its numerous forks will remain incompatible. That is not a technical decision and, thus, hard to resolve.
Implementing QUIC in BIND
For the reasons given above, I think that we should choose the ngtcp2 library as a basis for our QUIC transport. Additionally, it is much more mature than the implementation that OpenSSL will initially provide as and has been considered stable for a while (1.0.0 was released Oct 15, 2023). Moreover, before implementing the final IETF QUIC, it implemented a number of drafts, so it is safe to say that the authors have been tracking the development of the protocol very closely. Considering that the most recent RFCs have been implemented as well (like QUIC version 2, which, in fact, is a minor update of the protocol), it appears to still be the case. We can pair it with our own crypto API implementation inspired by the code from NGINX and currently provided crypto libraries. That will ensure that BIND still remains compatible with other OpenSSL forks, in particular on the platforms that use them by default (like OpenBSD). That should give us the flexibility of NGINX without the burden of maintaining our own QUIC implementation, as I cannot justify doing that as much as depending on any of the numerous OpenSSL forks.
I cannot justify waiting for OpenSSL to have their implementation completed either, as it is not clear how good the initial implementation will be, and I am not sure if the higher-level API they intend to provide is going to be the best fit for us. Ngtcp2 already looks more promising in that regard, not to mention that we have, in my opinion, a very good experience of using nghttp2 from the same author. Also, it is worth noting that at this stage, we have an internal subsystem for managing TLS contexts used by DoT and DoH, so it is very desirable to use it for QUIC as well and having our own ngtcp2 crypto API implementation should make it possible. We can use the code from NGINX and existing ngtcp2 crypto libraries as examples.
Apart from choosing the library which will serve as the foundation of our QUIC implementation, there are many other problems to solve, which are no less challenging and will require some thinking and trial and error, so it is better for us not to wait for OpenSSL so that we will have more time to iron out our QUIC-related code before the new release. That is, the code related to connections and stream management and, ideally, connection migration is the most challenging, in my opinion, though the choice of the library will affect its structure for sure.
I think that it would be best to use and scale the experience of structuring the transport code in a way similar to Stream DNS and PROXYv2. It is very desirable as it allows testing without direct dependence on the networking code. By such a design, I mean implementing the most important parts of the QUIC code as a black box to which we pass data, and it calls the necessary callbacks when required.
On the highest level, bi-directional QUIC streams (in which we are interested the most for now) should map well to Stream DNS or a very similar transport (a QUIC Stream based on isc_dnsstream_assembler_t
).
QUIC has some newer characteristics not present in other transports, like an ability by both end-points to create new streams at any moment as much as a concept of a generic multiplexed transport itself. These things are likely to be resolved in a manner similar to HTTP/2 - using a virtual connection per stream. Another thing to mention is connection migration. Our code is not ready for that (and, likely, never will), so we should set a QUIC Stream end-point information at the moment of creating and then never update that (from the point of view of the higher level code for compatibility reasons - in fact, we still can do the actual connection migration with all the open streams).
So, in short, it seems that we should attempt to use ngtcp2 with custom crypto API implementation first. The initial plan was to combine the client-side support for QUIC from OpenSSL and combine it with "OpenSSL Compatibility Layer" (as they named it) from NGINX (which turned out to be a complete internal only in-house QUIC implementation), but due to the reasons given above, that will not work as it is not possible to combine QUIC-related functionality in OpenSSL with any third-party QUIC implementation at all (and likely never will). Ngtcp2 is the only crypto library agnostic implementation of QUIC at the moment.
I would rather choose to depend on OpenSSL for our QUIC implementation as a backup plan.
Bridging the Gap: How We Could Make an Ngtcp2 Crypto API Implementation
Let's see how we can proceed with providing our own ngtcp2 crypto API implementation.
Ngtcp2 Crypto API Implementations Structure
As it was noted above, in order to work, ngtcp2 requires a crypto API implementation. The project provides multiple of them for the crypto libraries that have explicit support for QUIC. The list of these libraries includes (at the time of writing) QuicTLS, BoringSSL, GnuTLS, WolfSSL, and PicoTLS.
Each crypto API implementation library consists of two parts.
Firstly, it is a high-level, shared part. Among other things, it includes default implementations of all callbacks used by ngtcp2, and some important API calls, most notably ngtcp2_crypto_derive_and_install_rx_key()
and ngtcp2_crypto_derive_and_install_tx_key()
, which are mentioned in the ngtcp2 programmers guide.
Secondly, a low-level part that provides a foundation for the functionality of the shared part. There is an implementation for each of the above mentioned supported crypto libraries.
The shared part and a low-level part linked together is an ngtcp2 crypto API implementation.
Of these implementations, the most interesting for us is the one for QuicTLS and, to a somewhat lesser extent, BoringSSL, as both are OpenSSL forks. In fact, we can use most of the code from there without much adaptation, as QuicTLS is essentially OpenSSL + QUIC-related API (it is regularly rebased on top of the "mainline" OpenSSL).
The QuicTLS crypto API implementation does not use many QUIC-related API calls. I have identified only the following:
-
SSL_CTX_set_quic_method()
- we would be better using a very similar SSL_set_quic_method(); -
SSL_provide_quic_data()
; -
SSL_set_quic_transport_params()
/SSL_get_peer_quic_transport_params()
; -
SSL_process_quic_post_handshake()
- it passes any handshake-related data after the handshake was completed.
There is one problem with how the crypto API libraries are structured - as there is no way to use only the shared part without depending on one of the above-mentioned crypto libraries (which would not work for us anyway).
So, in our ngtcp2 crypto API implementation, we would need to provide a replacement for the shared part - at least for the callbacks and ngtcp2_crypto_derive_and_install_rx_key()
/ ngtcp2_crypto_derive_and_install_tx_key()
. That is doable but very unfortunate, as it includes a lot of crypto-related things, not to mention that it is extra code to maintain.
Regarding the missing QUIC API in OpenSSL, there is a solution from NGINX which might work for us as well.
NGINX OpenSSL Compatibility Layer
As noted above, NGINX includes a compatibility layer with NGINX as a part of its QUIC implementation. It includes the implementations of the following functions:
SSL_set_quic_method()
SSL_provide_quic_data()
-
SSL_set_quic_transport_params()
/SSL_get_peer_quic_transport_params()
NGINX's OpenSSL compatibility layer clearly provides an internal implementation of missing parts of BoringSSL QUIC API, which is not that different from QuicTLS API.
One missing thing is SSL_process_quic_post_handshake()
, it seems that it can be implemented with SSL_read(_ex)()
as it handles handshake-related data. HAProxy's compatibility layer uses a dummy stub for this.
It is worth noting that NGINX is a server application, so the provided QUIC API implementation might need some adjustments to work for client-side code.
There is an article from NGINX authors that describes the inner working of their compatibility code in detail. It is worth noting that HAProxy is using a similar approach as well.
At this point, it seems that despite some limitations, it is possible to provide a portable ngtcp2 crypto API implementation.
Source-code References:
- NGINX's OpenSSL QUIC compatibility layer: implementation, definitions;
- NGINX's QUIC/TLS Crypto Utilities code: implementation, definitions;
- HAProxy's OpenSSL QUIC compatibility layer (related to the similar code in NGINX): implementation, public definitions, internal definitions;
- HAProxy's QUIC/TLS Crypto Utilities code: implementation, public definitions, internal definitions;
- ngtcp2's Crypto API Library: public interface definitions, internal interface definitions, portable internal interface high-level code, QuicTLS-specific internal interface low-level code;
- QuicTLS QUIC integration API implementation: high-level code.
-
BoringSSL's
SSL_process_quic_post_handshake()
implementation.