BIND issueshttps://gitlab.isc.org/isc-projects/bind9/-/issues2024-02-28T10:46:36Zhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4606"dry-run" mode to help with dnssec-policy migration2024-02-28T10:46:36ZCarsten Strotmann"dry-run" mode to help with dnssec-policy migration### Description
For some users of BIND 9, esp. people are part time DNS admins only, migrating from manual DNSSEC key management with "auto-dnssec maintain;" towards "dnssec-policy" is difficult.
While the documentation provided by ISC...### Description
For some users of BIND 9, esp. people are part time DNS admins only, migrating from manual DNSSEC key management with "auto-dnssec maintain;" towards "dnssec-policy" is difficult.
While the documentation provided by ISC is good, there is currently no way to "verify" the new "dnssec-policy" configuration before enabling it. Experience has shown (in DNS training classes, but also in real world deployments) that there are many things that can go wrong:
- differences in the DNSSEC key configuration (old vs. new)
- file system permissions on the old key material
- file system location of the old key material
- issues with the time-events stored in the old key material
Going online with a slightly wrong configuration can cause an immediate key rollover, which might break the zone. Recovering from this situation is possible, but requires good knowledge of BIND 9 DNSSEC workings
### Request
Provide a "dnssec-policy dry-run" mode, where BIND 9 will log the next steps in the automatic DNSSEC management to the log files (e.g. category "DNSSEC"), but will not execute any changes to the DNSSEC signed zone or the key material. This will enable the user to test drive the new "dnssec-policy" to see if it will act as expected.
Admins can create a configuration with "dry-run" mode enabled, check the logfiles, and if the actions in the log-file match the expectations, the "dry-run" mode can be removed and the new configuration will become active.
### Links / referencesMatthijs Mekkingmatthijs@isc.orgMatthijs Mekkingmatthijs@isc.orghttps://gitlab.isc.org/isc-projects/bind9/-/issues/3579Add more debugging messages for network-level events2022-10-05T09:37:10ZMichał KępieńAdd more debugging messages for network-level eventsThe network manager code is currently not particularly verbose when it
comes to logging debug messages:
$ git grep isc_log_write lib/isc/netmgr/ | wc -l
9
In particular, this applies to "positive" events (non-errors), like
esta...The network manager code is currently not particularly verbose when it
comes to logging debug messages:
$ git grep isc_log_write lib/isc/netmgr/ | wc -l
9
In particular, this applies to "positive" events (non-errors), like
establishing a connection, correctly receiving data from a socket, etc.
This applies to both non-encrypted transports (like TCP) and encrypted
ones.
The problem for me as an administrator/troubleshooter is that I have
very limited visibility into what BIND 9 "sees" on its side of things
when things go south. For example, I recently experimented with getting
`systemd-resolved` to talk to `named` over DNS-over-TLS; the former
reported, well, *errors*, and I could not get a grasp of the point at
which things are failing without resorting to Wireshark ("Is it the TCP
connection on port 853 itself? Or maybe the TLS session negotiation?
Or is that part okay and it is something about the data that
`system-resolved` sends inside a properly-established TLS session that
makes `named` complain?" etc.)
I am opening this issue so that it can serve as a public acknowledgment
of this being a known deficiency. It would be nice to do something
about it in the log run. Obviously there will have to be performance
trade-offs, but I think even hiding certain log messages behind a
build-time switch is fine as long as there is *some* way of getting
`named` to become more talkative logging-wise when it comes to
network-level events.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4532An option to not have bind9/dnssec-settime (possibly other tools) reset permi...2024-01-16T20:30:36ZDan MahoneyAn option to not have bind9/dnssec-settime (possibly other tools) reset permissions on a .private file.### Description
The `named` process and `dnssec-settime` (perhaps other tools) will take it upon themselves to change the permissions of a private key on certain changes.
However, we track our key-directory (and other configs) using gi...### Description
The `named` process and `dnssec-settime` (perhaps other tools) will take it upon themselves to change the permissions of a private key on certain changes.
However, we track our key-directory (and other configs) using git, with a group-shared repository.
Typical permissions on .private files are bind:bind with mode 660, but because a normal user (in the bind group) diffs/commits/pushes the repository, these keys can also be user:bind mode 660.
(Noting as well that our tooling is not more comfortable running git tasks as root, complaining of other permissions issues. Also, the less we can do as root, the better.)
With bind's usual permissions model, one cannot do a git diff/git log if the file is owned by bind. If the file is owned by user:bind, bind loses access to it on the permissions change.
Changing the umask under which the process runs doesn't seem to fix this, we tried.
Running a periodic cron job to fix this is a possible workaround, but feels like it shouldn't be necessary.
### Request
For command line tools, an option to not do this.
For `named, an `options` statement that lets us turn this off.
Both retaining the current behavior by default.
### Links / referencesNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4345Debug messages logging network traffic only include the address of one peer2024-02-24T08:19:42ZMichał KępieńDebug messages logging network traffic only include the address of one peerEven with `-d 99` used on the command line, `named` only logs lines
like:
28-Sep-2023 14:31:23.212 sending packet to 2001:503:ba3e::2:30#53
or:
28-Sep-2023 14:31:23.232 received packet from 2001:503:ba3e::2:30#53
However, net...Even with `-d 99` used on the command line, `named` only logs lines
like:
28-Sep-2023 14:31:23.212 sending packet to 2001:503:ba3e::2:30#53
or:
28-Sep-2023 14:31:23.232 received packet from 2001:503:ba3e::2:30#53
However, network traffic is always sent **from** one socket **to**
another. The currently available debug messages do not include the
sender's address (first example) or the receiver's address (second
example). As a result, just bumping up the log level is often not
enough to diagnose certain issues and a network traffic sniffer has to
be used in order to learn the details of the packets being exchanged.
This lack of detail sometimes also makes debugging system test issues
harder than it has to be. With multiple tests being run in parallel,
knowing the exact addresses and ports that were used by each running
`named` instance is crucial for determining whether a test failure was
caused by an unexpected interaction between tests or not. (Such issues
happened more than once in the past, particularly when network code
and/or the system test framework were being worked on.)
Debug messages logging network traffic should be extended to include
information about both sides of each communication channel.
While this issue is technically only tangential to #4344, having
detailed network-level information available would greatly improve the
benefits of the feature proposed here.May 2024 (9.18.27, 9.18.27-S1, 9.19.24)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4344Enable extraction of exact local socket addresses2024-02-24T08:19:47ZMichał KępieńEnable extraction of exact local socket addressesThe Network Manager API is currently unable to expose the exact
address/port that a local wildcard/TCP socket is bound to. This limits
the level of detail available to all sorts of traffic-logging code
(debug messages, dnstap, etc.)
Th...The Network Manager API is currently unable to expose the exact
address/port that a local wildcard/TCP socket is bound to. This limits
the level of detail available to all sorts of traffic-logging code
(debug messages, dnstap, etc.)
This has been previously discussed (in dnstap context) in #3143. Back
then, it quickly [emerged][1] that extracting the exact address that a
local wildcard/TCP socket is bound to requires issuing a system call.
Unfortunately, the function that would be responsible for doing this is
[called on a hot path][2]. After running some performance tests, it
[became obvious][3] that doing that unconditionally is a non-starter
performance-wise. The proposal was scrapped and replaced with a [note
in documentation](!6472).
However, the problem persists and limits the capabilities of not just
dnstap, but also logging code. In some cases, more detailed logging is
preferred over raw performance and there should be some way for the user
to express their preference in that regard.
[1]: https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5816#note_272336
[2]: https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5816#note_272404
[3]: https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5816#note_272407May 2024 (9.18.27, 9.18.27-S1, 9.19.24)Michał KępieńMichał Kępieńhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4303dnstap logging based on rcode2023-09-13T14:09:41ZPetr Špačekpspacek@isc.orgdnstap logging based on rcode### Motivation
When FORMERR or SERVFAIL happen in the middle of resolution, we don't have exact information what we have sent and what exactly came back. We have to guess and attempt to reproduce the problem with `dig` or other tools.
#...### Motivation
When FORMERR or SERVFAIL happen in the middle of resolution, we don't have exact information what we have sent and what exactly came back. We have to guess and attempt to reproduce the problem with `dig` or other tools.
### Request
Add parameter to `dnstap` statement which would allow logging just selected RCODEs. Presumably FORMERR and/or SERVFAIL. I imagine that this could be so low-touch that it could be running in production indefinitely (as a cyclic buffer).
### Links / referencesNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4239Add dnstap-output support for tcp2023-08-02T08:00:29ZMr BenAdd dnstap-output support for tcp### Description
goals:Add dnstap-output support for tcp
### Request
###
Now the output of dnstap only supports file and unix domain socket to generate logs. I expect to output logs in the form of tcp, so that the log server can be on ...### Description
goals:Add dnstap-output support for tcp
### Request
###
Now the output of dnstap only supports file and unix domain socket to generate logs. I expect to output logs in the form of tcp, so that the log server can be on other hosts. Reduce the overhead of the dns server itself.
The reason for such a requirement is that in the production environment, even if the dns server itself generates logs, they will still be output to other places for analysis and detection through tcp/udp. It is better to output directly to other hosts.
###
### Links / references
###
Like coredns, it supports tcp output log.
dnstap /tmp/dnstap.sock
dnstap unix:///tmp/dnstap.sock full
dnstap tcp://127.0.0.1:6000 full
dnstap tcp://example.com:6000 full
###Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4233dig: add +human option2023-08-08T13:04:27ZJulia Evansdig: add +human option### Description
I've spent a lot of time explaining `dig` to newcomers to DNS over the years, and I've found that they generally find `dig`'s output format to be very inscrutable. Of course, there's always `+short` or `+noall +answer` f...### Description
I've spent a lot of time explaining `dig` to newcomers to DNS over the years, and I've found that they generally find `dig`'s output format to be very inscrutable. Of course, there's always `+short` or `+noall +answer` for a terser output, but I generally want to explain more advanced DNS concepts to people (like glue records or SOA records for example), and for that you do need the full output.
Some specific things that I find confusing in dig's default output: ([example output here](https://gist.github.com/jvns/2d252e3f54a86ee6b2ea001e733a8e78))
* There's some ASCII art decoration (`<<>> DiG 9.10.6 <<>>`, `->>HEADER<<-`) that feels very ad hoc and it's hard to tell initially if those symbols are supposed to have some technical meaning. (why is `->>HEADER<--` styled like that, but not `OPT PSEUDOSECTION`?)
* the header is split across 2 lines, and it's not completely clear that the second line is also part of the header
* overall, it's not obvious which pieces of information are part of the DNS response itself and which aren't. For example, is `global options: +cmd` part of the DNS response? (of course it isn't, I don't think that's immediately obvious)
* There's no newline between `OPT PSEUDOSECTION` and `QUESTION SECTION`, but there is a newline between `QUESTION SECTION` and `ANSWER SECTION`. There seems to be no reason for that inconsistency.
* In `MSG SIZE rcvd: 56`, why are there two spaces between `SIZE` and `rcvd`? Are there more fields in `MSG SIZE`? (I checked the source code and the answer is no, the code says `printf(";; MSG SIZE rcvd: %u\n", bytes);`, so it seems like this is just an ad hoc choice)
* The `;;` prefix is confusing to many people. I realize it's because `;` it's the comment character in a zone file, but I personally do not use `bind` or zone files and most DNS users I talk to don't either: they either use web-based admin consoles to administrate their DNS records or do it through an API like Route 53. So the `;` character isn't familiar.
These might sound a little nitpicky -- each of these things on its own is pretty minor, and of course most users of dig learn to ignore them. But taken together I've found that newcomers are often misled into thinking that DNS responses are much more complicated than they actually are, which is really unfortunate.
I find the way Wireshark displays DNS packets to be much more clear ([screenshot](https://jvns.ca/images/dns-wireshark.png)), even though they're both working at around the same level of detail.
### Request
I realize that `dig`'s default output format needs to remain relatively stable because people parse it in scripts. But would ISC be open to adding a `+human` (or something) option to dig that's designed to be more intuitive for newcomers to dig? Similarly to how `du` has an `-h` option.
I'm imagining something like this:
```
$ dig +human example.com
Received response from 192.168.1.1:53 (UDP), 68 bytes in 16ms
HEADER:
status: NOERROR
opcode: QUERY
id: 15451
flags: qr rd ra
records: QUESTION: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
OPT PSEUDOSECTION:
EDNS: version: 0, flags: None, udp: 4096
QUESTION SECTION:
example.com. IN A
ANSWER SECTION:
example.com. 78709 IN A 93.184.216.34
```
I think the important thing is to make it easy for a newcomer to see at a glance that there are 4 parts to this DNS response (the header, the EDNS record, the question, and the answer)
I created a very rough proof of concept at https://github.com/jvns/dig-pretty that parses dig's `+yaml` format and outputs a format like what I suggested above, with a tiny bit of syntax highlighting for the DNS status code.
### Alternatives I've considered
* `+short` or `+noall +answer` are great for a lot of use cases, but as I mentioned above, they don't work for more advanced usage like looking at the `SOA` record on a `NXDOMAIN` response.
* We already have `+yaml`, but I find `+yaml` to be extremely verbose (the output for `dig +yaml example.com` doesn't fit in my terminal window), and it's really a machine format and not a human format.
* There are also alternative DNS tools (like `dog`) that aim to be more user friendly, but in general I've found those tools to be missing important features that `dig` has.
Thanks for considering this! I love dig and would love to see it become a little more approachable.
### Links / referencesNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3896Add the TLS SNI +[no]tls-sni option to dig for DoT and DoH2023-11-02T16:30:30ZAlexey ChernyakAdd the TLS SNI +[no]tls-sni option to dig for DoT and DoH### Description
Currently, when doing DoT or DoH lookups, `dig` does not include the Server Name Indication (SNI) extension during its TLS handshake `ClientHello` request.
This makes dig unusable as a DoT or DoH client with TLS proxies...### Description
Currently, when doing DoT or DoH lookups, `dig` does not include the Server Name Indication (SNI) extension during its TLS handshake `ClientHello` request.
This makes dig unusable as a DoT or DoH client with TLS proxies and HTTPS proxies and DoT/DoH servers that implement:
* SNI based routing of requests to the correct backends as part of name-based virtual hosting, and/or
* SNI filtering based access control.
Currently `dig` also has very poor handling of the fatal-level SNI-related TLS handshake `Alert` response from server.
Example of such TLS Alert server response:
```
Transport Layer Security
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unrecognized Name)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Unrecognized Name (112)
```
Current dig behaviour is as follows when the server aborts the handshake by sending such a fatal-level `unrecognized_name(112)` alert:
```
~ $ dig +https @dns.example.com example.com A; echo $?
;; Connection to 192.0.2.53#443(192.0.2.53) for example.com failed: TLS error.
;; Connection to 192.0.2.53#443(192.0.2.53) for example.com failed: TLS error.
;; Connection to 192.0.2.53#443(192.0.2.53) for example.com failed: TLS error.
9
~ $ dig +tls @dns.example.com example.com A; echo $?
0
~ $
```
There is no dependency on implementing SNI support in `named` (which also would be nice for ACLs and for multiple views).
While `named` can be placed behind a SNI-capable proxy, there's no workarounds for `dig` other than replacing it with a SNI-capable client such as `kdig`.
### Request
Add `+[no]tls-sni=STR` option to `dig` similar to the [option](https://www.knot-dns.cz/docs/3.2/html/man_kdig.html#options) in `kdig`.
Expected behaviour is something roughly along the lines of:
```
if request is DoT or DoH
if +tls-sni option is specified
SNIHostName = option value
else if +notls-sni option is specified
SNIHostName = NULL
else if @server argument is a hostname
SNIHostName = server hostname FQDN
else if +tls-hostname option is specified
SNIHostName = option value
else
SNIHostName = NULL
if SNIHostName
remove trailing FQDN dot from SNIHostName if specified
Do IDN format conversion to ASCII if necessary
validate SNIHostName against SNI HostName format specification
if SNIHostName format is valid
include SNIHostName as a SNI in TLS ClientHello
include SNIHostName in the first line of dig query output?
else
exit with an error
else
Don't include SNI in TLS ClientHello
if ServerHello response includes "server_name" extension
include SNIHostName in SERVER line in dig query output
```
Better error handling of the fatal-level `unrecognized_name(112)` alert TLS responses from server:
* Start returning an error during DoT instead of a silent success.
* Consider a more descriptive error message than the current `TLS error`:
```
if a <SNI> value was included in ClientHello
Error message something along the lines of: `'<SNI>' SNI is unrecognised by the server`
else
Error message something along the lines of: `SNI is required by the server`
```
### Links / references
For the most recent background and guidance on SNI see [Section 3.7 of RFC 9325](https://www.rfc-editor.org/rfc/rfc9325#name-server-name-indication-sni).
For the latest SNI specification defining SNI `HostName` format and client/server behaviour see [Section 3 of RFC 6066](https://www.rfc-editor.org/rfc/rfc6066.html#section-3).
When implementing, consider application of Internationalized Domain Names (IDN) in SNI as per [RFC 5890](https://www.rfc-editor.org/rfc/rfc5890) and TLS Encrypted Client Hello (ECH) as per the latest [draft](thttps://www.ietf.org/archive/id/draft-ietf-tls-esni-15.html).Not plannedArtem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3695Improvement: Including query time in dnstap CLIENT_RESPONSE messages2023-01-11T12:15:33ZBorja Marcos EA2EKHImprovement: Including query time in dnstap CLIENT_RESPONSE messages### Description
While the dnstap specification recommends including the query time for AUTH_RESPONSE, RESOLVER_RESPONSE and
CLIENT_RESPONSE dnstap messages, the latter is excluded.
Having the query time in CLIENT_RESPONSE dnstap messa...### Description
While the dnstap specification recommends including the query time for AUTH_RESPONSE, RESOLVER_RESPONSE and
CLIENT_RESPONSE dnstap messages, the latter is excluded.
Having the query time in CLIENT_RESPONSE dnstap messages would be very useful when using dnstap to keep track
of response times.
### Request
In lib/dns/dnstap.c (both for 9.16 and 9.18) the dns_dt_send function accepts the qtime and rtime parameters.
However, when building the dnstap message, CLIENT_RESPONSE messages are prevented from using the qtime parameter.
` dm.m.has_response_time_sec = 1;
dm.m.response_time_nsec = isc_time_nanoseconds(t);
dm.m.has_response_time_nsec = 1;
/*
* Types CR, RR, and FR can fall through and get the query
* time set as well. Any other response type, break.
*/
if (msgtype != DNS_DTTYPE_RR && msgtype != DNS_DTTYPE_FR
&& msgtype != DNS_DTTYPE_CR) { // << I HAVE ADDED THIS!
break;
}
FALLTHROUGH;
case DNS_DTTYPE_AQ:
case DNS_DTTYPE_CQ:
case DNS_DTTYPE_FQ:
case DNS_DTTYPE_RQ:
case DNS_DTTYPE_SQ:
case DNS_DTTYPE_TQ:
case DNS_DTTYPE_UQ:
if (qtime != NULL) {
t = qtime;
}
dm.m.query_time_sec = isc_time_seconds(t);
dm.m.has_query_time_sec = 1;
dm.m.query_time_nsec = isc_time_nanoseconds(t);
dm.m.has_query_time_nsec = 1;
break;
`
I have tried making the simple change shown above (so that qtime is considered for
CLIENT_RESPONSE messages as well) and it works both for 9.16.35 and 9.18.9.
The change looks safe enough (it won´t crash because if qtime is NULL t will contain a
timestamp obtained when dns_dt_send() is invoked) and at worst it would contain a false
qtime.
A more correct alternative would be to include it for CLIENT_RESPONSE messages only if qtime != NULL. But
I don´t know whether it can happen or all the calls to dns_dt_send() will contain qtime.
Also, is it possible for qtime to be missing for a CLIENT_RESPONSE but not for a RESOLVER_RESPONSE? Because for a RESOLVER_RESPONSE it would mean that query time in the dnstap message would contain the timestamp obtained in dns_dt_send() and, being probably
greater than the response time itself that would botch a time difference calculation.
### Links / referencesNot plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3464Histograms for timing and memory statistics2023-11-02T17:05:05ZTony FinchHistograms for timing and memory statisticsBIND needs to be able to record statistics covering a wide range of possible values (several decimal orders of magnitude):
* latency times, from submilliseond queries on the same LAN to multi-minute zone transfers
* memory usage, fo...BIND needs to be able to record statistics covering a wide range of possible values (several decimal orders of magnitude):
* latency times, from submilliseond queries on the same LAN to multi-minute zone transfers
* memory usage, for zones from a handful of records to tens of millions
* message sizes, from 64 bytes to 64 kilobytes
In this issue I'm outlining a possible design for a general-purpose histogram data structure,
that could be added to `libisc` for collecting statistics efficiently in several places in BIND.
## existing histograms in BIND
The statistics channel has histograms for request and response sizes, which use buckets that
are defined manually with some tediously repetitive code. These could be replaced by the
proposed self-tuning histograms, although the bucketing will be somewhat different.
## examples of general-purpose histograms
It's possible to record histograms of values covering a wide range, with bucket sizes chosen automatically to provide a particular level of accuracy (e.g. 1% or 10%), and without using more than a few KiB for each histogram. Existing examples are:
* [circllhist, Circonus log-linear histogram](https://github.com/openhistogram/libcircllhist),
aka [OpenHistogram](https://openhistogram.io/)
Uses decimal floating point with two digits of mantissa and a 1 byte exponent,
to record values with 1% accuracy.
* [DDSketch from DataDog](https://www.datadoghq.com/blog/engineering/computing-accurate-percentiles-with-ddsketch/)
Uses the floating-point logarithm to a base derived from the required accuracy, rounded to an integer to make a bucket index.
Has an alternative "fast" mode more like HdrHistogram.
* [HdrHistogram, high dynamic range histogram](http://www.hdrhistogram.org/)
Uses low-precision floating point numbers as bucket indexes.
* [hg64, 64-bit histograms](https://github.com/fanf2/hg64)
My prototype implementation intended for use in BIND.
The DataDog blog article has a nice overview, and compares a quantile sketch implementation (that is designed for a particular rank error) with a histogram (designed for a particular value error). From my reading on this topic I concluded that histograms are both easier to understand, simpler to implement, and have similar or better CPU and memory usage compared to rank-error-based quantile sketches.
## key idea
The histogram counts how many measurements (time or space) have particular `uint64_t` value
or range of values, according to the histogram's configured precision (e.g. 1% or 10%).
Each range of values corresponds to a bucket or counter.
My prototype `hg64` uses a log-linear bucket spacing, which has two parts:
* a logarithm of the value to cover a large dynamic range with a few bits;
specifically, the log base 2 of a `uint64_t` varies from 0 to 63, which fits in 6 bits.
* linear, evenly spaced buckets between logarithms, to provide more precision
than you can get from just a power of 2 or 10. 4 buckets per log are enough
for 10% precision; 32 buckets per log gives 1% precision.
This log-linear bucketing is the same thing as decimal scientific notation,
like 1e9 (1 significant digit, 10% precision) or 2.2e8 (2 significant digits, 1% precision).
It's also the same as a (low-precision) binary floating point number:
the FP exponent is the logarithmic part, and the FP mantissa is the linear part.
## measurements and values
When counting time measurements, it makes sense for the `uint64_t` value to be the time measured in nanoseconds. This allows the histogram to count any time measurements we are likely to need, from submicrosecond up to a few centuries. There is no point using lower-precision time measurements because the histogram bucketing algorithm will reduce the precision as required.
Unlike nanosecond measurements, whose values are towards the logarithmic mid-range of `uint64_t`, memory measurements tend to cluster around zero. The `hg64` bucketing algorithm provides one counter for each distinct small integer; for instance, with 1% precision `hg64` has a counter for each value from 0 to 63, above which multiple values share each counter. To make the best use of these small-value counters, it makes sense to divide a memory measurement to get the desired resolution. For example, if the allocator quantum is 16 bytes, divide an allocation size by 16 before using it as a histogram value.
## incrementing counters quickly
It is very cheap to turn a `uint64_t` value into a bucket number, using CLZ to get the logarithm
with some bit shuffling to move things into place. The basic principle is
roughly the same as used by HdrHistogram and fast-mode DDSketch.
[Paul Khuong encouraged me to use his algorithm](https://twitter.com/pkhuong/status/1571831293335277573)
which is smaller and faster than the version I developed for my proof-of-concept.
As in BIND's existing statistics code, we use a relaxed atomic increment to update a counter.
When the histogram is in cache and uncontended, the whole operation (calculating the bucket
number and incrementing the counter) takes less than 2.5ns in my prototype code.
## efficient storage
The `hg64` bucket keys are small, e.g. 8 bits for 10% precision, or 11 bits for 1% precision.
We could store the buckets as a simple array of counters, which would use 2 KiB for 10%
precision, or 16 KiB for 1% precision. However a large fraction of that space will be
unused, because the values we are recording do not cover anywhere near 20 orders of
magnitude.
My prototype code has a 64 entry top-level array (one for each possible exponent)
and allocate each sub-array on demand (with a counter for each possible mantissa).
Most of the sub-arrays will remain unused. This layout supports lock-free multithreading.
## operations on histograms
* given a value, find its rank (or percentile)
* find the value at a given rank (or percentile)
* get the mean and standard deviation of the data recorded in the histogram
* merge two histograms (which may differ in precision)
* dump and load a histogram in text (e.g. csv, xml, json) and/or binary (for efficiency)
* export a histogram to a user-selected collection of buckets (e.g. for prometheus)
I have implementations of the first four.
The rank and percentile queries work on a snapshot of the working histogram, to avoid multithreading races and to make the calculations more efficient.
## exporting data
An important consumer for data recorded in histograms is Prometheus.
The docs <https://prometheus.io/docs/practices/histograms/> say it supports
* a "histogram" type (actually a cumulative frequency digest) where quantiles are calculated on the server
* a "summary" type, where quantiles are calculated on the client and the server aggregates them over a sliding window
Prometheus has its own textual format for exposing / ingesting data,
<https://prometheus.io/docs/instrumenting/exposition_formats/>.
It looks like it would be fairly easy for `hg64` and BIND to support it,
though it isn't clear whether the server is able to re-bucket data that
is exposed with a different bucketing than configured on the server.
## elsewhere on gitlab
Related issues #598 #2101 #3455Not plannedTony FinchTony Finchhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3277Follow-up from "Draft: Don't delete CDS DELETE after zone sign"2022-04-12T13:18:41ZMatthijs Mekkingmatthijs@isc.orgFollow-up from "Draft: Don't delete CDS DELETE after zone sign"The following discussion from !5706 should be addressed:
- [ ] @marka started a [discussion](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5706#note_259110):
While the changes are fine. I'm worried about a disconnect betwe...The following discussion from !5706 should be addressed:
- [ ] @marka started a [discussion](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/5706#note_259110):
While the changes are fine. I'm worried about a disconnect between parent and child over the signalling method (CDS vs CDNSKEY) in use. Always publishing matching CDS DELETE and CDNSKEY DELETE records avoids this.
When an update adds a CDS DELETE record, BIND should also add the corresponding CDNSKEY DELETE record (if not already done in the update), and vice versa. Also when an update removes a CDS DELETE record, BIND should remove the corresponding CDNSKEY DELETE record (if not already done in the update), and vice versa.
It is unclear what the desired behavior is if an update adds a CDS DELETE record and removes a CDNSKEY DELETE record at the same time.
Note that with `dnssec-policy` the CDS DELETE and CDNSKEY DELETE records are already synced (when setting `dnssec-policy` to `insecure`).Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3231forward queries via DoH2023-11-02T17:05:04ZDaniel Mittonforward queries via DoH### Description
How do you configure bind to forward queries, regardless of initial connection type, via DoH/DoT?
### Request
Forwarder configuration that specifies that forwarded queries are to be forwarded over DoH/Dot.
### Links /...### Description
How do you configure bind to forward queries, regardless of initial connection type, via DoH/DoT?
### Request
Forwarder configuration that specifies that forwarded queries are to be forwarded over DoH/Dot.
### Links / referencesNot plannedArtem BoldarievArtem Boldarievhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3136dnstap-read: add incremental packet number to the output2022-02-10T15:06:10Zmartinvonwittichdnstap-read: add incremental packet number to the output### Description
Currently, `dnstap-read` doesn't print any kind of unique identifier per packet. E.g. if I'm using it without any options to get the short summary format:
```
09-Feb-2022 02:33:36.294 CQ 127.0.0.1:41718 -> 127.0.0.1:0 U...### Description
Currently, `dnstap-read` doesn't print any kind of unique identifier per packet. E.g. if I'm using it without any options to get the short summary format:
```
09-Feb-2022 02:33:36.294 CQ 127.0.0.1:41718 -> 127.0.0.1:0 UDP 33b example.com/IN/MX
09-Feb-2022 02:33:36.334 RR 192.168.1.2:55163 <- 192.168.1.1:53 UDP 71b example.com/IN/MX
09-Feb-2022 02:33:36.294 RQ 192.168.1.2:55163 -> 192.168.1.1:53 UDP 33b example.com/IN/MX
09-Feb-2022 02:33:36.334 CR 127.0.0.1:41718 <- 127.0.0.1:0 UDP 102b example.com/IN/MX
09-Feb-2022 02:33:38.453 CQ 127.0.0.1:57293 -> 127.0.0.1:0 UDP 33b example.com/IN/MX
09-Feb-2022 02:33:38.453 CR 127.0.0.1:57293 <- 127.0.0.1:0 UDP 102b example.com/IN/MX
```
and then I want to lookup the details of one of these packets in the `-p` format, I have to search for the whole line to find it.
It's even worse in the `-y` format because contrary to the `-p` format, the YAML representation doesn't contain the original summary line, and while the summary and `-p` will print timestamps in the local timezone, `-y` will print UTC timestamps.
### Request
I would like `dnstap-read` to prefix each packet with an incremental number in the summary and in the `-p` output, so that the details for a packet can easily be searched. The YAML representation should contain the number in an additional YAML field.
### Links / references
I like the way it works in `tshark` - each line in the summary is prefixed with an incremental packet number:
```
server ~ # tshark -i ens3 -w test.pcap
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens3'
4 ^C
server ~ # tshark -r test.pcap
Running as user "root" and group "root". This could be dangerous.
1 2022-02-09 17:43:01,528756106 02:00:62:3e:71:f5 → 02:00:62:3e:71:f9 ARP 42 Who has 172.16.56.10? Tell 172.16.0.1
2 2022-02-09 17:43:01,528792938 02:00:62:3e:71:f9 → 02:00:62:3e:71:f5 ARP 42 172.16.56.10 is at 02:00:62:3e:71:f9
3 2022-02-09 17:43:02,068971390 172.16.56.10 → 172.21.0.10 SSH 102 Server: Encrypted packet (len=36)
4 2022-02-09 17:43:02,170798836 172.21.0.10 → 172.16.56.10 TCP 66 55798 → 22 [ACK] Seq=1 Ack=37 Win=990 Len=0 TSval=1691964300 TSecr=1370499909
```
When printing the capture file with the `-V` option, the first line of each frame is prefixed with `Frame n`, which makes it easy to search in a pager:
```
server ~ # tshark -r test.pcap -V | head -n 5
Running as user "root" and group "root". This could be dangerous.
Frame 1: 42 bytes on wire (336 bits), 42 bytes captured (336 bits) on interface 0
Interface id: 0 (ens3)
Interface name: ens3
Encapsulation type: Ethernet (1)
Arrival Time: Feb 9, 2022 17:43:01.528756106 CET
```Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/2996Migrate PKCS11 from "engine" to "provider"2022-12-26T19:20:52ZMark AndrewsMigrate PKCS11 from "engine" to "provider"OpenSSL 3.0.0 has deprecated the `engine` api in favour of the `provider` api. We currently use the `engine` api and OpenSC for pkcs11 access to HSMs. We need to move to using the `provider` api as soon as possible.
OpenSC has an open...OpenSSL 3.0.0 has deprecated the `engine` api in favour of the `provider` api. We currently use the `engine` api and OpenSC for pkcs11 access to HSMs. We need to move to using the `provider` api as soon as possible.
OpenSC has an open ticket for OpenSSL 3.0.0 https://github.com/OpenSC/OpenSC/issues/2308Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4370Check that a zone is served by IPv6 servers if it has AAAA records2023-12-05T09:04:32ZMark AndrewsCheck that a zone is served by IPv6 servers if it has AAAA recordsOne thing that is often forgotten when turning on an IPv6 service is to ensure that the zone holding the AAAA records for that service is also served over IPv6. This can be relatively easy be checked for by named-checkzone by looking fo...One thing that is often forgotten when turning on an IPv6 service is to ensure that the zone holding the AAAA records for that service is also served over IPv6. This can be relatively easy be checked for by named-checkzone by looking for AAAA records in the zone contents, including glue AAAA records, and then checking that there are AAAA records published for some of the nameservers if any are found (in zone or elsewhere). This can also sometimes be determined by named without needing to look beyond the zone's contents, but cannot be guaranteed.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4348implement QPDB databases2024-03-08T23:38:09ZEvan Huntimplement QPDB databasesCreate QP-trie based databases to take the place of RBTDB, for use as a:
- [ ] zone database
- [ ] cacheCreate QP-trie based databases to take the place of RBTDB, for use as a:
- [ ] zone database
- [ ] cacheBIND 9.21.xEvan HuntEvan Hunthttps://gitlab.isc.org/isc-projects/bind9/-/issues/4218Extract the no DS proof, if any, from the referral and save/validate it.2023-07-25T06:56:43ZMark AndrewsExtract the no DS proof, if any, from the referral and save/validate it.Insecure referrals contain a no DS proof. We are currently not using it and instead are make DS queries and validating those. Extracting the no DS proof should resolution for data in insecure zones.Insecure referrals contain a no DS proof. We are currently not using it and instead are make DS queries and validating those. Extracting the no DS proof should resolution for data in insecure zones.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/4216Report missing DNSSEC algorithms in returned RRSIGs2023-11-02T16:30:30ZMark AndrewsReport missing DNSSEC algorithms in returned RRSIGsWhile we only require one DNSSEC algorithm to validate a response reporting missing algorithms listed in the DS RRset will be useful for the overall health of the system.While we only require one DNSSEC algorithm to validate a response reporting missing algorithms listed in the DS RRset will be useful for the overall health of the system.Not plannedhttps://gitlab.isc.org/isc-projects/bind9/-/issues/3751Extend 'allow-*' options to support extended syntax that includes 'transport'...2023-11-02T17:05:06ZArtem BoldarievExtend 'allow-*' options to support extended syntax that includes 'transport' and 'port' parameters (like 'allow-transfer')While working on XoT we extended `allow-transfer` option with `port` and `transport` parameters (see !5587). As in 9.19 we support update forwarding over TLS (see !6710) and soon will support queries forwarding over TLS (see !7199 which ...While working on XoT we extended `allow-transfer` option with `port` and `transport` parameters (see !5587). As in 9.19 we support update forwarding over TLS (see !6710) and soon will support queries forwarding over TLS (see !7199 which is being worked on by @aram ), it seems that it is time to update other related options to support the extended syntax. The low-level functionality is there, it is a matter of enabling the syntax and extending the `transport-acl` system test.
We should not forget to extend these as we are improving our support for DNS transports other than UDP and TCP, as was originally intended while adding the initial support for these extra parameters.
This might end up being a "meta" issue referencing other related issues/MRs.
We should improve the documentation on that too, but that is a different matter.Not plannedArtem BoldarievArtem Boldariev