GitLab

Marka dns r badtsig map to formerr v9 16

See merge request !3914

(cherry picked from commit 88ff6b84)

Now that the log message has been printed set the result code to
DNS_R_FORMERR.  We don't do this via dns_result_torcode() as we
don't want upstream errors to produce FORMERR if that processing
end with DNS_R_BADTSIG.

(cherry picked from commit 20488d6a)

Resolve "Observed stats underflow in multiple stats"

See merge request !3866

The basic scenario for the problem was that in the process of
resolving a query, if any rrset was eligible for prefetching, then it
would trigger a call to query_prefetch(), this call would run in
parallel to the normal query processing.

The problem arises due to the fact that both query_prefetch(), and,
in the original thread, a call to ns_query_recurse(), try to attach
to the recursionquota, but recursing client stats counter is only
incremented if ns_query_recurse() attachs to it first.

Conversely, if fetch_callback() is called before prefetch_done(),
it would not only detach from recursionquota, but also decrement
the stats counter, if query_prefetch() attached to te quota first
that would result in a decrement not matched by an increment, as
expected.

To solve this issue an atomic bool was added, it is set once in
ns_query_recurse(), allowing fetch_callback() to check for it
and decrement stats accordingly.

For a more compreensive explanation check the thread comment below:
#1719 (comment 145857)

[v9_16] Restore placeholder entry at sequence number 5481

See merge request !3911

(cherry picked from commit 029e32c0)

Resolve "Bind not handling interfaces changes correctly when listen-on-v6  any  specified"

See merge request !3907

(cherry picked from commit 94eda43a)

When a new IPv6 interface/address appears it's first in a tentative
state - in which we cannot bind to it, yet it's already being reported
by the route socket. Because of that BIND9 is unable to listen on any
newly detected IPv6 addresses. Fix it by setting IP_FREEBIND option (or
equivalent option on other OSes) and then retrying bind() call.

(cherry picked from commit a0f7d289)

[v9_16] Only run system tests as root in developer mode

See merge request !3897

Running system tests with root privileges is potentially dangerous.
Only allow it when explicitly requested (by building with
--enable-developer).

(cherry picked from commit 3ef106f6)

Always check the return from isc_refcount_decrement.

See merge request !3901

Created isc_refcount_decrement_expect macro to test conditionally
the return value to ensure it is in expected range.  Converted
unchecked isc_refcount_decrement to use isc_refcount_decrement_expect.
Converted INSIST(isc_refcount_decrement()...) to isc_refcount_decrement_expect.

(cherry picked from commit bde5c763)

Refactor the code that counts the last log version to keep

See merge request !3900

When silencing the Coverity warning in remove_old_tsversions(), the code
was refactored to reduce the indentation levels and break down the long
code into individual functions.  This improve fix for [GL #1989].

(cherry picked from commit aca18b8b)

[v9_16] Various system test fixes

See merge request !3898

Merge branch '1775-resizing-growing-of-cache-hash-tables-causes-delays-in-processing-of-client-queries-v9_16' into 'v9_16'

Resolve "Resizing (growing) of cache hash tables causes delays in processing of client queries"

See merge request !3871

(cherry picked from commit 2b4f0f03)

Change the dns_hash_name() and dns_hash_fullname() functions to use
isc_hash32() as the maximum hashtable size in rbt is 0..UINT32_MAX
large.

(cherry picked from commit a9182c89)

As the names suggest the original isc_hash64 function returns 64-bit
long hash values and the isc_hash32() returns 32-bit values.

(cherry picked from commit f59fd49f)

The HalfSipHash implementation has 32-bit keys and returns 32-bit
value.

(cherry picked from commit 344d66aa)

Creation of EVP_MD_CTX and EVP_PKEY is quite expensive, so until we fix the code
to reuse the OpenSSL contexts and keys we'll use our own implementation of
siphash instead of trying to integrate with OpenSSL.

(cherry picked from commit 21d751df)

There were several problems with rbt hashtable implementation:

1. Our internal hashing function returns uint64_t value, but it was
   silently truncated to unsigned int in dns_name_hash() and
   dns_name_fullhash() functions.  As the SipHash 2-4 higher bits are
   more random, we need to use the upper half of the return value.

2. The hashtable implementation in rbt.c was using modulo to pick the
   slot number for the hash table.  This has several problems because
   modulo is: a) slow, b) oblivious to patterns in the input data.  This
   could lead to very uneven distribution of the hashed data in the
   hashtable.  Combined with the single-linked lists we use, it could
   really hog-down the lookup and removal of the nodes from the rbt
   tree[a].  The Fibonacci Hashing is much better fit for the hashtable
   function here.  For longer description, read "Fibonacci Hashing: The
   Optimization that the World Forgot"[b] or just look at the Linux
   kernel.  Also this will make Diego very happy :).

3. The hashtable would rehash every time the number of nodes in the rbt
   tree would exceed 3 * (hashtable size).  The overcommit will make the
   uneven distribution in the hashtable even worse, but the main problem
   lies in the rehashing - every time the database grows beyond the
   limit, each subsequent rehashing will be much slower.  The mitigation
   here is letting the rbt know how big the cache can grown and
   pre-allocate the hashtable to be big enough to actually never need to
   rehash.  This will consume more memory at the start, but since the
   size of the hashtable is capped to `1 << 32` (e.g. 4 mio entries), it
   will only consume maximum of 32GB of memory for hashtable in the
   worst case (and max-cache-size would need to be set to more than
   4TB).  Calling the dns_db_adjusthashsize() will also cap the maximum
   size of the hashtable to the pre-computed number of bits, so it won't
   try to consume more gigabytes of memory than available for the
   database.

   FIXME: What is the average size of the rbt node that gets hashed?  I
   chose the pagesize (4k) as initial value to precompute the size of
   the hashtable, but the value is based on feeling and not any real
   data.

For future work, there are more places where we use result of the hash
value modulo some small number and that would benefit from Fibonacci
Hashing to get better distribution.

Notes:
a. A doubly linked list should be used here to speedup the removal of
   the entries from the hashtable.
b. https://probablydance.com/2018/06/16/fibonacci-hashing-the-optimization-that-the-world-forgot-or-a-better-alternative-to-integer-modulo/

(cherry picked from commit e24bc324)

[v9_16] Fix idle timeout for connected TCP sockets

See merge request !3896

(cherry picked from commit 18efb245)

When named acting as a resolver connects to an authoritative server over
TCP, it sets the idle timeout for that connection to 20 seconds.  This
fixed timeout was picked back when the default processing timeout for
each client query was hardcoded to 30 seconds.  Commit
000a8970 made this processing timeout
configurable through "resolver-query-timeout" and decreased its default
value to 10 seconds, but the idle TCP timeout was not adjusted to
reflect that change.  As a result, with the current defaults in effect,
a single hung TCP connection will consistently cause the resolution
process for a given query to time out.

Set the idle timeout for connected TCP sockets to half of the client
query processing timeout configured for a resolver.  This allows named
to handle hung TCP connections more robustly and prevents the timeout
mismatch issue from resurfacing in the future if the default is ever
changed again.

(cherry picked from commit 953d704b)

report libuv version string in `named -V`

See merge request !3890

(cherry picked from commit 1036338a)

Resolve "RPZ wildcard passthru ignored"

See merge request !3889

Whenever an exact match is found by dns_rbt_findnode(),
the highest level node in the chain will not be put into
chain->levels[] array, but instead the chain->end
pointer will be adjusted to point to that node.

Suppose we have the following entries in a rpz zone:
example.com     CNAME rpz-passthru.
*.example.com   CNAME rpz-passthru.

A query for www.example.com would result in the
following chain object returned by dns_rbt_findnode():

chain->level_count = 2
chain->level_matches = 2
chain->levels[0] = .
chain->levels[1] = example.com
chain->levels[2] = NULL
chain->end = www

Since exact matches only care for testing rpz set bits,
we need to test for rpz wild bits through iterating the nodechain, and
that includes testing the rpz wild bits in the highest level node found.

In the case of an exact match, chain->levels[chain->level_matches]
will be NULL, to address that we must use chain->end as the start point,
then iterate over the remaining levels in the chain.

Merge branch '2043-dns_rdata_hip_next-fails-to-return-isc_r_nomore-at-the-right-time-v9_16' into 'v9_16'

Resolve "dns_rdata_hip_next() fails to return ISC_R_NOMORE at the right time."

See merge request !3885

(cherry picked from commit bcd049f1)

Also fixes extraneous white space at end of record when
there are no rendezvous servers.

(cherry picked from commit 78db46d7)