"timeouts" system test fails consistently for BIND 9.16(-S) on FreeBSD
TL;DR: the test_send_timeout()
check in the timeouts
system test
does not work on FreeBSD; it was previously not run at all in CI due to
missing dependencies.
Last Saturday, the timeouts
system test suddenly started failing for
v9_16
and v9_16_sub
on FreeBSD 12.3 & 13.0:
S:timeouts:2022-06-18T02:15:11+0000
T:timeouts:1:A
A:timeouts:System test timeouts
I:timeouts:PORTRANGE:13100 - 13199
I:timeouts:starting servers
D:timeouts:============================= test session starts ==============================
D:timeouts:platform freebsd13 -- Python 3.8.13, pytest-4.6.11, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python3.8
D:timeouts:hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/builds/isc-projects/bind9/bin/tests/system/timeouts/.hypothesis/examples')
D:timeouts:rootdir: /builds/isc-projects/bind9/bin/tests/system/timeouts
D:timeouts:plugins: hypothesis-6.39.4
D:timeouts:collecting ... collected 8 items
D:timeouts:
D:timeouts:tests-tcp.py::test_initial_timeout PASSED [ 12%]
D:timeouts:tests-tcp.py::test_idle_timeout PASSED [ 25%]
D:timeouts:tests-tcp.py::test_keepalive_timeout PASSED [ 37%]
D:timeouts:tests-tcp.py::test_pipelining_timeout PASSED [ 50%]
D:timeouts:tests-tcp.py::test_long_axfr PASSED [ 62%]
D:timeouts:tests-tcp.py::test_send_timeout FAILED [ 75%]
D:timeouts:tests-tcp.py::test_max_transfer_idle_out SKIPPED [ 87%]
D:timeouts:tests-tcp.py::test_max_transfer_time_out SKIPPED [100%]
D:timeouts:
D:timeouts:=================================== FAILURES ===================================
D:timeouts:______________________________ test_send_timeout _______________________________
D:timeouts:
D:timeouts:named_port = 13100
D:timeouts:
D:timeouts: def test_send_timeout(named_port):
D:timeouts: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
D:timeouts: sock.connect(("10.53.0.1", named_port))
D:timeouts:
D:timeouts: # Send and receive single large RDATA over TCP
D:timeouts: msg = create_msg("large.example.", "TXT")
D:timeouts: (sbytes, stime) = dns.query.send_tcp(sock, msg, timeout())
D:timeouts: (response, rtime) = dns.query.receive_tcp(sock, timeout())
D:timeouts:
D:timeouts: # Send and receive 28 large (~32k) DNS queries that should
D:timeouts: # fill the default maximum 208k TCP send buffer
D:timeouts: for n in range(28):
D:timeouts: (sbytes, stime) = dns.query.send_tcp(sock, msg, timeout())
D:timeouts:
D:timeouts: # configure idle interval is 5 seconds, sleep 6 to make sure we are
D:timeouts: # above the interval
D:timeouts: time.sleep(6)
D:timeouts:
D:timeouts: with pytest.raises(EOFError):
D:timeouts: try:
D:timeouts: (sbytes, stime) = dns.query.send_tcp(sock, msg, timeout())
D:timeouts: (response, rtime) = dns.query.receive_tcp(sock, timeout())
D:timeouts: except ConnectionError as e:
D:timeouts:> raise EOFError from e
D:timeouts:E Failed: DID NOT RAISE <class 'EOFError'>
D:timeouts:
D:timeouts:tests-tcp.py:211: Failed
D:timeouts:=========================== short test summary info ============================
D:timeouts:SKIPPED [1] tests-tcp.py:214: CI_ENABLE_ALL_TESTS not set
D:timeouts:SKIPPED [1] tests-tcp.py:248: CI_ENABLE_ALL_TESTS not set
D:timeouts:================ 1 failed, 5 passed, 2 skipped in 70.49 seconds ================
I:timeouts:stopping servers
R:timeouts:FAIL
E:timeouts:2022-06-18T02:16:26+0000
Note that:
-
the moment these failures started happening coincides with a QCOW2 image update triggered by merging isc-projects/images!175,
-
previously, the
timeouts
test was not run at all on FreeBSD due to dnspython 2.x not being available: -
the last time these images were touched was back in January.
Given all of the above, I assumed that the py38-dnspython
package got
updated from version 1.x to 2.x in the meantime and I subsequently
confirmed that this was indeed the case:
https://cgit.freebsd.org/ports/commit/?id=5d855cc1c57944dff3cfac0f498b38a4380a1a40
Links to the jobs exhibiting such failures:
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2586577
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2586578
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2586669
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2586670
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2587084
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2587085
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2587176
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2587177
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2588029
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/2588030
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2588121
- https://gitlab.isc.org/isc-private/bind9/-/jobs/2588122
We will need to fix this before the July code freeze.