broken checkpoint

the resolver needs to clean up after a connect failure and try
another server; that still isn't working now, but this commit fixes
some of the problems that turn up.
73 jobs for !4601 with 2401-use-netmgr-for-dispatch in 63 minutes and 42 seconds (queued for 8 seconds)
detached
Status Job ID Name Coverage
  Autoconf
passed #1539249
amd64 linux
autoreconf

00:00:18

 
  Precheck
passed #1539251
amd64 linux
clang-format

00:00:19

passed #1539252
amd64 linux
coccinelle

00:03:49

failed #1539253
amd64 linux
danger

00:00:22

passed #1539250
amd64 linux
misc

00:00:19

passed #1539254
amd64 linux
tarball-create

00:01:54

 
  Build
passed #1539266
amd64 linux
clang:asan

00:01:53

passed #1539270
amd64 linux
clang:buster:amd64

00:01:13

passed #1539273
amd64 libvirt
clang:freebsd11:amd64

00:01:22

passed #1539274
amd64 libvirt
clang:freebsd12:amd64

00:01:22

passed #1539275
amd64 libvirt
clang:openbsd:amd64

00:02:33

passed #1539268
amd64 linux
clang:tsan

00:01:21

passed #1539255
amd64 linux
gcc:alpine3.13:amd64

00:01:08

passed #1539265
amd64 linux
gcc:asan

00:01:42

passed #1539259
amd64 linux
gcc:buster:amd64

00:00:44

passed #1539256
amd64 linux
gcc:centos7:amd64

00:01:03

passed #1539257
amd64 linux
gcc:centos8:amd64

00:00:52

passed #1539264
amd64 linux
gcc:focal:amd64

00:01:08

passed #1539269
amd64 linux
gcc:mutexatomics

00:01:08

passed #1539261
amd64 linux
gcc:out-of-tree

00:01:04

passed #1539260
amd64 linux
gcc:sid:amd64

00:01:18

passed #1539271
amd64 linux
gcc:softhsm2.4

00:01:00

passed #1539272
amd64 linux
gcc:softhsm2.6

00:00:58

passed #1539258
amd64 linux
gcc:stretch:amd64

00:01:04

passed #1539267
amd64 linux
gcc:tsan

00:01:04

passed #1539262
amd64 linux
gcc:tumbleweed:amd64

00:01:07

passed #1539263
amd64 linux
gcc:xenial:amd64

00:01:10

passed #1539276
amd64 windows
msvc:windows:amd64

00:00:49

 
  Unit
failed #1539287
amd64 linux
unit:clang:asan

01:00:01

failed #1539291
amd64 linux
unit:clang:buster:amd64

01:00:01

failed #1539294
amd64 libvirt
unit:clang:freebsd11:amd64

00:02:02

failed #1539295
amd64 libvirt
unit:clang:freebsd12:amd64

00:01:49

failed #1539296
amd64 libvirt
unit:clang:openbsd:amd64

00:03:25

failed #1539289
amd64 linux
unit:clang:tsan

01:00:01

failed #1539277
amd64 linux
unit:gcc:alpine3.13:amd64

01:00:01

failed #1539286
amd64 linux
unit:gcc:asan

01:00:01

failed #1539281
amd64 linux
unit:gcc:buster:amd64

01:00:01

failed #1539278
amd64 linux
unit:gcc:centos7:amd64

01:00:02

failed #1539279
amd64 linux
unit:gcc:centos8:amd64

01:00:01

failed #1539285
amd64 linux
unit:gcc:focal:amd64

01:00:01

failed #1539290
amd64 linux
unit:gcc:mutexatomics

01:00:01

failed #1539282
amd64 linux
unit:gcc:sid:amd64

01:00:01

failed #1539292
amd64 linux
unit:gcc:softhsm2.4

01:00:01

failed #1539293
amd64 linux
unit:gcc:softhsm2.6

01:00:01

failed #1539280
amd64 linux
unit:gcc:stretch:amd64

01:00:01

failed #1539288
amd64 linux
unit:gcc:tsan

01:00:01

failed #1539283
amd64 linux
unit:gcc:tumbleweed:amd64

01:00:01

failed #1539284
amd64 linux
unit:gcc:xenial:amd64

01:00:01

 
  System
failed #1539307
amd64 linux
system:clang:asan

00:20:16

failed #1539311
amd64 linux
system:clang:buster:amd64

00:15:43

failed #1539314
amd64 libvirt
system:clang:freebsd11:amd64

00:10:19

failed #1539315
amd64 libvirt
system:clang:freebsd12:amd64

00:10:33

failed #1539309
amd64 linux allowed to fail
system:clang:tsan

00:25:46

failed #1539297
amd64 linux
system:gcc:alpine3.13:amd64

00:16:26

failed #1539306
amd64 linux
system:gcc:asan

00:17:46

skipped #1539301
amd64 linux
system:gcc:buster:amd64
failed #1539298
amd64 linux
system:gcc:centos7:amd64

00:15:39

failed #1539299
amd64 linux
system:gcc:centos8:amd64

00:14:28

failed #1539305
amd64 linux
system:gcc:focal:amd64

00:14:53

failed #1539310
amd64 linux
system:gcc:mutexatomics

00:15:24

failed #1539302
amd64 linux
system:gcc:sid:amd64

00:14:39

failed #1539312
amd64 linux
system:gcc:softhsm2.4

00:14:47

failed #1539313
amd64 linux
system:gcc:softhsm2.6

00:14:01

failed #1539300
amd64 linux
system:gcc:stretch:amd64

00:40:13

failed #1539308
amd64 linux allowed to fail
system:gcc:tsan

00:17:57

failed #1539303
amd64 linux
system:gcc:tumbleweed:amd64

00:14:07

failed #1539304
amd64 linux
system:gcc:xenial:amd64

00:16:38

 
  Docs
passed #1539316
amd64 linux
docs

00:01:05

 
  Postcheck
passed #1539320
amd64 linux
cppcheck

00:00:59

passed #1539317
amd64 linux
flake8

00:00:22

skipped #1539321
amd64 linux
gcov
passed #1539318
amd64 linux
pylint

00:00:32

failed #1539319
amd64 linux
scan-build

00:04:31

 
Name Stage Failure
failed
unit:clang:tsan Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:clang:asan Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:asan Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:clang:buster:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:sid:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: dnstap_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:mutexatomics Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:xenial:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: time_test
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:focal:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:tumbleweed:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:alpine3.13:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: dnstap_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:tsan Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:softhsm2.4 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:stretch:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:centos7:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: dnstap_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:softhsm2.6 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:centos8:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: tsig_test
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: rbt_test
PASS: db_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
unit:gcc:buster:amd64 Unit The script exceeded the maximum execution time set for the job
PASS: update_test
PASS: zonemgr_test
PASS: zt_test
PASS: master_test
PASS: geoip_test
PASS: dnstap_test
PASS: db_test
PASS: rbt_test
ERROR: Job failed: execution took longer than 1h0m0s seconds
failed
system:gcc:stretch:amd64 System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 6658 files

Uploading artifacts as "archive" to coordinator... ok
id=1539300 responseStatus=201 Created token=h9Bjj_w6
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:clang:tsan System
FAIL shutdown (exit status: 1)

$ find bin -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 10448 files

Uploading artifacts as "archive" to coordinator... ok
id=1539309 responseStatus=201 Created token=RE7PxGUG
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:clang:asan System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 8823 files

Uploading artifacts as "archive" to coordinator... ok
id=1539307 responseStatus=201 Created token=HcYKMmq7
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:asan System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 6516 files

Uploading artifacts as "archive" to coordinator... ok
id=1539306 responseStatus=201 Created token=xTmDtd_w
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:tsan System
FAIL shutdown (exit status: 1)

$ find bin -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 10183 files

Uploading artifacts as "archive" to coordinator... ok
id=1539308 responseStatus=201 Created token=SxpiJXd1
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:xenial:amd64 System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 7360 files

Uploading artifacts as "archive" to coordinator... ok
id=1539304 responseStatus=201 Created token=HfsT4rdz
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:alpine3.13:amd64 System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 6358 files

Uploading artifacts as "archive" to coordinator... ok
id=1539297 responseStatus=201 Created token=xH7ru2JV
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:clang:buster:amd64 System
FAIL shutdown (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 6063 files

Uploading artifacts as "archive" to coordinator... ok
id=1539311 responseStatus=201 Created token=59uz2LPW
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:mutexatomics System
FAIL shutdown (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 5006 files

Uploading artifacts as "archive" to coordinator... ok
id=1539310 responseStatus=201 Created token=c7ihTA5V
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:centos7:amd64 System
FAIL shutdown (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 3005 files

Uploading artifacts as "archive" to coordinator... ok
id=1539298 responseStatus=201 Created token=bAzhcubJ
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:focal:amd64 System
FAIL shutdown (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 4472 files

Uploading artifacts as "archive" to coordinator... ok
id=1539305 responseStatus=201 Created token=pNSic5xK
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:sid:amd64 System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 6627 files

Uploading artifacts as "archive" to coordinator... ok
id=1539302 responseStatus=201 Created token=yovs-qmz
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:softhsm2.4 System
FAIL pipelined (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 4198 files

Uploading artifacts as "archive" to coordinator... ok
id=1539312 responseStatus=201 Created token=EU4st69s
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:centos8:amd64 System
FAIL shutdown (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 5856 files

Uploading artifacts as "archive" to coordinator... ok
id=1539299 responseStatus=201 Created token=EyRAzR5Z
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:tumbleweed:amd64 System
FAIL dnssec (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 4440 files

Uploading artifacts as "archive" to coordinator... ok
id=1539303 responseStatus=201 Created token=tzhooG9d
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:gcc:softhsm2.6 System
FAIL qmin (exit status: 1)

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 3648 files

Uploading artifacts as "archive" to coordinator... ok
id=1539313 responseStatus=201 Created token=s76Rsfu2
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
system:clang:freebsd12:amd64 System

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=62074 revision=2ebc4dc4 version=13.9.0
untracked: found 6447 files

Uploading artifacts as "archive" to coordinator... ok
id=1539315 responseStatus=201 Created token=X4DvmhkB
Cleaning up file based variables
ERROR: Job failed: exit status 1
failed
system:clang:freebsd11:amd64 System

$ test -n "${OUT_OF_TREE_WORKSPACE}" && mv "${OUT_OF_TREE_WORKSPACE}" "${CI_PROJECT_DIR}"
Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=61124 revision=2ebc4dc4 version=13.9.0
untracked: found 6392 files

Uploading artifacts as "archive" to coordinator... ok
id=1539314 responseStatus=201 Created token=dexJNEJV
Cleaning up file based variables
ERROR: Job failed: exit status 1
failed
unit:clang:openbsd:amd64 Unit
`unit' not remade because of errors.
Errors while building
Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=openbsd pid=36822 revision=9ad2ddee version=12.4.0~beta.1928.g9ad2ddee
untracked: found 1571 files

Uploading artifacts to coordinator... ok
id=1539296 responseStatus=201 Created token=_zjLV9-F
Cleaning up file based variables
ERROR: Job failed: exit status 1
failed
scan-build Postcheck
scan-build: Analysis run complete.
scan-build: 1 bug found.
scan-build: Run 'scan-view /builds/isc-projects/bind9/scan-build.reports/2021-02-28-204203-7214-1' to examine bug reports.
Uploading artifacts for failed job
Uploading artifacts...
scan-build.reports/: found 6 matching files and directories

Uploading artifacts as "archive" to coordinator... ok
id=1539319 responseStatus=201 Created token=rfk4JpPH
Cleaning up file based variables
ERROR: Job failed: exit code 1
failed
unit:clang:freebsd11:amd64 Unit

make: stopped in /builds/isc-projects/bind9
Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=14843 revision=2ebc4dc4 version=13.9.0
untracked: found 1573 files

Uploading artifacts as "archive" to coordinator... ok
id=1539294 responseStatus=201 Created token=Xs1aTUD7
Cleaning up file based variables
ERROR: Job failed: exit status 1
failed
unit:clang:freebsd12:amd64 Unit
# ERROR: 0
============================================================================
Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=15045 revision=2ebc4dc4 version=13.9.0
untracked: found 1588 files

Uploading artifacts as "archive" to coordinator... ok
id=1539295 responseStatus=201 Created token=f4ozdgxP
Cleaning up file based variables
ERROR: Job failed: exit status 1
failed
danger Precheck
 * branch            main       -> FETCH_HEAD



Failing the build, there are 2 fails.
Feedback: https://gitlab.isc.org/isc-projects/bind9/merge_requests/4601#note_188963

Cleaning up file based variables
ERROR: Job failed: exit code 1