Break lock order loop by sending TAT in an event

The dotat() function has been changed to send the TAT
query asynchronously, so there's no lock order loop
because we initialize the data first and then we schedule
the TAT send to happen asynchronously.

This breaks following lock-order loops:

zone->lock (dns_zone_setviewcommit) while holding view->lock
(dns_view_setviewcommit)

keytable->lock (dns_keytable_find) while holding zone->lock
(zone_asyncload)

view->lock (dns_view_findzonecut) while holding keytable->lock
(dns_keytable_forall)
74 jobs for !4157 with 2156-threadsanitizer-lock-order-inversion-potential-deadlock-in-pthread_mutex_lock-2 in 19 minutes and 31 seconds (queued for 10 seconds)
latest detached
Status Job ID Name Coverage
  Autoconf
passed #1174551
amd64 linux
autoreconf

00:00:16

 
  Precheck
passed #1174553
amd64 linux
clang-format

00:00:16

passed #1174554
amd64 linux
coccinelle

00:03:15

passed #1174555
amd64 linux
danger

00:00:13

passed #1174552
amd64 linux
misc

00:00:10

passed #1174556
amd64 linux
tarball-create

00:01:31

 
  Build
passed #1174569
amd64 linux
clang:asan

00:02:04

passed #1174573
amd64 linux
clang:buster:amd64

00:01:00

passed #1174576
amd64 libvirt
clang:freebsd11:amd64

00:01:04

passed #1174577
amd64 libvirt
clang:freebsd12:amd64

00:01:04

passed #1174578
amd64 libvirt
clang:openbsd:amd64

00:02:11

passed #1174571
amd64 linux
clang:tsan

00:01:07

passed #1174557
amd64 linux
gcc:alpine3.12:amd64

00:00:45

passed #1174568
amd64 linux
gcc:asan

00:01:38

passed #1174561
amd64 linux
gcc:buster:amd64

00:00:45

passed #1174558
amd64 linux
gcc:centos7:amd64

00:00:44

passed #1174559
amd64 linux
gcc:centos8:amd64

00:00:45

passed #1174567
amd64 linux
gcc:focal:amd64

00:00:46

passed #1174572
amd64 linux
gcc:mutexatomics

00:00:54

passed #1174563
amd64 linux
gcc:out-of-tree

00:00:47

passed #1174562
amd64 linux
gcc:sid:amd64

00:00:58

passed #1174564
i386 linux
gcc:sid:i386

00:00:52

passed #1174574
amd64 linux
gcc:softhsm2.4

00:00:56

passed #1174575
amd64 linux
gcc:softhsm2.6

00:00:55

passed #1174560
amd64 linux
gcc:stretch:amd64

00:00:48

passed #1174570
amd64 linux
gcc:tsan

00:01:00

passed #1174565
amd64 linux
gcc:tumbleweed:amd64

00:00:47

passed #1174566
amd64 linux
gcc:xenial:amd64

00:00:49

passed #1174579
amd64 windows
msvc:windows:amd64

00:00:42

 
  Unit
passed #1174591
amd64 linux
unit:clang:asan

00:01:23

passed #1174595
amd64 linux
unit:clang:buster:amd64

00:00:59

passed #1174598
amd64 libvirt
unit:clang:freebsd11:amd64

00:01:30

passed #1174599
amd64 libvirt
unit:clang:freebsd12:amd64

00:01:24

passed #1174593
amd64 linux
unit:clang:tsan

00:03:16

passed #1174580
amd64 linux
unit:gcc:alpine3.12:amd64

00:01:04

passed #1174590
amd64 linux
unit:gcc:asan

00:01:14

passed #1174584
amd64 linux
unit:gcc:buster:amd64

00:00:56

passed #1174581
amd64 linux
unit:gcc:centos7:amd64

00:00:58

passed #1174582
amd64 linux
unit:gcc:centos8:amd64

00:00:55

passed #1174589
amd64 linux
unit:gcc:focal:amd64

00:00:59

passed #1174594
amd64 linux
unit:gcc:mutexatomics

00:00:55

passed #1174585
amd64 linux
unit:gcc:sid:amd64

00:00:57

passed #1174586
i386 linux
unit:gcc:sid:i386

00:00:58

passed #1174596
amd64 linux
unit:gcc:softhsm2.4

00:01:00

passed #1174597
amd64 linux
unit:gcc:softhsm2.6

00:00:58

passed #1174583
amd64 linux
unit:gcc:stretch:amd64

00:00:58

passed #1174592
amd64 linux
unit:gcc:tsan

00:03:58

passed #1174587
amd64 linux
unit:gcc:tumbleweed:amd64

00:00:55

passed #1174588
amd64 linux
unit:gcc:xenial:amd64

00:00:59

 
  System
passed #1174611
amd64 linux
system:clang:asan

00:12:17

passed #1174615
amd64 linux
system:clang:buster:amd64

00:10:33

failed #1174618
amd64 libvirt
system:clang:freebsd11:amd64

00:11:13

failed #1174619
amd64 libvirt
system:clang:freebsd12:amd64

00:10:19

failed #1174613
amd64 linux allowed to fail
system:clang:tsan

00:13:54

passed #1174600
amd64 linux
system:gcc:alpine3.12:amd64

00:11:41

passed #1174610
amd64 linux
system:gcc:asan

00:12:26

passed #1174604
amd64 linux
system:gcc:buster:amd64

00:11:26

passed #1174601
amd64 linux
system:gcc:centos7:amd64

00:11:42

passed #1174602
amd64 linux
system:gcc:centos8:amd64

00:11:17

passed #1174609
amd64 linux
system:gcc:focal:amd64

00:11:14

passed #1174614
amd64 linux
system:gcc:mutexatomics

00:10:05

passed #1174605
amd64 linux
system:gcc:sid:amd64

00:09:50

passed #1174606
i386 linux
system:gcc:sid:i386

00:11:46

passed #1174616
amd64 linux
system:gcc:softhsm2.4

00:10:56

passed #1174617
amd64 linux
system:gcc:softhsm2.6

00:10:32

passed #1174603
amd64 linux
system:gcc:stretch:amd64

00:10:41

failed #1174612
amd64 linux allowed to fail
system:gcc:tsan

00:14:08

passed #1174607
amd64 linux
system:gcc:tumbleweed:amd64

00:10:45

passed #1174608
amd64 linux
system:gcc:xenial:amd64

00:09:36

 
  Docs
passed #1174620
amd64 linux
docs

00:00:36

 
  Postcheck
passed #1174624
amd64 linux
cppcheck

00:00:37

passed #1174621
amd64 linux
flake8

00:00:16

passed #1174622
amd64 linux
pylint

00:00:20

passed #1174623
amd64 linux
scan-build

00:03:58

 
Name Stage Failure
failed
system:gcc:tsan System
E:shutdown:2020-09-22T12:59:50+0000
FAIL shutdown (exit status: 1)

$ find bin -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 9534 files

Uploading artifacts as "archive" to coordinator... ok
id=1174612 responseStatus=201 Created token=xVzNGTVU
ERROR: Job failed: exit code 1
failed
system:clang:tsan System
E:shutdown:2020-09-22T12:59:45+0000
FAIL shutdown (exit status: 1)

$ find bin -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts for failed job
Uploading artifacts...
untracked: found 9700 files

Uploading artifacts as "archive" to coordinator... ok
id=1174613 responseStatus=201 Created token=uCZM8WgM
ERROR: Job failed: exit code 1
failed
system:clang:freebsd11:amd64 System
E:rpz:2020-09-22T12:50:32+0000
FAIL rpz (exit status: 1)

Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=47777 revision=86ad88ea version=13.3.0
untracked: found 1565 files

Uploading artifacts as "archive" to coordinator... ok
id=1174618 responseStatus=201 Created token=jP6211WW
ERROR: Job failed: exit status 1
failed
system:clang:freebsd12:amd64 System
E:dnstap:2020-09-22T12:52:05+0000
FAIL dnstap (exit status: 1)

Uploading artifacts for failed job
Uploading artifacts...
Runtime platform
arch=amd64 os=freebsd pid=50568 revision=86ad88ea version=13.3.0
untracked: found 1565 files

Uploading artifacts as "archive" to coordinator... ok
id=1174619 responseStatus=201 Created token=VsUnqd91
ERROR: Job failed: exit status 1