Make isc_task_pause/isc_task_unpause thread safe.

isc_task_pause/unpause were inherently thread-unsafe - a task
could be paused only once by one thread, if the task was running
while we paused it it led to races. Fix it by making sure that
the task will pause if requested to, and by using a 'pause reference
counter' to count task pause requests - a task will be unpaused
iff all threads unpause it.

Don't remove from queue when pausing task - we lock the queue lock
(expensive), while it's unlikely that the task will be running -
and we'll remove it anyway in dispatcher
75 jobs for !2966 with wpk/task-pausing-thread-safety in 19 minutes and 55 seconds (queued for 4 seconds)
detached
Status Job ID Name Coverage
  Precheck
passed #674236
amd64 linux
🐞:sid:amd64

00:03:53

passed #674234
amd64 linux
autoreconf:sid:amd64

00:00:32

passed #674235
amd64 linux
misc:sid:amd64

00:00:17

 
  Build
passed #674251
amd64 linux
asan:sid:amd64

00:01:57

passed #674258
amd64 freebsd
clang:freebsd11.3:amd64

00:01:07

passed #674259
amd64 freebsd
clang:freebsd12.0:amd64

00:01:00

passed #674260
amd64 libvirt
clang:openbsd6.6:amd64

00:04:22

passed #674255
amd64 linux
clang:stretch:amd64

00:01:55

passed #674256
i386 linux
clang:stretch:i386

00:00:56

passed #674237
amd64 linux
gcc:alpine3.11:amd64

00:02:45

passed #674250
amd64 linux
gcc:bionic:amd64

00:01:52

passed #674243
amd64 linux
gcc:buster:amd64

00:01:44

passed #674238
amd64 linux
gcc:centos6:amd64

00:02:24

passed #674239
amd64 linux
gcc:centos7:amd64

00:02:32

passed #674240
amd64 linux
gcc:centos8:amd64

00:02:41

passed #674248
amd64 linux
gcc:fedora31:amd64

00:02:39

passed #674241
amd64 linux
gcc:jessie:amd64

00:01:26

passed #674244
amd64 linux
gcc:sid:amd64

00:02:42

passed #674246
i386 linux
gcc:sid:i386

00:02:42

passed #674242
amd64 linux
gcc:stretch:amd64

00:01:48

passed #674247
amd64 linux
gcc:tumbleweed:amd64

00:02:27

passed #674249
amd64 linux
gcc:xenial:amd64

00:01:41

passed #674262
amd64 windows
msvc:windows:amd64

00:02:23

passed #674254
amd64 linux
mutexatomics:sid:amd64

00:01:35

passed #674261
amd64 linux
nolibtool:sid:amd64

00:02:22

passed #674245
amd64 linux
oot:sid:amd64

00:01:45

passed #674257
amd64 linux
pkcs11:sid:amd64

00:01:37

passed #674253
amd64 linux
rwlock:sid:amd64

00:01:45

passed #674252
amd64 linux
tsan:buster:amd64

00:03:12

 
  Unit
passed #674276
amd64 linux
unit:asan:sid:amd64

00:01:15

passed #674282
amd64 freebsd
unit:clang:freebsd11.3:amd64

00:00:37

passed #674283
amd64 freebsd
unit:clang:freebsd12.0:amd64

00:00:37

passed #674280
amd64 linux
unit:clang:stretch:amd64

00:00:52

passed #674263
amd64 linux
unit:gcc:alpine3.11:amd64

00:01:10

passed #674275
amd64 linux
unit:gcc:bionic:amd64

00:00:55

passed #674269
amd64 linux
unit:gcc:buster:amd64

00:00:48

passed #674264
amd64 linux
unit:gcc:centos6:amd64

00:00:50

passed #674265
amd64 linux
unit:gcc:centos7:amd64

00:00:46

passed #674266
amd64 linux
unit:gcc:centos8:amd64

00:00:55

passed #674273
amd64 linux
unit:gcc:fedora31:amd64

00:01:06

passed #674267
amd64 linux
unit:gcc:jessie:amd64

00:00:34

passed #674270
amd64 linux
unit:gcc:sid:amd64

00:00:43

passed #674271
i386 linux
unit:gcc:sid:i386

00:01:06

passed #674268
amd64 linux
unit:gcc:stretch:amd64

00:01:01

passed #674272
amd64 linux
unit:gcc:tumbleweed:amd64

00:00:54

passed #674274
amd64 linux
unit:gcc:xenial:amd64

00:01:03

failed #674279
amd64 linux allowed to fail
unit:mutexatomics:sid:amd64

00:10:54

passed #674284
amd64 linux
unit:nolibtool:sid:amd64

00:01:14

passed #674281
amd64 linux
unit:pkcs11:sid:amd64

00:00:48

passed #674278
amd64 linux
unit:rwlock:sid:amd64

00:00:56

failed #674277
amd64 linux allowed to fail
unit:tsan:buster:amd64

00:01:49

 
  System
passed #674298
amd64 linux
system:asan:sid:amd64

00:13:27

passed #674303
amd64 freebsd
system:clang:freebsd11.3:amd64

00:12:35

passed #674304
amd64 freebsd
system:clang:freebsd12.0:amd64

00:12:34

passed #674285
amd64 linux
system:gcc:alpine3.11:amd64

00:12:38

passed #674297
amd64 linux
system:gcc:bionic:amd64

00:12:19

passed #674291
amd64 linux
system:gcc:buster:amd64

00:11:33

passed #674286
amd64 linux
system:gcc:centos6:amd64

00:12:14

passed #674287
amd64 linux
system:gcc:centos7:amd64

00:12:01

passed #674288
amd64 linux
system:gcc:centos8:amd64

00:11:23

passed #674295
amd64 linux
system:gcc:fedora31:amd64

00:12:31

passed #674289
amd64 linux
system:gcc:jessie:amd64

00:10:59

passed #674292
amd64 linux
system:gcc:sid:amd64

00:10:59

passed #674293
i386 linux
system:gcc:sid:i386

00:10:49

passed #674290
amd64 linux
system:gcc:stretch:amd64

00:11:31

passed #674294
amd64 linux
system:gcc:tumbleweed:amd64

00:11:36

passed #674296
amd64 linux
system:gcc:xenial:amd64

00:12:06

passed #674301
amd64 linux
system:mutexatomics:sid:amd64

00:11:37

passed #674305
amd64 linux
system:nolibtool:sid:amd64

00:12:33

passed #674302
amd64 linux
system:pkcs11:sid:amd64

00:12:03

passed #674300
amd64 linux
system:rwlock:sid:amd64

00:12:27

failed #674299
amd64 linux allowed to fail
system:tsan:buster:amd64

00:16:02

 
  Docs
passed #674306
amd64 linux
docs:sid:amd64

00:00:54

 
  Postcheck
passed #674308
amd64 linux
cppcheck:gcc:sid:amd64

00:01:54

passed #674307
amd64 linux
scan-build:buster:amd64

00:10:36

 
Name Stage Failure
failed
system:tsan:buster:amd64 System
make: *** [Makefile:486: test] Error 1
Running after script...
$ find bin -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts...
bin/tests/system/*/tsan.*: found 18 matching files
bin/tests/system/*/*/tsan.*: found 270 matching files

tsan/: found 185 matching files

Uploading artifacts to coordinator... ok
id=674299 responseStatus=201 Created token=7rY-E_8S
ERROR: Job failed: exit code 1
failed
unit:mutexatomics:sid:amd64 Unit
Generating kyua_html/lib_ns_tests_query_test_main.html
Generating kyua_html/report.css
Generating kyua_html/index.html
Uploading artifacts...
kyua.log: found 1 matching files
kyua.results: found 1 matching files

kyua_html/: found 69 matching files

Uploading artifacts to coordinator... ok
id=674279 responseStatus=201 Created token=7RBtK6uB
ERROR: Job failed: exit code 1
failed
unit:tsan:buster:amd64 Unit
$ find lib -name 'tsan.*' -exec python3 util/parse_tsan.py {} \;
Uploading artifacts...
WARNING: lib/*/tests/tsan.*: no matching files
WARNING: tsan/: no matching files
kyua.log: found 1 matching files

kyua.results: found 1 matching files

WARNING: kyua_html/: no matching files
Uploading artifacts to coordinator... ok
id=674277 responseStatus=201 Created token=-szFkias
ERROR: Job failed: exit code 1