"shutdown" system test fails intermittently with AttributeError
A quick investigation leads me to believe that this failure mode is caused by a threading issue in dnspython < 1.16.0:
D:shutdown:
D:shutdown:name = 'dns.rdtypes.IN.A'
D:shutdown:
D:shutdown:def import_module(name):
D:shutdown:mod = __import__(name)
D:shutdown:components = name.split('.')
D:shutdown:for comp in components[1:]:
D:shutdown:> mod = getattr(mod, comp)
D:shutdown:E AttributeError: module 'dns.rdtypes' has no attribute 'IN'
D:shutdown:
D:shutdown:/usr/lib/python3/dist-packages/dns/rdata.py:356: AttributeError
https://gitlab.isc.org/isc-private/bind9/-/jobs/1779765
D:shutdown:=================================== FAILURES ===================================
D:shutdown:_____________________________ test_named_shutdown ______________________________
D:shutdown:
D:shutdown:named_port = 24730, control_port = 24742
D:shutdown:
D:shutdown:@pytest.mark.dnspython
D:shutdown:def test_named_shutdown(named_port, control_port):
D:shutdown:# pylint: disable-msg=too-many-locals
D:shutdown:cfg_dir = os.path.join(os.getcwd(), "resolver")
D:shutdown:assert os.path.isdir(cfg_dir)
D:shutdown:
D:shutdown:cfg_file = os.path.join(cfg_dir, "named.conf")
D:shutdown:assert os.path.isfile(cfg_file)
D:shutdown:
D:shutdown:named = os.getenv("NAMED")
D:shutdown:assert named is not None
D:shutdown:
D:shutdown:rndc = os.getenv("RNDC")
D:shutdown:assert rndc is not None
D:shutdown:
D:shutdown:# rndc configuration resides in ../common/rndc.conf
D:shutdown:rndc_cfg = os.path.join("..", "common", "rndc.conf")
D:shutdown:assert os.path.isfile(rndc_cfg)
D:shutdown:
D:shutdown:# rndc command with default arguments.
D:shutdown:rndc_cmd = [rndc, "-c", rndc_cfg, "-p", str(control_port),
D:shutdown:"-s", "10.53.0.3"]
D:shutdown:
D:shutdown:# We create a resolver instance that will be used to send queries.
D:shutdown:resolver = dns.resolver.Resolver()
D:shutdown:resolver.nameservers = ['10.53.0.3']
D:shutdown:resolver.port = named_port
D:shutdown:
D:shutdown:# We test named shutting down using two methods:
D:shutdown:# Method 1: using rndc ctop
D:shutdown:# Method 2: killing with SIGTERM
D:shutdown:# In both methods named should exit gracefully.
D:shutdown:for kill_method in ("rndc", "sigterm"):
D:shutdown:named_cmdline = [named, "-c", cfg_file, "-f"]
D:shutdown:with subprocess.Popen(named_cmdline, cwd=cfg_dir) as named_proc:
D:shutdown:# Ensure named is running
D:shutdown:assert named_proc.poll() is None
D:shutdown:# wait for named to finish loading
D:shutdown:for _ in range(10):
D:shutdown:try:
D:shutdown:resolver.query('version.bind', 'TXT', 'CH')
D:shutdown:break
D:shutdown:except (dns.resolver.NoNameservers, dns.exception.Timeout):
D:shutdown:time.sleep(1)
D:shutdown:
D:shutdown:do_work(named_proc, resolver, rndc_cmd,
D:shutdown:> kill_method, n_workers=12, n_queries=16)
D:shutdown:
D:shutdown:tests-shutdown.py:174:
D:shutdown:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D:shutdown:tests-shutdown.py:109: in do_work
D:shutdown:result = future.result()
D:shutdown:/usr/lib/python3.5/concurrent/futures/_base.py:398: in result
D:shutdown:return self.__get_result()
D:shutdown:/usr/lib/python3.5/concurrent/futures/_base.py:357: in __get_result
D:shutdown:raise self._exception
D:shutdown:/usr/lib/python3.5/concurrent/futures/thread.py:55: in run
D:shutdown:result = self.fn(*self.args, **self.kwargs)
D:shutdown:/usr/lib/python3/dist-packages/dns/resolver.py:962: in query
D:shutdown:source_port=source_port)
D:shutdown:/usr/lib/python3/dist-packages/dns/query.py:257: in udp
D:shutdown:one_rr_per_rrset=one_rr_per_rrset)
D:shutdown:/usr/lib/python3/dist-packages/dns/message.py:807: in from_wire
D:shutdown:reader.read()
D:shutdown:/usr/lib/python3/dist-packages/dns/message.py:746: in read
D:shutdown:self._get_section(self.message.answer, ancount)
D:shutdown:/usr/lib/python3/dist-packages/dns/message.py:720: in _get_section
D:shutdown:self.message.origin)
D:shutdown:/usr/lib/python3/dist-packages/dns/rdata.py:457: in from_wire
D:shutdown:cls = get_rdata_class(rdclass, rdtype)
D:shutdown:/usr/lib/python3/dist-packages/dns/rdata.py:368: in get_rdata_class
D:shutdown:rdclass_text, rdtype_text]))
D:shutdown:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D:shutdown:
D:shutdown:name = 'dns.rdtypes.IN.A'
D:shutdown:
D:shutdown:def import_module(name):
D:shutdown:mod = __import__(name)
D:shutdown:components = name.split('.')
D:shutdown:for comp in components[1:]:
D:shutdown:> mod = getattr(mod, comp)
D:shutdown:E AttributeError: module 'dns.rdtypes' has no attribute 'IN'
D:shutdown:
D:shutdown:/usr/lib/python3/dist-packages/dns/rdata.py:356: AttributeError
I believe this problem is caused by the import_module()
closure not
locking the threading lock in dnspython < 1.16.0:
-
dnspython 1.15.0
def get_rdata_class(rdclass, rdtype): def import_module(name): mod = __import__(name) components = name.split('.') for comp in components[1:]: mod = getattr(mod, comp) return mod
-
dnspython 1.16.0
_import_lock = _threading.Lock() def get_rdata_class(rdclass, rdtype): def import_module(name): with _import_lock: mod = __import__(name) components = name.split('.') for comp in components[1:]: mod = getattr(mod, comp) return mod
The failure above was triggered on Debian 9 "stretch", which has dnspython 1.15.0.
I am opening this issue merely as a data point - nothing needs to be fixed on our side here.