Skip to content
GitLab

Resolve "make gtests, shell tests, python tests run sequentially (not concurrently)"

Merged Andrei Pavel requested to merge 1642-gtests-shell-tests-python-tests-are-not-run-sequentially into master

Closes #1642 (closed)

This change is mostly to not force make distcheck -j 1 and to allow make distcheck -j 2 for faster Jenkins testing.

Managed to do it without splitting into separate directories. All that needs to be done is to add the scripts to TESTS and to check_SCRIPTS. Both were missing.

https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html

If the special variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing.

https://www.gnu.org/software/automake/manual/html_node/Scripts.html

Scripts that need not be installed can be listed in noinst_SCRIPTS, and among them, those which are needed only by ‘make check’ should go in check_SCRIPTS.

Apart from that, I felt like these Makefile.ams could use more ordering. Here is the logic that I used:

  • some have SUBDIRS = ., some don't. For consistency, I added it everywhere. It shouldn't make a real diference.
  • ordered automake statements in the order that you would normally call upon them as an autotools user:
    • EXTRA_DIST - make distdir
    • TESTS_ENVIRONMENT - make check (needed when called with valgrind I think)
    • SHTESTS + check_SCRIPTS + TESTS - make check
    • DISTCLEANFILES - make clean
    • noinst_SCRIPTS - make install
    • and then for gtest (C++ tests):
    • PROGRAMTESTS - make check with gtest enabled
    • DISTCLEANFILES - make clean with gtest enabled
    • noinst_SCRIPTS - make install with gtest enabled

The shell and python tests are now considered proper automake tests instead of some random strings that we run alongside make check. You'll see in the libraries that contain shell tests or python tests that instead of 1 test passed you get All 2 tests passed refering to C++ tests (1) and shell/python tests (2). Not sure on what implications this will have, but I'm thinking it can only be good.

Apart from that, some tests now failed because they some variables different paths in make distcheck versus make check. In one instance, these variables pointed to the installation directory which was empty, and is most of the time so. So to avoid this error, I replaced a conditional assignment of KEA_*_DIR to always point to builddir. This seems to have solved the issue.

distcheck run in jenkins on this branch: https://jenkins.aws.isc.org/view/Kea-manual/job/kea-manual/job/distcheck/22/

Edited by Andrei Pavel