README 5.51 KB
Newer Older
Jelte Jansen's avatar
Jelte Jansen committed
1
2
3
4
5
6
7
8
BIND10 system testing with Lettuce
or: to BDD or not to BDD

In this directory, we define a set of behavioral tests for BIND 10. Currently,
these tests are specific for BIND10, but we are keeping in mind that RFC-related
tests could be separated, so that we can test other systems as well.

Prerequisites:
9
- BIND 10 must be compiled or installed
Jelte Jansen's avatar
Jelte Jansen committed
10
11
12
13
14
15
16
- dig
- lettuce (http://lettuce.it)

To install lettuce, if you have the python pip installation tool, simply do
pip install lettuce
See http://lettuce.it/intro/install.html

Jelte Jansen's avatar
Jelte Jansen committed
17
18
Most systems have the pip tool in a separate package; on Debian-based systems
it is called python-pip. On FreeBSD the port is devel/py-pip.
Jelte Jansen's avatar
Jelte Jansen committed
19
20
21
22

Running the tests
-----------------

Jelte Jansen's avatar
Jelte Jansen committed
23
At this moment, we have a fixed port for local tests in our setups, port 47806.
Jelte Jansen's avatar
Jelte Jansen committed
24
25
This port must be free. (TODO: can we make this run-time discovered?).
Port 47805 is used for cmdctl, and must also be available.
26
27
(note, we will need to extend this to a range, or if possible, we will need to
do some on-the-fly available port finding)
Jelte Jansen's avatar
Jelte Jansen committed
28

29
You can run the lettuce tests with the provided run_lettuce.sh script.
Jelte Jansen's avatar
Jelte Jansen committed
30

31
32
By default it will use the build tree, but you can use an installed version
of bind10 by passing -I as the first argument of run_lettuce.sh
Jelte Jansen's avatar
Jelte Jansen committed
33

34
35
The tool 'dig' must be in the default search path of your environment. If
you specified -I, so must the main bind10 program, as well as bindctl.
Jelte Jansen's avatar
Jelte Jansen committed
36

37
38
39
40
41
Due to the default way lettuce prints its output, it is advisable to run it
in a terminal that is wide than the default. If you see a lot of lines twice
in different colors, the terminal is not wide enough.

If you just want to run one specific feature test, use
42
run_lettuce.sh [-I] features/<feature file>
43
44

To run a specific scenario from a feature, use
45
run_lettuce.sh [-I] features/<feature file> -s <scenario number>
46

47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
We have set up the tests to assume that lettuce is run from this directory,
so even if you specify a specific feature file, you should do it from this
directory.

What to do when a test fails
----------------------------

First of all, look at the error it printed and see what step it occurred in.
If written well, the output should explain most of what went wrong.

The stacktrace that is printed is *not* of bind10, but of the testing
framework; this helps in finding more information about what exactly the test
tried to achieve when it failed (as well as help debug the tests themselves).

Furthermore, if any scenario fails, the output from long-running processes
will be stored in the directory output/. The name of the files will be
63
64
65
66
67
68
<Feature name>-<Scenario name>-<Process name>.stdout and
<Feature name>-<Scenario name>-<Process name>.stderr
Where spaces and other non-standard characters are replaced by an underscore.
The process name is either the standard name for said process (e.g. 'bind10'),
or the name given to it by the test ('when i run bind10 as <name>').

Jelte Jansen's avatar
Jelte Jansen committed
69
70
71
These files *will* be overwritten or deleted if the same scenarios are run
again, so if you want to inspect them after a failed test, either do so
immediately or move the files.
Jelte Jansen's avatar
Jelte Jansen committed
72

73
74
75
76
77
If you want to keep these output files even for successful runs, you can
specify the environment variable LETTUCE_KEEP_OUTPUT=1. The files will
still be overwritten by subsequent runs, but they will not automatically be
deleted.

78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
Adding and extending tests
--------------------------

If you want to add tests, it is advisable to first go through the examples to
see what is possible, and read the documentation on http://www.lettuce.it

There is also a README.tutorial file here.

We have a couple of conventions to keep things manageable.

Configuration files go into the configurations/ directory.
Data files go into the data/ directory.
Step definition go into the features/terrain/ directory (the name terrain is 
chosen for the same reason Lettuce chose terrain.py, this is the place the 
tests 'live' in).
Feature definitions go directly into the features/ directory.

These directories are currently not divided further; we may want to consider 
this as the set grows. Due to a (current?) limitation of Lettuce, for 
feature files this is currently not possible; the python files containing 
steps and terrain must be below or at the same level of the feature files.

Long-running processes should be started through the world.RunningProcesses
instance. If you want to add a process (e.g. bind9), create start, stop and
control steps in terrain/<base_name>_control.py, and let it use the
RunningProcesses API (defined in terrain.py). See bind10_control.py for an
example.

For sending queries and checking the results, steps have been defined in
terrain/querying.py. These use dig and store the results split up into text
strings. This is intentionally not parsed through our own library (as that way
we might run into a 'symmetric bug'). If you need something more advanced from
query results, define it here.

Some very general steps are defined in terrain/steps.py.
Initialization code, cleanup code, and helper classes are defined in
terrain/terrain.py.

Jelte Jansen's avatar
Jelte Jansen committed
116
117
118
119
120
To find the right steps, case insensitive matching is used. Parameters taken
from the steps are case-sensitive though. So a step defined as
'do foo with value (bar)' will be matched when using
'Do Foo with value xyz', but xyz will be taken as given.

121
122
123
124
125
126
127
If you need to add steps that are very particular to one test, create a new 
file with a name relevant for that test in terrain. We may want to consider 
creating a specific subdirectory for these, but at this moment it is unclear 
whether we need to.

We should try to keep steps as general as possible, while not making them to
complex and error-prone.
Jelte Jansen's avatar
Jelte Jansen committed
128