Commit 352a5899 authored by Wlodzimierz Wencel's avatar Wlodzimierz Wencel
Browse files

lettuce DNS tests removed

parent 4e9e28dc
/output/
/setup_intree_bind10.sh
noinst_SCRIPTS = setup_intree_bind10.sh
BIND10 system testing with Lettuce
or: to BDD or not to BDD
In this directory, we define a set of behavioral tests for BIND 10. Currently,
these tests are specific for BIND10, but we are keeping in mind that RFC-related
tests could be separated, so that we can test other systems as well.
Prerequisites:
- BIND 10 must be compiled or installed (even when testing in-tree build;
see below) with both DNS and DHCP components enabled
- dig
- lettuce (http://lettuce.it)
To install lettuce, if you have the python pip installation tool, simply do
pip install lettuce
See http://lettuce.it/intro/install.html
Most systems have the pip tool in a separate package; on Debian-based systems
it is called python-pip. On FreeBSD the port is devel/py-pip.
Running the tests
-----------------
At this moment, we have a fixed port for local tests in our setups, port 56176.
This port must be free. (TODO: can we make this run-time discovered?).
Port 56175 is used for cmdctl, and must also be available.
(note, we will need to extend this to a range, or if possible, we will need to
do some on-the-fly available port finding)
You can run the lettuce tests with the provided run_lettuce.sh script.
By default it will use the build tree, but you can use an installed version
of bind10 by passing -I as the first argument of run_lettuce.sh
The tool 'dig' must be in the default search path of your environment. If
you specified -I, so must the main BIND 10 programs. And, with or without
-I, some BIND 10 programs still have to be installed as they are invoked
from test tools. Those include bindctl and b10-loadzone.
Due to the default way lettuce prints its output, it is advisable to run it
in a terminal that is wide than the default. If you see a lot of lines twice
in different colors, the terminal is not wide enough.
If you just want to run one specific feature test, use
run_lettuce.sh [-I] features/<feature file>
To run a specific scenario from a feature, use
run_lettuce.sh [-I] features/<feature file> -s <scenario number>
We have set up the tests to assume that lettuce is run from this directory,
so even if you specify a specific feature file, you should do it from this
directory.
What to do when a test fails
----------------------------
First of all, look at the error it printed and see what step it occurred in.
If written well, the output should explain most of what went wrong.
The stacktrace that is printed is *not* of bind10, but of the testing
framework; this helps in finding more information about what exactly the test
tried to achieve when it failed (as well as help debug the tests themselves).
Furthermore, if any scenario fails, the output from long-running processes
will be stored in the directory output/. The name of the files will be
<Feature name>-<Scenario name>-<Process name>.stdout and
<Feature name>-<Scenario name>-<Process name>.stderr
Where spaces and other non-standard characters are replaced by an underscore.
The process name is either the standard name for said process (e.g. 'bind10'),
or the name given to it by the test ('when i run bind10 as <name>').
These files *will* be overwritten or deleted if the same scenarios are run
again, so if you want to inspect them after a failed test, either do so
immediately or move the files.
If you want to keep these output files even for successful runs, you can
specify the environment variable LETTUCE_KEEP_OUTPUT=1. The files will
still be overwritten by subsequent runs, but they will not automatically be
deleted.
Adding and extending tests
--------------------------
If you want to add tests, it is advisable to first go through the examples to
see what is possible, and read the documentation on http://www.lettuce.it
There is also a README.tutorial file here.
We have a couple of conventions to keep things manageable.
Configuration files go into the configurations/ directory.
Data files go into the data/ directory.
Step definition go into the features/terrain/ directory (the name terrain is
chosen for the same reason Lettuce chose terrain.py, this is the place the
tests 'live' in).
Feature definitions go directly into the features/ directory.
These directories are currently not divided further; we may want to consider
this as the set grows. Due to a (current?) limitation of Lettuce, for
feature files this is currently not possible; the python files containing
steps and terrain must be below or at the same level of the feature files.
Long-running processes should be started through the world.RunningProcesses
instance. If you want to add a process (e.g. bind9), create start, stop and
control steps in terrain/<base_name>_control.py, and let it use the
RunningProcesses API (defined in terrain.py). See bind10_control.py for an
example.
For sending queries and checking the results, steps have been defined in
terrain/querying.py. These use dig and store the results split up into text
strings. This is intentionally not parsed through our own library (as that way
we might run into a 'symmetric bug'). If you need something more advanced from
query results, define it here.
Some very general steps are defined in terrain/steps.py.
Initialization code, cleanup code, and helper classes are defined in
terrain/terrain.py.
To find the right steps, case insensitive matching is used. Parameters taken
from the steps are case-sensitive though. So a step defined as
'do foo with value (bar)' will be matched when using
'Do Foo with value xyz', but xyz will be taken as given.
If you need to add steps that are very particular to one test, create a new
file with a name relevant for that test in terrain. We may want to consider
creating a specific subdirectory for these, but at this moment it is unclear
whether we need to.
We should try to keep steps as general as possible, while not making them to
complex and error-prone.
Quick tutorial and overview
---------------------------
Lettuce is a framework for doing Behaviour Driven Development (BDD).
The idea behind BDD is that you first write down your requirements in
the form of scenarios, then implement their behaviour.
We do not plan on doing full BDD, but such a system should also help
us make system tests. And, hopefully, being able to better identify
what exactly is going wrong when a test fails.
Lettuce is a python implementation of the Cucumber framework, which is
a ruby system. So far we chose lettuce because we already need python
anyway, so chances are higher that any system we want to run it on
supports it. It only supports a subset of cucumber, but more cucumber
features are planned. As I do not know much details of cucumber, I
can't really say what is there and what is not.
A slight letdown is that the current version does not support python 3.
However, as long as the tool-calling glue is python2, this should not
cause any problems, since these aren't unit tests; We do not plan to use
our libraries directly, but only through the runnable scripts and
executables.
-----
Features, Scenarios, Steps.
Lettuce makes a distinction between features, scenarios, and steps.
Features are general, well, features. Each 'feature' has its own file
ending in .feature. A feature file contains a description and a number
of scenarios. Each scenario tests one or more particular parts of the
feature. Each scenario consists of a number of steps.
So let's open up a simple one.
-- example.feature
Feature: showing off BIND 10
This is to show BIND 10 running and that it answer queries
Scenario: Starting bind10
# steps go here
--
I have predefined a number of steps we can use, as we build test we
will need to expand these, but we will look at them shortly.
This file defines a feature, just under the feature name we can
provide a description of the feature.
The one scenario we have has no steps, so if we run it we should
see something like:
-- output
> ./run_lettuce.sh
Feature: showing off BIND 10
This is to show BIND 10 running and that it answer queries
Scenario: Starting bind10
1 feature (1 passed)
1 scenario (1 passed)
0 step (0 passed)
--
Let's first add some steps that send queries.
--
A query for www.example.com should have rcode REFUSED
A query for www.example.org should have rcode NOERROR
--
Since we didn't start any bind10, dig will time out and the result
should be an error saying it got no answer. Errors are in the
form of stack traces (trigger by failed assertions), so we can find
out easily where in the tests they occurred. Especially when the total
set of steps gets bigger we might need that.
So let's add a step that starts bind10.
--
When I start bind10 with configuration example.org.config
--
This is not good enough; it will start the process, but setting up
b10-auth may take a few moments, so we need to add a step to wait for
it to be started before we continue.
--
Then wait for bind10 auth to start
--
And let's run the tests again.
--
> ./run_lettuce.sh
Feature: showing off BIND 10
This is to show BIND 10 running and that it answer queries
Scenario: Starting bind10
When I start bind10 with configuration example.org.config
Then wait for bind10 auth to start
A query for www.example.com should have rcode REFUSED
A query for www.example.org should have rcode NOERROR
1 feature (1 passed)
1 scenario (1 passed)
4 steps (4 passed)
(finished within 2 seconds)
--
So take a look at one of those steps, let's pick the first one.
A step is defined through a python decorator, which in essence is a regular
expression; lettuce searches through all defined steps to find one that
matches. These are 'partial' matches (unless specified otherwise in the
regular expression itself), so if the step is defined with "do foo bar", the
scenario can add words for readability "When I do foo bar".
Each captured group will be passed as an argument to the function we define.
For bind10, I defined a configuration file, a cmdctl port, and a process
name. The first two should be self-evident, and the process name is an
optional name we give it, should we want to address it in the rest of the
tests. This is most useful if we want to start multiple instances. In the
next step (the wait for auth to start), I added a 'of <instance>'. So if we
define the bind10 'as b10_second_instance', we can specify that one here as
'of b10_second_instance'.
--
When I start bind10 with configuration second.config
with cmdctl port 12345 as b10_second_instance
--
(line wrapped for readability)
But notice how we needed two steps, which we probably always need (but
not entirely always)? We can also combine steps; for instance:
--
@step('have bind10 running(?: with configuration ([\w.]+))?')
def have_bind10_running(step, config_file):
step.given('start bind10 with configuration ' + config_file)
step.given('wait for bind10 auth to start')
--
Now we can replace the two steps with one:
--
Given I have bind10 running
--
That's it for the quick overview. For some more examples, with comments,
take a look at features/example.feature. You can read more about lettuce and
its features on http://www.lettuce.it, and if you plan on adding tests and
scenarios, please consult the last section of the main README first.
/bindctl_commands.config
/example.org.config
/generate.config
/root.config
/static.config
/auth_badzone.config
/auth_basic.config
{
"version": 3,
"Logging": {
"loggers": [{
"severity": "DEBUG",
"name": "*",
"debuglevel": 99
}]
},
"Auth": {
"listen_on": [{
"port": 56176,
"address": "127.0.0.1"
}]
},
"data_sources": {
"classes": {
"IN": [
{
"type": "MasterFiles",
"cache-enable": true,
"params": {
"example.org": "data/example.org",
"example.com": "data/example.com-broken",
"example.net": "data/example.net-empty",
"example.info": "data/example.info-doesnt-exist"
}
}
]
}
},
"Init": {
"components": {
"b10-auth": { "kind": "needed", "special": "auth" },
"b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
}
}
}
{
"version": 3,
"Logging": {
"loggers": [ {
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
} ]
},
"Auth": {
"listen_on": [ {
"port": 56176,
"address": "127.0.0.1"
} ]
},
"Init": {
"components": {
"b10-auth": { "kind": "needed", "special": "auth" },
"b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
}
}
}
{
"version": 3,
"Logging": {
"loggers": [ {
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
} ]
},
"Auth": {
"database_file": "data/example.org.sqlite3",
"listen_on": [ {
"port": 56176,
"address": "127.0.0.1"
} ]
},
"data_sources": {
"classes": {}
},
"Init": {
"components": {
"b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
}
}
}
{
"version": 3,
"Logging": {
"loggers": [ {
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
} ]
},
"Auth": {
"database_file": "data/example.org.sqlite3",
"listen_on": [ {
"port": 56176,
"address": "127.0.0.1"
} ]
},
"data_sources": {
"classes": {}
},
"StatsHttpd": {
"listen_on": [ {
"port": 47811,
"address": "127.0.0.1"
} ]
},
"Init": {
"components": {
"b10-auth": { "kind": "dispensable", "special": "auth" },
"b10-xfrin": { "address": "Xfrin", "kind": "dispensable" },
"b10-xfrout": { "address": "Xfrout", "kind": "dispensable" },
"b10-zonemgr": { "address": "Zonemgr", "kind": "dispensable" },
"b10-stats": { "address": "Stats", "kind": "dispensable" },
"b10-stats-httpd": { "address": "StatsHttpd", "kind": "dispensable" },
"b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
}
}
}
{
"version": 3,
"Logging": {
"loggers": [
{
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
}
]
},
"Zonemgr": {
"secondary_zones": [
{
"class": "IN",
"name": "secondary.org"
}
]
},
"Auth": {
"database_file": "data/ddns/example.org.sqlite3",
"listen_on": [
{
"port": 56176,
"address":
"127.0.0.1"
}
]
},
"data_sources": {
"classes": {
"IN": [
{
"type": "sqlite3",
"params": {
"database_file": "data/ddns/example.org.sqlite3"
}
}
]
}
},
"Init": {
"components": {
"b10-xfrout": {
"kind": "dispensable",
"address": "Xfrout"
},
"b10-zonemgr": {
"kind": "dispensable",
"address": "ZoneMgr"
},
"b10-ddns": {
"kind": "dispensable",
"address": "DDNS"
},
"b10-auth": {
"kind": "needed",
"special": "auth"
},
"b10-cmdctl": {
"kind": "needed",
"special": "cmdctl"
}
}
},
"DDNS": {
"zones": [
{
"origin": "example.org.",
"update_acl": [
{
"action": "ACCEPT",
"from": "127.0.0.1"
}
],
"class": "IN"
},
{
"origin": "secondary.org.",
"update_acl": [
{
"action": "ACCEPT",
"from": "127.0.0.1"
}
],
"class": "IN"
}
]
}
}
{
"version": 3,
"Logging": {
"loggers": [
{
"severity": "DEBUG",
"name": "*",
"debuglevel": 99
}
]
},
"DDNS": {"zones": []},
"Auth": {
"database_file": "data/ddns/example.org.sqlite3",
"listen_on": [
{
"port": 56176,
"address": "127.0.0.1"
}
]
},
"data_sources": {
"classes": {
"IN": [
{
"type": "sqlite3",
"params": {
"database_file": "data/ddns/example.org.sqlite3"
},
"cache-enable": true,
"cache-zones": [
"example.org"
]
}
]
}
},
"Init": {
"components": {
"b10-xfrout": {"kind": "dispensable"},
"b10-auth": {"kind": "needed", "special": "auth"},
"b10-zonemgr": {"kind": "dispensable", "address": "ZoneMgr" },
"b10-cmdctl": {"kind": "needed", "special": "cmdctl"}
}
}
}
{
"version": 3,
"Logging": {
"loggers": [ {
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
} ]
},
"StatsHttpd": {
"listen_on": [ {
"port": 47811,
"address": "127.0.0.1"
} ]
}
}
{
"version": 3,
"Logging": {
"loggers": [ {
"debuglevel": 99,
"severity": "DEBUG",
"name": "*"
} ]
},
"Auth": {
"database_file": "data/example.org.sqlite3",
"listen_on": [ {
"port": 56176,
"address": "127.0.0.1"
} ]
},
"data_sources": {
"classes": {
"IN": [
{
"type": "sqlite3",
"params": {
"database_file": "data/example.org.sqlite3"
}
}
]
}
},
"Init": {
"components": {
"b10-auth": { "kind": "needed", "special": "auth" },
"b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
}
}