This RFC describes the replacement of pycsw’s custom testing tools with pytest.
Pycsw’s current testing tool is not using any standard testing framework, it has been developed specifically for pycsw. It is usable and provides effective testing for pycsw. However, it has a number of weak areas that make its usage a bit cumbersome. Specifically:
pavement.py file, but
it is perhaps not ideal that test setup be dependent on additional tooling.
pytest is a mature, full-featured Python testing tool. It is widely used in the Python community. It is a robust tool, available for multiple Python versions and implementations. It is feature complete and also sports a number of extra plugins, allowing for a flexible usage.
Pytest can be used to replace pycsw’s own testing tool. Its usage will improve all of the aforementioned issues with the current tool:
Tests can be selected using pytest markers or by filtering based on the name of each test. It becomes possible to run individual tests or even arbitrary groups of tests. Running all tests, or individual test suites remains possible;
Pytest suppresses the output of passing tests by default, making it easy to focus only on the output of failed tests. This behaviour is easily configurable, and it is possible to show the full output of every test, if needed;
Since it is easy to run individual tests and also to suppress unneeded output, saving test results into new files is not needed anymore;
Pytest has a very flexible way of configuring setup and tear down of test resources, making it possible to start and stop pycsw instances in an integrated way, as part of the testing framework.
Integrating pytest and tox allows running tests in multiple isolated environments with different python versions and database backends.
In addition to the mentioned improvements, pytest brings some additional benefits:
In addition to pytest, it is very common in the python community to also use tox to test the same code base against multiple python versions and dependency combinations. Tox creates multiple virtual environments in a fully automatic way. It can be integrated with pytest and also works well under continuous integration. This enables devs to test pycsw locally against multiple configurations, the same way as it is done on pycsw’s current CI tool, travis
Pytest will be added as an extra requirement for pycsw development. This should not be problematic, as pytest is easily installable using pip and it is available on all platforms targeted by pycsw.
The existing testing tool will be refactored in order to remove the custom test runner code, replacing it with the pytest test runner. Small modficiations to the actual test suites will also be introduced, mostly in order to remove unneeded parameters and moving expected results into their own suite’s directory. The new testing framework will not remove any of the currently available features.
Tox will also be setup to run pycsw’s pytest tests against multiple Python
versions. In the future, this can be extended to also run tests against
multiple pycsw repository backends, such as
Pytest tests will be generated in a similar way to the current implementation:
For each test suite, a new database is always created on each test session,
and an individual pycsw instance is fed a test request; In contrast with the
current implementation, pycsw will not be called through a web server, but
directly by importing
pycsw.core.server. This allows collecting information
on code coverage and also negates the need to spawn an extra process for the
Those suites wich do not require custom data, will still get a new database
populated with data from the
Suites that do not specify a
data/ directory will get an empty database for
testing, as is the case in the current implementation;
A pytest configuration function scans the available suite files and automatically parametrizes a generic test class, creating individual tests for each POST or GET request in the available suites. Each test is named by combining three parameters:
This makes it easy to run only certain tests by using pytest’s
Each test is basically an HTTP request that gets sent to the pycsw instance followed by a comparison of the received response with an expected result. The way in which tests are evaluated is the same as in the current implementation;
Running of tests will be performed by using the
py.test tool, which is
available with pytest’s installation.
Pytest provides flexible mechanisms for configuration and parametrization of tests:
function will be used to automatically generate individual test cases for each
request in the existing
tests/suites directory. The name of each test
will combine the name of the suite with the identifier of each request,
making it trivial to select subsets of tests to run. For example:
py.test -k "cite and get"
The previous command can be used to run all tests from the
cite suite that
are made with an HTTP GET request;
The addition of extra command-line parameters to pytest’s runner will provide a way to specify the necessary configuration of tests. It will be possible to specify the same configuration parameters as with the current tool;
The definition of fixtures.
These are used to start, configure and stop local pycsw instances for each
suite. This means that it is not necessary to resort to an external tool for
configuring and running tests. A technical detail that is possible with this
kind of unification is the a priori definition of a server’s
configuration file, meaning that it is not necessary to use the
KVP parameter in test calls as in the current implementation.
Tox will be used as a frontend for running pytest tests in order to allow quick testing against multiple python versions. It will also be used in the configuration of the Travis CI script, so that there is a consistent way to run tests, locally and on the CI server. Regardless of tox, pytest can also be used standalone when hacking on pycsw’s code to get some quick feedback on test execution;
Since this RFC is about replacing the current testing tool with a new one, the new testing tool will active by default and becomes the only choice for running pycsw’s tests.
Tests will be run using the
py.test command line utility that becomes
available after pytest has been installed. Some example usage:
py.test -h # outputs help on pytest usage
py.test --verbose --exitfirst # be more verbose and exit as soon as one test fails
py.test --capture=no # do not suppress test's stdout
py.test -k cite # only run tests that have 'cite' in their identifiers
py.test -k "cite and post" # only run tests that have 'cite' and 'post' in their identifiers
As mentioned previously, it will also be possible to use tox to test against multiple python versions (the versions must be previously intalled somehow). Some example usage:
sudo pip install tox # must be installed on the base system
tox # runs all tests in python versions 2.6, 2.7, 3.4 and 3.5
tox -e py27-sqlite # runs all tests only on python2.7
tox -e py34-sqlite -- -k 'csw30' # runs only tests from the csw30 suite and only on python 3.4
docs - Relevant documentation will be updated to include information on how to run tests and also on how to add new tests
tests/expected/suites_<suite-name>_<request-type>_<test-id>.xml - All
files will be moved to their suite’s corresponding
tests/functionaltests/suites/<suite-name>/expected/ directory and each
file’s name is simplified to
tests/suites/<suite-name>/get/request.txt - The
PYCSW_SERVER?config=<path-to-config-file> portion of each GET request will
be deleted. This becomes unnecessary, as each suite is backed by a
previously setup pycsw instance that is already configured with the
appropriate configuration file.
tests/run_tests.py - To be deleted. Pytest’s
py.test script becomes the
new test runner
pavement.py - The paver tasks that relate to testing will be deleted.
tests/conftest.py - This is a new file that holds fixtures and
parametrization instructions for setting up the tests;
tests/functionaltests/ - This is a new directory that will contain the
existing tests. In the future it will be possible to add a
directory with new, smaller scoped tests;
tests/functionaltests/conftest.py - This is a new file that holds the
pytest code to automatically generate the functional tests;
tests/functionaltests/test_suites_functional.py - This is a new file that
contains the actual code for performing tests.
requirements-dev.txt - To include pytest and additional plugins as
.travis.yml - To be refactored with the new commands to run tests on
github’s travis CI
The proposed change is backwards incompatible because the whole testing infrastructure will be replaced. However, this change will not affect any user facing functionality. It is strictly oriented towards improving developer experience. Even if the existing test infrastructure will be refactored, the actual subjects under test will not change in any way. The same test requests, same test data and same test suites will continue to be used. As such, all tests will continue to pass, including CITE tests.
Recap of major new functionalities:
py.test command-line tool
Nothing will change on the actual
pycsw code, this change affects only the
As no change will be made to
pycsw’s code, performance will not be altered in
any way. However, the performance of tests will be improved due to the
lxml all the time;
Documentation will be updated as required.
Pull request at: