This RFC describes the replacement of pycsw’s custom testing tools with pytest.
Pycsw’s current testing tool is not using any standard testing framework, nor
unittest package. It has been developed specifically for pycsw.
It is usable and provides effective testing for pycsw. However, it has a
number of weak areas that make its usage a bit cumbersome. Specifically:
pavement.pyfile, but it is perhaps not ideal that test setup be dependent on additional tooling.
pytest is a mature, full-featured Python testing tool. It is widely used in the Python community. It is a robust tool, available for multiple Python versions and implementations. It is feature complete and also sports a number of extra plugins, allowing for a flexible usage.
Pytest can be used to replace pycsw’s own testing tool. Its usage will improve all of the aforementioned issues with the current tool:
In addition to the mentioned improvements, pytest brings some additional benefits:
Pytest will be added as an extra requirement for pycsw development. This should not be problematic, as pytest is easily installable using pip and it is available on all platforms targeted by pycsw.
The existing testing tool will be refactored in order to remove the custom test runner code, replacing it with the pytest test runner. No modification should be needed in the actual test suites, as they are mostly data files. The new testing framework will not remove any of the currently available features.
Pytest tests will be generated in a similar way to the current implementation:
Running of tests will be performed by using the
py.test tool, which is
available with pytest’s installation.
Pytest provides flexible mechanisms for configuration and parametrization of tests:
function will be used to automatically generate individual test cases for each
request in the existing
tests/suites directory. The name of each test
will combine the name of the suite with the identifier of each request, making
it trivial to select subsets of tests to run, using
py.test -k "expression" syntax;
The addition of extra command-line parameters to pytest’s runner will provide a way to specify the necessary configuration of tests. It will be possible to specify the same configuration parameters as with the current tool. These parameters can also be picked up from configuration files or specified as environment variables, which can be used effectively in pycsw’s continuous integration builds;
The definition of fixtures.
These are used to start, configure and stop local pycsw instances for each
suite. This means that it is not necessary to resort to an external tool for
configuring and running tests. A technical detail that is possible with this
kind of unification is the a priori definition of a server’s
configuration file, meaning that it is not necessary to use the
KVP parameter in test calls as in the current implementation.
Since this RFC is about replacing the current testing tool with a new one, the new testing tool will active by default and becomes the only choice for running pycsw’s tests.
tests/run_tests.py- To be deleted.
tests/gen_html.py- To be deleted. Pytest’s
pytest-htmlplugin will be used to provide similar functionality
pavement.py- The paver tasks that relate to testing will be deleted. Pytest’s
py.testtool will be used for running tests
tests/conftest.py- This is a new file that holds fixtures and parametrization instructions for setting up the tests;
tests/test_suites.py- This is a new file that contains the tests to be run
requirements-dev.txt- To include pytest and additional plugins as dependencies
.travis.yml- To be refactored with the new commands to run tests
The proposed change is backwards incompatible because the whole testing infrastructure will be replaced. However, this change will not affect any user facing functionality. It is strictly oriented towards improving developer experience. Even if the existing test infrastructure will be refactored, the actual subjects under test will not change in any way. The same test requests, same test data and same test suites will continue to be used. As such, all tests will continue to pass, including CITE tests.
Recap of major new functionalities:
Documentation will be updated as required.
Pull request has not been issued yet. Implementation is ongoing at: