RFC 8: Move to pytest testing tool

Overview

This RFC describes the replacement of pycsw’s custom testing tools with pytest.

Pycsw’s current testing tool is not using any standard testing framework, it has been developed specifically for pycsw. It is usable and provides effective testing for pycsw. However, it has a number of weak areas that make its usage a bit cumbersome. Specifically:

pytest is a mature, full-featured Python testing tool. It is widely used in the Python community. It is a robust tool, available for multiple Python versions and implementations. It is feature complete and also sports a number of extra plugins, allowing for a flexible usage.

Pytest can be used to replace pycsw’s own testing tool. Its usage will improve all of the aforementioned issues with the current tool:

In addition to the mentioned improvements, pytest brings some additional benefits:

In addition to pytest, it is very common in the python community to also use tox to test the same code base against multiple python versions and dependency combinations. Tox creates multiple virtual environments in a fully automatic way. It can be integrated with pytest and also works well under continuous integration. This enables devs to test pycsw locally against multiple configurations, the same way as it is done on pycsw’s current CI tool, travis

Proposed Solution

Pytest will be added as an extra requirement for pycsw development. This should not be problematic, as pytest is easily installable using pip and it is available on all platforms targeted by pycsw.

The existing testing tool will be refactored in order to remove the custom test runner code, replacing it with the pytest test runner. Small modficiations to the actual test suites will also be introduced, mostly in order to remove unneeded parameters and moving expected results into their own suite’s directory. The new testing framework will not remove any of the currently available features.

Tox will also be setup to run pycsw’s pytest tests against multiple Python versions. In the future, this can be extended to also run tests against multiple pycsw repository backends, such as sqlite, postgresql and mysql.

Pytest tests will be generated in a similar way to the current implementation:

Running of tests will be performed by using the py.test tool, which is available with pytest’s installation.

Implementation

Pytest provides flexible mechanisms for configuration and parametrization of tests:

Activating the feature

Since this RFC is about replacing the current testing tool with a new one, the new testing tool will active by default and becomes the only choice for running pycsw’s tests.

Tests will be run using the py.test command line utility that becomes available after pytest has been installed. Some example usage:

py.test -h  # outputs help on pytest usage
py.test --verbose --exitfirst  # be more verbose and exit as soon as one test fails
py.test --capture=no  # do not suppress test's stdout
py.test -k cite  # only run tests that have 'cite' in their identifiers
py.test -k "cite and post"  # only run tests that have 'cite' and 'post' in their identifiers

As mentioned previously, it will also be possible to use tox to test against multiple python versions (the versions must be previously intalled somehow). Some example usage:

sudo pip install tox  # must be installed on the base system
tox  # runs all tests in python versions 2.6, 2.7, 3.4 and 3.5
tox -e py27-sqlite  # runs all tests only on python2.7
tox -e py34-sqlite -- -k 'csw30'  # runs only tests from the csw30 suite and only on python 3.4

Files affected

Backwards Compatibility Issues

The proposed change is backwards incompatible because the whole testing infrastructure will be replaced. However, this change will not affect any user facing functionality. It is strictly oriented towards improving developer experience. Even if the existing test infrastructure will be refactored, the actual subjects under test will not change in any way. The same test requests, same test data and same test suites will continue to be used. As such, all tests will continue to pass, including CITE tests.

Recap of major new functionalities:

Internal Interface changes

Nothing will change on the actual pycsw code, this change affects only the testing infrastructure.

Performance Implications

As no change will be made to pycsw’s code, performance will not be altered in any way. However, the performance of tests will be improved due to the following:

TBD

Restrictions

None

Documentation

Documentation will be updated as required.

Miscellaneous

Issue Tracking ID

https://github.com/geopython/pycsw/issues/428

Pull Request

Pull request at:

https://github.com/geopython/pycsw/pull/495

Voting History

+1 from @amercader, @kalxas, @tomkralidis, @ricardogsilva*, @capooti

* non-PSC

Status

Adopted.

TBD