pycsw Blog

Using keywords from a thesaurus as queryables

Wed, 18 Sep 2024 14:59:00 -0400

Using keywords from a thesaurus as queryables

A common convention in catalogues is the use of keywords from a dedicated thesaurus. The assignment of these keywords can then later be used to filter or query the catalogue by these terms. To achieve this use case in pycsw, some configuration needs to be tailored. This blog post indicates the changes needed for this scenario.

For this example we’ll use a keyword from the INSPIRE Themes thesaurus. We will define a new queryable inspiretheme, which will be populated with the relevant keyword (if present).

You can repeat these steps for any other thesaurus.

Extra database column

Extend the records table in the database with an extra field for the selected thesaurus. This is usually a manual operation on the database.

ALTER TABLE records
ADD inspiretheme VARCHAR(255);

Add parameter to pycsw

In pycsw/core/config.py the newly created database column can be registered to pycsw.

'pycsw:InspireTheme': 'inspiretheme',

Add column to mapping

etc/mappings.py links the pycsw parameter to the columnname in the table.

'pycsw:InspireTheme': 'inspiretheme',

Define parameter as queryable

Which of the parameters are queryable is defined in pycsw/core/repository.py.

'inspiretheme': self.dataset.inspiretheme,

Add parameter to record results?

Keywords are already published in records, so there is generally no need to extend the record with the new parameter. If needed you can do so in pycsw/ogc/api/records.py (Line 1150).

Populate the parameter from record imports

We have 2 options here, either manage the population of the column within the database as part of an insert trigger on the record.themes field. Alternatively update pycsw/core/metadata.py so the column is populated when records are imported.

For the second option consider the following code. For each of the keyword blocks, it tries to match the thesaurus title or uri and, if matched, adds the keywords to the new parameter.

_set(context, recobj, 'pycsw:InspireTheme', ", ".join(
    [", ".join(k.name for k in t.keywords if k.name not in [None,'']) for t in md_identification.keywords if ( hasattr(t,'thesaurus') and 
        t.thesaurus not in [None,''] and ((
            'title' in t.thesaurus and t.thesaurus['title'] not in [None,''] and
            t.thesaurus['title'] in ['GEMET - INSPIRE themes, version 1.0','GEMET Themes, version 2.3']
        ) or (
            'uri' in t.thesaurus and t.thesaurus['uri'] not in [None,''] and
            t.thesaurus['uri'] == 'http://inspire.ec.europa.eu/theme')))]))

Facets enable to further limit search results. Keywords from thesauri are very useful to add as facet. Add the paremeter to default.yml.

facets:
    - type
    - inspiretheme

Dashboarding with pycsw and Apache Superset

Thu, 24 Nov 2022 07:59:00 -0400

Dashboarding with pycsw and Apache Superset

At ISRIC - World Soil Information we’re in the process of adopting pycsw in a data workflow related to the EJP Soil project. One of the project requirenements is to provide a dashboard with catalogue statistics. On one hand to better understand the content of the catalogue, but also to identify gaps in data availability, by location, date and/or topic.

At ISRIC we’re already using Apache Superset in some of our projects, so it was relatively easy to add the pycsw database and set up the initial dashboard vizualisations. The initiative was appreciated, so these days we include the dashboard in each of the pycsw deployments.

Superset is a tool to create dashboard vizualisations on (relational) databases. It is not as full featured as Kibana or Tableau. But it does offer a considerable set of features and fits well in the Open Source aspect of pycsw.

In this blog I’ll give a quick starter on how to get started. Note that for production systems I recommend to set up proper authentication and Redis Cache for improved performance.

The starting point is https://hub.docker.com/r/apache/superset which provides good guidance on how to get started with superset. The guidance suggests to install the World Bank's Health Nutrition and Population Stats samples. Which is a nice example, but it is probably better to continue with a clean environment. A SQLite driver is available by default, but you need to tweak the config file to allow connections to sqlite.

If you aim to connect to a pycsw PostGreSQL database, you need to add the psycopg2 driver to the docker image. Alexander Mencevice prepared some docker compose examples for setting up superset with postgres (and redis).

With a running superset instance you can now login and connect to the pycsw database. Setting up vizualisations on fields such as accessconstraints is quite straight forward.

On datasets create a new dataset based on the connected database and select the records table.
Then the dataset explore screen opens.
Select a chart type (for example pie-chart).
Select group by identifier.
On the simple tab in metrics, select accessconstraints and Count
Save and Run query to get your first chart.

You can now add the chart to a dashboard.

Sharing the dashboard to the public can be done by publishing the dashboard. But also you need to give public access to the datasets behind the charts:

Open the List Roles in Settings.
Edit the public role.
Start typing datasource access on [My database][records](id:7) and select the relevant dataset from autocomplete.

Unnesting json type pycsw fields

In recent versions pycsw introduced a number of fields which contain json. Superset currently is not able to parse json and use it in charts. However you can set up saved queries and use them as datasets. In a saved query you can place advanced PostGres SQL to parse and unnest the json. JSON support in postgres has evolved quite a bit in recent versions. Note that you have to parse the json to string of record(set). In case of record(set) you need to join the result of the json parsing back to the original resultset (unnest). An example of a query to unnest the pycsw links:

SELECT identifier, k.name, k.protocol
FROM 
  records r, 
  json_to_recordset(cast(r.links as json)) as k(name text, protocol text)

Saved queries are created in the SQL lab. Click Explore once the query is ok to save it and create a vizualisation on it. Note that you have to give access to public on the saved query for it to be available publicly.

pycsw May 2022 code sprint report

Sun, 29 May 2022 07:59:00 -0400

pycsw May 2022 code sprint report

The second pycsw code sprint was held on 23 - 25 May 2022. The sprint was held virtually (Jitsi/Gitter) and involved collaboration between both users and developers. A number of areas were discussed and addressed, which are elaborated on in detail below.

CSW Federated search fixes: issue 729 was tested and validated
OWSLib ISO constraint parsing: testing and discussion of OWSLib PR 799
XSLT support: implementation of issue 761 in
SOLR repository backends: initial SOLR backed support was implemented in https://github.com/epifanio/adc-pycsw
JSON storage: as per issue 704, initial JSON support was implemented along with an OGC API - Records parser / ingest
Security, access control and transactions

More information on the sprint can be found on the pycsw wiki.

Thank you to everyone who contributed their time/resources/facilities to the sprint - see you in Florence at FOSS4G 2022!

pycsw May 2022 code sprint

Mon, 18 Apr 2022 14:14:49 -0400

pycsw May 2022 code sprint

pycsw will hold a virtual sprint on 23 - 25 May 2022. The purpose of the sprint will be to focus on features for the 3.0 release (OGC API - Records support, JSON backend support, custom transformations, etc.).

More information on the sprint can be found on the pycsw wiki.

Looking forward to seeing you there!

pycsw turns 10!

Mon, 07 Dec 2020 05:34:39 -0500

10 years – Happy birthday pycsw! 🎉

This year (02 December) marks the 10th anniversary of the pycsw project. From the initial commit on SourceForge, and subsequent announcement, of the project, pycsw has evolved into a stable, production ready and core spatial data infrastructure (SDI) component for disseminating metadata via the OGC CSW specification.

pycsw powers numerous high profile open data / SDI activities, including US Data.gov, LINZ, Geodata.gov.gr, IOOS, NOAA, WMO WOUDC, UN WFP, Copernicus and GEOSS. The flexible nature of the project has also resulted in integration with leading open data management systems such as CKAN and GeoNode.

From day one, pycsw has always strived to be OGC Compliant. The project continues to be a leading OGC Reference Implementation of CSW. In addition, pycsw continues as a long standing OSGeo project and a core component of the growing ecosystem of geopython projects for the open source geospatial community.

Future plans

The project will continue to evolve into the future. pycsw’s flexible nature will continue to support new metadata formats, as well as integration with downstream applications. To support OGC CSW as a baseline OGC web service specification, as well as the evolving OGC API efforts to modernize web service APIs, pycsw will continue to support CSW 2/3, as well as the emerging OGC API - Records specification.

Thank you to the pycsw community, OSGeo, OGC and beyond for your support and use of our lightweight, flexible and OGC compliant metadata catalogue server – happy birthday pycsw!

Sincerly

The pycsw Project Steering Committee

pycsw 2.6.0 released

Sat, 05 Dec 2020 08:47:00 -0500

The pycsw team announces the release of pycsw 2.6.0.

This release brings various enhancements to OpenSearch temporal support, cloud enhancements, and 12 factor support. This release also drops all Python 2 support given the Python 2 end of life which occured on 01 January 2020. Users are strongly encouraged to update their deployments to Python 3 as soon as possible.

Source and binary downloads:

The source code is available at:

https://pycsw.org/download

PyPI packages are available at:

https://pypi.org/project/pycsw

Version 2.6.0 (2020-12-05):

[Bulleted list of enhancements / bug fixes]

fix GetRecords startposition empty parameter fixes
update OpenSearch temporal extent query support
add 12 factor support
support environment variables in configuration
add kubernetes and helm configurations
fix quoting for PostgreSQL backends
add logging switch to pycsw-admin.py CLI to stdout
safeguard XML as bytes to unicode
update core model xml column to Unicode on repository creation
handle different formats for CRS code input
add test for invalid gml:posList geometry
drop all Python 2 support

Testers and developers are welcome.

The pycsw developer team. https://pycsw.org

pycsw 2.4.0 released

Fri, 17 May 2019 13:34:41 -0500

The pycsw team announces the release of pycsw 2.4.0.

Note that though pycsw works with Python 2 and 3, we have turned off Python 2 testing given the Python 2 end of life scheduled for 01 January 2020. Users are strongly encouraged to update their deployments to Python 3 as soon as possible.

Source and binary downloads:

The source code is available at:

https://pycsw.org/download

PyPI packages are available at:

https://pypi.org/project/pycsw

Version 2.4.0 (2019-05-17):

[Bulleted list of enhancements / bug fixes]

fix CAT 3.0 schema locations
fix to handle plugin loading across various operating systems
bump of requirements
new project logos
safeguard WKT exceptions against newer versions of Shapely
updated Chinese translations
enhancements and fixes to large metadata harvesting workflows
safeguard async naming for Python 3.7
OpenSearch description document updates
startposition fixes to for GetRecords workflows

Testers and developers are welcome.

We would like to thank OSGeo and the 2019 Minneapolis Code Sprint organizers and sponsors for their support.

The pycsw developer team. https://pycsw.org/

pycsw 2.2.0 released

Tue, 20 Mar 2018 06:08:22 -0400

The pycsw team announces the release of pycsw 2.2.0.

The 2.2.0 release adds WMS 1.3.0 and WPS process harvesting as well as plugin support enhancements.

Source and binary downloads:

The source code is available at:

http://pycsw.org/download

Version 2.2.0 (2018-03-20):

[Bulleted list of enhancements / bug fixes]

Support overriding PYCSW_ROOT via environment variable
handle malformed basic service options
support Python import for plugins
support WMS 1.3.0 harvesting
implement CQL to Filter transforms
implement WPS process harvesting
fix CQL literals with spaces
include dct:alternative in CSW3 full output
update testing framework to py.test
implement OGC filter parsing as Python dict for easy parsing by repository plugins
do not silence exceptions on custom plugins
support CQL WKT ENVELOPE syntax
check forwarded ip address when pycsw is behind a proxy
add official Docker implementation
fix CSW service / version support (optional in 3.0.0)
fix CSW 3 GetRecords POST handling

Testers and developers are welcome.

We would like to thank OSGeo and the 2018 Bonn Code Sprint organizers and sponsors for their support.

The pycsw developer team. http://pycsw.org/

pycsw Security Releases

Mon, 14 Nov 2016 07:05:44 -0400

The pycsw team announces the release of pycsw 2.0.2, 1.10.5 and 1.8.6.

This is a security release to mitigate an SQL injection vulnerability when using CQL or SRU mode queries (CVE-2016-8640).

You are strongly recommended to update pycsw to the latest stable releases in the 2.0.x, 1.10.x and 1.8.x release lines.

pycsw 2.0.0 "Doug" released

Tue, 12 Jul 2016 13:13:43 +0000

12 July 2016

The pycsw team proudly announces the release of pycsw 2.0.0 “Doug”.

The 2.0.0 “Doug” release brings major features, enhancements and fixes to the codebase, including:

CSW 3 support (OGC Reference Implementation)
Python 3 support
WMTS harvesting (thanks @jfdickens)
JSON output improvements
XML output improvements
GM03 support for Swiss metadata
add temporal extent support to WMS layer harvesting

The full list of enhancements and bug fixes is available at https://github.com/geopython/pycsw/milestone/8. Users are strongly advised to review the migration guide.

The 2.0.0 release is codenamed “Doug” in honour of Doug Nebert of the FGDC. Doug was internationally recognized as a champion of metadata, discovery and interoperability. Involved in numerous international standards bodies and spatial data infrastructure initiatives, Doug was one of the editors of the CSW 3.0 specification and encouraged pycsw developers to adopt and implement CSW 3.0 as part of US data.gov efforts. Doug’s vision and expertise will always be remembered and appreciated by the pycsw development team.

pycsw is an OGC CSW server implementation written in Python.

pycsw fully implements the OpenGIS Catalogue Service Implementation Specification (Catalogue Service for the Web). Initial development started in 2010 (more formally announced in 2011). The project is certified OGC Compliant, and is an OGC Reference Implementation. Since 2015, pycsw is an official OSGeo Project.

pycsw allows for the publishing and discovery of geospatial metadata. Existing repositories of geospatial metadata can also be exposed via numerous APIs (CSW 2/CSW 3, OpenSearch, OAI-PMH, SRU), providing a standards-based metadata and catalogue component of spatial data infrastructures.

pycsw is Open Source, released under an MIT license, and runs on all major platforms (Windows, Linux, Mac OS X).

Source and binary downloads

The source code is available at: http://pycsw.org/download

Testers and developers are welcome.

The pycsw developer team. http://pycsw.org/

pycsw Blog

Using keywords from a thesaurus as queryables

Using keywords from a thesaurus as queryables

Extra database column

Add parameter to pycsw

Add column to mapping

Define parameter as queryable

Add parameter to record results?

Populate the parameter from record imports

Add parameter to OGC API - Records facets

Dashboarding with pycsw and Apache Superset

Dashboarding with pycsw and Apache Superset

Unnesting json type pycsw fields

pycsw May 2022 code sprint report

pycsw May 2022 code sprint report

pycsw May 2022 code sprint

pycsw May 2022 code sprint

pycsw turns 10!

10 years – Happy birthday pycsw! 🎉

Future plans

pycsw 2.6.0 released

Source and binary downloads:

Version 2.6.0 (2020-12-05):

pycsw 2.4.0 released

Source and binary downloads:

Version 2.4.0 (2019-05-17):

pycsw 2.2.0 released

Source and binary downloads:

Version 2.2.0 (2018-03-20):

pycsw Security Releases

pycsw 2.0.0 "Doug" released

Source and binary downloads