Introduction

Introduction

• pycsw is a OGC CSW server implementation written in Python
• Open Source project released under the MIT license
• Runs on all major platforms (Windows, Linux, Mac OS X)
• OSGeo Project since 11 March 2015

Introduction

• pycsw fully implements the OpenGIS Catalogue Service Implementation Specification [Catalogue Service for the Web]
• pycsw also implements OpenSearch, OAI-PMH, SRU
• pycsw allows for the publishing and discovery of geospatial metadata

Introduction

The project is certified OGC Compliant, and is an OGC Reference Implementation for both CSW 2.0.2 and 3.0.0

pycsw is an official OSGeo Project

Project History

• 2010: Development started by Tom Kralidis
• Feb 2011: Project officially announced
• Apr 2011: First official release (0.1) was released and already passed all CITE tests
• Jul 2011: Version 1.0 released
• Feb 2012: pycsw included in OSGeoLive
• Jan 2013: pycsw 1.4 certified as OGC Compliant
• Apr 2013: pycsw entered OSGeo Incubation
• Feb 2014: pycsw powers data.gov
• Mar 2015: pycsw graduates OSGeo Incubation
• Jul 2016: Latest stable release (2.0.0)
• Jul 2016: Reference implementation of OGC CSW 3.0.0
• Mar 2018: 2.2.0 released
• May 2019: 2.4.0 released

Goals

• Lightweight and easy to setup: a standalone catalogue, no GUI or metadata editing front end, designed for the use case of exposing ready-to-go metadata (files or in existing DB) through a CSW interface
• Extensible: the ability to add metadata formats and mapping them to a common information model and core/additional queryables
• OGC compliant: always pass CITE tests

Features

Features

• Certified OGC Compliant and OGC Reference Implementation for both CSW 2.0.2 and CSW 3.0.0
• Harvesting support for WMS, WFS, WCS, WPS, WAF, CSW, SOS
• Implements ISO Metadata Application Profile 1.0.0
• Implements FGDC CSDGM Application Profile for CSW 2.0
• Implements OGC OpenSearch Geo and Time Extensions
• Implements INSPIRE Discovery Services 3.0
• Supports ISO, Dublin Core, DIF, FGDC, Atom and GM03 metadata models
• Standalone of embedded deployment (CGI or WSGI)
• Transactional capabilities (CSW-T)
• Flexible repository configuration (SQLite, PostgreSQL, PostGIS, MySQL)
• Federated catalogue distributed searching
• Python 2 and 3 compatible

More features...

• Simple configuration
• Extensible plugin architecture (profiles, repositories/backends)
• Seamless integration with Python environments (e.g. GeoNode, HHypermap, Open Data Catalog)
• Integration with CKAN through ckanext-spatial and ckanext-publicamundi
• Includes commandline utility to administer the metadata repository
• Implements the Search/Retrieval via URL (SRU) search protocol
• Implements Open Archives Initiative Protocol for Metadata Harvesting
• Implements Full Text Search capabilities
• Realtime XML Schema validation

Standards Support

• OGC CSW 2.0.2, 3.0.0
• OGC Filter 1.1.0, 2.0.0
• OGC OWS Common 1.0.0, 2.0.0
• OGC GML 3.1.1
• OGC SFSQL 1.2.1
• Dublin Core 1.1
• SOAP 1.2
• ISO 19115 2003
• ISO 19139 2007
• ISO 19119 2005
• NASA DIF 9.7
• FGDC CSDGM 1998
• SRU 1.1
• OAI-PMH 2.0
• OGC OpenSearch 1.0

Supported CSW Operations

• GetCapabilities
• DescribeRecord
• GetRecords
• GetRecordById
• GetRepositoryItem
• GetDomain
• Harvest
• UnHarvest
• Transaction

What's new in pycsw 2.4.0

Harvesting Enhancements

• WMS 1.3.0
• WPS processes

Plugin enhancements

• Abstraction of query filters (previously bound to SQL)
• OGC Filter XML -> Python dict <-> repository query grammar
• Enables search engine backends (Elasticsearch, GraphQL, etc.)

Docker Support

• Official Docker image on Docker Hub
• Documentation

docker run -p 8000:8000 geopython/pycsw

New Branding

Thanks to Amanda Yoshiizumi!

Get your stickers at the OSGeo booth!

More improvements

• Updated testing framework (py.test)
• Translation updates (Chinese)
• Enhanced harvesting workflows
• paging improvements

Meetings

• July 2018: dev meeting
• June 2019: OGC API Hackathon

Architecture

Component Architecture

Software Architecture

Installation

Installation: the proper way

• Python Package Index (PyPI)
• Debian (non-free due to the OGC schema licensing: link1 link2)
• Ubuntu
• openSUSE Application:Geo
• Source code available on GitHub

Installation: 4 Minute Install

# Setup a virtual environment:
virtualenv pycsw && cd pycsw && . bin/activate

# Grab the pycsw source code:
git clone https://github.com/geopython/pycsw.git && cd pycsw
pip install -e . && pip install -r requirements-standalone.txt

# Create and adjust a configuration file:
cp default-sample.cfg default.cfg
vi default.cfg
# adjust paths in
# - server.home
# - repository.database
# set server.url to http://localhost:8000/

# Setup the database:
pycsw-admin.py -c setup_db -f default.cfg

# Load records by indicating a directory of XML files, use -r for recursive:
pycsw-admin.py -c load_records -f default.cfg -p /path/to/xml/

# Run the server:
python ./pycsw/wsgi.py

# See that it works!
curl http://localhost:8000/?service=CSW&version=2.0.2&request=GetCapabilities

		                

OSGeo-Live

• pycsw is available to test in OSGeoLive since version 5.5
• Project Overview and Quickstart Tutorial are included

Downstream Projects and Deployments

GeoNode

Open Source Geospatial Content Management System

• GeoNode is a web-based application and platform for developing geospatial information systems (GIS) and for deploying spatial data infrastructures (SDI).
• pycsw is embeded and enabled as the default CSW server

CKAN

• An Open Source Data Management System for powering data hubs and data portals
• Publishing, sharing, finding and using data
• pycsw is embeded in spatial extension (ckanext-spatial)
• Tight integration implemented in PublicaMundi extension

Open Data Catalog

• Code For America Application
• Open data publishing
• pycsw is embeded and enabled out of the box

HHypermap (Harvard Hypermap)

• HHypermap (Harvard Hypermap) Registry is an application that manages OWS, Esri REST, and other types of map service harvesting, and maintains uptime statistics for services and layers
• pycsw 2.0.0 is embedded

Data.gov

• US Government Open Data portal is based on CKAN
• Since 2014 pycsw is deployed as the data.gov CSW implementation serving 2 million datasets

Recent Deployments

• British Antarctic Survey
• Copernicus Atmosphere Monitoring Service
• Earth Observation Data Centre for Water Resources Monitoring

See more at the pycsw Live Deployments Map

Roadmap

OGC API efforts

• Evolution of OGC API standards
• REST/JSON/OpenAPI/Swagger
• See the pygeoapi presentation for more info
• pycsw is part of the OGC CAT4.0 standards working group

Future OGC standards

• OGC API - Common (OWS Common)
• OGC API - Features (WFS)
• OGC API - Coverages (WCS)
• OGC API - Maps and Tiles (WMS/WMTS)
• OGC API - Processing (WPS)
• OGC API - Records (CSW)

Community

Getting Involved

Community

https://pycsw.org/community.html

Mailing List and IRC

https://lists.osgeo.org/mailman/listinfo/pycsw-devel

Gitter channel, Freenode (#pycsw and #geopython)

Source Code, Wiki, Issues on GitHub

https://github.com/geopython/pycsw

Professional support

OSGeo Service providers

Workshop

Oregon Coastal & Marine Data Network pycsw Workshop materials available at http://www.coastalmarinedata.net/?p=229

Workshop source code available on https://github.com/geopython/pycsw-workshop