CDCAGG OAI

This package implements the CESSDA Metadata Aggregator OAI-PMH Repo Handler.

It relies heavily on Kuha OAI-PMH Repo Handler interfaces, but defines it’s own metadataformats and OAI-sets.

New metadataformats can be built as plugins by registering them for discovery via entrypoints. The entrypoints are loaded in cdcagg_oai.serve.

Note

For information on entrypoint discovery see https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata

serve.py

Entrypoint to start serving the OAI-PMH Repo Handler.

Handle command line arguments, application setup, discovery & load of plugins, server startup and critical exception logging.

cdcagg_oai.serve.app_setup(settings, mdformats)[source]

Setup and return Tornado web application

:param argparse.Namespace settings: Loaded settings :param list mdformats: Loaded & configured metadataformats :returns: Tornado web application instance

cdcagg_oai.serve.configure(mdformats)[source]

Configure application.

Define configuration options. Load settings. Configure metadataformats. Setup logging. Return loaded settings.

Parameters:: mdformats (list) – Loaded metadataformats.
Returns:: Loaded settings.
Return type:: argparse.Namespace

cdcagg_oai.serve.main()[source]

Starts the server.

Load metadataformats using entrypoint discovery group cdcagg.oai.metadataformats. Call configure() to define, load and setup configurations. Initiate controller and start server.

metadataformats.py

Define metadataformats and sets of the OAI-PMH Repo Handler.

class cdcagg_oai.metadataformats.AggDCMetadataFormat(oai, corr_id_header)[source]

Define metadataformat for OAI-DC.

Subclass of AggMetadataFormatBase.

This metadataformat is available using metadataprefix oai_dc

classmethod add_cli_args(parser)[source]

Add command line arguments to argument parser.

Parameters:: parser (configargparse.ArgumentParser) – Argument parser.

classmethod configure(settings)[source]

Configure metadataformat with loaded settings.

Parameters:: settings (argparse.Namespace) – Loaded settings.

async get_record(*args, **kwargs)

Get OAI-DC record and prepare metadata response.

Defines a genshi template used in response via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

async list_records(*args, **kwargs)

Get OAI-DC records and prepare metadata response.

Defines a genshi template used in response serialization via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

class cdcagg_oai.metadataformats.AggMetadataFormatBase(oai, corr_id_header)[source]

Base class for Aggregator metadataformat definitions.

Subclass of kuha_oai_pmh_repo_handler.metadataformats.MDFormat

Adds a new template folder, which is relative to this package install location. The folder is added to the list of lookup folders, so Genshi lookup may search for Kuha’s templates and Aggregator’s templates.

Overrides parent’s study_class with Aggregator’s Study record class.

Overrides parent’s sets to keep Kuha’s language and openaire_data OAI sets, and to include Aggregator’s SourceAggMDSet and ConfigurableAggMDSet OAI sets.

Overrides parent’s _header_fields() to include cdcagg_common.records.Study._aggregator_identifier and cdcagg_common.records.Study._provenance.

Overrides parents _get_identifier() to use cdcagg_common.records.Study._aggregator_identifier as local OAI-identifier.

Overrides parents _valid_record_filter() to use cdcagg_common.records.Study._aggregator_identifier for record lookup in DocStore query.

study_class: alias of Study

class cdcagg_oai.metadataformats.AggOAIDDI25MetadataFormat(oai, corr_id_header)[source]

Define metadataformat for OAI-DDI25.

Subclass of AggMetadataFormatBase.

This metadataformat is available using metadataprefix oai_ddi25

classmethod add_cli_args(parser)[source]

Add command line arguments to argument parser.

Parameters:: parser (configargparse.ArgumentParser) – Argument parser.

classmethod configure(settings)[source]

Configure metadataformat with loaded settings.

Parameters:: settings (argparse.Namespace) – Loaded settings.

async get_record(*args, **kwargs)

Get OAI-DDI25 record and prepare metadata response.

Defines a genshi template used in response via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

async list_records(*args, **kwargs)

Get OAI-DDI25 records and prepare metadata response.

Defines a genshi template used in response serialization via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

class cdcagg_oai.metadataformats.AggOAIDataciteMetadataFormat(oai, corr_id_header)[source]

Define metadataformat for OAI-Datacite.

Subclass of AggMetadataFormatBase.

This metadataformat is available using metadataprefix oai_datacite

classmethod add_cli_args(parser)[source]

Add command line arguments to argument parser.

Parameters:: parser (configargparse.ArgumentParser) – Argument parser.

classmethod configure(settings)[source]

Configure metadataformat with loaded settings.

Parameters:: settings (argparse.Namespace) – Loaded settings.

async get_record(*args, **kwargs)

Get OAI-Datacite record and prepare metadata response.

Defines a genshi template used in response via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

async list_records(*args, **kwargs)

Get OAI-Datacite records and prepare metadata response.

Defines a genshi template used in response serialization via GenPlate-decorator.

Returns:: Context used in Genshi XML template.
Return type:: dict

class cdcagg_oai.metadataformats.ConfigurableAggMDSet(mdformat)[source]

Configurable arbitrary OAI set

Groups records to arbitrary sets.

The grouping relies on a mapping file that maps OAI setspecs to record’s aggregator_identifiers. The mapping file is read everytime this class is used to query(), get() or filter() records.

The mapping file is expected to be valid YAML. A single spec-key must be found from top-level. The spec value is used as a top-level OAI setspec value and identifies this MDSet. The nodes-key contains a list of second-level set definitions. The second-level spec-values must be unique and contain list of identifiers that belong to that particular OAI set.

Configuration file syntax example:

spec: 'thematic'
name: 'Thematic'
description: 'Thematic grouping of records'
nodes:
  - spec: 'social_sciences'
    name: 'Social sciences'
    description: 'Studies in social sciences'
    identifiers:
    - id_1
    - id_2
  - spec: 'humanities'
    name: 'Humanities'
    description: 'Studies in humanities'
    identifiers:
    - id_2
    - id_3
    - id_4

The example is intepreted in following way:

thematic is the top-level setspec node.

The top-level node contains two child nodes: social_sciences and humanities

The set thematic:social_sciences contains two records identied by id_1 and id_2.

The set thematic:humanities contains three records identified by id_2, id_3 and id_4.

The identifier id_2 belongs to two sets.

Features & limitations:

Supports hierachical set of records with a single top-level node. Example setspec: top_level_node

Only direct child nodes are supported after the top-level node. Example setspec: top_level_node:child_node

The mapping file YAML syntax is checked on configure() and mandatory keys are validated. Since it is not loaded to memory, it is possible to modify the file contents while the server is operating. This may lead to errors during runtime.

The top-level spec node is used to identify this particular MDSet. For example if the configuration file declares spec: first a request with OAI setspec value first:second implies that the correct MDSet class to consult is this one.

A single node may alternatively specify a path key that points to an absolute path of an external configuration of sets. The external configuration must specify spec, name and identifiers keys and may have an optional description key. The external configuration file can specify a single node or multiple nodes in a list.

Main configuration file with path:

spec: 'thematic'
name: 'Thematic'
description: 'Thematic grouping of records'
nodes:
  - path: '/absolute/path/to/ext/conf.yaml'

External configuration file with a single node:

spec: 'history'
name: 'History'
description: 'Studies in history'
identifiers:
- id_5
- id_6

External configuration file with a list of nodes:

- spec: 'history'
  name: 'History'
  description: 'Studies in history'
  identifiers:
  - id_5
  - id_6
- spec: 'literature'
  name: 'Literature'
  description: 'Literature Studies'
  identifiers:
  - id_7
  - id_8

classmethod add_cli_args(parser)[source]

Add command line arguments to argument parser.

Parameters:: parser (configargparse.ArgumentParser) – Argument parser.

classmethod configure(settings)[source]

Configure set with loaded settings. Validate mapping file.

Parameters:: settings (argparse.Namespace) – Loaded settings.

async fields()[source]

Return list of fields to include when querying for record headers.

This is used when gathering all docstore fields that are needed to construct oai headers.

Returns:: list of fields
Return type:: list

async filter(value)[source]

Return a query filter that includes all studies matching ‘value’.

This is used when constructing docstore query that will include all records in this OAI-set group. In other words, in selective harvesting.

Parameters:: value (str or None) – Requested setspec after colon.
Returns:: query filter
Return type:: dict

async get(study)[source]

Get values from record used in setspec: ‘:<value>’. A None value will leave out the <value> part: ‘<key>’

This is used when constructing setspecs for a specific record.

Parameters:: study – study record to get set values from
Returns:: List of values

async query(on_set_cb)[source]

Query and add distinct values for setspecs

This is used when constructing ListSets OAI response.

Parameters:: on_set_cb – Async callback with signature (spec, name=None, description=None)
Returns:: None

exception cdcagg_oai.metadataformats.InvalidMappingConfig[source]: Raised on invalid config/mapping file contents

class cdcagg_oai.metadataformats.SourceAggMDSet(mdformat)[source]

OAI set grouping records by their originating source archive

The grouping relies on a mapping file that maps a record source url (OAI base url) to a source value. This file is read once on configure() and kept in-memory for the rest of the application run time.

Mapping file syntax:

-
  url: 'http://archive_1.url/oai'
  source: 'Archive_1'
  setname: 'a short human-readable string naming the set source:archive_1'
  description: 'an optional long description for the set source:archive_1'
-
  url: 'http://archive_2.url/oai'
  source: 'Archive_2'
  setname: 'a short human-readable string naming the set source:archive_2'

classmethod add_cli_args(parser)[source]

Add command line arguments to argument parser.

Parameters:: parser (configargparse.ArgumentParser) – Argument parser.

classmethod configure(settings)[source]

Configure set with loaded settings. Load mapping file to memory.

Parameters:: settings (argparse.Namespace) – Loaded settings.

async fields()[source]

Return list of fields to include when querying for record headers.

This is used when gathering all docstore fields that are needed to construct oai headers.

Returns:: list of fields
Return type:: list

async filter(value)[source]

Return a query filter that includes all studies matching ‘value’.

This is used when constructing docstore query that will include all records in this OAI-set group. In other words, in selective harvesting.

Parameters:: value (str or None) – Requested setspec value after colon.
Returns:: query filter
Return type:: dict

async get(study)[source]

Get values from record used in setspec: ‘<key>:<value>’. A None value will leave out the <value> part: ‘<key>’

This is used when constructing setspecs for a specific record.

Parameters:: study – study record to get set values from
Returns:: List of values

async query(on_set_cb)[source]

Query and add distinct values for setspecs

This is used when constructing ListSets OAI response.

Parameters:: on_set_cb – Async callback with signature (spec, name=None, description=None)
Returns:: None