CDCAGG DocStore

This package contains modules that build the DocStore server.

DocStore provides HTTP APIs for full CRUD support via REST API and flexible query support via Query API. The use of the DocStore HTTP APIs are documented in OpenAPI 3.0 format. OpenAPI documentation resides in package root folder in file openapi.json.

All responses from DocStore are streamed to the requester. Care must be taken to handle the streaming requests correctly. The Aggregator components use Kuha Common client (kuha_common.document_store.client) internally to support streaming responses. Python programs can use the same approach to support streaming responses.

cdcagg_docstore.iter_collections()

Iterates every defined mdb collection.

Currently only collection that is defines is cdcagg_docstore.mdb.studies_collection()

Returns:

Generator that yields collections.

Return type:

iterable

controller.py

Controller is responsible for handling the backend of Document Store.

class cdcagg_docstore.controller.CDCAggDatabase(collections, **kwargs)

CDC Aggregator Database class.

Subclass of DocumentStoreDatabase. Overrides methods _get_record_by_collection_name() and _prepare_validation_schema() to support different records that parent class.

cdcagg_docstore.controller.add_cli_args(parser)

Adds CLI arguments to argument parser.

Parameters:

parser (configargparse.ArgumentParser) – Argument parser

cdcagg_docstore.controller.db_from_settings(settings)

Instantiate CDCAggDatabase from loaded settings

Parameters:

settings (argparse.Namespace) – loaded settings

Returns:

Instance of CDCAggDatabase

Return type:

CDCAggDatabase

db_admin.py

Admin operations to setup and manage CDC OAI Aggregator MongoDB.

For initial setup run the following against a mongodb replicaset:

python -m cdcagg_docstore.db_admin initiate_replicaset setup_database setup_collections setup_users

Setup an empty database into an existing replicaset:

python -m cdcagg_docstore.db_admin setup_database setup_collections setup_users

Drop & re-create collections, for example when indexes have been altered:

python -m cdcagg_docstore.db_admin drop_collections setup_collections

class cdcagg_docstore.db_admin.DBOperations

DBOperations class used as a singleton to load arguments and setup connections. Use with @cli_operation decorated functions.

__init__()

Initiate DBOperations class into an object.

get(name)

Return the wrapped operations function.

Parameters:

name (str) – function name.

Returns:

function

setup(admin_username, admin_password, settings)

Setup the object for operations.

Parameters:

• admin_username (str) – MongoDB admin username

• admin_password (str) – MongoDB admin password

• settings (argparse.Namespace) – Loaded settings

class cdcagg_docstore.db_admin.OperationsSetup(admin_credentials, settings, client, app_db, admin_db)

Operations setup variables.

These are used for setup and operations performed in this module

Parameters:

• admin_credentials (tuple) – DB Admin credentials.

• settings (argparse.Namespace) – Loaded settings

• client (motor.motor_tornado.MotorClient) – MotorClient instance with valid connection uri.

• app_db (motor.motor_tornado.MotorDatabase) – Application database connection.

• admin_db (motor.motor_tornado.MotorDatabase) – Admin database connection.

admin_credentials

Alias for field number 0

admin_db

Alias for field number 4

app_db

Alias for field number 3

client

Alias for field number 2

settings

Alias for field number 1

cdcagg_docstore.db_admin.cli_operation(func)

Decorator for CLI operation functions.

Parameters:

func (function) – Decorated function

Returns:

Wrapper callable without arguments. Calls the decorated function and prints out the result.

cdcagg_docstore.db_admin.configure()

Define and load configuration.

Returns:

Loaded configuration

Return type:

argparse.Namespace

async cdcagg_docstore.db_admin.drop_collections()

CLI operation to drop (remove) collections.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Operation results.

async cdcagg_docstore.db_admin.drop_database()

CLI operation to drop (remove) database.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of MotorClient.drop_database()

async cdcagg_docstore.db_admin.initiate_replicaset()

CLI operation to initiate replicaset.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of ‘repsSetInitiate’

async cdcagg_docstore.db_admin.list_admin_users()

CLI operation to list admin users.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of command ‘usersInfo’ against the admin-database.

async cdcagg_docstore.db_admin.list_collection_indexes()

CLI operation to list collection indexes.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Operation results in dict format, where each key is a collection name and it’s value is a list of indexes.

async cdcagg_docstore.db_admin.list_collections()

CLI operation to list collection names.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of MotorClient[<db_name>].list_collection_names()

async cdcagg_docstore.db_admin.list_databases()

CLI operation to list database names.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of MotorClient.list_database_names()

async cdcagg_docstore.db_admin.list_users()

CLI operation to list application database users.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Results of commands ‘usersInfo’ against the application-database.

cdcagg_docstore.db_admin.main()

Starts db_admin script from command line.

Define configuration arguments & load them. Run every operation in IOLoop.

Returns:

0 on success

async cdcagg_docstore.db_admin.remove_users()

CLI operation to remove application database users.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Results of commands ‘dropUser’ against the application-database with reader and editor credentials.

async cdcagg_docstore.db_admin.setup_collections()

CLI operation to setup database collections.

Creates every collection and sets up it’s indexes and validation.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Operations results in dict, where each key is a name of a collection, value is a list of task results: {<coll_name>: [<task_result_1>, <task_result_2>]}

async cdcagg_docstore.db_admin.setup_database()

CLI operation to setup the application database.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of MotorClient.get_database()

async cdcagg_docstore.db_admin.setup_users()

CLI operation to setup application db users.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Results of commands ‘createUser’ against the application-database.

async cdcagg_docstore.db_admin.show_replicaset_config()

CLI operation to print out replicaset configs.

Parameters:

ops_setup (OperationsSetup) – Setup variables.

Returns:

Result of ‘repsSetGetConfig’

async cdcagg_docstore.db_admin.show_replicaset_status()

CLI operations to print out replicaset status.

Parameters:

ops_setup – Setup variables.

Type:

OperationsSetup

Returns:

Result of ‘repsSetGetStatus’

http_api.py

Defines the HTTP API for DocStore

cdcagg_docstore.http_api.get_app(api_version, collections, **kw)

Format and return WebApplication

Api version and collections are used to build routes to handlers. The handlers and optional keyword arguments are passed onto the WebApplication class.

WebApplication is a subclass of tornado.web.Application.

Parameters:

• api_version (str) – DocStore API version. Gets prepended to all routes.

• collections (list) – Available collections. Every collection gets its own route to REST and QUERY handlers.

Returns:

WebApplication object.

Return type:

kuha_common.server.WebApplication

mdb.py

MongoDB properties

class cdcagg_docstore.mdb.Collection(name, validators, indexes_unique, indexes, isodate_fields, object_id_fields)

Collection object contains properties of a MongoDB collection.

Parameters:

• name (str) – Collection name.

• validators (dict) – MongoDB validators for collection.

• indexes_unique (list) – List of unique MongoDB indexes for collection.

• indexes (list) – List of MongoDB indexes for collection.

• isodate_fields (list) – List of isodate fields for collection.

• object_id_fields (list) – List of collection’s object ID fields.

indexes

Alias for field number 3

indexes_unique

Alias for field number 2

isodate_fields

Alias for field number 4

name

Alias for field number 0

object_id_fields

Alias for field number 5

validators

Alias for field number 1

cdcagg_docstore.mdb.studies_collection()

Initiate and return studies collection.

Returns:

Studies collection object.

Return type:

Collection

serve.py

Entrypoint to start serving the DocStore.

Handle command line arguments, application setup, server startup and critical exception logging.

cdcagg_docstore.serve.configure()

Configure application.

Define configuration options. Load arguments. Setup logging and return loaded settings.

Returns:

Loaded settings.

Return type:

argparse.Namespace

cdcagg_docstore.serve.main()

Starts the server.

Load settings, initiate controller, setup and serve application.

Use as a command line entrypoint.

Returns:

0 on success

Return type:

int