kuha_common

High-level modules common for Kuha applications.

server.py

Common server functions & classes used in Kuha.

kuha_common.server.log_request(handler)[source]

Gets called after each completed HTTP request.

Note that even requests that end up in exceptions are usually responded, thus successfully handled.

:param tornado.web.RequestHandler handler: Current request handler. :returns: None

kuha_common.server.str_api_endpoint(api_version, suffix=None)[source]

Helper function to prepend endpoints with api_version.

Parameters:

  • api_version (str) – version of the api.

  • suffix (str) – api endpoint.

Returns:

str – endpoint prepended with api_version

kuha_common.server.serve(web_app, port, on_exit=None)[source]

Serve web application.

Parameters:

  • web_app (tornado.web.Application) – application to serve

  • port (int) – Port to listen to.

  • on_exit (function) – callback on server/ioloop stop.

class kuha_common.server.RequestHandler(*args, **kwargs)[source]

Common request handler for kuha server applications. Subclass in application specific handlers.

async prepare()[source]

Prepare each request.

Look for correlation id; create one if not found. Set correlation id to response header. Prepare logging context with HTTP request information.

property correlation_id

Public read-only interface to get response correlation ID.

To get request correlation ID, use request.headers.get(correlation_id.get_header_key())

Returns:

CorrelationID

Return type:

CorrelationID

set_output_content_type(ct, charset='UTF-8')[source]

Sets content type for responses.

Parameters:

  • ct (str) – content type for response.

  • charset (str) – charset definition for response content type.

log_exception(typ, value, tb)[source]

Overrides tornados exception logging. Sends HTTP errors as responses. For easier debugging it also calls tornado.web.RequestHandler.log_exception to output full traceback.

Parameters:

  • typ – type of exception

  • value – caught exception

  • tb – traceback

assert_request_content_type(supported_content_type)[source]

Assert request has correct content type header.

Parameters:

supported_content_type (str) – content type supported by endpoint.

Raises:

InvalidContentType – if request has invalid content type.

get_valid_query_arguments(*valid_keys_occurrences)[source]

Validate and return query arguments.

Validation is performed based on query argument keys. The value is not checked. Request can be checked to contain no query arguments by calling this method without arguments.

Parameters:

valid_keys_occurrences – optional positional arguments contain all valid keys and their expected occurrences. The parameter is a two-tuple: (<key>, <number_of_expected_occurrences>). If the number of expected occurrences is 0, no validation is performed based on occurrences. If the number of expected occurrences is 1, the value is popped out from the list container: {<key>: [<val>]} -> {<key>: <val>}

Returns:

validated query arguments.

Return type:

dict

Raises:

BadRequest for invalid query argument keys.

write_error(status_code, **kwargs)[source]

Overrides tornado.web.RequestHandler.write_error. Outputs error messages in preferred content type.

Parameters:

  • status_code (int) – HTTP status code.

  • **kwargs – keyword arguments are passed to tornado.web.RequestHandler.write_error if output content type is not application/json

class kuha_common.server.WebApplication(*args, **kwargs)[source]
exception kuha_common.server.InvalidContentType(requested_content_type, supported_content_type)[source]

Invalid content type HTTP-exception.

exception kuha_common.server.BadRequest(log_message=None)[source]

Bad request HTTP-exception.

exception kuha_common.server.ResourceNotFound(log_message=None, context=None)[source]

Resource not found HTTP-exception.

exception kuha_common.server.Conflict(log_message=None)[source]

Resource in conflicting state

query.py

Perform query operations against Kuha Document Store.

Offers High-level query methods to facilitate an easy access point with all necessary actions and properties needed to perform queries. To query the Document Store the caller only needs to use methods defined in class QueryController and records defined in kuha_common.document_store.records

class kuha_common.query.ResultHandler(record_constructor, on_record=None)[source]

Class which handles results and correct calls to callbacks, if any. Stores the result for later use.

Dynamically creates callable method handle() which receives result payload and calls on_record correctly.

Parameters:
Returns:

ResultHandler

class kuha_common.query.QueryController(headers=None, record_limit=0)[source]

Asynchronous controller to query the Document Store.

Use to build queries and automatically fetch responses using HTTP as a protocol and JSON as exchange format.

Optional record_limit parameter may be given at initialization time to limit the number of records that are requested in a single HTTP request.

Parameters:

  • headers (dict) – Optional headers parameter to store headers that are used in each query application wide as HTTP headers.

  • record_limit (int) – Optional record_limit parameter which is used to limit the number of records requested in a single HTTP request.

Example:

from kuha_common.document_store import Study
query_ctrl = QueryController()
study = await query_ctrl.query_single(
    Study,
    fields=[Study._metadata, Study.study_number],
    _filter={Study.study_number: 1234}
)
fk_constants

alias of FilterKeyConstants

async query_single(record, on_record=None, headers=None, **kwargs)[source]

Query single record.

Parameters:

  • record (Subclass of kuha_common.document_store.records.RecordBase) – class used to construct the record instance.

  • on_record (function or coroutinefunction) – Optional callback function that gets passed the returned and instantiated record object.

  • headers (dict) – Optional headers for this query. Headers get added to headers given for QueryController at initialization time. Note that it will overwrite headers with same key.

  • **kwargs – Keyword arguments contain parameters for the query. They get passed to kuha_common.document_store.query.Query.construct()

Returns:

None if passed on_record callback, else returns the initiated record object.

Raises:

QueryException – if query parameters given as keyword arguments contain limit-parameter.

async query_multiple(record, on_record, headers=None, **kwargs)[source]

Query multiple records.

Queries the document store for multiple records. Behaviour depends on whether record_limit has been set:

If there is a record_limit

If there is no record_limit

  • this method returns nothing.

  • The on_record callback gets called with each instantiated record object.

  • on_record may be a normal function or a coroutine.

Parameters:
Returns:

None if no record_limit, else kuha_common.document_store.client.JSONStreamClient.run_queued_requests()

async query_count(record, headers=None, **kwargs)[source]

Query the number of records.

Parameters:
Returns:

Number of records

Return type:

int

async query_distinct(record, headers=None, **kwargs)[source]

Query distinct values.

Parameters:
Returns:

distinct values: {fieldname : [value1, value2, value3]}. Note that contained values may be dictionaries or strings, depending on what is stored in requested field.

Return type:

dict

conf.py

Parse command line and store application wide settings.

Typically used to get configuration values using configargparse. However, the module can be used to manually setup configuration values.

Get configuration using configargparse:

>>> load('myprogram')
>>> add('--some-arg')
>>> settings = get_conf()

Manually set configuration values:

>>> add_conf('some_arg', 'some_value')
>>> settings = get_conf()
exception kuha_common.conf.CommandLineParseException[source]

Raised on exceptions during command line parsing

class kuha_common.conf.KuhaConfigFileParser[source]

Inherit to override configargparse.DefaultConfigFileParser.get_syntax_description()

get_syntax_description()[source]

Override syntax description of configargparse.DefaultConfigFileParser

Returns:

Syntax description for configuration file.

Return type:

str

kuha_common.conf.load(prog, package=None, env_var_prefix=None, **kw)[source]

Load configuration parser.

Parameters are passed to configargparse.get_arg_parser() and the parser is stored for later use.

Parameters:

  • prog (str) – Program name

  • package (str) – Python package. Use prog if not given.

  • env_var_prefix (str) – Prefix for environment variables. Optional.

  • default_config_files (list) – Default configuration files. Use CWD/PROG.ini if not given.

  • conflict_handler (str) – Conflict handler. Default to ‘error’. See configargparse and argparse for details.

  • config_file_parser_class (class) – Configuration file parser class. Defaults to KuhaConfigFileParser.

Returns:

Loaded parser

Return type:

configargparse.ArgumentParser

kuha_common.conf.add(*args, **kwargs)[source]

Shorthand adds argument to stored parser

All arguments are passed as is to stored parser. If there is a submitted env_var and load was given env_var_prefix, the environment variable keys are prefixed with loaded env_var_prefix.

Raises CommandLineParseException if configuration has already been loaded.

kuha_common.conf.get(key, default=None)[source]

Get configuration value for key or default.

Use to get a configuration key from loaded settings. If a key is not present, returns default instead.

Parameters:

  • key (str) – Get key from loaded settings

  • default – Return default if key not in settings

Returns:

Configuration value

kuha_common.conf.add_conf(key, value)[source]

Add configuration key-value pair.

Use this to bypass command line parsing.

Parameters:

  • key (str) – Configuration key

  • value – Configuration value

kuha_common.conf.get_conf(conf_key=None)[source]

Parse arguments, store and return them.

If arguments are already parsed and stored, return them.

Note

Throws errors for unrecognized command line arguments and configuration file keys (parse_args()). Use environment variables to enable a single source of truth in project wide scope.

Parameters:

conf_key (str) – Optional parameter to return a single configuration value.

Returns:

Parsed arguments

Return type:

argparse.Namespace

kuha_common.conf.print_conf()[source]

Print active configuration.

Returns:

None

cli_setup.py

Command line setup for Kuha applications

Parse command line for common configuration options and store loaded settings.

kuha_common.cli_setup.MOD_DS_CLIENT = 'document_store.client'

Constant for configuring kuha_common.document_store.client

kuha_common.cli_setup.MOD_DS_QUERY = 'document_store.query'

Constant for configuring kuha_common.document_store.query

kuha_common.cli_setup.MOD_SERVER = 'server'

Constant for configuring kuha_common.server

kuha_common.cli_setup.MOD_LOGGING = 'logging'

Constant for configuring logging

kuha_common.cli_setup.add_logging_configs()[source]

Wrapper to add logging-module configuration.

kuha_common.cli_setup.setup_logging(settings)[source]

Setup py12flogging module.

kuha_common.cli_setup.setup_common_modules(*modules)[source]

Setup command line parser.

Load modules, parse command line arguments, return loaded settings in argparse.Namespace

Parameters:

*modules (str) – common Kuha modules to load and include in parsing of command line arguments.

Returns:

Loaded settings.

Return type:

argparse.Namespace

document_store

Contains modules for interacting with Document Store.

document_store/client.py

kuha_common.document_store.client provides a http client interface to communicate with Kuha Document Store.

class kuha_common.document_store.client.JSONStreamClient[source]

Base class used for requests. Implements own queue to store requests, since tornado.httpclient.AsyncHTTPClient starts timers for request_timeout and connect_timeout at the moment we call client.fetch(). See https://github.com/tornadoweb/tornado/issues/1400 for more details.

Handles JSON decoding of incoming chunks and the encoding of body to JSON.

sleep_on_queue = 5

Sets sleep timer for queue.

request(url, **kwargs)[source]

Constucts a streaming request.

Parameters:

  • url (str) – url to request.

  • kwargs – keyword arguments passed to tornado.httpclient.HTTPRequest

Returns:

tornado.httpclient.HTTPRequest

wrap_streaming_callback(callback)[source]

Wrap streaming callback to support chunked JSON responses.

Parameters:

callback (callable.) – streaming callback. Gets called with response which is decoded to python object from JSON.

Returns:

Wrapped callback

Return type:

functools.partial

async execute_stored_callbacks()[source]

Executes asynchronous callbacks stored in _callbacks

async run_queued_requests(queued_requests=None)[source]

Run queued requests.

Calls queued requests asynchronically. Sleeps for sleep_on_queue if max_clients reached.

Parameters:

queued_requests (collections.deque) – Optionally pass queued_requests to run.

get_streaming_request(streaming_callback, url, body=None, method=None, headers=None, **kw)[source]

Get a streaming request.

Sets default headers Content-Type: application/json if not already given. Encodes body to JSON if given and is not string or bytes. If response is empty (for example query with no results) the streaming callback doesn’t get called.

Subclass and override to support arbitrary requests.

Parameters:

  • streaming_callback (callable) – callback which receives the response if any.

  • url (str) – URL to send request to.

  • body (str, dict, list, tuple, integer, float or None) – Optional request body. String will be supplied as is. Other values will be encoded to JSON.

  • method (str or None) – HTTP method. Defaults to POST.

  • headers (dict or None) – optional request headers. if Content-Type is not set, will set ‘Content-Type’: ‘application/json’ as default.

Returns:

HTTP request

Return type:

tornado.httpclient.HTTPRequest

async queue_request(*args, **kwargs)[source]

Queue request to be run aynchronously by calling run_queued_requests.

Parameters:

  • *args – arguments passed to get_streaming_request

  • **kwargs – keyword arguments passed to get_streaming_request.

Returns:

run_queued_requests() method to call to run the queued requests.

async fetch(*args, **kwargs)[source]

Run single query.

Parameters:

  • *args – arguments passed to queue_requests.

  • **kwargs – keyword arguments passed to queue_requests

document_store/query.py

Access query properties by convenience methods to help build valid queries against the Document Store.

class kuha_common.document_store.query.FilterKeyConstants[source]

Class used as a namespace to contain constants used in query filter.

exception kuha_common.document_store.query.QueryException(msg, context=None)[source]

Exception for errors raised by Query. Has an optional context parameter for giving some additional information about the context of the exception.

class kuha_common.document_store.query.Query(query, query_document, query_type='select')[source]

Manipulate query properties without compromising the validity of the constructed query. Build the correct url for different query types.

Note:

This class provides low-level operations. Use kuha_common.query.QueryController for easy access to query actions and properties.

Example:

from kuha_common.document_store import Study, Query
query = Query(Query.construct(_filter={Study.study_number:'123'}), Study.collection)
Parameters:

  • query (dict) – Actual query containing the properties such as _filter, fields, sort_by etc.

  • query_document (str) – One of the supported query documents declared in Query.supported_query_documents and specified in kuha_common.document_store.records.py

  • query_type (str) – Optional query_type parameter. Defaults to Query.query_type_select. Other valid values are Query.query_type_count and Query.query_type_distinct.

k_filter = '_filter'

Query parameter for filtering results.

k_fields = 'fields'

Query parameter for fields to contain in results.

k_limit = 'limit'

Query parameters for limiting returned results.

k_skip = 'skip'

Query parameter for skipping number of results from the beginning of the resultset.

k_sort_order = 'sort_order'

Query parameter for sort order.

k_sort_by = 'sort_by'

Query parameter for sorting by a certain field.

k_fieldname = 'fieldname'

Query parameter for distinct queries. Specifies a field from which the distinct values are to be fetched.

query_type_select = 'select'

Query type for select queries. Using this query type gets records as a response.

query_type_count = 'count'

Query type for count queries. Using this query type the query returns an integer.

query_type_distinct = 'distinct'

Query type for distinct queries. Using this query type the returning object contains all distinct values for a certain field.

classmethod as_supported_datetime_str(datetime_obj)[source]

Get datetime object as a supported datetime string.

Parameters:

datetime_obj (datetime-object.) – Python datetime-object to be convered to str.

Returns:

String represenation of the datetime-object that is suitable foe querying.

Return type:

str

classmethod construct(**kwargs)[source]

Construct valid query parameters.

Example:

from kuha_common.document_store import Query, Study
params = Query.construct(_filter={Study.study_number:'123'},
                         fields=[Study._metadata, Study._id, Study.abstract],
                         sort_by=Study._id)
query = Query(params, Study.collection)
Parameters:

**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.

Returns:

Valid query, ready to be sent to Document Store.

Return type:

dict

classmethod construct_distinct(**kwargs)[source]

Construct valid query parameters for distinct queries.

Parameters:

**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.

Returns:

Valid query, ready to be sent to Document Store.

Return type:

dict

classmethod build_query_for_date_range(from_=None, until=None)[source]

Build query filter for date-range.

Parameters:

  • from (datetime-object) – start of the date-range:

  • until (datetime-object) – end of the date-range:

Returns:

date-range query-filter with datetime-objects converted into string representation.

Return type:

dict

classmethod build_query_for_exists(exists)[source]

Build query for exists-query.

Parameters:

exists (bool) – whether the field should exists or not.

Returns:

valid exists query for filter.

Return type:

dict

Raises:

ValueError for invalid boolean values in exists-parameter.

classmethod get_valid_params(query_type=None)[source]

Return valid query parameters for the query type.

Parameters:

query_type (str) – Optional query_type for which the query-parameters should be valid for.

classmethod get_required_params(query_type)[source]

Get list of required parameters for query type

Parameters:

query_type (str) – Query type to get required parameters for

Returns:

List of required parameters

classmethod is_valid_query(query, query_type)[source]

Check the validity of query parameters.

Parameters:

  • query (dict) – Full query to validate.

  • query_type (str) – Query type to validate against.

Returns:

Whether or not the query-parameters given are valid.

Return type:

bool

classmethod is_valid_query_type(query_type)[source]

Check the validity of query_type.

Parameters:

query_type (str) – Query type to validate.

Returns:

Whether or not the query type is valid.

Return type:

bool

is_valid_query_document(query_document)[source]

Check the validity of query document.

Parameters:

query_document (str) – Query document to validate.

Returns:

Whether or not the query document is valid.

Return type:

bool

is_valid_param(parameter)[source]

Check the validity of a single query parameter.

Parameters:

parameter (str) – Query parameter to validate.

Returns:

Whether or not the parameter is valid.

Return type:

bool

validate_query(query)[source]

Validate query parameters.

Checks parameters’ validity for chosen query type. Raises QueryException if invalid.

Parameters:

query (dict) – Query parameters.

Returns:

Query parameters.

Return type:

dict

Raises:

QueryException if query parameters are invalid.

validate_query_type(query_type)[source]

Validate query type.

Checks that the query type is supported by Document Store. Raises QueryException for invalid query type.

Parameters:

query_type (str) – Query type to validate.

Returns:

Query type.

Return type:

str

Raises:

QueryException if query type is invalid.

validate_query_document(query_document)[source]

Validates query document.

Checks that the query document is supported by Document Store. Raises QueryException if invalid.

Parameters:

query_document (str) – Query document to validate.

Returns:

Query document

Return type:

str

Raises:

QueryException if query document is invalid.

get_endpoint()[source]

Get correct endpoint for querying the Document Store.

Builds the endpoint by consulting configured values and the instantiated query for query_type and query_document

Returns:

Full url to Document Store endpoint which handles the constructed query.

Return type:

str

get_query(strip_invalid_params=True)[source]

Returns the constructed query parameters.

If the query type has been changed after initialization, for example to get the count of records, this method strips the invalid query parameters from the returned query. When doing so, it does not change the stored query parameters, but rather makes a copy of them for manipulating and returning.

Parameters:

strip_invalid_params (bool) – Whether to strip the unsupported (=invalid) query parameters out of the returned query.

Returns:

Constructed query parameters ready to submit to Document Store.

Return type:

dict

get_limit()[source]

Get query limit parameter.

Returns:

Query limit (int) if set. None if not set.

Return type:

int or None

get_skip()[source]

Get query skip parameter.

Returns:

Query skip (int) if set. None if not set.

Return type:

int or None

set_limit(limit)[source]

Set limit parameter for query.

Limit controls how many results should be returned.

Parameters:

limit (int) – Limit parameter for query.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

set_skip(skip)[source]

Set skip parameter for query.

Skip conrols how many results should be skipped from the start (offset).

Parameters:

skip (int) – Skip parameter for query.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

set_fields(fields)[source]

Set fields parameter for query.

Field controls which fields of the record should be returned. fields can be a list of strings in the form used by MongoDB or a list of kuha_common.document_store.records class-variables.

Example:

from kuha_common.document_store import Query, Study
_params = Query.construct(_filter={Study.study_number:'123'})
_query = Query(_params, Study.collection)
_query.set_fields([Study.abstract, Study.study_number])
Parameters:

fields (list) – Fields parameter for query.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

set_sort_by(sort_by)[source]

Set sort_by parameter for query.

Determines sorting of the returned results. sort_by can be a string in the form used by MongoDB or a kuha_common.document_store.records class-variables.

Parameters:

sort_by (srt or class-variable of a record.) – Sort by parameter for query.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

set_sort_order(order)[source]

Set sort order for the query.

Determines the order which the returned results are to be sorted by.

Note:

Valid values come from pymongo. They actually depend on the mongodb driver, but since this is a caller API we don’t want to make pymongo a dependency.

Parameters:

order (int) – Sort order. Must be either 1 or -1.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

Raises:

QueryException for invalid order values.

set_query_type(query_type)[source]

Set query type.

Parameters:

query_type (str) – Valid query type for the query to be constructed.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

add_query_statement(field, statement)[source]

Add query statement.

Manipulates the _filter parameter of the query parameters. Raises a QueryException if the field already has a statement declared in _filter.

Parameters:

  • field (str) – Field to target the statement to.

  • statement (str) – Statement to filter the results by.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

add_query_statements(**kwargs)[source]

Add multiple query statements to filter the returned results.

Manipulates the _filter parameter of the query parameters.

Parameters:

**kwargs – key-value pairs that are to be added to the _filter parameter.

Returns:

self for easy aggregation of manipulation methods.

Return type:

instantiated Query()

document_store/field_types.py

Properties and actions for field types supported by records defined in kuha_common.document_store.records

Provides field types to be used not only for the construction of new records and updating exising records, but also to provide a format for fields of records that is interchangeable in a way that a receiver does not need to know the specifics of a field beforehand, but may use the field to gain knowledge of the properties of the field.

This module also provides factories which are used to fabricate the fieldtypes. The instantiated factories hold knowledge of the fields even thought the fields themselves are not yet instantiated. This knowledge is used for querying records, but also to dynamically fabricate fieldtypes for records in kuha_common.document_store.records

exception kuha_common.document_store.field_types.FieldTypeException[source]

Exception to raise on field type errors. Used for programming errors.

class kuha_common.document_store.field_types.Value(name, value=None)[source]

Value is the most simple type of field.

Field type with name and single value. Serves also as a baseclass for other field types.

Parameters:

  • name (str) – Name of the field.

  • value – Optional value for the field.

set_value(value)[source]

Set initial value.

Parameters:

value – Value to set.

add_value(value)[source]

Add value for the field.

Note:

Overrides existing value.

Parameters:

value – The value to set.

get_value()[source]

Get the value of the field.

Returns:

Value.

get_name()[source]

Get name of the field.

Returns:

The name of the field.

Return type:

str

export_dict()[source]

Exports Value as dictinary.

Returns:

{name:value}

Return type:

dict

import_records(record)[source]

Import record to field.

Parameters:

record – record to import.

updates(secondary_values)[source]

Update value.

Parameters:

secondary_values (str) – Value to update to.

class kuha_common.document_store.field_types.Set(name, value=None)[source]

Set is a field type with name and list of unique values.

Derived from Value Implements methods that differ from parent class.

Parameters:

  • name (str) – Name of the field.

  • value (list or None) – Optional value for the field.

set_value(value)[source]

Sets value.

Parameters:

value (list) – Value for field.

Raises:

FieldTypeException if submitted value is not a list.

add_value(value)[source]

Add value to field.

Appends a value to the list of values already set. Makes sure that the list holds no duplicates by silently discarding them.

Parameters:

value (list or str or None) – value or values to be appended. If value is None, empties the list.

import_records(record)[source]

Import records by adding the submitted records to contained values.

Parameters:

record (list or str or None) – Hold the values to be imported.

updates(secondary_values)[source]

Updates old values with values contained in this Set.

Looks for combination of secondary_values and values in this set. Discards duplicates and stores the updated values to value.

Parameters:

secondary_values (list) – list of old values to be updated with new ones.

class kuha_common.document_store.field_types.Element(name, attribute_names=None)[source]

Element is a field type with name, value and attributes.

Derived from Value.

Element is used to store fields that contain attributes in addition to a value. Each attribute in itself is an instance of Value and is dynamically stored in instance variable attr_<name>. When instantiated and populated with values and attributes, the element-instance can be used to get it’s value, value’s name, but also to get the attributes and their names, even thought the caller does not know the attribute names a priori.

Example of constructing an element (the source):

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)

Example of reading from an unknown element (the receiver):

>>> unknown_element.get_name()
'animal'
>>> unknown_element.get_value()
'cat'
>>> for att in unknown_element.iterate_attributes():
...     att.get_name() + ' : ' + str(att.get_value())
...
'height : 5'
'color : yellow'
'weight : 10'
>>> unknown_element.attr_color.get_value()
'yellow'

This is especially useful when using as an interchange format. The receiver does not need to know the attribute names beforehand. Instead the receiver can iterate throught every attribute to get their name-value pairs or if the receiver is interested in a single attribute, it may be called by the dynamically constructed instance-variable prefixed with attr_.

Parameters:

  • name (str) – Name of the field.

  • attribute_names (list) – Optional parameter for attribute names.

Raises:

FieldTypeException if attribute_names has duplicates.

is_pending()[source]

Is the element pending for values.

Returns:

True if pending, False if not.

Return type:

bool

new()[source]

Create a new element-instance with same name and attributes but without values.

Instantiates a new instance of itself. The new instance is pending for values.

Example:

>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> another_animal = animal.new()
>>> another_animal.add_value('dog', color='white', weight=30, height=15)
Returns:

new element.

Return type:

Element

add_value(value=None, **attributes)[source]

Add value with attributes as keyword arguments.

Note:

This may only be called once for each instance.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
Parameters:

  • value (str or int or None) – Value for the element.

  • **attributes – keyword arguments for attributes of the element.

Raises:

FieldTypeException if the element already has values or if submitted value is None and no attributes are given.

iterate_attributes()[source]

Generator function. Iterates element attributes.

Returns:

a generator object for iterating attributes.

get_attribute(name)[source]

Get attribute by attribute name.

Parameters:

name (str) – Name of the attribute to look for.

Returns:

attribute of the element or None if not found.

Return type:

Value or None

export_attributes_as_dict()[source]

Export element’s attributes as a dictionary.

Returns:

dictinary representing the attributes

Return type:

dict

export_dict()[source]

Export the element as a dictionary.

Returns a dictionary with key-value pairs given wrapped inside a another dictionary with the elements key as name.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> animal.export_dict()
{'animal': {'color': 'yellow', 'weight': 10, 'height': 5, 'animal': 'cat'}}
Returns:

dictinary representing the Element

Return type:

dict

import_records(record)[source]

Import records.

Import records by adding the submitted records to value and attributes.

Note:

the record parameter is emptied by this method.

Parameters:

record (dict) – Record to import.

updates(secondary_values)[source]

Updates attributes not found in this element with the ones found from secondary_values. Not found in this context means values that are None.

Note:

The parameter is iterated destructively.

Parameters:

secondary_values (dict) – Attributes from old element.

class kuha_common.document_store.field_types.LocalizableElement(name, attribute_names=None)[source]

LocalizableElement is a field type with name, value, language and attributes.

Derived from Element. Has an additional attribute for language. The language is special attribute that is used when updating elements.

Seealso:

Element

Parameters:

  • name (str) – Name of the element.

  • attribute_names (list) – Optional list of attribute names.

Raises:

FieldTypeException if attribute_names contain a name that is reserved for language.

set_language(language)[source]

Set language for element.

Parameters:

language (str) – language to set.

Raises:

FieldTypeException if language already set.

get_language()[source]

Get language of element.

Returns:

language

Return type:

str or None

add_value(value=None, language=None, **attributes)[source]

Add values for element.

Note:

This may only be called once for each instance.

Seealso:

Element.add_value()

Parameters:

  • value (str or int) – value to set.

  • language (str) – language of the element.

  • **attributes – keyword arguments for attributes of the element.

Raises:

TypeError if language is not given or is None.

export_dict()[source]

Export the element as a dictionary.

Seealso:

Element.export_dict()

Returns:

dictinary representation of the element.

Return type:

dict

class kuha_common.document_store.field_types.ElementContainer(name, sub_element)[source]

ElementContainer contains a list of single type of Element/LocalizableElement field types.

Receives mandatory parameters for name and sub_element. The sub_element describes the element types that this container can store. Every new element that a container can create will be an instance created from this sub_element.

Example:

>>> from kuha_common.document_store.field_types import ElementContainer, LocalizableElement
>>> animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> animals = ElementContainer('animals', animal)
>>> animals.add_value('cat', 'en', color='yellow', width=10, height=5)
>>> animals.add_value('kissa', 'fi', color='keltainen', width=10, height=5)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'width': 10,
     'language': 'en',
     'color': 'yellow',
     'height': 5,
     'animal': 'cat'},
    {'width': 10,
     'language': 'fi',
     'color': 'keltainen',
     'height': 5,
     'animal': 'kissa'}]
}

Elements can be iterated:

>>> for animal in animals:
...     animal.attr_color.get_value() + " for language: " + animal.get_language()
...
'yellow for language: en'
'keltainen for language: fi'

And updated with containers sharing name and attribute names:

>>> another_animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> more_animals = ElementContainer('animals', another_animal)
>>> more_animals.add_value('dog', 'en', color='white', width=20, height=10)
>>> more_animals.add_value('koira', 'fi', color='valkoinen', width=20, height=10)
>>> animals.updates(more_animals)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'language': 'en',
     'height': 5,
     'color': 'yellow',
     'animal': 'cat',
     'width': 10},
    {'language': 'fi',
     'height': 5,
     'color': 'keltainen',
     'animal': 'kissa',
     'width': 10},
    {'language': 'en',
     'height': 10,
     'color': 'white',
     'animal': 'dog',
     'width': 20},
    {'language': 'fi',
     'height': 10,
     'color': 'valkoinen',
     'animal': 'koira',
     'width': 20}]
}
Parameters:
Raises:

FieldTypeException for invalid sub_element.

import_records(record)[source]

Imports records from a list of dictionaries.

Note:

The dictionaries will lose information.

Parameters:

record (list) – list of dictionaries with records to import.

add_value(value=None, language=None, **kwargs)[source]

Add new element to list of elements

Parameters:

  • value (str or int or None) – value for the new element.

  • language (str or None) – language for the new element.

  • **kwargs – key-value pairs for attributes of the new element.

Raises:

FieldTypeException for invalid language parameter depending on whether the sub_element is localizable.

export_dict()[source]

Export container as dictionary.

Returns:

dictionary representing the container.

Return type:

dict

iterate_values_for_language(language)[source]

Generator for iterating contained elements by language.

Parameters:

language (str) – language which is used to filter yielded results.

Returns:

a generator object for iterating elements

get_available_languages()[source]

Get list of languages for this container.

Returns:

list of distinct languages.

Return type:

list

updates(secondary_values)[source]

Updates contained values with secondary_values.

Looks for values that are not currently contained, and appends them as contained values. Also appends different language versions. If a language version has the same value, looks for differences in attributes. If new value has not the same attributes as the old one, adds these attributes to the new value. If old value has same attribute name, it will be discarded.

Note:

Document Store uses MongoDB as a backend. MongoDB deals with JSON-like objects, which in turn are best represented in Python as dictionaries. The purpose of kuha_common.document_store.records is to be used as a global (in Kuha context) interchange format and so it will be best to support both dictionaries and ElementContainers for this operation. Therefore there is some flexibility in the type of parameter that this method accepts.

Note:

There is a logical difference in which type of parameter is submitted to this method. When using other types than ElementContainers, the parameter’s content will be changed.

Parameters:

secondary_values (instance of ElementContainer or dict or list) – Old values known to have the same container (must have the same name). If secondary_values is a list, it is assumed that the caller has explicitly checked that the parameter represents old values for this container. Otherwise the name of the container will be checked here and KeyError exceptions will be raised.

class kuha_common.document_store.field_types.FieldAttribute(name, parent=None)[source]

Common attributes for each field type.

Stores fields name, parent fields name and constructs a path for the field. This path can be used when building queries against Document Store. The name can be used to lookup values from objects returned from Document Store.

Used by FieldTypeFactory to store information of fields that can be used before the field has been fabricated.

Parameters:

  • name (str) – name of the field.

  • parent (str) – optional parameter parent. Used for sub-elements.

value_from_dict(_dict)[source]

Get value or values corresponding to path from parameter.

Note:

Returned values cannot be separated by language afterwards.

Parameters:

_dict (dict) – dictionary to lookup for path.

Returns:

value or values stored in path of the _dict.

Return type:

str or list or None

class kuha_common.document_store.field_types.FieldTypeFactory(name, sub_name=None, attrs=None, localizable=True, single_value=False)[source]

Factory for field types.

Stores information for each field, that can be used before the field actually has been initiated. This is useful for building queries against Document Store, because the caller needs to know the names and paths of the fields about to be queried.

The attributes stored here are also used to fabricate each field type. This means that each of the field types supported by kuha_common.document_store.records are to be initiated throught this factory.

Seealso:

ElementContainer

Example:

>>> from kuha_common.document_store.field_types import FieldTypeFactory
>>> animals_factory = FieldTypeFactory('animals', 'animal', ['color', 'width', 'height'])
>>> animals_factory.attr_color.name
'color'
>>> animals_factory.attr_color.path
'animals.color'
>>> animals = animals_factory.fabricate()
>>> animals.add_value('cat', 'en', color='yellow', height=10, width=5)
>>> animals.export_dict()
{'animals': [{'color': 'yellow', 'animal': 'cat', 'height': 10, 'width': 5, 'language': 'en'}]}
Parameters:

  • name (str) – name of the field.

  • sub_name (str) – name of the sub field, if any.

  • attrs (list or str) – field attributes, if any. Multiple attributes in list.

  • localizable (bool) – is the field localizable.

  • single_value (bool) – The fabricated field can contain only a single value.

Raises:

ValueError if attribute has same name as the element or sub_element.

Raises:

FieldTypeException for parameter combinations that are not supported.

fabricate()[source]

Fabricate field type by factory attributes.

Returns the correct type of field type based on attributes given to the factory at initialization time.

Returns:

Instance of one of the fields types.

document_store/records.py

Models for records supported by Document Store.

Due to its schemaless design, the document store relies heavily on these models. Use these models when building importers.

kuha_common.document_store.records.datetime_to_datestamp(_datetime)[source]

Convert datetime object to datestamp string supported by Document Store.

Parameters:

datetime (datetime.datetime) – datetime to convert.

Returns:

converted datestamp.

Return type:

str

kuha_common.document_store.records.datestamp_to_datetime(datestamp)[source]

Convert datestamp string to datetime.datetime object.

Parameters:

datestamp (str) – datestamp to convert.

Returns:

converted datetime.

Return type:

datetime.datetime

kuha_common.document_store.records.datetime_now()[source]

Get current datetime in supported format.

Returns:

Supported datetime object representing current time.

Return type:

datetime.datetime

kuha_common.document_store.records.path_split(path)[source]

Split path into steps.

Parameters:

path (str) – record attribute path.

Returns:

path steps

Return type:

list

kuha_common.document_store.records.path_join(step, *steps)[source]

Join two or more steps into a record attribute path by inserting a separator between the steps.

Parameters:

  • step (str) – step used to create path.

  • *steps (str) –

    steps used to create path.

Returns:

Record attribute path.

Return type:

str

kuha_common.document_store.records.dig_and_set(target, attr, replacement, allow_none=False)[source]

Convert value corresponding to attr found from target dict / list of dicts.

Dig into target by consulting attr path. attr may either be an instance of Attribute or a string. If replacement is a callable call it with the value found from path. If it is a fixed value, insert it in place of value found from path. If allow_none is set, raise exception if path contains no value (is None or target has no corresponding key).

Parameters:

target (dict or list) – Target of replacement. May be a list or a list of dicts

:param Attribute or str attr: Record Class attribute or path string :param callable or value replacement: callable returns new value or fixed value. If its a

callable, it receives old value as positional argument.

Parameters:

allow_none (bool) – False raises ValueError if target has no value for attr.

Returns:

None

class kuha_common.document_store.records.RecordBase(document_store_dictionary=None)[source]

Baseclass for each record.

Provides methods used to import, export, create and update records. Dynamically fabricates each class variable of type FieldTypeFactory into an instance variable overriding the class variable.

Note:

Use this class throught subclasses only.

Parameters:

document_store_dictionary (dict) – Optional parameter for creating a record at initialization time. Note that this dictionary will be iterated destructively.

classmethod get_collection()[source]

Get record collection.

Collection is used for queries against the Document Store.

Returns:

collection of the record.

Return type:

str

classmethod iterate_record_fields()[source]

Iterate class attributes used as record fields.

Iteration returns tuples: (attribute_name, attribute)

Returns:

generator for iterating record fields.

export_metadata_dict(as_datestamps=True)[source]

Export record metadata as dictionary.

Parameters:

as_datestamps (bool) – Convert metadata datetime-objects to string timestamps.

Returns:

record’s metadata

Return type:

dict

export_dict(include_metadata=True, include_id=True)[source]

Return JSON serializable dictionary representation of a record.

Return JSON serializable dictionary. Datetimes will be converted to datestamps.

Parameters:

  • include_metadata (bool) – export includes metadata

  • include_id (bool) – export includes id

Returns:

record

Return type:

dict

set_updated(value=None)[source]

Set updated timestamp.

Sets updated metadata attribute.

Note:

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters:

value (datetime.datetime or str) – Optional timestamp to set.

set_created(value=None)[source]

Set created timestamp.

Sets created metadata attribute.

Note:

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters:

value (datetime.datetime or str) – Optional timestamp to set.

set_deleted(value=None)[source]

Set deleted timestamp.

Sets deleted metadata attribute.

Note:

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters:

value (datetime.datetime or str) – Optional timestamp to set.

set_cmm_type(value=None)[source]

Set cmm type.

Parameters:

value (str) – Optional type to set.

set_id(value)[source]

Set ID.

Parameters:

value (str) – id to set.

get_updated()[source]

Get updated value.

Note:

The timestamp is stored as a datetime.datetime in _metadata.attr_updated, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns:

updated timestamp.

Return type:

str

get_created()[source]

Get created value.

Note:

The timestamp is stored as a datetime.datetime in _metadata.attr_created, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns:

created timestamp.

Return type:

str

get_deleted()[source]

Get deleted value.

Note:

The timestamp is stored as a datetime.datetime in _metadata.attr_deleted, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns:

deleted timestamp.

Return type:

str

get_id()[source]

Get record ID.

Id comes from the backend storage system.

Returns:

record ID in storage.

Return type:

str or None

is_deleted()[source]

Return True if record is logically deleted

Returns:

True if record has been deleted.

bypass_update(*fields)[source]

Add fields to be bypassed on update operation.

Parameters:

*fields (str) – fieldnames to bypass.

bypass_create(*fields)[source]

Add fields to be bypassed on create operation.

Parameters:

*fields (str) – fieldnames to bypass.

updates_record(old_record_dict)[source]

Update record by appending old values that are not present in current record. Use old record’s _id and _metadata.created if present.

Note:

parameter is a dictionary since MongoDB returns records as JSON-like objects, which in turn are best represented as dictionaries in python.

Parameters:

old_record_dict (dict) – Old record as a dictionary.

updates(secondary_record)[source]

Update record by appending values from secondary which are not present in this record.

Parameters:

secondary_record (Record instance subclassed from RecordBase) – lookup values from this record.

class kuha_common.document_store.records.Study(study_dict=None)[source]

Study record.

Derived from RecordBase. Used to store and manipulate Study records. Study number is a special attribute and it cannot be updated.

All attributes of the record are declared as class variables initiated from kuha_common.document_store.field_types.FieldTypeFactory. Instance methods defined in this class are used to add/set values to record attributes. The signatures of the methods are dynamically constructed by the definition of the FieldTypeFactory instances. If, for example, there is a class variable definition:

animals = FieldTypeFactory('animals', 'animal', ['color', 'weight', 'height'])

The correct method signature should be:

def add_animals(self, value, language, color=None, weight=None, height=None):

For the dynamic nature of the record-model these signatures are left open, and python’s *args and **kwargs are used instead. Note that the field type used will raise exceptions if keyword argument key is not found in the initial definition of the field type.

Create a new study record:

>>> study = Study()
>>> study.add_study_number(1234)
>>> study.add_study_titles('Study about animals', 'en')
>>> study.add_principal_investigators('investigator', 'en', organization='Big organization ltd.')

Import existing study record from dictionary:

>>> study_dict = {'study_number': 1234,
... 'study_titles': [{'study_title': 'Study about animals', 'language': 'en'}],
... 'principal_investigators': [{'principal_investigator': 'investigator',
... 'language': 'en', 'organization': 'Big organization ltd.'}]}
>>> study = Study(study_dict)

Iterate attributes:

>>> for pi in study.principal_investigators:
...     pi.attr_organization.get_value()
...
'Big organization ltd.'
Seealso:

RecordBase and kuha_common.document_store.field_types

Parameters:

study_dict (dict) – Optional study record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number is used to identify a study. It must be unique within records, not localizable and contain only a single value. It cannot be updated.

persistent_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Persistent identifiers. Multivalue-field with unique values.

identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Identifiers. Localizable field with agency-attribute. This needs to be localizable for the sake of agency-attribute.

study_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study titles. Localizable, multivalue-field without attributoes.

document_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document titles. Localizable, multivalue-field without attributoes.

parallel_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Parallele study titles. Localizable, multivalue-field without attributes.

principal_investigators = <kuha_common.document_store.field_types.FieldTypeFactory object>

Pricipal investigators. Localizable, multivalue-field with organization-attribute.

publishers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publishers. Localizable, multivalue-field with abbreviation-attribute.

distributors = <kuha_common.document_store.field_types.FieldTypeFactory object>

Distributors. Localizable, multivalue-field with abbreviation and uri attributes.

document_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document URIs. Localizable, multivalue-field with location and description attributes.

study_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study URIs. Localizable, multivalue-field with location and description attributes.

publication_dates = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication dates. Localizable, multivalue-field without attributes. Note that these are treated as strings, not datetime-objects.

publication_years = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication years. Localizable, multivalue-field with distribution date attribute.

distribution_dates = <kuha_common.document_store.field_types.FieldTypeFactory object>

Distribution dates. Localizable, multivalue-field with description attribute.

abstract = <kuha_common.document_store.field_types.FieldTypeFactory object>

Abstract. Localizable, multivalue-field.

classifications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Classifications. Localizable, multivalue-field with system name, uri and description attributes.

keywords = <kuha_common.document_store.field_types.FieldTypeFactory object>

Keywords. Localizable, multivalue-field with system name, uri and description attributes.

time_methods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Time methods. Localizable, multivalue-field with system name, uri and description attribute.

sampling_procedures = <kuha_common.document_store.field_types.FieldTypeFactory object>

Sampling procedures. Localizable, multivalue-field with description, system name and uri attibutes.

collection_modes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection modes. Localizable, multivalue-field with system name and uri attritubes.

analysis_units = <kuha_common.document_store.field_types.FieldTypeFactory object>

Analysis units. Localizable, multivalue-field with system name, uri and description attributes.

research_instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Research instruments. Localizable, multivalue-field with system name, uri and description attributes.

collection_periods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection periods. Localizable, multivalue-field with event and description attributes.

data_kinds = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data kinds. Localizable, multivalue-field.

study_area_countries = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study area countries. Localizable, multivalue-field with abbreviation attribute.

geographic_coverages = <kuha_common.document_store.field_types.FieldTypeFactory object>

Geographic coverages. Localizable, multivalue-field.

universes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Universes. Localizable, multivalue-field with included attribute.

data_access = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access. Localizable, multivalue-field.

data_access_descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access descriptions. Localizable, multivalue-field with element_version attribute.

citation_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Citation requirements. Localizable, multivalue-field.

deposit_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Deposit requirements. Localizable, multivalue-field.

file_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

File names. Localizable, multivalue-field.

instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Instruments. Localizable, multivalue-field with instrument name attribute.

related_publications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Related publications. Localizable multivalue-field

study_groups = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study groups. Localizable, multivalue-field with name and description attributes.

copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

data_collection_copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

funding_agencies = <kuha_common.document_store.field_types.FieldTypeFactory object>

Funding agencies. Localizable, multiple-field with role, agency, abbreviation, description and grant number attributes.

grant_numbers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Grant numbers. Localizable, multivalue-field with agency and role attributes.

collection = 'studies'

Database collection (table) for persistent storage.

cmm_type = 'study'

CMM type for Study.

add_study_number(value)[source]

Add study number.

Note:

despite the name, the value does not need to be a number.

Parameters:

value (str or int) – study number.

add_persistent_identifiers(value)[source]

Add persistent identifiers

Parameters:

value (str or int) – persistent identifier

add_identifiers(value, *args, **kwargs)[source]

Add identifiers.

Parameters:
add_study_titles(value, *args, **kwargs)[source]

Add study titles.

Parameters:
add_document_titles(value, *args, **kwargs)[source]

Add document titles.

Parameters:
add_parallel_titles(value, *args, **kwargs)[source]

Add parallel titles.

Parameters:
add_principal_investigators(value, *args, **kwargs)[source]

Add principal investigators.

Parameters:
add_publishers(value, *args, **kwargs)[source]

Add publishers.

Parameters:
add_distributors(value, *args, **kwargs)[source]

Add distributors.

Parameters:
add_document_uris(value, *args, **kwargs)[source]

Add document URIs.

Parameters:
add_study_uris(value, *args, **kwargs)[source]

Add study URIs.

Parameters:
add_publication_dates(value, *args, **kwargs)[source]

Add publication dates.

Parameters:
add_publication_years(value, *args, **kwargs)[source]

Add publication dates.

Parameters:
add_distribution_dates(value, *args, **kwargs)[source]

Add distribution dates.

Parameters:
add_abstract(value, *args, **kwargs)[source]

Add abstract.

Parameters:
add_classifications(value, *args, **kwargs)[source]

Add classifications.

Parameters:
add_keywords(value, *args, **kwargs)[source]

Add keywords.

Parameters:
add_time_methods(value, *args, **kwargs)[source]

Add time methods.

Parameters:
add_sampling_procedures(value, *args, **kwargs)[source]

Add sampling procedures

Parameters:
add_collection_modes(value, *args, **kwargs)[source]

Add collection modes

Parameters:
add_analysis_units(value, *args, **kwargs)[source]

Add analysis units.

Parameters:
add_research_instruments(value, *args, **kwargs)[source]

Add research instruments.

Parameters:
add_collection_periods(value, *args, **kwargs)[source]

Add collection periods.

Parameters:
add_data_kinds(value, *args)[source]

Add data kinds.

Parameters:
add_study_area_countries(value, *args, **kwargs)[source]

Add study area countries.

Parameters:
add_geographic_coverages(value, *args)[source]

Add geographic coverages

Parameters:
add_universes(value, *args, **kwargs)[source]

Add universes.

Parameters:
add_data_access(value, *args, **kwargs)[source]

Add data access.

Parameters:
add_data_access_descriptions(value, *args, **kwargs)[source]

Add data access descriptions.

Parameters:
add_citation_requirements(value, *args, **kwargs)[source]

Add citation requirements.

Parameters:
add_deposit_requirements(value, *args, **kwargs)[source]

Add deposit requirements.

Parameters:
add_file_names(value, *args, **kwargs)[source]

Add file name.

Parameters:
add_instruments(value, *args, **kwargs)[source]

Add instrument.

Parameters:

Add related publications.

Parameters:
add_study_groups(value, *args, **kwargs)[source]

Add study group.

Parameters:
add_copyrights(value, *args, **kwargs)[source]

Add copyright.

Parameters:
add_data_collection_copyrights(value, *args, **kwargs)[source]

Add data collection copyrights.

Parameters:
add_funding_agencies(value, *args, **kwargs)[source]

Add funding_agencies.

Parameters:
add_grant_numbers(value, *args, **kwargs)[source]

Add grant numbers.

Parameters:
add_document_description_notes(value, *args, **kwargs)[source]

Add document description notes.

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:

secondary (Study) – Lookup values to update from secondary record.

Returns:

True if record updated, False if not.

Return type:

bool

class kuha_common.document_store.records.Variable(variable_dict=None)[source]

Variable record.

Derived from RecordBase. Used to store and manipulate variable records. Study number and variable name are special attributes and cannot be updated.

Seealso:

Study documentation for more information.

Parameters:

variable_dict (dict) – Optional variable record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and variable name are used to identify a variable within variable records. Their combination must be unique withing variable records, they cannot be localizable, and they can only contain a single value. They also cannot be updated.

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name within a study. See also study_number

question_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifiers, if variable refers to a question. Not localizable, multiple unique values.

variable_labels = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable labels. Localizable, multivalue-field.

codelist_codes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist codes. Localizable, multivalue-field with label and missing attributes.

collection = 'variables'

Database collection for persistent storage.

cmm_type = 'variable'

CMM type for variable.

add_study_number(value)[source]

Add study number.

Parameters:

value (str or int.) – study number.

add_variable_name(value)[source]

Add variable name.

Parameters:

value (str) – variable name.

add_question_identifiers(value)[source]

Add question identifier

Parameters:

value (str or int.) – question identifier.

add_variable_labels(value, *args, **kwargs)[source]

Add variable label

Parameters:
add_codelist_codes(value, *args, **kwargs)[source]

Add codelist code

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:

secondary (Variable) – Lookup values to update from secondary record.

Returns:

True if record updated, False if not.

Return type:

bool

class kuha_common.document_store.records.Question(question_dict=None)[source]

Question record.

Derived from RecordBase. Used to store and manipulate question records. study_number and question_idenntifier are special attributes and cannot be updated.

Seealso:

Study documentation for more information.

Parameters:

question_dict (dict) – Optional question record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and question identifier are used to identify a question. Their combination must be unique withing records, they must not be localizable and they can only contain a single value. They also cannot be updated.

question_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifier within a study. See also study_number

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name that specifies the variable for the question. Not localizable, single value.

question_texts = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question texts. Localizable, multivalue-field.

research_instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Research instruments. Localizable, multivalue-field.

codelist_references = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist references. Localizable, multivalue-field.

collection = 'questions'

Database collection for persistent storage.

cmm_type = 'question'

CMM type for question

add_study_number(value)[source]

Add study number.

Parameters:

value (str or int.) – study number.

add_question_identifier(value)[source]

Add question identifier

Parameters:

value (str or int.) – question identifier.

add_variable_name(value)[source]

Add variable name.

Parameters:

value (str) – variable name.

add_question_texts(value, *args, **kwargs)[source]

Add question text

Parameters:
add_research_instruments(value, *args, **kwargs)[source]

Add research instrument

Parameters:
add_codelist_references(value, *args, **kwargs)[source]

Add codelist reference

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:

secondary (Question) – Lookup values to update from secondary record.

Returns:

True if record updated, False if not.

Return type:

bool

class kuha_common.document_store.records.StudyGroup(study_group_dict=None)[source]

Study group record.

Derived from RecordBase. Used to store and manipulate study group records. study_group_identifier is a arpecial attribute and cannot be updated.

Seealso:

Study documentation for more information.

Parameters:

study_group_dict (dict) – Optional study group record as dictionary used for constructing a record instance.

study_group_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group identifier. Used to identify study group. Must be unique within study groups, cannot be localizable and can contain only a single value. This value cannot be updated.

study_group_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group names. Localizable, multivalue-field.

descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group descriptions. Localizable, multivalue-field.

uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group URIs. Localizable, multivalue-field.

study_numbers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study numbers. Multivalue-field with unique values.

collection = 'study_groups'

Database collection for persistent storage.

cmm_type = 'study_group'

CMM type for study groups.

add_study_group_identifier(value)[source]

Add study group identifier.

Parameters:

value (str or int) – Study group identifier.

add_study_group_names(value, *args, **kwargs)[source]

Add study group names

Parameters:
add_descriptions(value, *args)[source]

Add study group descriptions

Parameters:
add_uris(value, *args)[source]

Add study group URIs

Parameters:
add_study_numbers(value)[source]

Add study number.

Parameters:

value (str or int) – study number.

updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:

secondary (StudyGroup) – Lookup values to update from secondary record.

Returns:

True if record updated, False if not.

Return type:

bool

kuha_common.document_store.records.record_factory(ds_record_dict)[source]

Dynamically construct record instance based on given document store dictionary.

Looks up the correct record by the cmm type found from ds_record_dict metadata.

Parameters:

ds_record_dict (dict) – record received from Document Store.

Returns:

Record instance.

Return type:

Study or Variable or Question or StudyGroup

kuha_common.document_store.records.record_by_collection(collection)[source]

Finds a record class by the given collection.

Parameters:

collection (str) – collection of the record.

Returns:

record class

Return type:

Study or Variable or Question or StudyGroup

Raises:

KeyError if collection is not found in any record.

document_store/mappings

document_store/mappings/exceptions.py

Exceptions for mapping package.

exception kuha_common.document_store.mappings.exceptions.InvalidMapperParameters[source]

Raise for invalid configuration of a mapper - Coding error.

For example trying to add attributes to a mapper which does not support attributes. These errors are coding errors and should be treated as such.

exception kuha_common.document_store.mappings.exceptions.MappingError[source]

Raise for errors while mapping input - User error.

Subclass to create more precise error classes.

exception kuha_common.document_store.mappings.exceptions.ParseError[source]

Unable to parse source XML.

Note:

Mask ElementTree ParseError so caller may use MappingError to catch all user-errors when mapping.

exception kuha_common.document_store.mappings.exceptions.UnknownXMLRoot(unknown, expected, *expecteds)[source]

Unknown root element.

exception kuha_common.document_store.mappings.exceptions.MissingRequiredAttribute(*xpaths, msg=None)[source]

Source does not contain a required attribute.

exception kuha_common.document_store.mappings.exceptions.InvalidContent[source]

Attribute found but contains invalid data.

document_store/mappings/xmlbase.py

Components to use for XML parsing & mapping to Document Store records.

Contains a base class to use for parsing XML to Document Store records. Provides common functions useful in parsing XML data.

class kuha_common.document_store.mappings.xmlbase.MappedParams(value)[source]

Contains parameters ready to pass to record’s add-methods.

XMLMapper retrieves parameters from XML record and stores them in an instance of this class. The record instances add-methods get called with stored parameters by using tuple and dict unpacking.

Example:

mapped_params = MappedParams('study_identifier')
mapped_params.set_language('en')
mapped_params.keyword_arguments.update({'agency': 'archive'})
study = Study()
study.add_identifiers(*mapped_params.arguments, **mapped_params.keyword_arguments)
Parameters:

value (str or None) – value used as the first argument

has_language()[source]

True if MappedParams has language argument

Returns:

True if has language, False if not.

Return type:

bool

set_language(language)[source]

Set language argument. Will overwrite if previously set.

Parameters:

language (str) – Language to set.

get_language()[source]

Get language argument.

Returns:

Language

Return type:

str

get_value()[source]

Get value argument.

Returns:

value.

Return type:

str or None

copy()[source]

Make a copy of the object with contents and return the copy.

Returns:

copy of this MappedParams

Return type:

MappedParams

has_arguments()[source]

Return True if MappedParams has arguments or keyword_arguments.

Returns:

True if object has arguments or keyword_arguments.

Return type:

bool

class kuha_common.document_store.mappings.xmlbase.XMLMapper(xpath, from_attribute=None, required=False, localizable=True)[source]

XMLMapper populates MappedParams instances from XML.

Parameters:

  • xpath (str) – XPath where to look for element containing value.

  • from_attribute (str or None) – Look value from attribute of the element.

  • required (bool) – raises MissingRequiredAttribute if value is not found.

  • localizable (bool) – True if the value is localizable, False if not.

set_value_conversion(conv_func)[source]

Set conversion callable.

Note:

conv_func must accept a string or None as a parameter and return the converted value.

Parameters:

conv_func (callable.) – Callable used for conversion.

Returns:

self

set_value_getter(getter_func)[source]

Set value getter callable.

Note:

getter_func must accept an XML element xml.etree.ElementTree.Element as a parameter and return the value.

Parameters:

getter_func (callable.) – Callable used for getting a value from XML element.

Returns:

self

expect_single_value()[source]

This mapper will be expected to return a single value.

Returns:

self

expect_multiple_values()[source]

This mapper will be expected to return multiple values.

Returns:

self

disable_attributes()[source]

This mapper will not contain any attributes.

Returns:

self

iterate_attributes(*relations)[source]

Iterate attributes to map.

Parameters:

*relations (str) – optional parameters to iterate only attributes with a certain relation.

Returns:

A generator yielding tuples of each attribute in the format: (attribute_name, attribute_mapper, attribute_provides_main_lang)

Return type:

generator

as_params(element, default_language, xml_namespaces)[source]

Use mapping to construct a MappedParams from XML element.

Use mapper’s _value_getter and _value_conversion to get value from XML element. Construct a MappedParams from the value. If mapping localizable is True add language from XML elements xml:lang attribute.

Parameters:

  • element (xml.etree.ElementTree.Element) – XML element.

  • default_language (str) – default language if element has none.

  • xml_namespaces (dict) – XML namespaces for the element.

Returns:

mapped parameters ready to pass to records add-method.

Return type:

MappedParams

add_attribute(att_name, mapper, relative=True, provides_main_lang=False)[source]

Add attribute to mapper.

Counts the correct xpath if attribute’s mapper’s xpath is a parent element (starting with ‘/..’). Includes all needed information of the attribute to a list of tuples contained in attributes.

Parameters:

  • att_name (str) – attribute name

  • mapper (XMLMapper) – mapper instance for mapping value for the attribute.

  • relative (bool) – Is the attribute map’s xpath relative to this element. Defaults to True.

  • provides_main_lang (bool) – Should the language of the attribute be used as a language when mapping this value. Defaults to False.

Raises:

InvalidMapperParameters for conflicting parameters such as: 1. Calling this method on a mapper which has disabled use of attributes. 2. Using provides_main_lang for a non-localizable mapper. 3. Setting relative to False on a mapper whose xpath refers to parent element.

Returns:

self

value_params(source_xml_element, default_language, xml_namespaces, position=None)[source]

Generate sinle MappedParams object from source XML.

Parameters:

  • source_xml_element (xml.etree.ElementTree.Element) – XML element.

  • default_language (str) – Default language.

  • xml_namespaces (dict) – XML namespaces.

  • position (int or None) – Optional position for parent xpaths.

Returns:

generator yielding MappedParams.

Return type:

generator

Raises:

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.

values_params(source_xml_element, default_language, xml_namespaces)[source]

Generate multiple MappedParams objects from source XML.

The generated MappedParams will contain attributes as keyword_arguments.

Parameters:
Returns:

generator yielding MappedParams.

Return type:

generator

Raises:

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.

class kuha_common.document_store.mappings.xmlbase.XMLParserBase(root_element)[source]

Base class where parsers get derived from.

Declares the public API to be used in callers.

Input:

  • from_file()

  • from_string()

Output:

  • studies

  • variables

  • questions

  • study_groups

  • all

  • select(collection=None)

Provides common functionality to be used within subclasses which map XML-data to Document Store records. Subclasses must implement necessary generators that generate document store records.

Use in subclass:

class XMLRecordParser(XMLParserBase):
    @property
    def studies(self):
        maps = [(Study.add_study_number, self._map_single(xpath_to_study_number, required=True)),
                (Study.add_study_titles, self._map_multi(xpath_to_study_title))]
        for study_element in self.root_element.findall(xpath_to_study_element, self.NS):
            study = Study()
            self._map_to_record(study, study_element, maps)
            yield study
Parameters:

root_element (xml.etree.ElementTree.Element) – XML root.

NS = {'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

default_language = 'en'

Default language.

classmethod from_string(xml_body)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters:

xml_body (str) – XML Document as a string. This may come directly from HTTP request body.

Returns:

parser for iteratively parsing XML and generating Document Store records.

Return type:

XMLParserBase

classmethod from_file(filepath)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters:

filepath (str) – Path for the XML file.

Returns:

parser for iteratively parsing XML and generating Document Store records.

Return type:

XMLParserBase

classmethod child_text(xpath)[source]

Returns a function which will lookup a child element from given xpath. The returned function takes a single element as a parameter which should be an xml.etree.ElementTree.Element or similar. When executed the function returns the child element’s text contents or None if child element cannot be found.

Parameters:

xpath – xpath to child. relative to parent.

Returns:

function which accepts the parent element as a parameter.

Return type:

function

property root_element

Get root element.

Returns:

Root element

Return type:

xml.etree.ElementTree.Element

property root_language

Get language of the root element. If root does not have a language, returns self.default_language.

Returns:

root element language.

Return type:

str

property study_number

Get study number as formatted in source XML.

Seealso:

self.study_number_identifier

Returns:

Study number from source XML.

Return type:

str

property study_number_identifier

Get study number converted as a valid Document Store identifier.

Returns:

Study number as valid Document Store identifier.

Return type:

str

property studies

Studies generator. Must be implemented in subclass.

Returns:

Generator which yields Document Store studies.

property variables

Variables generator. Must be implemented in subclass.

Returns:

Generator which yields Document Store variables.

property questions

Questions generator. Must be implemented in subclass.

Returns:

Generator which yields Document Store questions.

property study_groups

Study groups generator. Must be implemented in subclass.

Returns:

Generator which yields Document Store study groups.

property all

Iterate all records found from source XML.

Returns:

Generator which yields Document Store records.

Return type:

Generator

select(collection=None)[source]

Returns a selective parser. Call with a Document Store collection as parameter to select records only for certain collection.

Note

The returned attributes are defined in subclasses, so they may or may not be generators.

Parameters:

collection (str or None) – Document Store collection to select only records belonging to this collection.

Returns:

Generator which yields Document Store records.

Return type:

Generator

kuha_common.document_store.mappings.xmlbase.as_valid_identifier(candidate)[source]

Convert candidate to a string that conforms the rules of validation.

Indentifier must match regex: [a-zA-Z0-9]+[a-zA-Z0-9?_()-.]*’”]

Note

Regex is defined in Document Store. Should it be moved to kuha_common?

Returns:

identifier which conforms the rules of validation.

Return type:

str

kuha_common.document_store.mappings.xmlbase.str_equals(correct, default=None)[source]

Conversion function wrapper to compare strings for equality.

Wrapper function that formats comparison value and default value for returned comparison function.

Check if string found from element value or element attribute equals to correct.

Parameters:

  • correct (str) – comparison string.

  • default (str) – If the value parameter of the comparison function is None, return this value.

Returns:

function which accepts a single parameter for comparison. Returns True or False, or default if the parameter is None.

Return type:

function

kuha_common.document_store.mappings.xmlbase.fixed_value(fixed)[source]

Fixed value.

Parameters:

fixed – Use this value

Returns:

function which accepts a single argument value. The function always returns fixed.

Return type:

function

kuha_common.document_store.mappings.xmlbase.element_remove_whitespaces(element)[source]

Conversion function to remove extra whitespace from end of element text.

Iterates element’s inner text using xml.etree.ElementTree.Element.itertext() which iterates over this element and all subelements. Removes extra whitespaces so paragraphs of text will only have one separating whitespace character.

Parameters:

element (xml.etree.ElementTree.Element) – Element from which to get text.

Returns:

Element’s inner text without extra whitespace.

Return type:

str

kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text(element, tags_to_remove=None)[source]

Conversion function to remove inner elements and their contents.

Parameters:

  • element (xml.etree.ElementTree.Element) – Element for lookup.

  • tags_to_remove (list) – Optional list of tags to remove from results text content. If given, will only remove elements that match tags that are found from this list. If not given, will remove all child elements. Tags that are given in list but not found from element children will be silently ignored.

Returns:

Element’s inner text without text from descendants and without extra whitespace.

Return type:

str

kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text_by_tag(tag, *tags)[source]

Conversion function to remove inner elements by tag name.

Acts as a closure function that formats the tags_to_remove parameter and proxies to element_strip_descendant_text().

Parameters:

tag (str) – Tag name to remove. Repeat for multiple tags.

Returns:

lambda that proxies to element_strip_descendant_text()

kuha_common.document_store.mappings.xmlbase.get_preferred_publication_id_agency_pair(ids_agencys)[source]

Get preferred id + agency two-tuple.

Due to limitations in Kuha data model, a Study.related_publication item can only contain a single identifier + agency pair. This function will search the preferred id + agency pair by consulting the agency values.

Preference by priority:

  1. DOI

  2. Handle

  3. URN

  4. ARK

  5. <first ID with agency>

  6. <first ID without agency>

Params list ids_agencys:

list of id + agency two-tuples

Returns:

id + agency two-tuple.

Return type:

tuple

document_store/mappings/ddi

Package containing DDI mappings to Kuha records.

document_store/mappings/ddi/codebook.py

Mapping profiles for DDI-codebook versions.

class kuha_common.document_store.mappings.ddi.codebook.DDI122RecordParser(root_element)[source]

Parse Document Store records from DDI 1.2.2. XML.

property studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns:

Generator to Populate Document Store Study record.

property variables

Parse XML to create and populate multiple kuha_common.document_store.records.Variable instances.

Returns:

Generator to populate multiple Document Store Variable records.

property questions

Parse XML to create and populate multiple kuha_common.document_store.records.Question instances.

Returns:

Generator to populate multiple Document Store Question records.

property study_groups

Parse XML to create and populate multiple kuha_common.document_store.records.StudyGroup instances.

Returns:

Generator to populate multiple Document Store StudyGroup records.

class kuha_common.document_store.mappings.ddi.codebook.DDI122NesstarRecordParser(root_element)[source]
class kuha_common.document_store.mappings.ddi.codebook.DDI25RecordParser(root_element)[source]

Parse Document Store records from DDI 2.5 XML.

NS = {'ddi': 'ddi:codebook:2_5', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

document_store/mappings/ddi/lifecycle.py

Common base for DDI Lifecycle (version 3.x) mapping profiles

class kuha_common.document_store.mappings.ddi.lifecycle.DDILifecycleParserBase(root_element)[source]
property variables

Parse XML to create and populate kuha_common.document_store.records.Variable.

Returns:

Generator to Populate Document Store Variable records.

property questions

Parse XML to create and populate kuha_common.document_store.records.Question.

Returns:

Generator to Populate Document Store Question records.

property study_groups

Parse XML to create and populate kuha_common.document_store.records.StudyGroup.

Returns:

Generator to Populate Document Store StudyGroup records.

class kuha_common.document_store.mappings.ddi.lifecycle.DDILifecycle32ParserBase(root_element)[source]

Common base for DDI 3.2 & DDI 3.3

DDI 3.2 changes how references work when comparing to DDI 3.1. It also introduces FragmentInstances.

property studies

Studies generator. Must be implemented in subclass.

Returns:

Generator which yields Document Store studies.

document_store/mappings/ddi/ddi31.py

Mapping profiles for DDI 3.1.

class kuha_common.document_store.mappings.ddi.ddi31.DDI31RecordParser(root_element)[source]

Parse Document Store records from DDI 3.1. XML

Check the root element. Expects either ddi:DDIInstance or s:StudyUnit. Currently supports only single s:StudyUnit element within the root.

Parameters:

root_element (xml.etree.ElementTree.Element) – XML root element.

Raises:

UnknownXMLRoot for unexpected root element.

Raises:

MappingError if root contains more or less that exactly one s:StudyUnit child.

NS = {'a': 'ddi:archive:3_1', 'c': 'ddi:conceptualcomponent:3_1', 'dc': 'ddi:datacollection:3_1', 'ddi': 'ddi:instance:3_1', 'g': 'ddi:group:3_1', 'l': 'ddi:logicalproduct:3_1', 'pd': 'ddi:physicaldataproduct:3_1', 'pi': 'ddi:physicalinstance:3_1', 'r': 'ddi:reusable:3_1', 's': 'ddi:studyunit:3_1', 'xhtml': 'http://www.w3.org/1999/xhtml', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

property studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns:

Generator to Populate Document Store Study record.

document_store/mappings/ddi/ddi33.py

Mapping profiles for DDI 3.3

class kuha_common.document_store.mappings.ddi.ddi33.DDI33RecordParser(root_element)[source]

XML Record Parser for DDI 3.3 files

DDI 3.3 introduces OtherMaterialSchemes and uses attributes controlledVocabularyName and controlledVocabularyURN attributes to define controlled vocabularies.

NS = {'a': 'ddi:archive:3_3', 'c': 'ddi:conceptualcomponent:3_3', 'dc': 'ddi:datacollection:3_3', 'ddi': 'ddi:instance:3_3', 'g': 'ddi:group:3_3', 'l': 'ddi:logicalproduct:3_3', 'pd': 'ddi:physicaldataproduct:3_3', 'pi': 'ddi:physicalinstance:3_3', 'r': 'ddi:reusable:3_3', 's': 'ddi:studyunit:3_3', 'xhtml': 'http://www.w3.org/1999/xhtml', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

testing

Package for common testing functions and classes.

kuha_common.testing.time_me(func)[source]

Decorate function to print its execution time to stdout.

Note:

test runner may capture the output.

Parameters:

func – Function to decorate. Count execution time of function.

kuha_common.testing.mock_coro(dummy_rval=None, func=None)[source]

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters:

  • dummy_rval – return value of dummy function.

  • func – function to call instead of original mocked out function.

Returns:

coroutine function.

kuha_common.testing.MockCoro(dummy_rval=None, func=None)

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters:

  • dummy_rval – return value of dummy function.

  • func – function to call instead of original mocked out function.

Returns:

coroutine function.

testing/testcases.py

Test cases for Kuha

class kuha_common.testing.testcases.KuhaUnitTestCase(methodName='runTest')[source]

Base class for unittests.

  • Assertion methods to check record equality.

  • Helper methods to provide access to dummydata.

dummydata_dir = '/home/docs/checkouts/readthedocs.org/user_builds/kuha2/envs/latest/src/kuha-common/kuha_common/testing/dummydata'

Override in sublass to lookup dummydata from different directory.

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass()[source]

Hook method for deconstructing the class fixture after running all tests in the class.

classmethod get_dummydata_path(path)[source]

Get absolute path to dummydatafile

Parameters:

path – Path. Gets turned into an absolute if it isn’t

Returns:

Absolute path.

Return type:

str

classmethod get_dummydata(path)[source]

Get dummydata by reading file from path

Parameters:

path – path to file.

Returns:

Contents of the file.

classmethod remove_dummyfile_if_exists(path)[source]

Remove dummyfile from path if it exists.

Parameters:

path – path to dummyfile.

Returns:

None

classmethod set_val(value)[source]

Assign value as dummyvalue.

Parameters:

value – Value to assign

Returns:

value

classmethod gen_val(lenght=None, unique=False, chars=None)[source]

Generate & assign dummyvalue.

Parameters:

  • lenght (int or None) – lenght of the value

  • unique (bool) – should the value be unique

  • chars (str or None.) – use specific characters.

Returns:

generated value

Return type:

str

classmethod gen_id()[source]

Generate Id.

Returns:

Generated id.

Return type:

str

classmethod generate_dummy_study()[source]

Generate and return a Study with dummydata.

Returns:

study with dummydata

Return type:

kuha_common.document_store.records.Study

classmethod generate_dummy_variable()[source]

Generate and return a Variable with dummydata.

Returns:

variable with dummydata

Return type:

kuha_common.document_store.records.Variable

classmethod generate_dummy_question()[source]

Generate and return a Question with dummydata.

Returns:

question with dummydata

Return type:

kuha_common.document_store.records.Question

classmethod generate_dummy_studygroup()[source]

Generate and return a StudyGroup with dummydata.

Returns:

studygroup with dummydata.

Return type:

kuha_common.document_store.records.StudyGroup

static run_until_complete(coro)[source]

Run coroutine until it’s completed

Parameters:

coro – Coroutine function.

setUp()[source]

Format testcase values and initialize event loop.

Call asynchronous code synchronously:

self.run_until_complete(coro())
tearDown()[source]

Stop patchers.

async await_and_store_result(coro)[source]

Await coroutine and store returning result.

Example:

self.run_until_complete(self.await_and_store_result(coro()))
Parameters:

coro – Coroutine or Future to await

init_patcher(patcher)[source]

Initialize patcher, store for later use, return it.

Parameters:

patcher (unittest.mock._patch) – Patch to start.

Returns:

MagicMock acting as patched object.

Return type:

unittest.mock.MagicMock

assert_records_are_equal(first, second, msg=None)[source]

Assert two Document Store records are equal.

Parameters:

  • first – First record to compare.

  • second – Second record to compare.

  • msg – Optional message to output on assertion.

assert_records_are_not_equal(first, second, msg=None)[source]

Assert two Document Store records are not equal.

Parameters:

  • first – First record to compare.

  • second – Second record to compare.

  • msg – Optional message to output on assertion.

assert_mock_meth_has_calls(mock_meth, call, *calls)[source]

Assert moch_meth was called with arguments.

This calls Mock.assert_has_calls and tests for call count. The actual benefit of using this method over the built-in assert_has_calls is that this method tries to pinpoint the actual call that was missing when assert_has_calls raised AssertionError. This is useful when mock_meth has had multiple calls. The built-in assert_has_calls will notify of all calls that the mock_meth has had, while this method will notify of the actual call that was missing.

Parameters:

  • mock_meth – Mocked method that is target of testing.

  • call – Call that should be found. Instance of unittest.mock._Call Repeat this argument to test for multiple calls.

Raises:

AssertionError if calls not found.

class kuha_common.testing.testcases.KuhaEndToEndTestCase(methodName='runTest')[source]

Base class for end-to-end tests.

  • HTTPClient for interacting with Document Store.

  • Assertion methods to check returning payload and status codes.

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass()[source]

Hook method for deconstructing the class fixture after running all tests in the class.

static get_record_url(rec_or_coll, _id=None)[source]

Get URL to Document Store records or single record.

Parameters:

  • rec_or_coll – record, record class or collection

  • _id (str or None) – Optional record ID.

Returns:

URL to Document Store collection or single record.

Return type:

str

static get_query_url(rec_or_coll, query_type=None)[source]

Get URL to Document Store query endpoint for collection

Parameters:

  • rec_or_coll (str, record, or record class) – Collection to query.

  • query_type – Optional query type

Returns:

URL to query endpoint.

Return type:

str

classmethod GET_to_document_store(rec_or_coll, _id=None)[source]

GET to Document Store returns record(s).

Parameters:

  • rec_or_coll – record or collection to get.

  • _id – Optional ObjectId. Will take precedence over rec_or_coll id.

Returns:

response body

classmethod POST_to_document_store(record)[source]

POST to Document Store creates record.

Parameters:

record – Record to post.

Returns:

response body

classmethod DELETE_to_document_store(rec_or_coll=None, _id=None, hard_delete=True)[source]

DELETE to Document Store deletes record(s).

Call without arguments to delete all records from all collections.

Parameters:

  • rec_or_coll (str or None) – Collection to delete from.

  • _id (str or None) – ID of the record to delete.

  • hard_delete (bool) – Set to False to do a logical delete instead.

Returns:

Response body or bodies wrapped in dict ({<collection>: <resp_body>})

classmethod query_document_store(rec_or_coll, query, query_type=None)[source]

Execute query against Document Store query API.

Parameters:

  • rec_or_coll (str or record class or record instance) – Collection to query.

  • query – Query.

  • query_type – Type of Query.

Returns:

query results

Return type:

None if query returned no results, dict for results.

classmethod get_collection_record_count(rec_or_coll, exclude_deleted=True)[source]

Return number or records for collection in Document Store.

Parameters:

  • rec_or_coll – Document Store record, Document Store record class or collection.

  • exclude_deleted (bool) – exclude logically deleted records from count.

Returns:

record count in Document Store.

Return type:

int

assert_document_store_is_empty(exclude_deleted=True)[source]

Assert Document Store contains no records.

Parameters:

exclude_deleted (bool) – exclude logically deleted records from count.

Raises:

AssertionError if Document Store has records.