kuha_common

High-level modules common for Kuha applications.

server.py

Common server functions & classes used in Kuha.

kuha_common.server.log_request(handler)[source]

Gets called after each completed HTTP request.

Note that even requests that end up in exceptions are usually responded, thus successfully handled.

:param tornado.web.RequestHandler handler: Current request handler. :returns: None

kuha_common.server.str_api_endpoint(api_version, suffix=None)[source]

Helper function to prepend endpoints with api_version.

Parameters

  • api_version (str) – version of the api.

  • suffix (str) – api endpoint.

Returns

str – endpoint prepended with api_version

kuha_common.server.serve(web_app, port, on_exit=None)[source]

Serve web application.

Parameters

  • web_app (tornado.web.Application) – application to serve

  • port (int) – Port to listen to.

  • on_exit (function) – callback on server/ioloop stop.

class kuha_common.server.RequestHandler(*args, **kwargs)[source]

Common request handler for kuha server applications. Subclass in application specific handlers.

async prepare()[source]

Prepare each request.

Look for correlation id; create one if not found. Set correlation id to response header. Prepare logging context with HTTP request information.

property correlation_id

Public read-only interface to get response correlation ID.

To get request correlation ID, use request.headers.get(correlation_id.get_header_key())

Returns

CorrelationID

Return type

CorrelationID

set_output_content_type(ct, charset='UTF-8')[source]

Sets content type for responses.

Parameters

  • ct (str) – content type for response.

  • charset (str) – charset definition for response content type.

log_exception(typ, value, tb)[source]

Overrides tornados exception logging. Sends HTTP errors as responses. For easier debugging it also calls tornado.web.RequestHandler.log_exception to output full traceback.

Parameters

  • typ – type of exception

  • value – caught exception

  • tb – traceback

assert_request_content_type(supported_content_type)[source]

Assert request has correct content type header.

Parameters

supported_content_type (str) – content type supported by endpoint.

Raises

InvalidContentType – if request has invalid content type.

get_valid_query_arguments(*valid_keys_occurrences)[source]

Validate and return query arguments.

Validation is performed based on query argument keys. The value is not checked. Request can be checked to contain no query arguments by calling this method without arguments.

Parameters

valid_keys_occurrences – optional positional arguments contain all valid keys and their expected occurrences. The parameter is a two-tuple: (<key>, <number_of_expected_occurrences>). If the number of expected occurrences is 0, no validation is performed based on occurrences. If the number of expected occurrences is 1, the value is popped out from the list container: {<key>: [<val>]} -> {<key>: <val>}

Returns

validated query arguments.

Return type

dict

Raises

BadRequest for invalid query argument keys.

write_error(status_code, **kwargs)[source]

Overrides tornado.web.RequestHandler.write_error. Outputs error messages in preferred content type.

Parameters

  • status_code (int) – HTTP status code.

  • **kwargs – keyword arguments are passed to tornado.web.RequestHandler.write_error if output content type is not application/json

class kuha_common.server.WebApplication(*args, **kwargs)[source]
exception kuha_common.server.InvalidContentType(requested_content_type, supported_content_type)[source]

Invalid content type HTTP-exception.

exception kuha_common.server.BadRequest(log_message=None)[source]

Bad request HTTP-exception.

exception kuha_common.server.ResourceNotFound(log_message=None, context=None)[source]

Resource not found HTTP-exception.

exception kuha_common.server.Conflict(log_message=None)[source]

Resource in conflicting state

query.py

Perform query operations against Kuha Document Store.

Offers High-level query methods to facilitate an easy access point with all necessary actions and properties needed to perform queries. To query the Document Store the caller only needs to use methods defined in class QueryController and records defined in kuha_common.document_store.records

class kuha_common.query.ResultHandler(record_constructor, on_record=None)[source]

Class which handles results and correct calls to callbacks, if any. Stores the result for later use.

Dynamically creates callable method handle() which receives result payload and calls on_record correctly.

Parameters
Returns

ResultHandler

class kuha_common.query.QueryController(headers=None, record_limit=0)[source]

Asynchronous controller to query the Document Store.

Use to build queries and automatically fetch responses using HTTP as a protocol and JSON as exchange format.

Optional record_limit parameter may be given at initialization time to limit the number of records that are requested in a single HTTP request.

Parameters

  • headers (dict) – Optional headers parameter to store headers that are used in each query application wide as HTTP headers.

  • record_limit (int) – Optional record_limit parameter which is used to limit the number of records requested in a single HTTP request.

Example:

from kuha_common.document_store import Study
query_ctrl = QueryController()
study = await query_ctrl.query_single(
    Study,
    fields=[Study._metadata, Study.study_number],
    _filter={Study.study_number: 1234}
)
fk_constants

alias of kuha_common.document_store.query.FilterKeyConstants

async query_single(record, on_record=None, headers=None, **kwargs)[source]

Query single record.

Parameters

  • record (Subclass of kuha_common.document_store.records.RecordBase) – class used to construct the record instance.

  • on_record (function or coroutinefunction) – Optional callback function that gets passed the returned and instantiated record object.

  • headers (dict) – Optional headers for this query. Headers get added to headers given for QueryController at initialization time. Note that it will overwrite headers with same key.

  • **kwargs – Keyword arguments contain parameters for the query. They get passed to kuha_common.document_store.query.Query.construct()

Returns

None if passed on_record callback, else returns the initiated record object.

Raises

QueryException – if query parameters given as keyword arguments contain limit-parameter.

async query_multiple(record, on_record, headers=None, **kwargs)[source]

Query multiple records.

Queries the document store for multiple records. Behaviour depends on whether record_limit has been set:

If there is a record_limit

If there is no record_limit

  • this method returns nothing.

  • The on_record callback gets called with each instantiated record object.

  • on_record may be a normal function or a coroutine.

Parameters
Returns

None if no record_limit, else kuha_common.document_store.client.JSONStreamClient.run_queued_requests()

async query_count(record, headers=None, **kwargs)[source]

Query the number of records.

Parameters
Returns

Number of records

Return type

int

async query_distinct(record, headers=None, **kwargs)[source]

Query distinct values.

Parameters
Returns

distinct values: {fieldname : [value1, value2, value3]}. Note that contained values may be dictionaries or strings, depending on what is stored in requested field.

Return type

dict

conf.py

Parse command line and store application wide settings.

Typically used to get configuration values using configargparse. However, the module can be used to manually setup configuration values.

Get configuration using configargparse:

>>> load('myprogram')
>>> add('--some-arg')
>>> settings = get_conf()

Manually set configuration values:

>>> add_conf('some_arg', 'some_value')
>>> settings = get_conf()
exception kuha_common.conf.CommandLineParseException[source]

Raised on exceptions during command line parsing

class kuha_common.conf.KuhaConfigFileParser[source]

Inherit to override configargparse.DefaultConfigFileParser.get_syntax_description()

get_syntax_description()[source]

Override syntax description of configargparse.DefaultConfigFileParser

Returns

Syntax description for configuration file.

Return type

str

kuha_common.conf.load(prog, package=None, env_var_prefix=None, **kw)[source]

Load configuration parser.

Parameters are passed to configargparse.get_arg_parser() and the parser is stored for later use.

Parameters

  • prog (str) – Program name

  • package (str) – Python package. Use prog if not given.

  • env_var_prefix (str) – Prefix for environment variables. Optional.

  • default_config_files (list) – Default configuration files. Use CWD/PROG.ini if not given.

  • conflict_handler (str) – Conflict handler. Default to ‘error’. See configargparse and argparse for details.

  • config_file_parser_class (class) – Configuration file parser class. Defaults to KuhaConfigFileParser.

Returns

Loaded parser

Return type

configargparse.ArgumentParser

kuha_common.conf.add(*args, **kwargs)[source]

Shorthand adds argument to stored parser

All arguments are passed as is to stored parser. If there is a submitted env_var and load was given env_var_prefix, the environment variable keys are prefixed with loaded env_var_prefix.

Raises CommandLineParseException if configuration has already been loaded.

kuha_common.conf.get(key, default=None)[source]

Get configuration value for key or default.

Use to get a configuration key from loaded settings. If a key is not present, returns default instead.

Parameters

  • key (str) – Get key from loaded settings

  • default – Return default if key not in settings

Returns

Configuration value

kuha_common.conf.add_conf(key, value)[source]

Add configuration key-value pair.

Use this to bypass command line parsing.

Parameters

  • key (str) – Configuration key

  • value – Configuration value

kuha_common.conf.get_conf(conf_key=None)[source]

Parse arguments, store and return them.

If arguments are already parsed and stored, return them.

Note

Throws errors for unrecognized command line arguments and configuration file keys (parse_args()). Use environment variables to enable a single source of truth in project wide scope.

Parameters

conf_key (str) – Optional parameter to return a single configuration value.

Returns

Parsed arguments

Return type

argparse.Namespace

kuha_common.conf.print_conf()[source]

Print active configuration.

Returns

None

cli_setup.py

Command line setup for Kuha applications

Parse command line for common configuration options and store loaded settings.

kuha_common.cli_setup.MOD_DS_CLIENT = 'document_store.client'

Constant for configuring kuha_common.document_store.client

kuha_common.cli_setup.MOD_DS_QUERY = 'document_store.query'

Constant for configuring kuha_common.document_store.query

kuha_common.cli_setup.MOD_SERVER = 'server'

Constant for configuring kuha_common.server

kuha_common.cli_setup.MOD_LOGGING = 'logging'

Constant for configuring logging

kuha_common.cli_setup.add_logging_configs()[source]

Wrapper to add logging-module configuration.

kuha_common.cli_setup.setup_logging(settings)[source]

Setup py12flogging module.

kuha_common.cli_setup.setup_common_modules(*modules)[source]

Setup command line parser.

Load modules, parse command line arguments, return loaded settings in argparse.Namespace

Parameters

*modules (str) – common Kuha modules to load and include in parsing of command line arguments.

Returns

Loaded settings.

Return type

argparse.Namespace

document_store

Contains modules for interacting with Document Store.

document_store/client.py

kuha_common.document_store.client provides a http client interface to communicate with Kuha Document Store.

class kuha_common.document_store.client.JSONStreamClient[source]

Base class used for requests. Implements own queue to store requests, since tornado.httpclient.AsyncHTTPClient starts timers for request_timeout and connect_timeout at the moment we call client.fetch(). See https://github.com/tornadoweb/tornado/issues/1400 for more details.

Handles JSON decoding of incoming chunks and the encoding of body to JSON.

sleep_on_queue = 5

Sets sleep timer for queue.

request(url, **kwargs)[source]

Constucts a streaming request.

Parameters

  • url (str) – url to request.

  • kwargs – keyword arguments passed to tornado.httpclient.HTTPRequest

Returns

tornado.httpclient.HTTPRequest

wrap_streaming_callback(callback)[source]

Wrap streaming callback to support chunked JSON responses.

Parameters

callback (callable.) – streaming callback. Gets called with response which is decoded to python object from JSON.

Returns

Wrapped callback

Return type

functools.partial

async execute_stored_callbacks()[source]

Executes asynchronous callbacks stored in _callbacks

async run_queued_requests(queued_requests=None)[source]

Run queued requests.

Calls queued requests asynchronically. Sleeps for sleep_on_queue if max_clients reached.

Parameters

queued_requests (collections.deque) – Optionally pass queued_requests to run.

get_streaming_request(streaming_callback, url, body=None, method=None, headers=None, **kw)[source]

Get a streaming request.

Sets default headers Content-Type: application/json if not already given. Encodes body to JSON if given and is not string or bytes. If response is empty (for example query with no results) the streaming callback doesn’t get called.

Subclass and override to support arbitrary requests.

Parameters

  • streaming_callback (callable) – callback which receives the response if any.

  • url (str) – URL to send request to.

  • body (str, dict, list, tuple, integer, float or None) – Optional request body. String will be supplied as is. Other values will be encoded to JSON.

  • method (str or None) – HTTP method. Defaults to POST.

  • headers (dict or None) – optional request headers. if Content-Type is not set, will set ‘Content-Type’: ‘application/json’ as default.

Returns

HTTP request

Return type

tornado.httpclient.HTTPRequest

async queue_request(*args, **kwargs)[source]

Queue request to be run aynchronously by calling run_queued_requests.

Parameters

  • *args – arguments passed to get_streaming_request

  • **kwargs – keyword arguments passed to get_streaming_request.

Returns

run_queued_requests() method to call to run the queued requests.

async fetch(*args, **kwargs)[source]

Run single query.

Parameters

  • *args – arguments passed to queue_requests.

  • **kwargs – keyword arguments passed to queue_requests

document_store/query.py

Access query properties by convenience methods to help build valid queries against the Document Store.

class kuha_common.document_store.query.FilterKeyConstants[source]

Class used as a namespace to contain constants used in query filter.

exception kuha_common.document_store.query.QueryException(msg, context=None)[source]

Exception for errors raised by Query. Has an optional context parameter for giving some additional information about the context of the exception.

class kuha_common.document_store.query.Query(query, query_document, query_type='select')[source]

Manipulate query properties without compromising the validity of the constructed query. Build the correct url for different query types.

Note

This class provides low-level operations. Use kuha_common.query.QueryController for easy access to query actions and properties.

Example:

from kuha_common.document_store import Study, Query
query = Query(Query.construct(_filter={Study.study_number:'123'}), Study.collection)
Parameters

  • query (dict) – Actual query containing the properties such as _filter, fields, sort_by etc.

  • query_document (str) – One of the supported query documents declared in Query.supported_query_documents and specified in kuha_common.document_store.records.py

  • query_type (str) – Optional query_type parameter. Defaults to Query.query_type_select. Other valid values are Query.query_type_count and Query.query_type_distinct.

k_filter = '_filter'

Query parameter for filtering results.

k_fields = 'fields'

Query parameter for fields to contain in results.

k_limit = 'limit'

Query parameters for limiting returned results.

k_skip = 'skip'

Query parameter for skipping number of results from the beginning of the resultset.

k_sort_order = 'sort_order'

Query parameter for sort order.

k_sort_by = 'sort_by'

Query parameter for sorting by a certain field.

k_fieldname = 'fieldname'

Query parameter for distinct queries. Specifies a field from which the distinct values are to be fetched.

query_type_select = 'select'

Query type for select queries. Using this query type gets records as a response.

query_type_count = 'count'

Query type for count queries. Using this query type the query returns an integer.

query_type_distinct = 'distinct'

Query type for distinct queries. Using this query type the returning object contains all distinct values for a certain field.

classmethod as_supported_datetime_str(datetime_obj)[source]

Get datetime object as a supported datetime string.

Parameters

datetime_obj (datetime-object.) – Python datetime-object to be convered to str.

Returns

String represenation of the datetime-object that is suitable foe querying.

Return type

str

classmethod construct(**kwargs)[source]

Construct valid query parameters.

Example:

from kuha_common.document_store import Query, Study
params = Query.construct(_filter={Study.study_number:'123'},
                         fields=[Study._metadata, Study._id, Study.abstract],
                         sort_by=Study._id)
query = Query(params, Study.collection)
Parameters

**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.

Returns

Valid query, ready to be sent to Document Store.

Return type

dict

classmethod construct_distinct(**kwargs)[source]

Construct valid query parameters for distinct queries.

Parameters

**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.

Returns

Valid query, ready to be sent to Document Store.

Return type

dict

classmethod build_query_for_date_range(from_=None, until=None)[source]

Build query filter for date-range.

Parameters

  • from (datetime-object) – start of the date-range:

  • until (datetime-object) – end of the date-range:

Returns

date-range query-filter with datetime-objects converted into string representation.

Return type

dict

classmethod build_query_for_exists(exists)[source]

Build query for exists-query.

Parameters

exists (bool) – whether the field should exists or not.

Returns

valid exists query for filter.

Return type

dict

Raises

ValueError for invalid boolean values in exists-parameter.

classmethod get_valid_params(query_type=None)[source]

Return valid query parameters for the query type.

Parameters

query_type (str) – Optional query_type for which the query-parameters should be valid for.

classmethod is_valid_query(query, query_type)[source]

Check the validity of query parameters.

Parameters

  • query (dict) – Full query to validate.

  • query_type (str) – Query type to validate against.

Returns

Whether or not the query-parameters given are valid.

Return type

bool

classmethod is_valid_query_type(query_type)[source]

Check the validity of query_type.

Parameters

query_type (str) – Query type to validate.

Returns

Whether or not the query type is valid.

Return type

bool

is_valid_query_document(query_document)[source]

Check the validity of query document.

Parameters

query_document (str) – Query document to validate.

Returns

Whether or not the query document is valid.

Return type

bool

is_valid_param(parameter)[source]

Check the validity of a single query parameter.

Parameters

parameter (str) – Query parameter to validate.

Returns

Whether or not the parameter is valid.

Return type

bool

validate_query(query)[source]

Validate query parameters.

Checks parameters’ validity for chosen query type. Raises QueryException if invalid.

Parameters

query (dict) – Query parameters.

Returns

Query parameters.

Return type

dict

Raises

QueryException if query parameters are invalid.

validate_query_type(query_type)[source]

Validate query type.

Checks that the query type is supported by Document Store. Raises QueryException for invalid query type.

Parameters

query_type (str) – Query type to validate.

Returns

Query type.

Return type

str

Raises

QueryException if query type is invalid.

validate_query_document(query_document)[source]

Validates query document.

Checks that the query document is supported by Document Store. Raises QueryException if invalid.

Parameters

query_document (str) – Query document to validate.

Returns

Query document

Return type

str

Raises

QueryException if query document is invalid.

get_endpoint()[source]

Get correct endpoint for querying the Document Store.

Builds the endpoint by consulting configured values and the instantiated query for query_type and query_document

Returns

Full url to Document Store endpoint which handles the constructed query.

Return type

str

get_query(strip_invalid_params=True)[source]

Returns the constructed query parameters.

If the query type has been changed after initialization, for example to get the count of records, this method strips the invalid query parameters from the returned query. When doing so, it does not change the stored query parameters, but rather makes a copy of them for manipulating and returning.

Parameters

strip_invalid_params (bool) – Whether to strip the unsupported (=invalid) query parameters out of the returned query.

Returns

Constructed query parameters ready to submit to Document Store.

Return type

dict

get_limit()[source]

Get query limit parameter.

Returns

Query limit (int) if set. None if not set.

Return type

int or None

get_skip()[source]

Get query skip parameter.

Returns

Query skip (int) if set. None if not set.

Return type

int or None

set_limit(limit)[source]

Set limit parameter for query.

Limit controls how many results should be returned.

Parameters

limit (int) – Limit parameter for query.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

set_skip(skip)[source]

Set skip parameter for query.

Skip conrols how many results should be skipped from the start (offset).

Parameters

skip (int) – Skip parameter for query.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

set_fields(fields)[source]

Set fields parameter for query.

Field controls which fields of the record should be returned. fields can be a list of strings in the form used by MongoDB or a list of kuha_common.document_store.records class-variables.

Example:

from kuha_common.document_store import Query, Study
_params = Query.construct(_filter={Study.study_number:'123'})
_query = Query(_params, Study.collection)
_query.set_fields([Study.abstract, Study.study_number])
Parameters

fields (list) – Fields parameter for query.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

set_sort_by(sort_by)[source]

Set sort_by parameter for query.

Determines sorting of the returned results. sort_by can be a string in the form used by MongoDB or a kuha_common.document_store.records class-variables.

Parameters

sort_by (srt or class-variable of a record.) – Sort by parameter for query.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

set_sort_order(order)[source]

Set sort order for the query.

Determines the order which the returned results are to be sorted by.

Note

Valid values come from pymongo. They actually depend on the mongodb driver, but since this is a caller API we don’t want to make pymongo a dependency.

Parameters

order (int) – Sort order. Must be either 1 or -1.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

Raises

QueryException for invalid order values.

set_query_type(query_type)[source]

Set query type.

Parameters

query_type (str) – Valid query type for the query to be constructed.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

add_query_statement(field, statement)[source]

Add query statement.

Manipulates the _filter parameter of the query parameters. Raises a QueryException if the field already has a statement declared in _filter.

Parameters

  • field (str) – Field to target the statement to.

  • statement (str) – Statement to filter the results by.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

add_query_statements(**kwargs)[source]

Add multiple query statements to filter the returned results.

Manipulates the _filter parameter of the query parameters.

Parameters

**kwargs – key-value pairs that are to be added to the _filter parameter.

Returns

self for easy aggregation of manipulation methods.

Return type

instantiated Query()

document_store/field_types.py

Properties and actions for field types supported by records defined in kuha_common.document_store.records

Provides field types to be used not only for the construction of new records and updating exising records, but also to provide a format for fields of records that is interchangeable in a way that a receiver does not need to know the specifics of a field beforehand, but may use the field to gain knowledge of the properties of the field.

This module also provides factories which are used to fabricate the fieldtypes. The instantiated factories hold knowledge of the fields even thought the fields themselves are not yet instantiated. This knowledge is used for querying records, but also to dynamically fabricate fieldtypes for records in kuha_common.document_store.records

exception kuha_common.document_store.field_types.FieldTypeException[source]

Exception to raise on field type errors. Used for programming errors.

class kuha_common.document_store.field_types.Value(name, value=None)[source]

Value is the most simple type of field.

Field type with name and single value. Serves also as a baseclass for other field types.

Parameters

  • name (str) – Name of the field.

  • value – Optional value for the field.

set_value(value)[source]

Set initial value.

Parameters

value – Value to set.

add_value(value)[source]

Add value for the field.

Note

Overrides existing value.

Parameters

value – The value to set.

get_value()[source]

Get the value of the field.

Returns

Value.

get_name()[source]

Get name of the field.

Returns

The name of the field.

Return type

str

export_dict()[source]

Exports Value as dictinary.

Returns

{name:value}

Return type

dict

import_records(record)[source]

Import record to field.

Parameters

record – record to import.

updates(secondary_values)[source]

Update value.

Parameters

secondary_values (str) – Value to update to.

class kuha_common.document_store.field_types.Set(name, value=None)[source]

Set is a field type with name and list of unique values.

Derived from Value Implements methods that differ from parent class.

Parameters

  • name (str) – Name of the field.

  • value (list or None) – Optional value for the field.

set_value(value)[source]

Sets value.

Parameters

value (list) – Value for field.

Raises

FieldTypeException if submitted value is not a list.

add_value(value)[source]

Add value to field.

Appends a value to the list of values already set. Makes sure that the list holds no duplicates by silently discarding them.

Parameters

value (list or str or None) – value or values to be appended. If value is None, empties the list.

import_records(record)[source]

Import records by adding the submitted records to contained values.

Parameters

record (list or str or None) – Hold the values to be imported.

updates(secondary_values)[source]

Updates old values with values contained in this Set.

Looks for combination of secondary_values and values in this set. Discards duplicates and stores the updated values to value.

Parameters

secondary_values (list) – list of old values to be updated with new ones.

class kuha_common.document_store.field_types.Element(name, attribute_names=None)[source]

Element is a field type with name, value and attributes.

Derived from Value.

Element is used to store fields that contain attributes in addition to a value. Each attribute in itself is an instance of Value and is dynamically stored in instance variable attr_<name>. When instantiated and populated with values and attributes, the element-instance can be used to get it’s value, value’s name, but also to get the attributes and their names, even thought the caller does not know the attribute names a priori.

Example of constructing an element (the source):

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)

Example of reading from an unknown element (the receiver):

>>> unknown_element.get_name()
'animal'
>>> unknown_element.get_value()
'cat'
>>> for att in unknown_element.iterate_attributes():
...     att.get_name() + ' : ' + str(att.get_value())
...
'height : 5'
'color : yellow'
'weight : 10'
>>> unknown_element.attr_color.get_value()
'yellow'

This is especially useful when using as an interchange format. The receiver does not need to know the attribute names beforehand. Instead the receiver can iterate throught every attribute to get their name-value pairs or if the receiver is interested in a single attribute, it may be called by the dynamically constructed instance-variable prefixed with attr_.

Parameters

  • name (str) – Name of the field.

  • attribute_names (list) – Optional parameter for attribute names.

Raises

FieldTypeException if attribute_names has duplicates.

is_pending()[source]

Is the element pending for values.

Returns

True if pending, False if not.

Return type

bool

new()[source]

Create a new element-instance with same name and attributes but without values.

Instantiates a new instance of itself. The new instance is pending for values.

Example:

>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> another_animal = animal.new()
>>> another_animal.add_value('dog', color='white', weight=30, height=15)
Returns

new element.

Return type

Element

add_value(value=None, **attributes)[source]

Add value with attributes as keyword arguments.

Note

This may only be called once for each instance.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
Parameters

  • value (str or int or None) – Value for the element.

  • **attributes – keyword arguments for attributes of the element.

Raises

FieldTypeException if the element already has values or if submitted value is None and no attributes are given.

iterate_attributes()[source]

Generator function. Iterates element attributes.

Returns

a generator object for iterating attributes.

get_attribute(name)[source]

Get attribute by attribute name.

Parameters

name (str) – Name of the attribute to look for.

Returns

attribute of the element or None if not found.

Return type

Value or None

export_attributes_as_dict()[source]

Export element’s attributes as a dictionary.

Returns

dictinary representing the attributes

Return type

dict

export_dict()[source]

Export the element as a dictionary.

Returns a dictionary with key-value pairs given wrapped inside a another dictionary with the elements key as name.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> animal.export_dict()
{'animal': {'color': 'yellow', 'weight': 10, 'height': 5, 'animal': 'cat'}}
Returns

dictinary representing the Element

Return type

dict

import_records(record)[source]

Import records.

Import records by adding the submitted records to value and attributes.

Note

the record parameter is emptied by this method.

Parameters

record (dict) – Record to import.

updates(secondary_values)[source]

Updates attributes not found in this element with the ones found from secondary_values. Not found in this context means values that are None.

Note

The parameter is iterated destructively.

Parameters

secondary_values (dict) – Attributes from old element.

class kuha_common.document_store.field_types.LocalizableElement(name, attribute_names=None)[source]

LocalizableElement is a field type with name, value, language and attributes.

Derived from Element. Has an additional attribute for language. The language is special attribute that is used when updating elements.

Seealso

Element

Parameters

  • name (str) – Name of the element.

  • attribute_names (list) – Optional list of attribute names.

Raises

FieldTypeException if attribute_names contain a name that is reserved for language.

set_language(language)[source]

Set language for element.

Parameters

language (str) – language to set.

Raises

FieldTypeException if language already set.

get_language()[source]

Get language of element.

Returns

language

Return type

str or None

add_value(value=None, language=None, **attributes)[source]

Add values for element.

Note

This may only be called once for each instance.

Seealso

Element.add_value()

Parameters

  • value (str or int) – value to set.

  • language (str) – language of the element.

  • **attributes – keyword arguments for attributes of the element.

Raises

TypeError if language is not given or is None.

export_dict()[source]

Export the element as a dictionary.

Seealso

Element.export_dict()

Returns

dictinary representation of the element.

Return type

dict

class kuha_common.document_store.field_types.ElementContainer(name, sub_element)[source]

ElementContainer contains a list of single type of Element/LocalizableElement field types.

Receives mandatory parameters for name and sub_element. The sub_element describes the element types that this container can store. Every new element that a container can create will be an instance created from this sub_element.

Example:

>>> from kuha_common.document_store.field_types import ElementContainer, LocalizableElement
>>> animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> animals = ElementContainer('animals', animal)
>>> animals.add_value('cat', 'en', color='yellow', width=10, height=5)
>>> animals.add_value('kissa', 'fi', color='keltainen', width=10, height=5)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'width': 10,
     'language': 'en',
     'color': 'yellow',
     'height': 5,
     'animal': 'cat'},
    {'width': 10,
     'language': 'fi',
     'color': 'keltainen',
     'height': 5,
     'animal': 'kissa'}]
}

Elements can be iterated:

>>> for animal in animals:
...     animal.attr_color.get_value() + " for language: " + animal.get_language()
...
'yellow for language: en'
'keltainen for language: fi'

And updated with containers sharing name and attribute names:

>>> another_animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> more_animals = ElementContainer('animals', another_animal)
>>> more_animals.add_value('dog', 'en', color='white', width=20, height=10)
>>> more_animals.add_value('koira', 'fi', color='valkoinen', width=20, height=10)
>>> animals.updates(more_animals)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'language': 'en',
     'height': 5,
     'color': 'yellow',
     'animal': 'cat',
     'width': 10},
    {'language': 'fi',
     'height': 5,
     'color': 'keltainen',
     'animal': 'kissa',
     'width': 10},
    {'language': 'en',
     'height': 10,
     'color': 'white',
     'animal': 'dog',
     'width': 20},
    {'language': 'fi',
     'height': 10,
     'color': 'valkoinen',
     'animal': 'koira',
     'width': 20}]
}
Parameters
Raises

FieldTypeException for invalid sub_element.

import_records(record)[source]

Imports records from a list of dictionaries.

Note

The dictionaries will lose information.

Parameters

record (list) – list of dictionaries with records to import.

add_value(value=None, language=None, **kwargs)[source]

Add new element to list of elements

Parameters

  • value (str or int or None) – value for the new element.

  • language (str or None) – language for the new element.

  • **kwargs – key-value pairs for attributes of the new element.

Raises

FieldTypeException for invalid language parameter depending on whether the sub_element is localizable.

export_dict()[source]

Export container as dictionary.

Returns

dictionary representing the container.

Return type

dict

iterate_values_for_language(language)[source]

Generator for iterating contained elements by language.

Parameters

language (str) – language which is used to filter yielded results.

Returns

a generator object for iterating elements

get_available_languages()[source]

Get list of languages for this container.

Returns

list of distinct languages.

Return type

list

updates(secondary_values)[source]

Updates contained values with secondary_values.

Looks for values that are not currently contained, and appends them as contained values. Also appends different language versions. If a language version has the same value, looks for differences in attributes. If new value has not the same attributes as the old one, adds these attributes to the new value. If old value has same attribute name, it will be discarded.

Note

Document Store uses MongoDB as a backend. MongoDB deals with JSON-like objects, which in turn are best represented in Python as dictionaries. The purpose of kuha_common.document_store.records is to be used as a global (in Kuha context) interchange format and so it will be best to support both dictionaries and ElementContainers for this operation. Therefore there is some flexibility in the type of parameter that this method accepts.

Note

There is a logical difference in which type of parameter is submitted to this method. When using other types than ElementContainers, the parameter’s content will be changed.

Parameters

secondary_values (instance of ElementContainer or dict or list) – Old values known to have the same container (must have the same name). If secondary_values is a list, it is assumed that the caller has explicitly checked that the parameter represents old values for this container. Otherwise the name of the container will be checked here and KeyError exceptions will be raised.

class kuha_common.document_store.field_types.FieldAttribute(name, parent=None)[source]

Common attributes for each field type.

Stores fields name, parent fields name and constructs a path for the field. This path can be used when building queries against Document Store. The name can be used to lookup values from objects returned from Document Store.

Used by FieldTypeFactory to store information of fields that can be used before the field has been fabricated.

Parameters

  • name (str) – name of the field.

  • parent (str) – optional parameter parent. Used for sub-elements.

value_from_dict(_dict)[source]

Get value or values corresponding to path from parameter.

Note

Returned values cannot be separated by language afterwards.

Parameters

_dict (dict) – dictionary to lookup for path.

Returns

value or values stored in path of the _dict.

Return type

str or list or None

class kuha_common.document_store.field_types.FieldTypeFactory(name, sub_name=None, attrs=None, localizable=True, single_value=False)[source]

Factory for field types.

Stores information for each field, that can be used before the field actually has been initiated. This is useful for building queries against Document Store, because the caller needs to know the names and paths of the fields about to be queried.

The attributes stored here are also used to fabricate each field type. This means that each of the field types supported by kuha_common.document_store.records are to be initiated throught this factory.

Seealso

ElementContainer

Example:

>>> from kuha_common.document_store.field_types import FieldTypeFactory
>>> animals_factory = FieldTypeFactory('animals', 'animal', ['color', 'width', 'height'])
>>> animals_factory.attr_color.name
'color'
>>> animals_factory.attr_color.path
'animals.color'
>>> animals = animals_factory.fabricate()
>>> animals.add_value('cat', 'en', color='yellow', height=10, width=5)
>>> animals.export_dict()
{'animals': [{'color': 'yellow', 'animal': 'cat', 'height': 10, 'width': 5, 'language': 'en'}]}
Parameters

  • name (str) – name of the field.

  • sub_name (str) – name of the sub field, if any.

  • attrs (list or str) – field attributes, if any. Multiple attributes in list.

  • localizable (bool) – is the field localizable.

  • single_value (bool) – The fabricated field can contain only a single value.

Raises

ValueError if attribute has same name as the element or sub_element.

Raises

FieldTypeException for parameter combinations that are not supported.

fabricate()[source]

Fabricate field type by factory attributes.

Returns the correct type of field type based on attributes given to the factory at initialization time.

Returns

Instance of one of the fields types.

document_store/records.py

Models for records supported by Document Store.

Due to its schemaless design, the document store relies heavily on these models. Use these models when building importers.

kuha_common.document_store.records.datetime_to_datestamp(_datetime)[source]

Convert datetime object to datestamp string supported by Document Store.

Parameters

datetime (datetime.datetime) – datetime to convert.

Returns

converted datestamp.

Return type

str

kuha_common.document_store.records.datestamp_to_datetime(datestamp)[source]

Convert datestamp string to datetime.datetime object.

Parameters

datestamp (str) – datestamp to convert.

Returns

converted datetime.

Return type

datetime.datetime

kuha_common.document_store.records.datetime_now()[source]

Get current datetime in supported format.

Returns

Supported datetime object representing current time.

Return type

datetime.datetime

kuha_common.document_store.records.path_split(path)[source]

Split path into steps.

Parameters

path (str) – record attribute path.

Returns

path steps

Return type

list

kuha_common.document_store.records.path_join(step, *steps)[source]

Join two or more steps into a record attribute path by inserting a separator between the steps.

Parameters

  • step (str) – step used to create path.

  • *steps (str) –

    steps used to create path.

Returns

Record attribute path.

Return type

str

kuha_common.document_store.records.dig_and_set(target, attr, replacement, allow_none=False)[source]

Convert value corresponding to attr found from target dict / list of dicts.

Dig into target by consulting attr path. attr may either be an instance of Attribute or a string. If replacement is a callable call it with the value found from path. If it is a fixed value, insert it in place of value found from path. If allow_none is set, raise exception if path contains no value (is None or target has no corresponding key).

Parameters

target (dict or list) – Target of replacement. May be a list or a list of dicts

:param Attribute or str attr: Record Class attribute or path string :param callable or value replacement: callable returns new value or fixed value. If its a

callable, it receives old value as positional argument.

Parameters

allow_none (bool) – False raises ValueError if target has no value for attr.

Returns

None

class kuha_common.document_store.records.RecordBase(document_store_dictionary=None)[source]

Baseclass for each record.

Provides methods used to import, export, create and update records. Dynamically fabricates each class variable of type FieldTypeFactory into an instance variable overriding the class variable.

Note

Use this class throught subclasses only.

Parameters

document_store_dictionary (dict) – Optional parameter for creating a record at initialization time. Note that this dictionary will be iterated destructively.

classmethod get_collection()[source]

Get record collection.

Collection is used for queries against the Document Store.

Returns

collection of the record.

Return type

str

classmethod iterate_record_fields()[source]

Iterate class attributes used as record fields.

Iteration returns tuples: (attribute_name, attribute)

Returns

generator for iterating record fields.

export_metadata_dict(as_datestamps=True)[source]

Export record metadata as dictionary.

Parameters

as_datestamps (bool) – Convert metadata datetime-objects to string timestamps.

Returns

record’s metadata

Return type

dict

export_dict(include_metadata=True, include_id=True)[source]

Return JSON serializable dictionary representation of a record.

Return JSON serializable dictionary. Datetimes will be converted to datestamps.

Parameters

  • include_metadata (bool) – export includes metadata

  • include_id (bool) – export includes id

Returns

record

Return type

dict

set_updated(value=None)[source]

Set updated timestamp.

Sets updated metadata attribute.

Note

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters

value (datetime.datetime or str) – Optional timestamp to set.

set_created(value=None)[source]

Set created timestamp.

Sets created metadata attribute.

Note

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters

value (datetime.datetime or str) – Optional timestamp to set.

set_deleted(value=None)[source]

Set deleted timestamp.

Sets deleted metadata attribute.

Note

The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.

Parameters

value (datetime.datetime or str) – Optional timestamp to set.

set_cmm_type(value=None)[source]

Set cmm type.

Parameters

value (str) – Optional type to set.

set_id(value)[source]

Set ID.

Parameters

value (str) – id to set.

get_updated()[source]

Get updated value.

Note

The timestamp is stored as a datetime.datetime in _metadata.attr_updated, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns

updated timestamp.

Return type

str

get_created()[source]

Get created value.

Note

The timestamp is stored as a datetime.datetime in _metadata.attr_created, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns

created timestamp.

Return type

str

get_deleted()[source]

Get deleted value.

Note

The timestamp is stored as a datetime.datetime in _metadata.attr_deleted, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.

Returns

deleted timestamp.

Return type

str

get_id()[source]

Get record ID.

Id comes from the backend storage system.

Returns

record ID in storage.

Return type

str or None

is_deleted()[source]

Return True if record is logically deleted

Returns

True if record has been deleted.

bypass_update(*fields)[source]

Add fields to be bypassed on update operation.

Parameters

*fields (str) – fieldnames to bypass.

bypass_create(*fields)[source]

Add fields to be bypassed on create operation.

Parameters

*fields (str) – fieldnames to bypass.

updates_record(old_record_dict)[source]

Update record by appending old values that are not present in current record. Use old record’s _id and _metadata.created if present.

Note

parameter is a dictionary since MongoDB returns records as JSON-like objects, which in turn are best represented as dictionaries in python.

Parameters

old_record_dict (dict) – Old record as a dictionary.

updates(secondary_record)[source]

Update record by appending values from secondary which are not present in this record.

Parameters

secondary_record (Record instance subclassed from RecordBase) – lookup values from this record.

class kuha_common.document_store.records.Study(study_dict=None)[source]

Study record.

Derived from RecordBase. Used to store and manipulate Study records. Study number is a special attribute and it cannot be updated.

All attributes of the record are declared as class variables initiated from kuha_common.document_store.field_types.FieldTypeFactory. Instance methods defined in this class are used to add/set values to record attributes. The signatures of the methods are dynamically constructed by the definition of the FieldTypeFactory instances. If, for example, there is a class variable definition:

animals = FieldTypeFactory('animals', 'animal', ['color', 'weight', 'height'])

The correct method signature should be:

def add_animals(self, value, language, color=None, weight=None, height=None):

For the dynamic nature of the record-model these signatures are left open, and python’s *args and **kwargs are used instead. Note that the field type used will raise exceptions if keyword argument key is not found in the initial definition of the field type.

Create a new study record:

>>> study = Study()
>>> study.add_study_number(1234)
>>> study.add_study_titles('Study about animals', 'en')
>>> study.add_principal_investigators('investigator', 'en', organization='Big organization ltd.')

Import existing study record from dictionary:

>>> study_dict = {'study_number': 1234,
... 'study_titles': [{'study_title': 'Study about animals', 'language': 'en'}],
... 'principal_investigators': [{'principal_investigator': 'investigator',
... 'language': 'en', 'organization': 'Big organization ltd.'}]}
>>> study = Study(study_dict)

Iterate attributes:

>>> for pi in study.principal_investigators:
...     pi.attr_organization.get_value()
...
'Big organization ltd.'
Seealso

RecordBase and kuha_common.document_store.field_types

Parameters

study_dict (dict) – Optional study record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number is used to identify a study. It must be unique within records, not localizable and contain only a single value. It cannot be updated.

persistent_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Persistent identifiers. Multivalue-field with unique values.

identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Identifiers. Localizable field with agency-attribute. This needs to be localizable for the sake of agency-attribute.

study_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study titles. Localizable, multivalue-field without attributoes.

document_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document titles. Localizable, multivalue-field without attributoes.

parallel_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Parallele study titles. Localizable, multivalue-field without attributes.

principal_investigators = <kuha_common.document_store.field_types.FieldTypeFactory object>

Pricipal investigators. Localizable, multivalue-field with organization-attribute.

publishers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publishers. Localizable, multivalue-field with abbreviation-attribute.

distributors = <kuha_common.document_store.field_types.FieldTypeFactory object>

Distributors. Localizable, multivalue-field with abbreviation and uri attributes.

document_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document URIs. Localizable, multivalue-field with location and description attributes.

study_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study URIs. Localizable, multivalue-field with location and description attributes.

publication_dates = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication dates. Localizable, multivalue-field without attributes. Note that these are treated as strings, not datetime-objects.

publication_years = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication years. Localizable, multivalue-field with distribution date attribute.

abstract = <kuha_common.document_store.field_types.FieldTypeFactory object>

Abstract. Localizable, multivalue-field.

classifications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Classifications. Localizable, multivalue-field with system name, uri and description attributes.

keywords = <kuha_common.document_store.field_types.FieldTypeFactory object>

Keywords. Localizable, multivalue-field with system name, uri and description attributes.

time_methods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Time methods. Localizable, multivalue-field with system name, uri and description attribute.

sampling_procedures = <kuha_common.document_store.field_types.FieldTypeFactory object>

Sampling procedures. Localizable, multivalue-field with description, system name and uri attibutes.

collection_modes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection modes. Localizable, multivalue-field with system name and uri attritubes.

analysis_units = <kuha_common.document_store.field_types.FieldTypeFactory object>

Analysis units. Localizable, multivalue-field with system name, uri and description attributes.

collection_periods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection periods. Localizable, multivalue-field with event-attribute.

data_kinds = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data kinds. Localizable, multivalue-field.

study_area_countries = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study area countries. Localizable, multivalue-field with abbreviation attribute.

geographic_coverages = <kuha_common.document_store.field_types.FieldTypeFactory object>

Geographic coverages. Localizable, multivalue-field.

universes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Universes. Localizable, multivalue-field with included attribute.

data_access = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access. Localizable, multivalue-field.

data_access_descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access descriptions. Localizable, multivalue-field.

citation_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Citation requirements. Localizable, multivalue-field.

deposit_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Deposit requirements. Localizable, multivalue-field.

file_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

File names. Localizable, multivalue-field.

instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Instruments. Localizable, multivalue-field with instrument name attribute.

related_publications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Related publications. Localizable multivalue-field

study_groups = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study groups. Localizable, multivalue-field with name and description attributes.

copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

data_collection_copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

funding_agencies = <kuha_common.document_store.field_types.FieldTypeFactory object>

Funding agencies. Localizable, multiple-field with role, agency, abbreviation, description and grant number attributes.

grant_numbers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Grant numbers. Localizable, multivalue-field with agency and role attributes.

collection = 'studies'

Database collection (table) for persistent storage.

cmm_type = 'study'

CMM type for Study.

add_study_number(value)[source]

Add study number.

Note

despite the name, the value does not need to be a number.

Parameters

value (str or int) – study number.

add_persistent_identifiers(value)[source]

Add persistent identifiers

Parameters

value (str or int) – persistent identifier

add_identifiers(value, *args, **kwargs)[source]

Add identifiers.

Parameters
add_study_titles(value, *args, **kwargs)[source]

Add study titles.

Parameters
add_document_titles(value, *args, **kwargs)[source]

Add document titles.

Parameters
add_parallel_titles(value, *args, **kwargs)[source]

Add parallel titles.

Parameters
add_principal_investigators(value, *args, **kwargs)[source]

Add principal investigators.

Parameters
add_publishers(value, *args, **kwargs)[source]

Add publishers.

Parameters
add_distributors(value, *args, **kwargs)[source]

Add distributors.

Parameters
add_document_uris(value, *args, **kwargs)[source]

Add document URIs.

Parameters
add_study_uris(value, *args, **kwargs)[source]

Add study URIs.

Parameters
add_publication_dates(value, *args, **kwargs)[source]

Add publication dates.

Parameters
add_publication_years(value, *args, **kwargs)[source]

Add publication dates.

Parameters
add_abstract(value, *args, **kwargs)[source]

Add abstract.

Parameters
add_classifications(value, *args, **kwargs)[source]

Add classifications.

Parameters
add_keywords(value, *args, **kwargs)[source]

Add keywords.

Parameters
add_time_methods(value, *args, **kwargs)[source]

Add time methods.

Parameters
add_sampling_procedures(value, *args, **kwargs)[source]

Add sampling procedures

Parameters
add_collection_modes(value, *args, **kwargs)[source]

Add collection modes

Parameters
add_analysis_units(value, *args, **kwargs)[source]

Add analysis units.

Parameters
add_collection_periods(value, *args, **kwargs)[source]

Add collection periods.

Parameters
add_data_kinds(value, *args)[source]

Add data kinds.

Parameters
add_study_area_countries(value, *args, **kwargs)[source]

Add study area countries.

Parameters
add_geographic_coverages(value, *args)[source]

Add geographic coverages

Parameters
add_universes(value, *args, **kwargs)[source]

Add universes.

Parameters
add_data_access(value, *args, **kwargs)[source]

Add data access.

Parameters
add_data_access_descriptions(value, *args, **kwargs)[source]

Add data access descriptions.

Parameters
add_citation_requirements(value, *args, **kwargs)[source]

Add citation requirements.

Parameters
add_deposit_requirements(value, *args, **kwargs)[source]

Add deposit requirements.

Parameters
add_file_names(value, *args, **kwargs)[source]

Add file name.

Parameters
add_instruments(value, *args, **kwargs)[source]

Add instrument.

Parameters

Add related publications.

Parameters
add_study_groups(value, *args, **kwargs)[source]

Add study group.

Parameters
add_copyrights(value, *args, **kwargs)[source]

Add copyright.

Parameters
add_data_collection_copyrights(value, *args, **kwargs)[source]

Add data collection copyrights.

Parameters
add_funding_agencies(value, *args, **kwargs)[source]

Add funding_agencies.

Parameters
add_grant_numbers(value, *args, **kwargs)[source]

Add grant numbers.

Parameters
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters

secondary (Study) – Lookup values to update from secondary record.

Returns

True if record updated, False if not.

Return type

bool

class kuha_common.document_store.records.Variable(variable_dict=None)[source]

Variable record.

Derived from RecordBase. Used to store and manipulate variable records. Study number and variable name are special attributes and cannot be updated.

Seealso

Study documentation for more information.

Parameters

variable_dict (dict) – Optional variable record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and variable name are used to identify a variable within variable records. Their combination must be unique withing variable records, they cannot be localizable, and they can only contain a single value. They also cannot be updated.

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name within a study. See also study_number

question_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifiers, if variable refers to a question. Not localizable, multiple unique values.

variable_labels = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable labels. Localizable, multivalue-field.

codelist_codes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist codes. Localizable, multivalue-field with label and missing attributes.

collection = 'variables'

Database collection for persistent storage.

cmm_type = 'variable'

CMM type for variable.

add_study_number(value)[source]

Add study number.

Parameters

value (str or int.) – study number.

add_variable_name(value)[source]

Add variable name.

Parameters

value (str) – variable name.

add_question_identifiers(value)[source]

Add question identifier

Parameters

value (str or int.) – question identifier.

add_variable_labels(value, *args, **kwargs)[source]

Add variable label

Parameters
add_codelist_codes(value, *args, **kwargs)[source]

Add codelist code

Parameters
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters

secondary (Variable) – Lookup values to update from secondary record.

Returns

True if record updated, False if not.

Return type

bool

class kuha_common.document_store.records.Question(question_dict=None)[source]

Question record.

Derived from RecordBase. Used to store and manipulate question records. study_number and question_idenntifier are special attributes and cannot be updated.

Seealso

Study documentation for more information.

Parameters

question_dict (dict) – Optional question record as dictionary used for constructing a record instance.

study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and question identifier are used to identify a question. Their combination must be unique withing records, they must not be localizable and they can only contain a single value. They also cannot be updated.

question_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifier within a study. See also study_number

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name that specifies the variable for the question. Not localizable, single value.

question_texts = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question texts. Localizable, multivalue-field.

research_instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Research instruments. Localizable, multivalue-field.

codelist_references = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist references. Localizable, multivalue-field.

collection = 'questions'

Database collection for persistent storage.

cmm_type = 'question'

CMM type for question

add_study_number(value)[source]

Add study number.

Parameters

value (str or int.) – study number.

add_question_identifier(value)[source]

Add question identifier

Parameters

value (str or int.) – question identifier.

add_variable_name(value)[source]

Add variable name.

Parameters

value (str) – variable name.

add_question_texts(value, *args, **kwargs)[source]

Add question text

Parameters
add_research_instruments(value, *args, **kwargs)[source]

Add research instrument

Parameters
add_codelist_references(value, *args, **kwargs)[source]

Add codelist reference

Parameters
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters

secondary (Question) – Lookup values to update from secondary record.

Returns

True if record updated, False if not.

Return type

bool

class kuha_common.document_store.records.StudyGroup(study_group_dict=None)[source]

Study group record.

Derived from RecordBase. Used to store and manipulate study group records. study_group_identifier is a arpecial attribute and cannot be updated.

Seealso

Study documentation for more information.

Parameters

study_group_dict (dict) – Optional study group record as dictionary used for constructing a record instance.

study_group_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group identifier. Used to identify study group. Must be unique within study groups, cannot be localizable and can contain only a single value. This value cannot be updated.

study_group_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group names. Localizable, multivalue-field.

descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group descriptions. Localizable, multivalue-field.

uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group URIs. Localizable, multivalue-field.

study_numbers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study numbers. Multivalue-field with unique values.

collection = 'study_groups'

Database collection for persistent storage.

cmm_type = 'study_group'

CMM type for study groups.

add_study_group_identifier(value)[source]

Add study group identifier.

Parameters

value (str or int) – Study group identifier.

add_study_group_names(value, *args, **kwargs)[source]

Add study group names

Parameters
add_descriptions(value, *args)[source]

Add study group descriptions

Parameters
add_uris(value, *args)[source]

Add study group URIs

Parameters
add_study_numbers(value)[source]

Add study number.

Parameters

value (str or int) – study number.

updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters

secondary (StudyGroup) – Lookup values to update from secondary record.

Returns

True if record updated, False if not.

Return type

bool

kuha_common.document_store.records.record_factory(ds_record_dict)[source]

Dynamically construct record instance based on given document store dictionary.

Looks up the correct record by the cmm type found from ds_record_dict metadata.

Parameters

ds_record_dict (dict) – record received from Document Store.

Returns

Record instance.

Return type

Study or Variable or Question or StudyGroup

kuha_common.document_store.records.record_by_collection(collection)[source]

Finds a record class by the given collection.

Parameters

collection (str) – collection of the record.

Returns

record class

Return type

Study or Variable or Question or StudyGroup

Raises

KeyError if collection is not found in any record.

document_store/mappings

document_store/mappings/exceptions.py

Exceptions for mapping package.

exception kuha_common.document_store.mappings.exceptions.InvalidMapperParameters[source]

Raise for invalid configuration of a mapper - Coding error.

For example trying to add attributes to a mapper which does not support attributes. These errors are coding errors and should be treated as such.

exception kuha_common.document_store.mappings.exceptions.MappingError[source]

Raise for errors while mapping input - User error.

Subclass to create more precise error classes.

exception kuha_common.document_store.mappings.exceptions.ParseError[source]

Unable to parse source XML.

Note

Mask ElementTree ParseError so caller may use MappingError to catch all user-errors when mapping.

exception kuha_common.document_store.mappings.exceptions.UnknownXMLRoot(unknown, expected, *expecteds)[source]

Unknown root element.

exception kuha_common.document_store.mappings.exceptions.MissingRequiredAttribute(*xpaths, msg=None)[source]

Source does not contain a required attribute.

exception kuha_common.document_store.mappings.exceptions.InvalidContent[source]

Attribute found but contains invalid data.

document_store/mappings/xmlbase.py

Components to use for XML parsing & mapping to Document Store records.

Contains a base class to use for parsing XML to Document Store records. Provides common functions useful in parsing XML data.

class kuha_common.document_store.mappings.xmlbase.MappedParams(value)[source]

Contains parameters ready to pass to record’s add-methods.

XMLMapper retrieves parameters from XML record and stores them in an instance of this class. The record instances add-methods get called with stored parameters by using tuple and dict unpacking.

Example:

mapped_params = MappedParams('study_identifier')
mapped_params.set_language('en')
mapped_params.keyword_arguments.update({'agency': 'archive'})
study = Study()
study.add_identifiers(*mapped_params.arguments, **mapped_params.keyword_arguments)
Parameters

value (str or None) – value used as the first argument

has_language()[source]

True if MappedParams has language argument

Returns

True if has language, False if not.

Return type

bool

set_language(language)[source]

Set language argument. Will overwrite if previously set.

Parameters

language (str) – Language to set.

get_language()[source]

Get language argument.

Returns

Language

Return type

str

get_value()[source]

Get value argument.

Returns

value.

Return type

str or None

copy()[source]

Make a copy of the object with contents and return the copy.

Returns

copy of this MappedParams

Return type

MappedParams

has_arguments()[source]

Return True if MappedParams has arguments or keyword_arguments.

Returns

True if object has arguments or keyword_arguments.

Return type

bool

class kuha_common.document_store.mappings.xmlbase.XMLMapper(xpath, from_attribute=None, required=False, localizable=True)[source]

XMLMapper populates MappedParams instances from XML.

Parameters

  • xpath (str) – XPath where to look for element containing value.

  • from_attribute (str or None) – Look value from attribute of the element.

  • required (bool) – raises MissingRequiredAttribute if value is not found.

  • localizable (bool) – True if the value is localizable, False if not.

set_value_conversion(conv_func)[source]

Set conversion callable.

Note

conv_func must accept a string or None as a parameter and return the converted value.

Parameters

conv_func (callable.) – Callable used for conversion.

Returns

self

set_value_getter(getter_func)[source]

Set value getter callable.

Note

getter_func must accept an XML element xml.etree.ElementTree.Element as a parameter and return the value.

Parameters

getter_func (callable.) – Callable used for getting a value from XML element.

Returns

self

expect_single_value()[source]

This mapper will be expected to return a single value.

Returns

self

expect_multiple_values()[source]

This mapper will be expected to return multiple values.

Returns

self

disable_attributes()[source]

This mapper will not contain any attributes.

Returns

self

iterate_attributes(*relations)[source]

Iterate attributes to map.

Parameters

*relations (str) – optional parameters to iterate only attributes with a certain relation.

Returns

A generator yielding tuples of each attribute in the format: (attribute_name, attribute_mapper, attribute_provides_main_lang)

Return type

generator

as_params(element, default_language, xml_namespaces)[source]

Use mapping to construct a MappedParams from XML element.

Use mapper’s _value_getter and _value_conversion to get value from XML element. Construct a MappedParams from the value. If mapping localizable is True add language from XML elements xml:lang attribute.

Parameters

  • element (xml.etree.ElementTree.Element) – XML element.

  • default_language (str) – default language if element has none.

  • xml_namespaces (dict) – XML namespaces for the element.

Returns

mapped parameters ready to pass to records add-method.

Return type

MappedParams

add_attribute(att_name, mapper, relative=True, provides_main_lang=False)[source]

Add attribute to mapper.

Counts the correct xpath if attribute’s mapper’s xpath is a parent element (starting with ‘/..’). Includes all needed information of the attribute to a list of tuples contained in attributes.

Parameters

  • att_name (str) – attribute name

  • mapper (XMLMapper) – mapper instance for mapping value for the attribute.

  • relative (bool) – Is the attribute map’s xpath relative to this element. Defaults to True.

  • provides_main_lang (bool) – Should the language of the attribute be used as a language when mapping this value. Defaults to False.

Raises

InvalidMapperParameters for conflicting parameters such as: 1. Calling this method on a mapper which has disabled use of attributes. 2. Using provides_main_lang for a non-localizable mapper. 3. Setting relative to False on a mapper whose xpath refers to parent element.

Returns

self

value_params(source_xml_element, default_language, xml_namespaces, position=None)[source]

Generate sinle MappedParams object from source XML.

Parameters

  • source_xml_element (xml.etree.ElementTree.Element) – XML element.

  • default_language (str) – Default language.

  • xml_namespaces (dict) – XML namespaces.

  • position (int or None) – Optional position for parent xpaths.

Returns

generator yielding MappedParams.

Return type

generator

Raises

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.

values_params(source_xml_element, default_language, xml_namespaces)[source]

Generate multiple MappedParams objects from source XML.

The generated MappedParams will contain attributes as keyword_arguments.

Parameters
Returns

generator yielding MappedParams.

Return type

generator

Raises

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.

class kuha_common.document_store.mappings.xmlbase.XMLParserBase(root_element)[source]

Base class where parsers get derived from.

Declares the public API to be used in callers.

Input:

  • from_file()

  • from_string()

Output:

  • studies

  • variables

  • questions

  • study_groups

  • all

  • select(collection=None)

Provides common functionality to be used within subclasses which map XML-data to Document Store records. Subclasses must implement necessary generators that generate document store records.

Use in subclass:

class XMLRecordParser(XMLParserBase):
    @property
    def studies(self):
        maps = [(Study.add_study_number, self._map_single(xpath_to_study_number, required=True)),
                (Study.add_study_titles, self._map_multi(xpath_to_study_title))]
        for study_element in self.root_element.findall(xpath_to_study_element, self.NS):
            study = Study()
            self._map_to_record(study, study_element, maps)
            yield study
Parameters

root_element (xml.etree.ElementTree.Element) – XML root.

NS = {'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

default_language = 'en'

Default language.

classmethod from_string(xml_body)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters

xml_body (str) – XML Document as a string. This may come directly from HTTP request body.

Returns

parser for iteratively parsing XML and generating Document Store records.

Return type

XMLParserBase

classmethod from_file(filepath)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters

filepath (str) – Path for the XML file.

Returns

parser for iteratively parsing XML and generating Document Store records.

Return type

XMLParserBase

classmethod child_text(xpath)[source]

Returns a function which will lookup a child element from given xpath. The returned function takes a single element as a parameter which should be an xml.etree.ElementTree.Element or similar. When executed the function returns the child element’s text contents or None if child element cannot be found.

Parameters

xpath – xpath to child. relative to parent.

Returns

function which accepts the parent element as a parameter.

Return type

function

property root_element

Get root element.

Returns

Root element

Return type

xml.etree.ElementTree.Element

property root_language

Get language of the root element. If root does not have a language, returns self.default_language.

Returns

root element language.

Return type

str

property study_number

Get study number as formatted in source XML.

Seealso

self.study_number_identifier

Returns

Study number from source XML.

Return type

str

property study_number_identifier

Get study number converted as a valid Document Store identifier.

Returns

Study number as valid Document Store identifier.

Return type

str

property studies

Studies generator. Must be implemented in subclass.

Returns

Generator which yields Document Store studies.

property variables

Variables generator. Must be implemented in subclass.

Returns

Generator which yields Document Store variables.

property questions

Questions generator. Must be implemented in subclass.

Returns

Generator which yields Document Store questions.

property study_groups

Study groups generator. Must be implemented in subclass.

Returns

Generator which yields Document Store study groups.

property all

Iterate all records found from source XML.

Returns

Generator which yields Document Store records.

Return type

Generator

select(collection=None)[source]

Returns a selective parser. Call with a Document Store collection as parameter to select records only for certain collection.

Note

The returned attributes are defined in subclasses, so they may or may not be generators.

Parameters

collection (str or None) – Document Store collection to select only records belonging to this collection.

Returns

Generator which yields Document Store records.

Return type

Generator

kuha_common.document_store.mappings.xmlbase.as_valid_identifier(candidate)[source]

Convert candidate to a string that conforms the rules of validation.

Indentifier must match regex: [a-zA-Z0-9]+[a-zA-Z0-9?_()-.]*’”]

Note

Regex is defined in Document Store. Should it be moved to kuha_common?

Returns

identifier which conforms the rules of validation.

Return type

str

kuha_common.document_store.mappings.xmlbase.str_equals(correct, default=None)[source]

Conversion function wrapper to compare strings for equality.

Wrapper function that formats comparison value and default value for returned comparison function.

Check if string found from element value or element attribute equals to correct.

Parameters

  • correct (str) – comparison string.

  • default (str) – If the value parameter of the comparison function is None, return this value.

Returns

function which accepts a single parameter for comparison. Returns True or False, or default if the parameter is None.

Return type

function

kuha_common.document_store.mappings.xmlbase.fixed_value(fixed)[source]

Fixed value.

Parameters

fixed – Use this value

Returns

function which accepts a single argument value. The function always returns fixed.

Return type

function

kuha_common.document_store.mappings.xmlbase.element_remove_whitespaces(element)[source]

Conversion function to remove extra whitespace from end of element text.

Iterates element’s inner text using xml.etree.ElementTree.Element.itertext() which iterates over this element and all subelements. Removes extra whitespaces so paragraphs of text will only have one separating whitespace character.

Parameters

element (xml.etree.ElementTree.Element) – Element from which to get text.

Returns

Element’s inner text without extra whitespace.

Return type

str

kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text(element, tags_to_remove=None)[source]

Conversion function to remove inner elements and their contents.

Parameters

  • element (xml.etree.ElementTree.Element) – Element for lookup.

  • tags_to_remove (list) – Optional list of tags to remove from results text content. If given, will only remove elements that match tags that are found from this list. If not given, will remove all child elements. Tags that are given in list but not found from element children will be silently ignored.

Returns

Element’s inner text without text from descendants and without extra whitespace.

Return type

str

kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text_by_tag(tag, *tags)[source]

Conversion function to remove inner elements by tag name.

Acts as a closure function that formats the tags_to_remove parameter and proxies to element_strip_descendant_text().

Parameters

tag (str) – Tag name to remove. Repeat for multiple tags.

Returns

lambda that proxies to element_strip_descendant_text()

kuha_common.document_store.mappings.xmlbase.get_preferred_publication_id_agency_pair(ids_agencys)[source]

Get preferred id + agency two-tuple.

Due to limitations in Kuha data model, a Study.related_publication item can only contain a single identifier + agency pair. This function will search the preferred id + agency pair by consulting the agency values.

Preference by priority:

  1. DOI

  2. Handle

  3. URN

  4. ARK

  5. <first ID with agency>

  6. <first ID without agency>

Params list ids_agencys

list of id + agency two-tuples

Returns

id + agency two-tuple.

Return type

tuple

document_store/mappings/ddi

Package containing DDI mappings to Kuha records.

document_store/mappings/ddi/codebook.py

Mapping profiles for DDI-codebook versions.

class kuha_common.document_store.mappings.ddi.codebook.DDI122RecordParser(root_element)[source]

Parse Document Store records from DDI 1.2.2. XML.

property studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns

Generator to Populate Document Store Study record.

property variables

Parse XML to create and populate multiple kuha_common.document_store.records.Variable instances.

Returns

Generator to populate multiple Document Store Variable records.

property questions

Parse XML to create and populate multiple kuha_common.document_store.records.Question instances.

Returns

Generator to populate multiple Document Store Question records.

property study_groups

Parse XML to create and populate multiple kuha_common.document_store.records.StudyGroup instances.

Returns

Generator to populate multiple Document Store StudyGroup records.

class kuha_common.document_store.mappings.ddi.codebook.DDI122NesstarRecordParser(root_element)[source]
class kuha_common.document_store.mappings.ddi.codebook.DDI25RecordParser(root_element)[source]

Parse Document Store records from DDI 2.5 XML.

NS = {'ddi': 'ddi:codebook:2_5', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

document_store/mappings/ddi/ddi31.py

Mapping profiles for DDI codebook versions.

class kuha_common.document_store.mappings.ddi.ddi31.DDI31RecordParser(root_element)[source]

Parse Document Store records from DDI 3.1. XML

Check the root element. Expects either ddi:DDIInstance or s:StudyUnit. Currently supports only single s:StudyUnit element within the root.

Parameters

root_element (xml.etree.ElementTree.Element) – XML root element.

Raises

UnknownXMLRoot for unexpected root element.

Raises

MappingError if root contains more or less that exactly one s:StudyUnit child.

NS = {'a': 'ddi:archive:3_1', 'c': 'ddi:conceptualcomponent:3_1', 'dc': 'ddi:datacollection:3_1', 'ddi': 'ddi:instance:3_1', 'g': 'ddi:group:3_1', 'l': 'ddi:logicalproduct:3_1', 'pd': 'ddi:physicaldataproduct:3_1', 'pi': 'ddi:physicalinstance:3_1', 'r': 'ddi:reusable:3_1', 's': 'ddi:studyunit:3_1', 'xhtml': 'http://www.w3.org/1999/xhtml', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

property studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns

Generator to Populate Document Store Study record.

property variables

Parse XML to create and populate kuha_common.document_store.records.Variable.

Returns

Generator to Populate Document Store Variable records.

property questions

Parse XML to create and populate kuha_common.document_store.records.Question.

Returns

Generator to Populate Document Store Question records.

property study_groups

Parse XML to create and populate kuha_common.document_store.records.StudyGroup.

Returns

Generator to Populate Document Store StudyGroup records.

document_store/mappings/ddi/ddi33.py

Mapping profiles for DDI 3.3

class kuha_common.document_store.mappings.ddi.ddi33.DDI33RecordParser(root_element)[source]
NS = {'a': 'ddi:archive:3_3', 'c': 'ddi:conceptualcomponent:3_3', 'dc': 'ddi:datacollection:3_3', 'ddi': 'ddi:instance:3_3', 'g': 'ddi:group:3_3', 'l': 'ddi:logicalproduct:3_3', 'pd': 'ddi:physicaldataproduct:3_3', 'pi': 'ddi:physicalinstance:3_3', 'r': 'ddi:reusable:3_3', 's': 'ddi:studyunit:3_3', 'xhtml': 'http://www.w3.org/1999/xhtml', 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}

XML namespaces

property studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns

Generator to Populate Document Store Study record.

testing

Package for common testing functions and classes.

kuha_common.testing.time_me(func)[source]

Decorate function to print its execution time to stdout.

Note

test runner may capture the output.

Parameters

func – Function to decorate. Count execution time of function.

kuha_common.testing.mock_coro(dummy_rval=None, func=None)[source]

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters

  • dummy_rval – return value of dummy function.

  • func – function to call instead of original mocked out function.

Returns

coroutine function.

kuha_common.testing.MockCoro(dummy_rval=None, func=None)

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters

  • dummy_rval – return value of dummy function.

  • func – function to call instead of original mocked out function.

Returns

coroutine function.

testing/testcases.py

Test cases for Kuha

class kuha_common.testing.testcases.KuhaUnitTestCase(methodName='runTest')[source]

Base class for unittests.

  • Assertion methods to check record equality.

  • Helper methods to provide access to dummydata.

dummydata_dir = '/home/docs/checkouts/readthedocs.org/user_builds/kuha2/envs/1.x.x_a/src/kuha-common/kuha_common/testing/dummydata'

Override in sublass to lookup dummydata from different directory.

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass()[source]

Hook method for deconstructing the class fixture after running all tests in the class.

classmethod get_dummydata_path(path)[source]

Get absolute path to dummydatafile

Parameters

path – Path. Gets turned into an absolute if it isn’t

Returns

Absolute path.

Return type

str

classmethod get_dummydata(path)[source]

Get dummydata by reading file from path

Parameters

path – path to file.

Returns

Contents of the file.

classmethod remove_dummyfile_if_exists(path)[source]

Remove dummyfile from path if it exists.

Parameters

path – path to dummyfile.

Returns

None

classmethod set_val(value)[source]

Assign value as dummyvalue.

Parameters

value – Value to assign

Returns

value

classmethod gen_val(lenght=None, unique=False, chars=None)[source]

Generate & assign dummyvalue.

Parameters

  • lenght (int or None) – lenght of the value

  • unique (bool) – should the value be unique

  • chars (str or None.) – use specific characters.

Returns

generated value

Return type

str

classmethod gen_id()[source]

Generate Id.

Returns

Generated id.

Return type

str

classmethod generate_dummy_study()[source]

Generate and return a Study with dummydata.

Returns

study with dummydata

Return type

kuha_common.document_store.records.Study

classmethod generate_dummy_variable()[source]

Generate and return a Variable with dummydata.

Returns

variable with dummydata

Return type

kuha_common.document_store.records.Variable

classmethod generate_dummy_question()[source]

Generate and return a Question with dummydata.

Returns

question with dummydata

Return type

kuha_common.document_store.records.Question

classmethod generate_dummy_studygroup()[source]

Generate and return a StudyGroup with dummydata.

Returns

studygroup with dummydata.

Return type

kuha_common.document_store.records.StudyGroup

static run_until_complete(coro)[source]

Run coroutine until it’s completed

Parameters

coro – Coroutine function.

setUp()[source]

Format testcase values and initialize event loop.

Call asynchronous code synchronously:

self.run_until_complete(coro())
tearDown()[source]

Stop patchers.

async await_and_store_result(coro)[source]

Await coroutine and store returning result.

Example:

self.run_until_complete(self.await_and_store_result(coro()))
Parameters

coro – Coroutine or Future to await

init_patcher(patcher)[source]

Initialize patcher, store for later use, return it.

Parameters

patcher (unittest.mock._patch) – Patch to start.

Returns

MagicMock acting as patched object.

Return type

unittest.mock.MagicMock

assert_records_are_equal(first, second, msg=None)[source]

Assert two Document Store records are equal.

Parameters

  • first – First record to compare.

  • second – Second record to compare.

  • msg – Optional message to output on assertion.

assert_records_are_not_equal(first, second, msg=None)[source]

Assert two Document Store records are not equal.

Parameters

  • first – First record to compare.

  • second – Second record to compare.

  • msg – Optional message to output on assertion.

assert_mock_meth_has_calls(mock_meth, call, *calls)[source]

Assert moch_meth was called with arguments.

This calls Mock.assert_has_calls and tests for call count. The actual benefit of using this method over the built-in assert_has_calls is that this method tries to pinpoint the actual call that was missing when assert_has_calls raised AssertionError. This is useful when mock_meth has had multiple calls. The built-in assert_has_calls will notify of all calls that the mock_meth has had, while this method will notify of the actual call that was missing.

Parameters

  • mock_meth – Mocked method that is target of testing.

  • call – Call that should be found. Instance of unittest.mock._Call Repeat this argument to test for multiple calls.

Raises

AssertionError if calls not found.

class kuha_common.testing.testcases.KuhaEndToEndTestCase(methodName='runTest')[source]

Base class for end-to-end tests.

  • HTTPClient for interacting with Document Store.

  • Assertion methods to check returning payload and status codes.

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass()[source]

Hook method for deconstructing the class fixture after running all tests in the class.

static get_record_url(rec_or_coll, _id=None)[source]

Get URL to Document Store records or single record.

Parameters

  • rec_or_coll – record, record class or collection

  • _id (str or None) – Optional record ID.

Returns

URL to Document Store collection or single record.

Return type

str

static get_query_url(rec_or_coll, query_type=None)[source]

Get URL to Document Store query endpoint for collection

Parameters

  • rec_or_coll (str, record, or record class) – Collection to query.

  • query_type – Optional query type

Returns

URL to query endpoint.

Return type

str

classmethod GET_to_document_store(rec_or_coll, _id=None)[source]

GET to Document Store returns record(s).

Parameters

  • rec_or_coll – record or collection to get.

  • _id – Optional ObjectId. Will take precedence over rec_or_coll id.

Returns

response body

classmethod POST_to_document_store(record)[source]

POST to Document Store creates record.

Parameters

record – Record to post.

Returns

response body

classmethod DELETE_to_document_store(rec_or_coll=None, _id=None, hard_delete=True)[source]

DELETE to Document Store deletes record(s).

Call without arguments to delete all records from all collections.

Parameters

  • rec_or_coll (str or None) – Collection to delete from.

  • _id (str or None) – ID of the record to delete.

  • hard_delete (bool) – Set to False to do a logical delete instead.

Returns

Response body or bodies wrapped in dict ({<collection>: <resp_body>})

classmethod query_document_store(rec_or_coll, query, query_type=None)[source]

Execute query against Document Store query API.

Parameters

  • rec_or_coll (str or record class or record instance) – Collection to query.

  • query – Query.

  • query_type – Type of Query.

Returns

query results

Return type

None if query returned no results, dict for results.

classmethod get_collection_record_count(rec_or_coll, exclude_deleted=True)[source]

Return number or records for collection in Document Store.

Parameters

  • rec_or_coll – Document Store record, Document Store record class or collection.

  • exclude_deleted (bool) – exclude logically deleted records from count.

Returns

record count in Document Store.

Return type

int

assert_document_store_is_empty(exclude_deleted=True)[source]

Assert Document Store contains no records.

Parameters

exclude_deleted (bool) – exclude logically deleted records from count.

Raises

AssertionError if Document Store has records.