kuha_common

High-level modules common for Kuha applications.

server.py

Common server functions & classes used in Kuha.

kuha_common.server.log_request(handler)[source]

Log request output to JSON. Gets called after each successful response.

Parameters:handler (subclass of tornado.web.RequestHandler) – handler for the current request.
kuha_common.server.log_exception(typ, value, tb, correlation_id)[source]

Log exception output to JSON. Gets called from RequestHandler.

Parameters:
  • typ – type of exception
  • value – caught exception
  • tb – traceback
  • correlation_id – correlation id of the request that ended in exception.
kuha_common.server.str_api_endpoint(api_version, suffix=None)[source]

Helper function to prepend endpoints with api_version.

Parameters:
  • api_version (str) – version of the api.
  • suffix (str) – api endpoint.
Returns:

str – endpoint prepended with api_version
kuha_common.server.serve(web_app, port, process_count=0, on_exit=None)[source]

Serve web application.

Parameters:
  • web_app (tornado.web.Application) – application to serve
  • port (int) – Port to listen to.
  • process_count (int) – number of processes to spawn. 0 = forks one process per cpu.
  • on_exit (function) – callback on server/ioloop stop.
class kuha_common.server.RequestHandler(*args, **kwargs)[source]

Common request handler for kuha server applications. Subclass in application specific handlers.

prepare()[source]

Prepare each request.

Look for correlation id; create one if not found. Set correlation id to response header.

set_output_content_type(ct, charset='UTF-8')[source]

Sets content type for responses.

Parameters:
  • ct (str) – content type for response.
  • charset (str) – charset definition for response content type.
log_exception(typ, value, tb)[source]

Overrides tornados exception logging. Sends HTTP errors as responses. Calls customised log_exception() which outputs JSON encoded log messages with request correlation ids. For easier debugging it also calls tornado.web.RequestHandler.log_exception to output full traceback.

Parameters:
  • typ – type of exception
  • value – caught exception
  • tb – traceback
assert_request_content_type(supported_content_type)[source]

Assert request has correct content type header.

Parameters:supported_content_type (str) – content type supported by endpoint.
Raises:InvalidContentType – if request has invalid content type.
write_error(status_code, **kwargs)[source]

Overrides tornado.web.RequestHandler.write_error. Outputs error messages in preferred content type.

Parameters:
  • status_code (int) – HTTP status code.
  • **kwargs – keyword arguments are passed to tornado.web.RequestHandler.write_error if output content type is not application/json
class kuha_common.server.WebApplication(handlers)[source]

Override tornado.web.Application to make sure server applications are using correct initialization parameters.

log_request(handler)[source]

Override tornado.web.Application.log_request. Server uses it’s own implementation of log_request.

Parameters:handler (Subclass of tornado.web.RequestHandler) – Handler of current request.
exception kuha_common.server.KuhaServerError(msg, status_code=500, context=None)[source]

Base class for common HTTP-exceptions

exception kuha_common.server.InvalidContentType(requested_content_type, supported_content_type)[source]

Invalid content type HTTP-exception.

exception kuha_common.server.BadRequest(msg=None)[source]

Bad request HTTP-exception.

exception kuha_common.server.ResourceNotFound(msg=None, context=None)[source]

Resource not found HTTP-exception.

query.py

Perform query operations against Kuha Document Store.

Offers High-level query methods to facilitate an easy access point with all necessary actions and properties needed to perform queries. To query the Document Store the caller only needs to use methods defined in class QueryController and records defined in kuha_common.document_store.records

class kuha_common.query.ResultHandler(record_constructor, on_record=None)[source]

Class which handles results and correct calls to callbacks, if any. Stores the result for later use.

Dynamically creates callable method handle() which receives result payload and calls on_record correctly.

Parameters:
Returns:

ResultHandler
class kuha_common.query.QueryController(headers=None, record_limit=0)[source]

Asynchronous controller to query the Document Store.

Use to build queries and automatically fetch responses using HTTP as a protocol and JSON as exchange format.

Optional record_limit parameter may be given at initialization time to limit the number of records that are requested in a single HTTP request.

Parameters:
  • headers (dict) – Optional headers parameter to store headers that are used in each query application wide as HTTP headers.
  • record_limit (int) – Optional record_limit parameter which is used to limit the number of records requested in a single HTTP request.

Example:

from kuha_common.document_store import Study
query_ctrl = QueryController()
study = await query_ctrl.query_single(
    Study,
    fields=[Study._metadata, Study.study_number],
    _filter={Study.study_number: 1234}
)
fk_constants

alias of FilterKeyConstants

query_single(record, on_record=None, headers=None, **kwargs)[source]

Query single record.

Parameters:
  • record (Subclass of kuha_common.document_store.records.RecordBase) – class used to construct the record instance.
  • on_record (function or coroutinefunction) – Optional callback function that gets passed the returned and instantiated record object.
  • headers (dict) – Optional headers for this query. Headers get added to headers given for QueryController at initialization time. Note that it will overwrite headers with same key.
  • **kwargs – Keyword arguments contain parameters for the query. They get passed to kuha_common.document_store.query.Query.construct()
Returns:

None if passed on_record callback, else returns the initiated record object.
Raises:

QueryException – if query parameters given as keyword arguments contain limit-parameter.
query_multiple(record, on_record, headers=None, **kwargs)[source]

Query multiple records.

Queries the document store for multiple records. Behaviour depends on whether record_limit has been set:

If there is a record_limit

If there is no record_limit

  • this method returns nothing.
  • The on_record callback gets called with each instantiated record object.
  • on_record may be a normal function or a coroutine.
Parameters:
Returns:

None if no record_limit, else kuha_common.document_store.client.JSONStreamClient.run_queued_requests()
query_count(record, headers=None, **kwargs)[source]

Query the number of records.

Parameters:
Returns:

Number of records
Return type:

int
query_distinct(record, headers=None, **kwargs)[source]

Query distinct values.

Parameters:
Returns:

distinct values: {fieldname : [value1, value2, value3]}. Note that contained values may be dictionaries or strings, depending on what is stored in requested field.
Return type:

dict

cli_setup.py

Command line setup for Kuha applications

Parse command line for common configuration options and store loaded settings.

Load modules for querying Document Store:

import os
from kuha_common import cli_setup
cli_setup.load(os.getcwd())
settings = cli_setup.setup(cli_setup.MOD_DS_CLIENT, cli_setup.MOD_DS_QUERY)
kuha_common.cli_setup.MOD_DS_CLIENT = 'document_store.client'

Constant for configuring kuha_common.document_store.client

kuha_common.cli_setup.MOD_DS_QUERY = 'document_store.query'

Constant for configuring kuha_common.document_store.query

kuha_common.cli_setup.MOD_LOGGING = 'logging'

Constant for configuring logging

kuha_common.cli_setup.add_kuha_loglevel(parser=None)[source]

Add loglevel to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_kuha_logformat(parser=None)[source]

Add logformat to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_document_store_url(parser=None, **kw)[source]

Add document store url to parser

Parameters:
  • parser (ArgumentParser instance) – command line parser.
  • **kw – keyword arguments get passes to parser.add
kuha_common.cli_setup.add_document_store_host(parser=None)[source]

Add document store host to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_document_store_port(parser=None)[source]

Add document store port to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_document_store_api_version(parser=None)[source]

Add document store api-version to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_document_store_client_request_timeout(parser=None)[source]

Add document store client request timeout to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_document_store_client_connect_timeout(parser=None)[source]

Add document store client connect timeout to parser

Parameters:parser (ArgumentParser instance) – command line parser
kuha_common.cli_setup.add_document_store_client_max_clients(parser=None)[source]

Add document store client max clients timeout to parser

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_print_configuration(parser=None)[source]

Add print configuration helper for testing configuration options.

Parameters:parser (ArgumentParser instance) – command line parser.
kuha_common.cli_setup.add_server_process_count_configuration(parser=None)[source]

Add tornado server process count to configuration options parser.

Parameters:parser (ArgumentParser instance) – command line parser.
class kuha_common.cli_setup.KuhaConfigFileParser[source]

Inherit to override configargparse.DefaultConfigFileParser.get_syntax_description()

get_syntax_description()[source]

Override syntax description of configargparse.DefaultConfigFileParser

Returns:Syntax description for configuration file.
Return type:str
class kuha_common.cli_setup.Settings[source]

Class for command line settings.

is_parser_loaded()[source]

Check is parser loaded.

Return type:bool
is_settings_loaded()[source]

Check is settings loaded.

Return type:bool
set_abs_dir_path(path)[source]

Set absolute directory path of configurable kuha application.

Parameters:path (str) – absolute path to kuha application directory.
get_abs_dir_path()[source]

Return absolute directory path of configurable kuha application.

Returns:absolute path to directory.
Return type:str
add_logging_configs()[source]

Wrapper to add logging-module configuration.

add_document_store_query_configs()[source]

Wrapper to add document_store_query configuration.

add_document_store_client_configs()[source]

Wrapper to add document_store_client configuration.

setup_logging()[source]

Setup logging module.

setup_document_store_query()[source]

Setup kuha_common.document_store.query module.

setup_document_store_client()[source]

Setup kuha_common.document_store.client module.

load_parser(config_file=None, **kw)[source]

Load command line parser.

Additional keyword arguments are passed to configargparse.ArgumentParser.

Parameters:config_file (str) – Name of configuration file.
set(parsed_opts)[source]

Assign parser options to settings.

Parameters:parsed_opts (argparse.Namespace) – parser options.
load_cli_args()[source]

Load command line arguments.

get()[source]

Return active settings.

Returns:active settings.
Return type:argparse.Namespace
add(*args, **kwargs)[source]

Add item for parser. Settings must not yet be loaded but parser must be loaded.

Parameters:
  • *args – arguments passed to configargparse.ArgumentParser
  • **kwargs – keyword arguments passed to configargparse.ArgumentParser
kuha_common.cli_setup.setup(*modules)[source]

Setup command line parser.

Load modules, parse command line arguments, return loaded settings in argparse.Namespace

Parameters:*modules (str) – common Kuha modules to load and include in parsing of command line arguments.
Returns:Loaded settings.
Return type:argparse.Namespace
kuha_common.cli_setup.get_settings()[source]

Get loaded settings stored in Settings.

Returns:Loaded settings.
Return type:argparse.Namespace
kuha_common.cli_setup.add(*args, **kwargs)[source]

Module level function to add items to be parsed in Settings singleton.

Parameters:
kuha_common.cli_setup.load(abs_dir_path, **kwargs)[source]

Module level function to load parser to Settings singleton.

Parameters:
  • abs_dir_path (str) – absolute path to directory of the kuha application to be configured.
  • **kwargs – keyword arguments passed to Settings.load_parser().
kuha_common.cli_setup.prepend_abs_dir_path(path)[source]

Helper function to prepend the stored absolute directory path to given argument.

Parameters:path – end of the path.
Returns:absolute path ending to path.
Return type:str

document_store

Contains modules for interacting with Document Store.

document_store/client.py

kuha_common.document_store.client provides a http client interface to communicate with Kuha Document Store.

class kuha_common.document_store.client.JSONStreamClient[source]

Base class used for requests. Implements own queue to store requests, since tornado.httpclient.AsyncHTTPClient starts timers for request_timeout and connect_timeout at the moment we call client.fetch(). See https://github.com/tornadoweb/tornado/issues/1400 for more details.

Handles JSON decoding of incoming chunks and the encoding of body to JSON.

max_clients = 10

Controls the maximum number of concurrent clients.

request_timeout = 120

Sets timeout for a request.

connect_timeout = 120

Sets timeout for establishing a connection.

sleep_on_queue = 5

Sets sleep timer for queue.

classmethod set_max_clients(max_clients)[source]

Set maximum concurrent clients

classmethod set_request_timeout(request_timeout)[source]

Set timeout per request

classmethod set_connect_timeout(connect_timeout)[source]

Set timeout per connection

classmethod request(url, **kwargs)[source]

Constucts a streaming request.

Parameters:
  • url (str) – url to request.
  • kwargs – keyword arguments passed to tornado.httpclient.HTTPRequest
Returns:

tornado.httpclient.HTTPRequest
wrap_streaming_callback(callback)[source]

Wrap streaming callback to support chunked JSON responses.

Parameters:callback (callable.) – streaming callback. Gets called with response which is decoded to python object from JSON.
Returns:Wrapped callback
Return type:functools.partial
execute_stored_callbacks()[source]

Executes asynchronous callbacks stored in _callbacks

run_queued_requests(queued_requests=None)[source]

Run queued requests.

Calls queued requests asynchronically. Sleeps for sleep_on_queue if max_clients reached.

Parameters:queued_requests (collections.deque) – Optionally pass queued_requests to run.
get_streaming_request(streaming_callback, url, body=None, method=None, headers=None, **kw)[source]

Get a streaming request.

Sets default headers Content-Type: application/json if not already given. Encodes body to JSON if given and is not string or bytes. If response is empty (for example query with no results) the streaming callback doesn’t get called.

Subclass and override to support arbitrary requests.

Parameters:
  • streaming_callback (callable) – callback which receives the response if any.
  • url (str) – URL to send request to.
  • body (str, dict, list, tuple, integer, float or None) – Optional request body. String will be supplied as is. Other values will be encoded to JSON.
  • method (str or None) – HTTP method. Defaults to POST.
  • headers (dict or None) – optional request headers. if Content-Type is not set, will set ‘Content-Type’: ‘application/json’ as default.
Returns:

HTTP request
Return type:

tornado.httpclient.HTTPRequest
queue_request(*args, **kwargs)[source]

Queue request to be run aynchronously by calling run_queued_requests.

Parameters:
  • *args – arguments passed to get_streaming_request
  • **kwargs – keyword arguments passed to get_streaming_request.
Returns:

run_queued_requests() method to call to run the queued requests.
fetch(*args, **kwargs)[source]

Run single query.

Parameters:
  • *args – arguments passed to queue_requests.
  • **kwargs – keyword arguments passed to queue_requests

document_store/query.py

Access query properties by convenience methods to help build valid queries against the Document Store.

class kuha_common.document_store.query.FilterKeyConstants[source]

Class used as a namespace to contain constants used in query filter.

exception kuha_common.document_store.query.QueryException(msg, context=None)[source]

Exception for errors raised by Query. Has an optional context parameter for giving some additional information about the context of the exception.

class kuha_common.document_store.query.Query(query, query_document, query_type='select')[source]

Manipulate query properties without compromising the validity of the constructed query. Build the correct url for different query types.

Note:This class provides low-level operations. Use kuha_common.query.QueryController for easy access to query actions and properties.

Example:

from kuha_common.document_store import Study, Query
query = Query(Query.construct(_filter={Study.study_number:'123'}), Study.collection)
Parameters:
  • query (dict) – Actual query containing the properties such as _filter, fields, sort_by etc.
  • query_document (str) – One of the supported query documents declared in Query.supported_query_documents and specified in kuha_common.document_store.records.py
  • query_type (str) – Optional query_type parameter. Defaults to Query.query_type_select. Other valid values are Query.query_type_count and Query.query_type_distinct.
k_filter = '_filter'

Query parameter for filtering results.

k_fields = 'fields'

Query parameter for fields to contain in results.

k_limit = 'limit'

Query parameters for limiting returned results.

k_skip = 'skip'

Query parameter for skipping number of results from the beginning of the resultset.

k_sort_order = 'sort_order'

Query parameter for sort order.

k_sort_by = 'sort_by'

Query parameter for sorting by a certain field.

k_fieldname = 'fieldname'

Query parameter for distinct queries. Specifies a field from which the distinct values are to be fetched.

query_type_select = 'select'

Query type for select queries. Using this query type gets records as a response.

query_type_count = 'count'

Query type for count queries. Using this query type the query returns an integer.

query_type_distinct = 'distinct'

Query type for distinct queries. Using this query type the returning object contains all distinct values for a certain field.

classmethod set_base_url(base_url)[source]

Configure document store url used as a base when constructing the endpoint url for queries.

classmethod as_supported_datetime_str(datetime_obj)[source]

Get datetime object as a supported datetime string.

Parameters:datetime_obj (datetime-object.) – Python datetime-object to be convered to str.
Returns:String represenation of the datetime-object that is suitable foe querying.
Return type:str
classmethod construct(**kwargs)[source]

Construct valid query parameters.

Example:

from kuha_common.document_store import Query, Study
params = Query.construct(_filter={Study.study_number:'123'},
                         fields=[Study._metadata, Study._id, Study.abstract],
                         sort_by=Study._id)
query = Query(params, Study.collection)
Parameters:**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.
Returns:Valid query, ready to be sent to Document Store.
Return type:dict
classmethod construct_distinct(**kwargs)[source]

Construct valid query parameters for distinct queries.

Parameters:**kwargs – keys should be valid query properties, while values should hold corresponding query values supported by the key.
Returns:Valid query, ready to be sent to Document Store.
Return type:dict
classmethod build_query_for_date_range(from_=None, until=None)[source]

Build query filter for date-range.

Parameters:
  • from (datetime-object) – start of the date-range:
  • until (datetime-object) – end of the date-range:
Returns:

date-range query-filter with datetime-objects converted into string representation.
Return type:

dict
classmethod build_query_for_exists(exists)[source]

Build query for exists-query.

Parameters:exists (bool) – whether the field should exists or not.
Returns:valid exists query for filter.
Return type:dict
Raises:ValueError for invalid boolean values in exists-parameter.
classmethod get_valid_params(query_type=None)[source]

Return valid query parameters for the query type.

Parameters:query_type (str) – Optional query_type for which the query-parameters should be valid for.
classmethod is_valid_query(query, query_type)[source]

Check the validity of query parameters.

Parameters:
  • query (dict) – Full query to validate.
  • query_type (str) – Query type to validate against.
Returns:

Whether or not the query-parameters given are valid.
Return type:

bool
classmethod is_valid_query_type(query_type)[source]

Check the validity of query_type.

Parameters:query_type (str) – Query type to validate.
Returns:Whether or not the query type is valid.
Return type:bool
is_valid_query_document(query_document)[source]

Check the validity of query document.

Parameters:query_document (str) – Query document to validate.
Returns:Whether or not the query document is valid.
Return type:bool
is_valid_param(parameter)[source]

Check the validity of a single query parameter.

Parameters:parameter (str) – Query parameter to validate.
Returns:Whether or not the parameter is valid.
Return type:bool
validate_query(query)[source]

Validate query parameters.

Checks parameters’ validity for chosen query type. Raises QueryException if invalid.

Parameters:query (dict) – Query parameters.
Returns:Query parameters.
Return type:dict
Raises:QueryException if query parameters are invalid.
validate_query_type(query_type)[source]

Validate query type.

Checks that the query type is supported by Document Store. Raises QueryException for invalid query type.

Parameters:query_type (str) – Query type to validate.
Returns:Query type.
Return type:str
Raises:QueryException if query type is invalid.
validate_query_document(query_document)[source]

Validates query document.

Checks that the query document is supported by Document Store. Raises QueryException if invalid.

Parameters:query_document (str) – Query document to validate.
Returns:Query document
Return type:str
Raises:QueryException if query document is invalid.
get_endpoint()[source]

Get correct endpoint for querying the Document Store.

Builds the endpoint by consulting configured values and the instantiated query for query_type and query_document

Returns:Full url to Document Store endpoint which handles the constructed query.
Return type:str
get_query(strip_invalid_params=True)[source]

Returns the constructed query parameters.

If the query type has been changed after initialization, for example to get the count of records, this method strips the invalid query parameters from the returned query. When doing so, it does not change the stored query parameters, but rather makes a copy of them for manipulating and returning.

Parameters:strip_invalid_params (bool) – Whether to strip the unsupported (=invalid) query parameters out of the returned query.
Returns:Constructed query parameters ready to submit to Document Store.
Return type:dict
get_limit()[source]

Get query limit parameter.

Returns:Query limit (int) if set. None if not set.
Return type:int or None
get_skip()[source]

Get query skip parameter.

Returns:Query skip (int) if set. None if not set.
Return type:int or None
set_limit(limit)[source]

Set limit parameter for query.

Limit controls how many results should be returned.

Parameters:limit (int) – Limit parameter for query.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
set_skip(skip)[source]

Set skip parameter for query.

Skip conrols how many results should be skipped from the start (offset).

Parameters:skip (int) – Skip parameter for query.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
set_fields(fields)[source]

Set fields parameter for query.

Field controls which fields of the record should be returned. fields can be a list of strings in the form used by MongoDB or a list of kuha_common.document_store.records class-variables.

Example:

from kuha_common.document_store import Query, Study
_params = Query.construct(_filter={Study.study_number:'123'})
_query = Query(_params, Study.collection)
_query.set_fields([Study.abstract, Study.study_number])
Parameters:fields (list) – Fields parameter for query.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
set_sort_by(sort_by)[source]

Set sort_by parameter for query.

Determines sorting of the returned results. sort_by can be a string in the form used by MongoDB or a kuha_common.document_store.records class-variables.

Parameters:sort_by (srt or class-variable of a record.) – Sort by parameter for query.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
set_sort_order(order)[source]

Set sort order for the query.

Determines the order which the returned results are to be sorted by.

Note:Valid values come from pymongo. They actually depend on the mongodb driver, but since this is a caller API we don’t want to make pymongo a dependency.
Parameters:order (int) – Sort order. Must be either 1 or -1.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
Raises:QueryException for invalid order values.
set_query_type(query_type)[source]

Set query type.

Parameters:query_type (str) – Valid query type for the query to be constructed.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()
add_query_statement(field, statement)[source]

Add query statement.

Manipulates the _filter parameter of the query parameters. Raises a QueryException if the field already has a statement declared in _filter.

Parameters:
  • field (str) – Field to target the statement to.
  • statement (str) – Statement to filter the results by.
Returns:

self for easy aggregation of manipulation methods.
Return type:

instantiated Query()
add_query_statements(**kwargs)[source]

Add multiple query statements to filter the returned results.

Manipulates the _filter parameter of the query parameters.

Parameters:**kwargs – key-value pairs that are to be added to the _filter parameter.
Returns:self for easy aggregation of manipulation methods.
Return type:instantiated Query()

document_store/field_types.py

Properties and actions for field types supported by records defined in kuha_common.document_store.records

Provides field types to be used not only for the construction of new records and updating exising records, but also to provide a format for fields of records that is interchangeable in a way that a receiver does not need to know the specifics of a field beforehand, but may use the field to gain knowledge of the properties of the field.

This module also provides factories which are used to fabricate the fieldtypes. The instantiated factories hold knowledge of the fields even thought the fields themselves are not yet instantiated. This knowledge is used for querying records, but also to dynamically fabricate fieldtypes for records in kuha_common.document_store.records

exception kuha_common.document_store.field_types.FieldTypeException[source]

Exception to raise on field type errors. Used for programming errors.

class kuha_common.document_store.field_types.Value(name, value=None)[source]

Value is the most simple type of field.

Field type with name and single value. Serves also as a baseclass for other field types.

Parameters:
  • name (str) – Name of the field.
  • value – Optional value for the field.
set_value(value)[source]

Set initial value.

Parameters:value – Value to set.
add_value(value)[source]

Add value for the field.

Note:Overrides existing value.
Parameters:value – The value to set.
get_value()[source]

Get the value of the field.

Returns:Value.
get_name()[source]

Get name of the field.

Returns:The name of the field.
Return type:str
export_dict()[source]

Exports Value as dictinary.

Returns:{name:value}
Return type:dict
import_records(record)[source]

Import record to field.

Parameters:record – record to import.
updates(secondary_values)[source]

Update value.

Parameters:secondary_values (str) – Value to update to.
class kuha_common.document_store.field_types.Set(name, value=None)[source]

Set is a field type with name and list of unique values.

Derived from Value Implements methods that differ from parent class.

Parameters:
  • name (str) – Name of the field.
  • value (list or None) – Optional value for the field.
set_value(value)[source]

Sets value.

Parameters:value (list) – Value for field.
Raises:FieldTypeException if submitted value is not a list.
add_value(value)[source]

Add value to field.

Appends a value to the list of values already set. Makes sure that the list holds no duplicates by silently discarding them.

Parameters:value (list or str or None) – value or values to be appended. If value is None, empties the list.
import_records(record)[source]

Import records by adding the submitted records to contained values.

Parameters:record (list or str or None) – Hold the values to be imported.
updates(secondary_values)[source]

Updates old values with values contained in this Set.

Looks for combination of secondary_values and values in this set. Discards duplicates and stores the updated values to value.

Parameters:secondary_values (list) – list of old values to be updated with new ones.
class kuha_common.document_store.field_types.Element(name, attribute_names=None)[source]

Element is a field type with name, value and attributes.

Derived from Value.

Element is used to store fields that contain attributes in addition to a value. Each attribute in itself is an instance of Value and is dynamically stored in instance variable attr_<name>. When instantiated and populated with values and attributes, the element-instance can be used to get it’s value, value’s name, but also to get the attributes and their names, even thought the caller does not know the attribute names a priori.

Example of constructing an element (the source):

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)

Example of reading from an unknown element (the receiver):

>>> unknown_element.get_name()
'animal'
>>> unknown_element.get_value()
'cat'
>>> for att in unknown_element.iterate_attributes():
...     att.get_name() + ' : ' + str(att.get_value())
...
'height : 5'
'color : yellow'
'weight : 10'
>>> unknown_element.attr_color.get_value()
'yellow'

This is especially useful when using as an interchange format. The receiver does not need to know the attribute names beforehand. Instead the receiver can iterate throught every attribute to get their name-value pairs or if the receiver is interested in a single attribute, it may be called by the dynamically constructed instance-variable prefixed with attr_.

Parameters:
  • name (str) – Name of the field.
  • attribute_names (list) – Optional parameter for attribute names.
Raises:

FieldTypeException if attribute_names has duplicates.
is_pending()[source]

Is the element pending for values.

Returns:True if pending, False if not.
Return type:bool
new()[source]

Create a new element-instance with same name and attributes but without values.

Instantiates a new instance of itself. The new instance is pending for values.

Example:

>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> another_animal = animal.new()
>>> another_animal.add_value('dog', color='white', weight=30, height=15)
Returns:new element.
Return type:Element
add_value(value=None, **attributes)[source]

Add value with attributes as keyword arguments.

Note:This may only be called once for each instance.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
Parameters:
  • value (str or int or None) – Value for the element.
  • **attributes – keyword arguments for attributes of the element.
Raises:

FieldTypeException if the element already has values or if submitted value is None and no attributes are given.
iterate_attributes()[source]

Generator function. Iterates element attributes.

Returns:a generator object for iterating attributes.
get_attribute(name)[source]

Get attribute by attribute name.

Parameters:name (str) – Name of the attribute to look for.
Returns:attribute of the element or None if not found.
Return type:Value or None
set_attribute(name, value)[source]

Sets new value for attribute.

Note:

The element must have an attribute with the name.
Parameters:
  • name (str) – attribute name.
  • value (str or int or None) – new value.
Raises:

FieldTypeException if element does not have an attribute with submitted name.
export_attributes_as_dict()[source]

Export element’s attributes as a dictionary.

Returns:dictinary representing the attributes
Return type:dict
export_dict()[source]

Export the element as a dictionary.

Returns a dictionary with key-value pairs given wrapped inside a another dictionary with the elements key as name.

Example:

>>> from kuha_common.document_store.field_types import Element
>>> animal = Element('animal', ['color', 'weight', 'height'])
>>> animal.add_value('cat', color='yellow', weight=10, height=5)
>>> animal.export_dict()
{'animal': {'color': 'yellow', 'weight': 10, 'height': 5, 'animal': 'cat'}}
Returns:dictinary representing the Element
Return type:dict
import_records(record)[source]

This object does not support importing records.

Raises:FieldTypeException
updates(secondary_values)[source]

Updates attributes not found in this element with the ones found from secondary_values.

Parameters:secondary_values (dict) – Attributes from old element.
class kuha_common.document_store.field_types.LocalizableElement(name, attribute_names=None)[source]

LocalizableElement is a field type with name, value, language and attributes.

Derived from Element. Has an additional attribute for language. The language is special attribute that is used when updating elements.

Seealso:

Element
Parameters:
  • name (str) – Name of the element.
  • attribute_names (list) – Optional list of attribute names.
Raises:

FieldTypeException if attribute_names contain a name that is reserved for language.
set_language(language)[source]

Set language for element.

Parameters:language (str) – language to set.
Raises:FieldTypeException if language already set.
get_language()[source]

Get language of element.

Returns:language
Return type:str or None
add_value(value=None, language=None, **attributes)[source]

Add values for element.

Note:

This may only be called once for each instance.
Seealso:

Element.add_value()
Parameters:
  • value (str or int) – value to set.
  • language (str) – language of the element.
  • **attributes – keyword arguments for attributes of the element.
Raises:

TypeError if language is not given or is None.
export_dict()[source]

Export the element as a dictionary.

Seealso:Element.export_dict()
Returns:dictinary representation of the element.
Return type:dict
class kuha_common.document_store.field_types.ElementContainer(name, sub_element)[source]

ElementContainer contains a list of single type of Element/LocalizableElement field types.

Receives mandatory parameters for name and sub_element. The sub_element describes the element types that this container can store. Every new element that a container can create will be an instance created from this sub_element.

Example:

>>> from kuha_common.document_store.field_types import ElementContainer, LocalizableElement
>>> animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> animals = ElementContainer('animals', animal)
>>> animals.add_value('cat', 'en', color='yellow', width=10, height=5)
>>> animals.add_value('kissa', 'fi', color='keltainen', width=10, height=5)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'width': 10,
     'language': 'en',
     'color': 'yellow',
     'height': 5,
     'animal': 'cat'},
    {'width': 10,
     'language': 'fi',
     'color': 'keltainen',
     'height': 5,
     'animal': 'kissa'}]
}

Elements can be iterated:

>>> for animal in animals:
...     animal.attr_color.get_value() + " for language: " + animal.get_language()
...
'yellow for language: en'
'keltainen for language: fi'

And updated with containers sharing name and attribute names:

>>> another_animal = LocalizableElement('animal', ['color', 'width', 'height'])
>>> more_animals = ElementContainer('animals', another_animal)
>>> more_animals.add_value('dog', 'en', color='white', width=20, height=10)
>>> more_animals.add_value('koira', 'fi', color='valkoinen', width=20, height=10)
>>> animals.updates(more_animals)
>>> animals.export_dict()  # result formatted for better readability
{'animals': [
    {'language': 'en',
     'height': 5,
     'color': 'yellow',
     'animal': 'cat',
     'width': 10},
    {'language': 'fi',
     'height': 5,
     'color': 'keltainen',
     'animal': 'kissa',
     'width': 10},
    {'language': 'en',
     'height': 10,
     'color': 'white',
     'animal': 'dog',
     'width': 20},
    {'language': 'fi',
     'height': 10,
     'color': 'valkoinen',
     'animal': 'koira',
     'width': 20}]
}
Parameters:
Raises:

FieldTypeException for invalid sub_element.
import_records(record)[source]

Imports records from a list of dictionaries.

Note:The dictionaries will lose information.
Parameters:record (list) – list of dictionaries with records to import.
add_value(value=None, language=None, **kwargs)[source]

Add new element to list of elements

Parameters:
  • value (str or int or None) – value for the new element.
  • language (str or None) – language for the new element.
  • **kwargs – key-value pairs for attributes of the new element.
Raises:

FieldTypeException for invalid language parameter depending on whether the sub_element is localizable.
export_dict()[source]

Export container as dictionary.

Returns:dictionary representing the container.
Return type:dict
iterate_values_for_language(language)[source]

Generator for iterating contained elements by language.

Parameters:language (str) – language which is used to filter yielded results.
Returns:a generator object for iterating elements
get_available_languages()[source]

Get list of languages for this container.

Returns:list of distinct languages.
Return type:list
updates(secondary_values)[source]

Updates contained values with secondary_values.

Looks for values that are not currently contained, and appends them as contained values. Also appends different language versions. If a language version has the same value, looks for differences in attributes. If new value has not the same attributes as the old one, adds these attributes to the new value. If old value has same attribute name, it will be discarded.

Note:Document Store uses MongoDB as a backend. MongoDB deals with JSON-like objects, which in turn are best represented in Python as dictionaries. The purpose of kuha_common.document_store.records is to be used as a global (in Kuha context) interchange format and so it will be best to support both dictionaries and ElementContainers for this operation. Therefore there is some flexibility in the type of parameter that this method accepts.
Note:There is a logical difference in which type of parameter is submitted to this method. When using other types than ElementContainers, the parameter’s content will be changed.
Parameters:secondary_values (instance of ElementContainer or dict or list) – Old values known to have the same container (must have the same name). If secondary_values is a list, it is assumed that the caller has explicitly checked that the parameter represents old values for this container. Otherwise the name of the container will be checked here and KeyError exceptions will be raised.
class kuha_common.document_store.field_types.FieldAttribute(name, parent=None)[source]

Common attributes for each field type.

Stores fields name, parent fields name and constructs a path for the field. This path can be used when building queries against Document Store. The name can be used to lookup values from objects returned from Document Store.

Used by FieldTypeFactory to store information of fields that can be used before the field has been fabricated.

Parameters:
  • name (str) – name of the field.
  • parent (str) – optional parameter parent. Used for sub-elements.
value_from_dict(_dict)[source]

Get value or values corresponding to path from parameter.

Note:Returned values cannot be separated by language afterwards.
Parameters:_dict (dict) – dictionary to lookup for path.
Returns:value or values stored in path of the _dict.
Return type:str or list or None
class kuha_common.document_store.field_types.FieldTypeFactory(name, sub_name=None, attrs=None, localizable=True, single_value=False)[source]

Factory for field types.

Stores information for each field, that can be used before the field actually has been initiated. This is useful for building queries against Document Store, because the caller needs to know the names and paths of the fields about to be queried.

The attributes stored here are also used to fabricate each field type. This means that each of the field types supported by kuha_common.document_store.records are to be initiated throught this factory.

Seealso:ElementContainer

Example:

>>> from kuha_common.document_store.field_types import FieldTypeFactory
>>> animals_factory = FieldTypeFactory('animals', 'animal', ['color', 'width', 'height'])
>>> animals_factory.attr_color.name
'color'
>>> animals_factory.attr_color.path
'animals.color'
>>> animals = animals_factory.fabricate()
>>> animals.add_value('cat', 'en', color='yellow', height=10, width=5)
>>> animals.export_dict()
{'animals': [{'color': 'yellow', 'animal': 'cat', 'height': 10, 'width': 5, 'language': 'en'}]}
Parameters:
  • name (str) – name of the field.
  • sub_name (str) – name of the sub field, if any.
  • attrs (list or str) – field attributes, if any. Multiple attributes in list.
  • localizable (bool) – is the field localizable.
  • single_value (bool) – The fabricated field can contain only a single value.
Raises:

ValueError if attribute has same name as the element or sub_element.
Raises:

FieldTypeException for parameter combinations that are not supported.
fabricate()[source]

Fabricate field type by factory attributes.

Returns the correct type of field type based on attributes given to the factory at initialization time.

Returns:Instance of one of the fields types.

document_store/records.py

Models for records supported by Document Store.

Due to its schemaless design, the document store relies heavily on these models. Use these models when building importers.

kuha_common.document_store.records.datetime_to_datestamp(_datetime)[source]

Convert datetime object to datestamp string supported by Document Store.

Parameters:datetime (datetime.datetime) – datetime to convert.
Returns:converted datestamp.
Return type:str
kuha_common.document_store.records.datestamp_to_datetime(datestamp)[source]

Convert datestamp string to datetime.datetime object.

Parameters:datestamp (str) – datestamp to convert.
Returns:converted datetime.
Return type:datetime.datetime
kuha_common.document_store.records.datetime_now()[source]

Get current datetime in supported format.

Returns:Supported datetime object representing current time.
Return type:datetime.datetime
class kuha_common.document_store.records.RecordBase(document_store_dictionary=None)[source]

Baseclass for each record.

Provides methods used to import, export, create and update records. Dynamically fabricates each class variable of type FieldTypeFactory into an instance variable overriding the class variable.

Note:Use this class throught subclasses only.
Parameters:document_store_dictionary (dict) – Optional parameter for creating a record at initialization time. Note that this dictionary will be iterated destructively.
classmethod get_collection()[source]

Get record collection.

Collection is used for queries against the Document Store.

Returns:collection of the record.
Return type:str
classmethod iterate_record_fields()[source]

Iterate class attributes used as record fields.

Iteration returns tuples: (attribute_name, attribute)

Returns:generator for iterating record fields.
export_metadata_dict()[source]

Export record metadata as dictionary.

Returns:record’s metadata
Return type:dict
export_dict(include_metadata=True, include_id=True)[source]

Return dictionary representation of record.

Parameters:
  • include_metadata (bool) – export includes metadata
  • include_id (bool) – export includes id
Returns:

record
Return type:

dict
set_updated(value=None)[source]

Set updated timestamp.

Sets updated metadata attribute.

Note:The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.
Parameters:value (datetime.datetime or str) – Optional timestamp to set.
set_created(value=None)[source]

Set created timestamp.

Sets created metadata attribute.

Note:The timestamp is always stored as datetime.datetime, but for convenience it is accepted as a string that is formatted accordingly.
Parameters:value (datetime.datetime or str) – Optional timestamp to set.
set_cmm_type(value=None)[source]

Set cmm type.

Parameters:value (str) – Optional type to set.
set_id(value)[source]

Set ID.

Parameters:value (str) – id to set.
get_updated()[source]

Get updated value.

Note:The timestamp is stored as a datetime.datetime in _metadata.attr_updated, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.
Returns:updated timestamp.
Return type:str
get_created()[source]

Get created value.

Note:The timestamp is stored as a datetime.datetime in _metadata.attr_created, but is returned as a string datestamp when using this method. If there is need to access the datetime.datetime object, use get_value() of the field.
Returns:created timestamp.
Return type:str
get_id()[source]

Get record ID.

Id comes from the backend storage system.

Returns:record ID in storage.
Return type:str or None
bypass_update(*fields)[source]

Add fields to be bypassed on update operation.

Parameters:*fields (str) – fieldnames to bypass.
bypass_create(*fields)[source]

Add fields to be bypassed on create operation.

Parameters:*fields (str) – fieldnames to bypass.
updates_record(old_record_dict)[source]

Update record by appending old values that are not present in current record. Use old record’s _id and _metadata.created if present.

Note:parameter is a dictionary since MongoDB returns records as JSON-like objects, which in turn are best represented as dictionaries in python.
Parameters:old_record_dict (dict) – Old record as a dictionary.
updates(secondary_record)[source]

Update record by appending values from secondary which are not present in this record.

Parameters:secondary_record (Record instance subclassed from RecordBase) – lookup values from this record.
class kuha_common.document_store.records.Study(study_dict=None)[source]

Study record.

Derived from RecordBase. Used to store and manipulate Study records. Study number is a special attribute and it cannot be updated.

All attributes of the record are declared as class variables initiated from kuha_common.document_store.field_types.FieldTypeFactory. Instance methods defined in this class are used to add/set values to record attributes. The signatures of the methods are dynamically constructed by the definition of the FieldTypeFactory instances. If, for example, there is a class variable definition:

animals = FieldTypeFactory('animals', 'animal', ['color', 'weight', 'height'])

The correct method signature should be:

def add_animals(self, value, language, color=None, weight=None, height=None):

For the dynamic nature of the record-model these signatures are left open, and python’s *args and **kwargs are used instead. Note that the field type used will raise exceptions if keyword argument key is not found in the initial definition of the field type.

Create a new study record:

>>> study = Study()
>>> study.add_study_number(1234)
>>> study.add_study_titles('Study about animals', 'en')
>>> study.add_principal_investigators('investigator', 'en', organization='Big organization ltd.')

Import existing study record from dictionary:

>>> study_dict = {'study_number': 1234,
... 'study_titles': [{'study_title': 'Study about animals', 'language': 'en'}],
... 'principal_investigators': [{'principal_investigator': 'investigator',
... 'language': 'en', 'organization': 'Big organization ltd.'}]}
>>> study = Study(study_dict)

Iterate attributes:

>>> for pi in study.principal_investigators:
...     pi.attr_organization.get_value()
...
'Big organization ltd.'
Seealso:RecordBase and kuha_common.document_store.field_types
Parameters:study_dict (dict) – Optional study record as dictionary used for constructing a record instance.
study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number is used to identify a study. It must be unique within records, not localizable and contain only a single value. It cannot be updated.

persistent_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Persistent identifiers. Multivalue-field with unique values.

identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Identifiers. Localizable field with agency-attribute. This needs to be localizable for the sake of agency-attribute. Note that two identical identifiers with same locale cannot exists at same time. The latter agency will overwrite the former on update.

study_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study titles. Localizable, multivalue-field without attributoes.

document_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document titles. Localizable, multivalue-field without attributoes.

parallel_titles = <kuha_common.document_store.field_types.FieldTypeFactory object>

Parallele study titles. Localizable, multivalue-field without attributes.

principal_investigators = <kuha_common.document_store.field_types.FieldTypeFactory object>

Pricipal investigators. Localizable, multivalue-field with organization-attribute.

publishers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publishers. Localizable, multivalue-field with abbreviation-attribute.

distributors = <kuha_common.document_store.field_types.FieldTypeFactory object>

Distributors. Localizable, multivalue-field with abbreviation and uri attributes.

document_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Document URIs. Localizable, multivalue-field with location and description attributes.

study_uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study URIs. Localizable, multivalue-field with location and description attributes.

publication_dates = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication dates. Localizable, multivalue-field without attributes. Note that these are treated as strings, not datetime-objects.

publication_years = <kuha_common.document_store.field_types.FieldTypeFactory object>

Publication years. Localizable, multivalue-field with distribution date attribute.

abstract = <kuha_common.document_store.field_types.FieldTypeFactory object>

Abstract. Localizable, multivalue-field.

classifications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Classifications. Localizable, multivalue-field with system name, uri and description attributes.

keywords = <kuha_common.document_store.field_types.FieldTypeFactory object>

Keywords. Localizable, multivalue-field with system name, uri and description attributes.

time_methods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Time methods. Localizable, multivalue-field with system name, uri and description attribute.

sampling_procedures = <kuha_common.document_store.field_types.FieldTypeFactory object>

Sampling procedures. Localizable, multivalue-field with description, system name and uri attibutes.

collection_modes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection modes. Localizable, multivalue-field with system name and uri attritubes.

analysis_units = <kuha_common.document_store.field_types.FieldTypeFactory object>

Analysis units. Localizable, multivalue-field with system name, uri and description attributes.

collection_periods = <kuha_common.document_store.field_types.FieldTypeFactory object>

Collection periods. Localizable, multivalue-field with event-attribute.

data_kinds = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data kinds. Localizable, multivalue-field.

study_area_countries = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study area countries. Localizable, multivalue-field with abbreviation attribute.

geographic_coverages = <kuha_common.document_store.field_types.FieldTypeFactory object>

Geographic coverages. Localizable, multivalue-field.

universes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Universes. Localizable, multivalue-field with included attribute.

data_access = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access. Localizable, multivalue-field.

data_access_descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Data access descriptions. Localizable, multivalue-field.

citation_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Citation requirements. Localizable, multivalue-field.

deposit_requirements = <kuha_common.document_store.field_types.FieldTypeFactory object>

Deposit requirements. Localizable, multivalue-field.

file_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

File names. Localizable, multivalue-field.

instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Instruments. Localizable, multivalue-field with instrument name attribute.

related_publications = <kuha_common.document_store.field_types.FieldTypeFactory object>

Related publications. Localizable multivalue-field

study_groups = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study groups. Localizable, multivalue-field with name and description attributes.

copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

data_collection_copyrights = <kuha_common.document_store.field_types.FieldTypeFactory object>

Copyrights. Localizable, multivalue-field.

collection = 'studies'

Database collection (table) for persistent storage.

cmm_type = 'study'

CMM type for Study.

add_study_number(value)[source]

Add study number.

Note:despite the name, the value does not need to be a number.
Parameters:value (str or int) – study number.
add_persistent_identifiers(value)[source]

Add persistent identifiers

Parameters:value (str or int) – persistent identifier
add_identifiers(value, *args, **kwargs)[source]

Add identifiers.

Parameters:
add_study_titles(value, *args, **kwargs)[source]

Add study titles.

Parameters:
add_document_titles(value, *args, **kwargs)[source]

Add document titles.

Parameters:
add_parallel_titles(value, *args, **kwargs)[source]

Add parallel titles.

Parameters:
add_principal_investigators(value, *args, **kwargs)[source]

Add principal investigators.

Parameters:
add_publishers(value, *args, **kwargs)[source]

Add publishers.

Parameters:
add_distributors(value, *args, **kwargs)[source]

Add distributors.

Parameters:
add_document_uris(value, *args, **kwargs)[source]

Add document URIs.

Parameters:
add_study_uris(value, *args, **kwargs)[source]

Add study URIs.

Parameters:
add_publication_dates(value, *args, **kwargs)[source]

Add publication dates.

Parameters:
add_publication_years(value, *args, **kwargs)[source]

Add publication dates.

Parameters:
add_abstract(value, *args, **kwargs)[source]

Add abstract.

Parameters:
add_classifications(value, *args, **kwargs)[source]

Add classifications.

Parameters:
add_keywords(value, *args, **kwargs)[source]

Add keywords.

Parameters:
add_time_methods(value, *args, **kwargs)[source]

Add time methods.

Parameters:
add_sampling_procedures(value, *args, **kwargs)[source]

Add sampling procedures

Parameters:
add_collection_modes(value, *args, **kwargs)[source]

Add collection modes

Parameters:
add_analysis_units(value, *args, **kwargs)[source]

Add analysis units.

Parameters:
add_collection_periods(value, *args, **kwargs)[source]

Add collection periods.

Parameters:
add_data_kinds(value, *args)[source]

Add data kinds.

Parameters:
add_study_area_countries(value, *args, **kwargs)[source]

Add study area countries.

Parameters:
add_geographic_coverages(value, *args)[source]

Add geographic coverages

Parameters:
add_universes(value, *args, **kwargs)[source]

Add universes.

Parameters:
add_data_access(value, *args, **kwargs)[source]

Add data access.

Parameters:
add_data_access_descriptions(value, *args, **kwargs)[source]

Add data access descriptions.

Parameters:
add_citation_requirements(value, *args)[source]

Add citation requirements.

Parameters:
add_deposit_requirements(value, *args)[source]

Add deposit requirements.

Parameters:
add_file_names(value, *args, **kwargs)[source]

Add file name.

Parameters:
add_instruments(value, *args, **kwargs)[source]

Add instrument.

Parameters:

Add related publications.

Parameters:
add_study_groups(value, *args, **kwargs)[source]

Add study group.

Parameters:
add_copyrights(value, *args, **kwargs)[source]

Add copyright.

Parameters:
add_data_collection_copyrights(value, *args)[source]

Add data collection copyrights.

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:secondary (Study) – Lookup values to update from secondary record.
Returns:True if record updated, False if not.
Return type:bool
class kuha_common.document_store.records.Variable(variable_dict=None)[source]

Variable record.

Derived from RecordBase. Used to store and manipulate variable records. Study number and variable name are special attributes and cannot be updated.

Seealso:Study documentation for more information.
Parameters:variable_dict (dict) – Optional variable record as dictionary used for constructing a record instance.
study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and variable name are used to identify a variable within variable records. Their combination must be unique withing variable records, they cannot be localizable, and they can only contain a single value. They also cannot be updated.

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name within a study. See also study_number

question_identifiers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifiers, if variable refers to a question. Not localizable, multiple unique values.

variable_labels = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable labels. Localizable, multivalue-field.

codelist_codes = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist codes. Localizable, multivalue-field with label and missing attributes.

collection = 'variables'

Database collection for persistent storage.

cmm_type = 'variable'

CMM type for variable.

add_study_number(value)[source]

Add study number.

Parameters:value (str or int.) – study number.
add_variable_name(value)[source]

Add variable name.

Parameters:value (str) – variable name.
add_question_identifiers(value)[source]

Add question identifier

Parameters:value (str or int.) – question identifier.
add_variable_labels(value, *args, **kwargs)[source]

Add variable label

Parameters:
add_codelist_codes(value, *args, **kwargs)[source]

Add codelist code

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:secondary (Variable) – Lookup values to update from secondary record.
Returns:True if record updated, False if not.
Return type:bool
class kuha_common.document_store.records.Question(question_dict=None)[source]

Question record.

Derived from RecordBase. Used to store and manipulate question records. study_number and question_idenntifier are special attributes and cannot be updated.

Seealso:Study documentation for more information.
Parameters:question_dict (dict) – Optional question record as dictionary used for constructing a record instance.
study_number = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study number and question identifier are used to identify a question. Their combination must be unique withing records, they must not be localizable and they can only contain a single value. They also cannot be updated.

question_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question identifier within a study. See also study_number

variable_name = <kuha_common.document_store.field_types.FieldTypeFactory object>

Variable name that specifies the variable for the question. Not localizable, single value.

question_texts = <kuha_common.document_store.field_types.FieldTypeFactory object>

Question texts. Localizable, multivalue-field.

research_instruments = <kuha_common.document_store.field_types.FieldTypeFactory object>

Research instruments. Localizable, multivalue-field.

codelist_references = <kuha_common.document_store.field_types.FieldTypeFactory object>

Codelist references. Localizable, multivalue-field.

collection = 'questions'

Database collection for persistent storage.

cmm_type = 'question'

CMM type for question

add_study_number(value)[source]

Add study number.

Parameters:value (str or int.) – study number.
add_question_identifier(value)[source]

Add question identifier

Parameters:value (str or int.) – question identifier.
add_variable_name(value)[source]

Add variable name.

Parameters:value (str) – variable name.
add_question_texts(value, *args, **kwargs)[source]

Add question text

Parameters:
add_research_instruments(value, *args, **kwargs)[source]

Add research instrument

Parameters:
add_codelist_references(value, *args, **kwargs)[source]

Add codelist reference

Parameters:
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:secondary (Question) – Lookup values to update from secondary record.
Returns:True if record updated, False if not.
Return type:bool
class kuha_common.document_store.records.StudyGroup(study_group_dict=None)[source]

Study group record.

Derived from RecordBase. Used to store and manipulate study group records. study_group_identifier is a arpecial attribute and cannot be updated.

Seealso:Study documentation for more information.
Parameters:study_group_dict (dict) – Optional study group record as dictionary used for constructing a record instance.
study_group_identifier = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group identifier. Used to identify study group. Must be unique within study groups, cannot be localizable and can contain only a single value. This value cannot be updated.

study_group_names = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group names. Localizable, multivalue-field.

descriptions = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group descriptions. Localizable, multivalue-field.

uris = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study group URIs. Localizable, multivalue-field.

study_numbers = <kuha_common.document_store.field_types.FieldTypeFactory object>

Study numbers. Multivalue-field with unique values.

collection = 'study_groups'

Database collection for persistent storage.

cmm_type = 'study_group'

CMM type for study groups.

add_study_group_identifier(value)[source]

Add study group identifier.

Parameters:value (str or int) – Study group identifier.
add_study_group_names(value, *args, **kwargs)[source]

Add study group names

Parameters:
add_descriptions(value, *args)[source]

Add study group descriptions

Parameters:
add_uris(value, *args)[source]

Add study group URIs

Parameters:
add_study_numbers(value)[source]

Add study number.

Parameters:value (str or int) – study number.
updates(secondary)[source]

Check that records have common unique keys. Update record by appending values from secondary which are not present in this record.

Parameters:secondary (StudyGroup) – Lookup values to update from secondary record.
Returns:True if record updated, False if not.
Return type:bool
kuha_common.document_store.records.record_factory(ds_record_dict)[source]

Dynamically construct record instance based on given document store dictionary.

Looks up the correct record by the cmm type found from ds_record_dict metadata.

Parameters:ds_record_dict (dict) – record received from Document Store.
Returns:Record instance.
Return type:Study or Variable or Question or StudyGroup
kuha_common.document_store.records.record_by_collection(collection)[source]

Finds a record class by the given collection.

Parameters:collection (str) – collection of the record.
Returns:record class
Return type:Study or Variable or Question or StudyGroup
Raises:KeyError if collection is not found in any record.

document_store/mappings

document_store/mappings/exceptions.py

Exceptions for mapping package.

exception kuha_common.document_store.mappings.exceptions.InvalidMapperParameters[source]

Raise for invalid configuration of a mapper - Coding error.

For example trying to add attributes to a mapper which does not support attributes. These errors are coding errors and should be treated as such.

exception kuha_common.document_store.mappings.exceptions.MappingError[source]

Raise for errors while mapping input - User error.

Subclass to create more precise error classes.

exception kuha_common.document_store.mappings.exceptions.ParseError[source]

Unable to parse source XML.

Note:Mask ElementTree ParseError so caller may use MappingError to catch all user-errors when mapping.
exception kuha_common.document_store.mappings.exceptions.UnknownXMLRoot(expected=None, unknown=None)[source]

Unknown root element.

exception kuha_common.document_store.mappings.exceptions.MissingRequiredAttribute(*xpaths, msg=None)[source]

Source does not contain a required attribute.

exception kuha_common.document_store.mappings.exceptions.InvalidContent[source]

Attribute found but contains invalid data.

document_store/mappings/xmlbase.py

Components to use for XML parsing & mapping to Document Store records.

Contains a base class to use for parsing XML to Document Store records. Provides common functions useful in parsing XML data.

class kuha_common.document_store.mappings.xmlbase.MappedParams(value)[source]

Contains parameters ready to pass to record’s add-methods.

XMLMapper retrieves parameters from XML record and stores them in an instance of this class. The record instances add-methods get called with stored parameters by using tuple and dict unpacking.

Example:

mapped_params = MappedParams('study_identifier')
mapped_params.set_language('en')
mapped_params.keyword_arguments.update({'agency': 'archive'})
study = Study()
study.add_identifiers(*mapped_params.arguments, **mapped_params.keyword_arguments)
Parameters:value (str or None) – value used as the first argument
has_language()[source]

True if MappedParams has language argument

Returns:True if has language, False if not.
Return type:bool
set_language(language)[source]

Set language argument. Will overwrite if previously set.

Parameters:language (str) – Language to set.
get_language()[source]

Get language argument.

Returns:Language
Return type:str
get_value()[source]

Get value argument.

Returns:value.
Return type:str or None
copy()[source]

Make a copy of the object with contents and return the copy.

Returns:copy of this MappedParams
Return type:MappedParams
has_arguments()[source]

Return True if MappedParams has arguments or keyword_arguments.

Returns:True if object has arguments or keyword_arguments.
Return type:bool
class kuha_common.document_store.mappings.xmlbase.XMLMapper(xpath, from_attribute=None, required=False, localizable=True)[source]

XMLMapper populates MappedParams instances from XML.

Parameters:
  • xpath (str) – XPath where to look for element containing value.
  • from_attribute (str or None) – Look value from attribute of the element.
  • required (bool) – raises MissingRequiredAttribute if value is not found.
  • localizable (bool) – True if the value is localizable, False if not.
set_value_conversion(conv_func)[source]

Set conversion callable.

Note:conv_func must accept a string or None as a parameter and return the converted value.
Parameters:conv_func (callable.) – Callable used for conversion.
Returns:self
set_value_getter(getter_func)[source]

Set value getter callable.

Note:getter_func must accept an XML element xml.etree.ElementTree.Element as a parameter and return the value.
Parameters:getter_func (callable.) – Callable used for getting a value from XML element.
Returns:self
expect_single_value()[source]

This mapper will be expected to return a single value.

Returns:self
expect_multiple_values()[source]

This mapper will be expected to return multiple values.

Returns:self
disable_attributes()[source]

This mapper will not contain any attributes.

Returns:self
iterate_attributes(*relations)[source]

Iterate attributes to map.

Parameters:*relations (str) – optional parameters to iterate only attributes with a certain relation.
Returns:A generator yielding tuples of each attribute in the format: (attribute_name, attribute_mapper, attribute_provides_main_lang)
Return type:generator
as_params(element, default_language, xml_namespaces)[source]

Use mapping to construct a MappedParams from XML element.

Use mapper’s _value_getter and _value_conversion to get value from XML element. Construct a MappedParams from the value. If mapping localizable is True add language from XML elements xml:lang attribute.

Parameters:
  • element (xml.etree.ElementTree.Element) – XML element.
  • default_language (str) – default language if element has none.
  • xml_namespaces (dict) – XML namespaces for the element.
Returns:

mapped parameters ready to pass to records add-method.
Return type:

MappedParams
add_attribute(att_name, mapper, relative=True, provides_main_lang=False)[source]

Add attribute to mapper.

Counts the correct xpath if attribute’s mapper’s xpath is a parent element (starting with ‘/..’). Includes all needed information of the attribute to a list of tuples contained in attributes.

Parameters:
  • att_name (str) – attribute name
  • mapper (XMLMapper) – mapper instance for mapping value for the attribute.
  • relative (bool) – Is the attribute map’s xpath relative to this element. Defaults to True.
  • provides_main_lang (bool) – Should the language of the attribute be used as a language when mapping this value. Defaults to False.
Raises:

InvalidMapperParameters for conflicting parameters such as: 1. Calling this method on a mapper which has disabled use of attributes. 2. Using provides_main_lang for a non-localizable mapper. 3. Setting relative to False on a mapper whose xpath refers to parent element.
Returns:

self
value_params(source_xml_element, default_language, xml_namespaces, position=None)[source]

Generate sinle MappedParams object from source XML.

Parameters:
  • source_xml_element (xml.etree.ElementTree.Element) – XML element.
  • default_language (str) – Default language.
  • xml_namespaces (dict) – XML namespaces.
  • position (int or None) – Optional position for parent xpaths.
Returns:

generator yielding MappedParams.
Return type:

generator
Raises:

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.
values_params(source_xml_element, default_language, xml_namespaces)[source]

Generate multiple MappedParams objects from source XML.

The generated MappedParams will contain attributes as keyword_arguments.

Parameters:
Returns:

generator yielding MappedParams.
Return type:

generator
Raises:

MissingRequiredAttribute if mapper’s required is True, but xpath provides no element or the element provides no value.
class kuha_common.document_store.mappings.xmlbase.XMLParserBase(root_element)[source]

Base class where parsers get derived from.

Declares the public API to be used in callers.

Input:

  • from_file()
  • from_string()

Output:

  • studies
  • variables
  • questions
  • study_groups
  • all
  • select(collection=None)

Provides common functionality to be used within subclasses which map XML-data to Document Store records. Subclasses must implement necessary generators that generate document store records.

Use in subclass:

class XMLRecordParser(XMLParserBase):
    @property
    def studies(self):
        maps = [(Study.add_study_number, self._map_single(xpath_to_study_number, required=True)),
                (Study.add_study_titles, self._map_multi(xpath_to_study_title))]
        for study_element in self.root_element.findall(xpath_to_study_element, self.NS):
            study = Study()
            self._map_to_record(study, study_element, maps)
            yield study
Parameters:root_element (xml.etree.ElementTree.Element) – XML root.
NS = {'xsi': 'http://www.w3.org/2001/XMLSchema-instance', 'xml': 'http://www.w3.org/XML/1998/namespace'}

XML namespaces

default_language = 'en'

Default language.

classmethod from_string(xml_body)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters:xml_body (str) – XML Document as a string. This may come directly from HTTP request body.
Returns:parser for iteratively parsing XML and generating Document Store records.
Return type:XMLParserBase
classmethod from_file(filepath)[source]

Get parser that iteratively parses XML and generates populated Document Store record instances.

Parameters:filepath (str) – Path for the XML file.
Returns:parser for iteratively parsing XML and generating Document Store records.
Return type:XMLParserBase
classmethod child_text(xpath)[source]

Returns a function which will lookup a child element from given xpath. The returned function takes a single element as a parameter which should be an xml.etree.ElementTree.Element or similar. When executed the function returns the child element’s text contents or None if child element cannot be found.

Parameters:xpath – xpath to child. relative to parent.
Returns:function which accepts the parent element as a parameter.
Return type:function
root_element

Get root element.

Returns:Root element
Return type:xml.etree.ElementTree.Element
root_language

Get language of the root element. If root does not have a language, returns self.default_language.

Returns:root element language.
Return type:str
study_number

Get study number as formatted in source XML.

Seealso:self.study_number_identifier
Returns:Study number from source XML.
Return type:str
study_number_identifier

Get study number converted as a valid Document Store identifier.

Returns:Study number as valid Document Store identifier.
Return type:str
studies

Studies generator. Must be implemented in subclass.

Returns:Generator which yields Document Store studies.
variables

Variables generator. Must be implemented in subclass.

Returns:Generator which yields Document Store variables.
questions

Questions generator. Must be implemented in subclass.

Returns:Generator which yields Document Store questions.
study_groups

Study groups generator. Must be implemented in subclass.

Returns:Generator which yields Document Store study groups.
all

Iterate all records found from source XML.

Returns:Generator which yields Document Store records.
Return type:Generator
select(collection=None)[source]

Returns a selective parser. Call with a Document Store collection as parameter to select records only for certain collection.

Note

The returned attributes are defined in subclasses, so they may or may not be generators.

Parameters:collection (str or None) – Document Store collection to select only records belonging to this collection.
Returns:Generator which yields Document Store records.
Return type:Generator
kuha_common.document_store.mappings.xmlbase.as_valid_identifier(candidate)[source]

Convert candidate to a string that conforms the rules of validation.

Indentifier must match regex: [a-zA-Z0-9]+[a-zA-Z0-9?_()-.]*’”]

Note

Regex is defined in Document Store. Should it be moved to kuha_common?

Returns:identifier which conforms the rules of validation.
Return type:str
kuha_common.document_store.mappings.xmlbase.str_equals(correct, default=None)[source]

Conversion function wrapper to compare strings for equality.

Wrapper function that formats comparison value and default value for returned comparison function.

Check if string found from element value or element attribute equals to correct.

Parameters:
  • correct (str) – comparison string.
  • default (str) – If the value parameter of the comparison function is None, return this value.
Returns:

function which accepts a single parameter for comparison. Returns True or False, or default if the parameter is None.
Return type:

function
kuha_common.document_store.mappings.xmlbase.fixed_value(fixed)[source]

Fixed value.

Parameters:fixed – Use this value
Returns:function which accepts a single argument value. The function always returns fixed.
Return type:function
kuha_common.document_store.mappings.xmlbase.element_remove_whitespaces(element)[source]

Conversion function to remove extra whitespace from end of element text.

Iterates element’s inner text using xml.etree.ElementTree.Element.itertext() which iterates over this element and all subelements. Removes extra whitespaces so paragraphs of text will only have one separating whitespace character.

Parameters:element (xml.etree.ElementTree.Element) – Element from which to get text.
Returns:Element’s inner text without extra whitespace.
Return type:str
kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text(element)[source]

Conversion function to remove inner elements and their contents.

Parameters:element (xml.etree.ElementTree.Element) – Element for lookup.
Returns:Element’s inner text without text from descendants and without extra whitespace.
Return type:str

document_store/mappings/ddi.py

Mapping profiles for DDI.

Note

has strict dependency to kuha_common.document_store.records

class kuha_common.document_store.mappings.ddi.DDI122RecordParser(root_element)[source]

Parse Document Store records from DDI 1.2.2. XML.

studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns:Generator to Populate Document Store Study record.
variables

Parse XML to create and populate multiple kuha_common.document_store.records.Variable instances.

Returns:Generator to populate multiple Document Store Variable records.
questions

Parse XML to create and populate multiple kuha_common.document_store.records.Question instances.

Returns:Generator to populate multiple Document Store Question records.
study_groups

Parse XML to create and populate multiple kuha_common.document_store.records.StudyGroup instances.

Returns:Generator to populate multiple Document Store StudyGroup records.
class kuha_common.document_store.mappings.ddi.DDI25RecordParser(root_element)[source]

Parse Document Store records from DDI 2.5 XML.

NS = {'ddi': 'ddi:codebook:2_5', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance', 'xml': 'http://www.w3.org/XML/1998/namespace'}

XML namespaces

class kuha_common.document_store.mappings.ddi.DDI31RecordParser(root_element)[source]

Parse Document Store records from DDI 3.1. XML

Check the root element. Expects either ddi:DDIInstance or s:StudyUnit. Currently supports only single s:StudyUnit element within the root.

Parameters:root_element (xml.etree.ElementTree.Element) – XML root element.
Raises:UnknownXMLRoot for unexpected root element.
Raises:MappingError if root contains more or less that exactly one s:StudyUnit child.
NS = {'l': 'ddi:logicalproduct:3_1', 'xml': 'http://www.w3.org/XML/1998/namespace', 'pd': 'ddi:physicaldataproduct:3_1', 'c': 'ddi:conceptualcomponent:3_1', 'xhtml': 'http://www.w3.org/1999/xhtml', 'pi': 'ddi:physicalinstance:3_1', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance', 'dc': 'ddi:datacollection:3_1', 'g': 'ddi:group:3_1', 's': 'ddi:studyunit:3_1', 'ddi': 'ddi:instance:3_1', 'a': 'ddi:archive:3_1', 'r': 'ddi:reusable:3_1'}

XML namespaces

studies

Parse XML to create and populate kuha_common.document_store.records.Study.

Returns:Generator to Populate Document Store Study record.
variables

Parse XML to create and populate kuha_common.document_store.records.Variable.

Returns:Generator to Populate Document Store Variable records.
questions

Parse XML to create and populate kuha_common.document_store.records.Question.

Returns:Generator to Populate Document Store Question records.
study_groups

Parse XML to create and populate kuha_common.document_store.records.StudyGroup.

Returns:Generator to Populate Document Store StudyGroup records.

testing

Package for common testing functions and classes.

kuha_common.testing.time_me(func)[source]

Decorate function to print its execution time to stdout.

Note:test runner may capture the output.
Parameters:func – Function to decorate. Count execution time of function.
kuha_common.testing.mock_coro(dummy_rval=None, func=None)[source]

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters:
  • dummy_rval – return value of dummy function.
  • func – function to call instead of original mocked out function.
Returns:

coroutine function.
kuha_common.testing.MockCoro(dummy_rval=None, func=None)

Mock out a coroutine function.

Accepts either keyword argument but not both. Submitting both will raise TypeError.

Mock out coroutine and set return value:

>>> coro = mock_coro(dummy_rval='return_value')
>>> rval = await coro()
>>> assert rval == 'return_value'

Mock out coroutine with custom function:

>>> call_args = []
>>> async def custom_func(*args):
>>>     call_args.append(args)
>>> coro = mock_coro(func=custom_func)
>>> await coro()
>>> assert call_args == [('expected', 'args')]

Use as a side_effect when patching:

>>> @mock.patch.object(pkg.Class, 'async_method', side_effect=mock_coro())
>>> def test_something(mock_method):
>>>     inst = pkg.Class()
>>>     eventloop.run_until_complete(inst.async_method())
>>>     mock_method.assert_called_once_with()
Parameters:
  • dummy_rval – return value of dummy function.
  • func – function to call instead of original mocked out function.
Returns:

coroutine function.

testing/testcases.py

Test cases for Kuha

class kuha_common.testing.testcases.KuhaUnitTestCase(methodName='runTest')[source]

Base class for unittests.

  • Assertion methods to check record equality.
  • Helper methods to provide access to dummydata.
dummydata_dir = '/home/docs/checkouts/readthedocs.org/user_builds/kuha2/envs/0.x.x/src/kuha-common/kuha_common/testing/dummydata'

Override in sublass to lookup dummydata from different directory.

classmethod get_dummydata_path(path)[source]

Get absolute path to dummydatafile

Parameters:path – Path. Gets turned into an absolute if it isn’t
Returns:Absolute path.
Return type:str
classmethod get_dummydata(path)[source]

Get dummydata by reading file from path

Parameters:path – path to file.
Returns:Contents of the file.
classmethod remove_dummyfile_if_exists(path)[source]

Remove dummyfile from path if it exists.

Parameters:path – path to dummyfile.
Returns:None
classmethod set_val(value)[source]

Assign value as dummyvalue.

Parameters:value – Value to assign
Returns:value
classmethod gen_val(lenght=None, unique=False, chars=None)[source]

Generate & assign dummyvalue.

Parameters:
  • lenght (int or None) – lenght of the value
  • unique (bool) – should the value be unique
  • chars (str or None.) – use specific characters.
Returns:

generated value
Return type:

str
classmethod gen_id()[source]

Generate Id.

Returns:Generated id.
Return type:str
classmethod generate_dummy_study()[source]

Generate and return a Study with dummydata.

Returns:study with dummydata
Return type:kuha_common.document_store.records.Study
classmethod generate_dummy_variable()[source]

Generate and return a Variable with dummydata.

Returns:variable with dummydata
Return type:kuha_common.document_store.records.Variable
classmethod generate_dummy_question()[source]

Generate and return a Question with dummydata.

Returns:question with dummydata
Return type:kuha_common.document_store.records.Question
classmethod generate_dummy_studygroup()[source]

Generate and return a StudyGroup with dummydata.

Returns:studygroup with dummydata.
Return type:kuha_common.document_store.records.StudyGroup
setUp()[source]

Format testcase values and initialize event loop.

Call asynchronous code synchronously:

self._loop.run_until_complete(coro())
tearDown()[source]

Stop patchers.

await_and_store_result(coro)[source]

Await coroutine and store returning result.

Example:

self._loop.run_until_complete(self.await_future_and_store_result(coro()))
Parameters:coro – Coroutine or Future to await
init_patcher(patcher)[source]

Initialize patcher, store for later use, return it.

Parameters:patcher (unittest.mock._patch) – Patch to start.
Returns:MagicMock acting as patched object.
Return type:unittest.mock.MagicMock
assert_records_are_equal(first, second, msg=None)[source]

Assert two Document Store records are equal.

Parameters:
  • first – First record to compare.
  • second – Second record to compare.
  • msg – Optional message to output on assertion.
assert_records_are_not_equal(first, second, msg=None)[source]

Assert two Document Store records are not equal.

Parameters:
  • first – First record to compare.
  • second – Second record to compare.
  • msg – Optional message to output on assertion.
assert_mock_meth_has_calls(mock_meth, call, *calls)[source]

Assert moch_meth was called with arguments.

This calls Mock.assert_has_calls and tests for call count. The actual benefit of using this method over the built-in assert_has_calls is that this method tries to pinpoint the actual call that was missing when assert_has_calls raised AssertionError. This is useful when mock_meth has had multiple calls. The built-in assert_has_calls will notify of all calls that the mock_meth has had, while this method will notify of the actual call that was missing.

Parameters:
  • mock_meth – Mocked method that is target of testing.
  • call – Call that should be found. Instance of unittest.mock._Call Repeat this argument to test for multiple calls.
Raises:

AssertionError if calls not found.
class kuha_common.testing.testcases.KuhaEndToEndTestCase(methodName='runTest')[source]

Base class for end-to-end tests.

  • HTTPClient for interacting with Document Store.
  • Assertion methods to check returning payload and status codes.
static load_cli_args(sysexit_to_skiptest=False)[source]

Load command line arguments. Setup Document Store URL.

Parameters:sysexit_to_skiptest (bool) – Mask SystemExit as unittest.SkipTest. Useful when missing command line arguments should not terminate the test run, but skip tests requiring the arguments.
Returns:arguments not known to kuha_common.cli_setup.settings (= arguments external to Kuha)
Return type:list
static get_record_url(rec_or_coll, _id=None)[source]

Get URL to Document Store records or single record.

Parameters:
  • rec_or_coll – record, record class or collection
  • _id (str or None) – Optional record ID.
Returns:

URL to Document Store collection or single record.
Return type:

str
static get_query_url(rec_or_coll, query_type=None)[source]

Get URL to Document Store query endpoint for collection

Parameters:
  • rec_or_coll (str, record, or record class) – Collection to query.
  • query_type – Optional query type
Returns:

URL to query endpoint.
Return type:

str
classmethod GET_to_document_store(rec_or_coll, _id=None)[source]

GET to Document Store returns record(s).

Parameters:
  • rec_or_coll – record or collection to get.
  • _id – Optional ObjectId. Will take precedence over rec_or_coll id.
Returns:

response body
classmethod POST_to_document_store(record)[source]

POST to Document Store creates record.

Parameters:record – Record to post.
Returns:response body
classmethod DELETE_to_document_store(rec_or_coll=None, _id=None)[source]

DELETE to Document Store deletes record(s).

Call without arguments to delete all records from all collections.

Parameters:
  • rec_or_coll (str or None) – Collection to delete from.
  • _id (str or None) – ID of the record to delete.
Returns:

None
classmethod query_document_store(rec_or_coll, query, query_type=None)[source]

Execute query against Document Store query API.

Parameters:
  • rec_or_coll (str or record class or record instance) – Collection to query.
  • query – Query.
  • query_type – Type of Query.
Returns:

query results
Return type:

None if query returned no results, dict for results.
classmethod get_collection_record_count(rec_or_coll)[source]

Return number or records for collection in Document Store.

Parameters:rec_or_coll – Document Store record, Document Store record class or collection.
Returns:record count in Document Store.
Return type:int
assert_document_store_is_empty()[source]

Assert Document Store contains no records.

Raises:AssertionError if Document Store has records.