Kuha Common Changelog

0.15.1 (2021-09-06)

• Add missing mapping from DDI 2.5 and DDI 1.2.2 to Study.study_uris.

0.15.0 (2021-09-03)

This release ensures future compatibility with CESSDA Data Catalogue.

• Add following attributes to kuha_common.document_store.records.Study:
  • document_titles to contain metadata document titles.
  • study_uris to contain URIs linking to study.
• Add DDI mappings to new attributes in kuha_common.document_store.mappings.ddi:
  • DDI 2.5 and DDI 1.2.2 use codeBook/docDscr/citation/titlStmt/titl to populate Study.document_titles
  • DDI 3.1 uses ddi:DDIInstance/r:Citation/r:Title to populate Study.document_titles
  • DDI 2.5 and DDI 1.2.2 use codeBook/stdyDscr/citation/holdings/@URI to populate Study.study_uris.
  • DDI 3.1 uses s:StudyUnit/a:Archive/a:ArchiveSpecific/a:Collection/a:URI to populate Study.study_uris. This XPATH was used to populate Study.document_uris, but it seems more appropriate to use it for Study.study_uris. There is currently no mapping to populate Study.document_uris from DDI 3.1.
• Jenkinsfile uses python-latest to build test environments. In FSD Jenkins python-latest always points to the latest installed python interpreter.
• Add py39 to test environments.
• Remove py35, py36 and py37 from tox test environments.
• Remove tasks running tests with py35, py36 and py37 interpreters from Jenkinsfile.

0.14.0 (2020-06-09)

Python 3.8 compatibility

• kuha_common.testing.MockCoro was not working properly with Python 3.8.2 AsyncMock. Switch to a closure function with pruned functionality. Introduce alias MockCoro to keep changes more backwards compatible. This is, however, backwards incompatible change.
  • MockCoro is now a closure function. Prior 0.14.0 it was a class.
  • MockCoro now returns a coroutine function. Prior 0.14.0 it returned a MockCoro instance.
  • MockCoro func does not get MockCoro instance as the first parameter. Instead the signature can now be exactly same as the mocked out function’s. Prior 0.14.0 MockCoro.func was called with MockCoro instance as the first argument.
  • Since mock_coro (alias MockCoro) is no longer a class, it won’t hold any attributes. Code relying in mock_coro.func or mock_coro.dummy_rval needs to change.
• Use list(element_copy) instead of deprecated element_copy.getchildren() in kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text()
• Add py38 to tox.ini environments. The code should now be Python 3.8. compatible.

Other changes to public APIs

These are backwards compatible changes.

• Add kuha_common.testing.testcases.KuhaUnitTestCase.assert_mock_meth_has_calls() to try to pinpoint the exact call that was missing from mocked method.
• Add kuha_common.cli_setup.add_server_process_count_configuration() which adds tornado server process count to configuration options.
• Change every add_* function signature in kuha_common.cli_setup to make parser an optional argument. If omitted the functions will use singleton stored in kuha_common.cli_setup.settings.parser.

0.13.0 (2020-05-06)

• DDI 2.5 and DDI 1.2.2 mappings use relpubl/citation/holdings/@URI and relpubl/citation/distStmt/distDate elements that map into Study.related_publications attributes distribution_date and uri.

0.12.0 (2020-04-28)

• Add related_publications to Study
• Add DDI 2.5 and DDI 1.2.2 mappings to Study.related_publications

0.11.2 (2020-01-24)

• Fix logic in kuha_common.document_store.field_types.ElementContainer._updates() where secondary values were compared to other secondary values. This lead to faulty contained values, when a secondary value matched with another secondary value.

0.11.1 (2020-01-09)

• Reset settings in kuha_common.testing.testcases.load_cli_args() if tests get skipped, since skipping won’t call tearDownClass method

0.11.0 (2020-01-08)

• Initialize argumentparser using configargparse.ArgumentParser directly instead of calling configargparse.get_arg_parser() in kuha_common.cli_setup. This way the settings get stored only in kuha_common.cli_setup.settings singleton.
• Reset settings in kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass().
• Fix faulty call to parent’s setUpClass in kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass().

0.10.0 (2019-10-22)

Warning

Breaks backwards compatibility in DDI 1.2.2. and DDI 2.5. mapping of attributes for Study.study_groups.study_group and StudyGroup.study_group_identifier

Change mappings for DDI 1.2.2. and DDI 2.5.

• Change mappings for Study.study_groups.study_group of DDI 2.5. and DDI 1.2.2. The value for the contained element is now taken from serStmt/@ID instead of serName/@ID. Breaks backwards compatibility
• Change mapping for StudyGroup.study_group_identifier of DDI 2.5. and DDI 1.2.2. The value is now taken from serStmt/@ID instead of serName/@ID. Breaks backwards compatibility

Add new attributes to kuha_common.document_store.records

• Study.study_groups.attr_description
• Study.study_groups.attr_uri
• Study.data_kinds
• Study.data_collection_copyrights
• Study.citation_requirements
• Study.deposit_requirements
• Study.geographic_coverages
• StudyGroup.descriptions
• StudyGroup.uris

Add mappings for new attributes

DDI 1.2.2. and DDI 2.5.:

• /codeBook/stdyDscr/citation/serStmt/serInfo maps to Study.study_groups.attr_description
• /codeBook/stdyDscr/citation/serStmt/serInfo maps to StudyGroups.descriptions
• /codeBook/stdyDscr/citation/serStmt/@URI maps to Study.study_groups.attr_uri
• /codeBook/stdyDscr/citation/serStmt/@URI maps to StudyGroup.uris
• /codeBook/stdyDscr/citation/prodStmt/copyright maps to Study.data_collection_copyrights
• /codeBook/stdyDscr/dataAccs/useStmt/citReq maps to Study.citation_requirements
• /codeBook/stdyDscr/dataAccs/useStmt/deposReq maps to Study.deposit_requirements
• /codeBook/stdyDscr/stdyInfo/sumDscr/geogCover maps to Study.geographic_coverages
• /codeBook/stdyDscr/stdyInfo/sumDscr/dataKind maps to Study.data_kinds

DDI 3.1.

• g:Group/g:Abstract/r:Content maps to StudyGroups.descriptions
• g:Group/@externalReferenceDefaultURI maps to StudyGroups.uris

0.9.1 (2019-04-03)

• DDI mappings: kuha_common.document_store.mappings.xmlbase.MappedParams.has_arguments() treat MappedParams objects which don’t have a value and all keyword arguments’ values are None as MappedParams objects that don’t have any arguments. This way all empty XML elements will be discarded in the mapping phase before they get turned into Document Store reocrds. (Fixes #26)
• Treat attributes-dictionaries with only None values as invalid if the Element’s value is also None in kuha_common.document_store.field_types.Element.add_value().
• Servers exception logging kuha_common.server.log_exception() will now log all traceback lines, instead of the last four.

0.9.0 (2019-03-14)

• Maintenance release with no new functionality besides bug fixes.
• Support tornado > 5
  • Remove callbacks used with tornado client.
  • requirements.txt declares tornado==6.0.1, which is currently the latest.
• kuha_common.query.QueryController._query_multiple_with_limit() now correctly handles limit and skip query parameters.
• kuha_common.query.QueryController.query_single() now correctly raises kuha_common.document_store.query.QueryException if passed a limit parameter.
• Add validation functions to kuha_common.document_store.query to validate query parameters.
• Add common testcase functionality to kuha_common.testing.testcases.KuhaUnitTestCase.
• Fix kuha_common.document_store.client.JSONStreamClient.fetch() so it won’t interfere with queue created by kuha_common.document_store.client.JSONStreamClient.queue_request().
• Update copyright headers to 2019.

0.8.0 (2018-12-18)

• Refactor kuha_common.server.
  • kuha_common.server.serve() replaces run_server() function. It takes the web-application as a parameter and does not handle application settings. Setting up the application should be handled by the caller who instantiates the application and knows what settings the application needs in order to operate.
• Add kuha_common.testing.MockCoro to help mocking out coroutine functions.

0.7.1 (2018-11-20)

• kuha_common.document_store.mappings.xmlbase.XMLParserBase._get_study_number_from_study_unit_element() Change priority of elements when looking up Study.study_number:
  1. a:Archive/a:ArchiveSpecific/a:Collection/a:CallNumber
  2. a:Archive/a:ArchiveSpecific/a:Item/a:CallNumber
  3. s:StudyUnit/r:UserID

0.7.0 (2018-11-19)

• Breaks backward compatibility Refactored DDI mapping profiles’ API. Package structure of kuha_common.document_store.mappings changed.
  • kuha_common.document_store.mappings.xmlbase.XMLParserBase now mandates how subclasses should be built by providing the public API for callers. The implementation of the actual mapping of resources is an implementation detail of subclasses and does not need to be public.
  • Refactor all DDI mapping profiles to a single module kuha_common.document_store.mappings.ddi
  • Add more mapping exceptions to differentiate error conditions in kuha_common.document_store.mappings.exceptions
  • Add mapping profile for DDI 3.1. to a new class kuha_common.document_store.mapping.ddi.DDI31RecordParser (Implements #24)

0.6.0 (2018-07-18)

• Add methods to kuha_common.testing.testcases
• Add support for parsing StudyGroups from DDI 1.2.2. in kuha_common.document_store.mappings.ddi_122_nesstar.DDI122RecordParser.p_study_groups() (Implements #20)
• Omit logging of request body to request log if the body is larger than kuha_common.server.REQUEST_LOG_BODY_MAXLEN. (Implements #22)
• Fix possible JSONDecodeError in kuha_common.server.log_request() (Fixes #23)

0.5.1 (2018-07-11)

• Declare testing.testcases.KuhaUnitTestCase.gen_id() as a classmethod, since it uses class’s method.

0.5.0 (2018-07-10)

• Package for common test functions and classes kuha_common.testing. (Implements #12)

0.4.1 (2018-07-04)

• kuha_common.document_store.mappings.xmlbase.XMLMapper._values_from_parent() resets the state of the attribute mapper if it needs to manipulate the mapper. (Fixes #19)

0.4.0 (2018-07-03)

• Relax record schema by decreasing the number of mandatory attributes. It should be possible to populate an element with attributes but have no value. (Implements #7)
  • kuha_common.document_store.field_types.Element.add_value() can be called without a value-parameter.
  • kuha_common.document_store.field_types.Value.export_dict() returns an empty dict if value is None.
  • kuha_common.document_store.field_types.ElementContainer._updates() If sec_value_value is None and no match is found, create and append new sub-element.
  • kuha_common.document_store.field_types.ElementContainer._updates() If sec_value_value is None and a matching value and attributes is found, discard the sec_value completely. (Implements #15)
  • Fix possible KeyError in kuha_common.document_store.field_types.FieldAttribute.value_from_dict() This was regression introduced by #7. (Fixes #16)
  • kuha_common.document_store.field_types.FieldAttribute.value_from_dict() Now always returns None if nothing was found. Previously it was possible to get an empty list.
• Relax DDI-C mappings: allow record field’s value to be None, if there are attributes for the field. (Implements #8)
• Add support for DDI from Nesstar Publisher. (Implements #9)
  • New mapping file ddi_122_nesstar.py to kuha_common.document_store.mappings package.
  • Refactor document_store/mappings/ddi_c.py and separate common XML classes & functions to a new module: xmlbase.py.
    • kuha_common.document_store.mappings.xmlbase.Value.from_element() converts empty string values to None.
• Correctly handle mapping from DDI-C if Codebook instance contains multiple series separated by ID. (Fixes #10)
• Fix DDI-C mapping for localized codelists for variables. (Fixes #11)
• DDI-C mapping for Question now removes extra whitepace around research_instruments.
• Fix fetching multiple parent elements for mapped parameters which have no main element. (Fixes #18)

Known Issues

• Mapping is unable to handle DDI-XML (DDI 2.5 and DDI 1.2.2 Nesstar) which contains inconsistent use of conceptual elements. See issue #17. For instance the following structure for anlyUnit:

  <anlyUnit>Description for analysis unit.
    <concept>concept.of.analysis.unit</concept>
  </anlyUnit>
  <anlyUnit>Description for another analysis unit.</anlyUnit>
  

  Will be interpreted as:

  [{'analysis_unit': 'concept.of.analysis.unit',
   'description': 'Description for analysis unit.',
   'language': 'en'}]
  

0.3.1 (2018-04-19)

• Add optional keyword arguments to kuha_common.cli_setup.add_document_store_url() that are passed to parser.add().
• Declare kuha_common.cli_setup.Settings.set() as public function.
• kuha_common.document_store.field_types.Value.updates() now actually updates the value.
• Support query filter for record’s _id.
• Use document-store-url with full document store base url for kuha_common.document_store.query.Query. (Fixes #5)
• Include request body for PUT and PATCH in kuha_common.server.log_request().
• Add kuha_common.document_store.records.RecordBase.get_id().
• Add kuha_common.document_store.records.RecordBase.updates().
• Add updates method for each subclass of kuha_common.document_store.records.RecordBase.
• Changes in kuha_common.document_store.mappings.ddi_c:
  • Fix default value for universe.attr_clusion.
  • Introduce new class kuha_common.document_store.mappings.ddi_c.RecordParser which stores default language; this way parsing can be done simultaneously to multiple XML-documents.
  • Default language is now a mandatory parameter for mapping methods:
    • kuha_common.document_store.mappings.ddi_c.Value.from_element()
    • kuha_common.document_store.mappings.ddi_c.ValueMap.__call__()
    • kuha_common.document_store.mappings.ddi_c.MultiValueMap.__call__()
  • Remove kuha_common.document_store.mappings.ddi_c.parse() to streamline the process.
  • Add global to store each parser and corresponding record collection.
  • Declare kuha_common.document_store.mappings.ddi_c.check_root() as public function.
  • Add kuha_common.document_store.mappings.ddi_c.get_root_language() function.

0.3.0 (2018-03-06)

• Move ddi_c.py mapping module (DDI-C -> Document Store records) from kuha_document_store to kuha_common.document_store.mappings package.
• Forward keyword arguments from Settings.load_parser to configargparse.get_arg_parser in cli_setup.py.
• Make JSONStreamClient._get_request a public method JSONStreamClient.get_request
• Forward keyword arguments from JSONStreamClient.get_request to DocumentStoreClient.streaming_query_request to support more options specifically more HTTP-methods than POST.
• Assert _log_request() in server.py will not raise UnicodeDecodeError if request.body is not utf-8 encoded.
• Add Study.document_uris
• Add abbreviation-attribute to Study.publishers.
• Add DDI-C mappings to Study.document_uris and Study.publishers.attr_abbreviation.

0.2.3 (2018-01-26)

• Implement support for non-localizable containerized elements.
• Add more fields to Study record.
• Add more unit & integration tests.

0.2.2 (2017-11-10)

• Update documentation

0.2.1 (2017-11-09)

• Fix referring variables to questions. Variable may refer multiple questions.
• Fix server.py log_request function. Call RequestHandler.CONTENT_TYPE_JSON, rather than handler-object.
• Partial support for coroutine callbacks in QueryController

0.2.0 (2017-11-01)

• Support referring variables to questions and vice versa.
• Add tox.ini to support running tests with tox.

0.1.0 (2017-10-25)

• Initial release