Kuha Common Changelog

2.2.0 (2023-05-19)

Changed

  • Do not raise MappingError when encountered a DDI-C file with relPubl/citation that has no titlStmt/titl child. Instead log a warning and continue iterating relPubl/citation elements. (Implements #47)

2.1.0 (2022-11-28)

Added

Fixed

  • DDI-C mapping will filter out inner ExtLink contents from /codeBook/stdyDscr/citation/rspStmt/AuthEnty values stored to Study-object in DocStore. (Fixes #46)

2.0.1 (2022-11-18)

Fixed

  • Read DDI-Codebook XML for /codeBook/stdyDscr/stdyInfo/sumDscr/anlyUnit/concept/@vocabURI correctly. (Fixes #44)

  • Read DDI-Codebook XML for /codeBook/stdyDscr/method/dataColl/collMode/concept/@vocabURI correcly. (Fixes #45)

2.0.0 (2022-10-24)

Added

  • Add localizable, multivalue-field Study.funding_agencies with role, agency, abbreviation, description and grant number attributes. (Implements #36)

  • Add localizable, multivalue-field Study.grant_numbers with agency and role attributes. (Implements #36)

  • Map DDI-C stdyDscr/citation/prodStmt/fundAg to Study.funding_agencies. (Implements #37)

  • Map DDI-L (3.1. and 3.3.) FundingInformation structure to Study.funding_agencies. (Implements #38)

  • Map DDI-C stdyDscr/citation/prodStmt/grantNo to Study.grant_numbers. (Implements #39)

  • Map DDI-L (3.1. and 3.3.) FundingInformation structure to Study.grant_numbers. (Implements #40)

  • Add identifier and identifier_agency attributes to Study.related_publications. (Implements #41)

  • Map DDI-C stdyDscr/othrStdyMat/relPubl/citation/titlStmt/IDNo to Study.related_publication.attr_identifier and IDNo@agency to Study.related_publication.attr_identifier_agency. (Implements #42)

  • Map related publication identifier info from DDI-L to identifier and identifier_agency attributes of Study.related_publications. (Implements #43)

Changed

  • Do not use eventloop directly in testcases. This fixes a DeprecationWarning “There is no current event loop”, when running with Python 3.10.

    • KuhaUnitTestCase.run_until_complete() replaces the practice of using the event loop directly with self._loop.run_until_complete. This is reflected to existing test cases.

  • Require Tornado 6.2.0 in requirements.txt.

  • Change initialization signature of exception kuha_common.document_store.mappings.exceptions.UnknownXMLRoot. Both unknown and expected are mandatory positional arguments. The argument expected can be repeated if multiple elements can be used. This is a backwards incompatible API change.

1.1.0 (2022-06-29)

Added

  • Mapping study-level data from DDI3.3 to Kuha study record. (Implements #34)

Changed

  • Move kuha_common.document_store.mappings.ddi module contents to a package kuha_common.document_store.mappings.ddi by splitting up the original module to version spesific modules: codebook.py and ddi31.py. The original ddi.py module was already over 1000 lines long and became hard to manage.

Fixed

  • Invalid interpretation for content language in DDI Codebook concept-elements. (Fixes #35)

1.0.0 (2022-03-15)

Added

  • Add more metadata to records: deleted-timestamp, records-status, schema-version. This is a backwards incompatible change and requires migration of database data. Kuha does not support such migrations. Operators are required to roll their own migration solution or to rebuild the database after upgrade.

  • Require py12flogging. Use it to provide JSON structured logging.

    • cli_setup.setup_logging() sets up py12flogging configuration.

    • cli_setup.setup_common_modules() sets up a logging context populator by registering server.serverlog_ctx_populator() when called with MOD_LOGGING and MOD_SERVER module setup constants.

  • Add kuha_common.conf to load, define and store application wide configuration.

  • kuha_common.document_store.query._build_filter() now supports logical operators for filter. Add constants for MongoDB logical operators. Add FilterKeyConstants.is_logical_operator().

    • Support not_equal ($ne), in ($in) and elem_match ($elemMatch) MongoDB-operators.

  • Add kuha_common.document_store.records.dig_and_set() helper method, which digs into a dictionary and converts values in certain paths.

  • Implement kuha_common.document_store.field_types.Element.import_records()

  • Add kuha_common.server.RequestHandler.get_valid_query_arguments() which validates and returns query arguments for current request.

  • Add kuha_common.document_store.mappings.ddi.DDI122NesstarRecordParser which uses an XML namespace for DDI1.2.2.

Changed

Removed

  • Drop test support for python versions below Python 3.8. From now on 3.8. is the oldest supported Python version.

  • Remove base class kuha_common.server.KuhaServerError. Inherit Kuha HTTP exceptions straight from tornado.web.HTTPError.

Fixed

  • In case a DDI-C XML codeBook/stdyDscr/othrStdyMat/relPubl/citation does not have a titlStmt child, a MappingError is raised. Previously the situation lead to an AttributeError. (Fixes #29)

0.15.1 (2021-09-06)

  • Add missing mapping from DDI 2.5 and DDI 1.2.2 to Study.study_uris.

0.15.0 (2021-09-03)

This release ensures future compatibility with CESSDA Data Catalogue.

  • Add following attributes to kuha_common.document_store.records.Study:

    • document_titles to contain metadata document titles.

    • study_uris to contain URIs linking to study.

  • Add DDI mappings to new attributes in kuha_common.document_store.mappings.ddi:

    • DDI 2.5 and DDI 1.2.2 use codeBook/docDscr/citation/titlStmt/titl to populate Study.document_titles

    • DDI 3.1 uses ddi:DDIInstance/r:Citation/r:Title to populate Study.document_titles

    • DDI 2.5 and DDI 1.2.2 use codeBook/stdyDscr/citation/holdings/@URI to populate Study.study_uris.

    • DDI 3.1 uses s:StudyUnit/a:Archive/a:ArchiveSpecific/a:Collection/a:URI to populate Study.study_uris. This XPATH was used to populate Study.document_uris, but it seems more appropriate to use it for Study.study_uris. There is currently no mapping to populate Study.document_uris from DDI 3.1.

  • Jenkinsfile uses python-latest to build test environments. In FSD Jenkins python-latest always points to the latest installed python interpreter.

  • Add py39 to test environments.

  • Remove py35, py36 and py37 from tox test environments.

  • Remove tasks running tests with py35, py36 and py37 interpreters from Jenkinsfile.

0.14.0 (2020-06-09)

Python 3.8 compatibility

  • kuha_common.testing.MockCoro was not working properly with Python 3.8.2 AsyncMock. Switch to a closure function with pruned functionality. Introduce alias MockCoro to keep changes more backwards compatible. This is, however, backwards incompatible change.

    • MockCoro is now a closure function. Prior 0.14.0 it was a class.

    • MockCoro now returns a coroutine function. Prior 0.14.0 it returned a MockCoro instance.

    • MockCoro func does not get MockCoro instance as the first parameter. Instead the signature can now be exactly same as the mocked out function’s. Prior 0.14.0 MockCoro.func was called with MockCoro instance as the first argument.

    • Since mock_coro (alias MockCoro) is no longer a class, it won’t hold any attributes. Code relying in mock_coro.func or mock_coro.dummy_rval needs to change.

  • Use list(element_copy) instead of deprecated element_copy.getchildren() in kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text()

  • Add py38 to tox.ini environments. The code should now be Python 3.8. compatible.

Other changes to public APIs

These are backwards compatible changes.

0.13.0 (2020-05-06)

  • DDI 2.5 and DDI 1.2.2 mappings use relpubl/citation/holdings/@URI and relpubl/citation/distStmt/distDate elements that map into Study.related_publications attributes distribution_date and uri.

0.12.0 (2020-04-28)

  • Add related_publications to Study

  • Add DDI 2.5 and DDI 1.2.2 mappings to Study.related_publications

0.11.2 (2020-01-24)

  • Fix logic in kuha_common.document_store.field_types.ElementContainer._updates() where secondary values were compared to other secondary values. This lead to faulty contained values, when a secondary value matched with another secondary value.

0.11.1 (2020-01-09)

  • Reset settings in kuha_common.testing.testcases.load_cli_args() if tests get skipped, since skipping won’t call tearDownClass method

0.11.0 (2020-01-08)

0.10.0 (2019-10-22)

Warning

Breaks backwards compatibility in DDI 1.2.2. and DDI 2.5. mapping of attributes for Study.study_groups.study_group and StudyGroup.study_group_identifier

Change mappings for DDI 1.2.2. and DDI 2.5.

  • Change mappings for Study.study_groups.study_group of DDI 2.5. and DDI 1.2.2. The value for the contained element is now taken from serStmt/@ID instead of serName/@ID. Breaks backwards compatibility

  • Change mapping for StudyGroup.study_group_identifier of DDI 2.5. and DDI 1.2.2. The value is now taken from serStmt/@ID instead of serName/@ID. Breaks backwards compatibility

Add new attributes to kuha_common.document_store.records

  • Study.study_groups.attr_description

  • Study.study_groups.attr_uri

  • Study.data_kinds

  • Study.data_collection_copyrights

  • Study.citation_requirements

  • Study.deposit_requirements

  • Study.geographic_coverages

  • StudyGroup.descriptions

  • StudyGroup.uris

Add mappings for new attributes

DDI 1.2.2. and DDI 2.5.:

  • /codeBook/stdyDscr/citation/serStmt/serInfo maps to Study.study_groups.attr_description

  • /codeBook/stdyDscr/citation/serStmt/serInfo maps to StudyGroups.descriptions

  • /codeBook/stdyDscr/citation/serStmt/@URI maps to Study.study_groups.attr_uri

  • /codeBook/stdyDscr/citation/serStmt/@URI maps to StudyGroup.uris

  • /codeBook/stdyDscr/citation/prodStmt/copyright maps to Study.data_collection_copyrights

  • /codeBook/stdyDscr/dataAccs/useStmt/citReq maps to Study.citation_requirements

  • /codeBook/stdyDscr/dataAccs/useStmt/deposReq maps to Study.deposit_requirements

  • /codeBook/stdyDscr/stdyInfo/sumDscr/geogCover maps to Study.geographic_coverages

  • /codeBook/stdyDscr/stdyInfo/sumDscr/dataKind maps to Study.data_kinds

DDI 3.1.

  • g:Group/g:Abstract/r:Content maps to StudyGroups.descriptions

  • g:Group/@externalReferenceDefaultURI maps to StudyGroups.uris

0.9.1 (2019-04-03)

0.9.0 (2019-03-14)

0.8.0 (2018-12-18)

  • Refactor kuha_common.server.

    • kuha_common.server.serve() replaces run_server() function. It takes the web-application as a parameter and does not handle application settings. Setting up the application should be handled by the caller who instantiates the application and knows what settings the application needs in order to operate.

  • Add kuha_common.testing.MockCoro to help mocking out coroutine functions.

0.7.1 (2018-11-20)

  • kuha_common.document_store.mappings.xmlbase.XMLParserBase._get_study_number_from_study_unit_element() Change priority of elements when looking up Study.study_number:

    1. a:Archive/a:ArchiveSpecific/a:Collection/a:CallNumber

    2. a:Archive/a:ArchiveSpecific/a:Item/a:CallNumber

    3. s:StudyUnit/r:UserID

0.7.0 (2018-11-19)

0.6.0 (2018-07-18)

  • Add methods to kuha_common.testing.testcases

  • Add support for parsing StudyGroups from DDI 1.2.2. in kuha_common.document_store.mappings.ddi_122_nesstar.DDI122RecordParser.p_study_groups() (Implements #20)

  • Omit logging of request body to request log if the body is larger than kuha_common.server.REQUEST_LOG_BODY_MAXLEN. (Implements #22)

  • Fix possible JSONDecodeError in kuha_common.server.log_request() (Fixes #23)

0.5.1 (2018-07-11)

  • Declare testing.testcases.KuhaUnitTestCase.gen_id() as a classmethod, since it uses class’s method.

0.5.0 (2018-07-10)

0.4.1 (2018-07-04)

  • kuha_common.document_store.mappings.xmlbase.XMLMapper._values_from_parent() resets the state of the attribute mapper if it needs to manipulate the mapper. (Fixes #19)

0.4.0 (2018-07-03)

  • Relax record schema by decreasing the number of mandatory attributes. It should be possible to populate an element with attributes but have no value. (Implements #7)

  • Relax DDI-C mappings: allow record field’s value to be None, if there are attributes for the field. (Implements #8)

  • Add support for DDI from Nesstar Publisher. (Implements #9)

    • New mapping file ddi_122_nesstar.py to kuha_common.document_store.mappings package.

    • Refactor document_store/mappings/ddi_c.py and separate common XML classes & functions to a new module: xmlbase.py.

      • kuha_common.document_store.mappings.xmlbase.Value.from_element() converts empty string values to None.

  • Correctly handle mapping from DDI-C if Codebook instance contains multiple series separated by ID. (Fixes #10)

  • Fix DDI-C mapping for localized codelists for variables. (Fixes #11)

  • DDI-C mapping for Question now removes extra whitepace around research_instruments.

  • Fix fetching multiple parent elements for mapped parameters which have no main element. (Fixes #18)

Known Issues

  • Mapping is unable to handle DDI-XML (DDI 2.5 and DDI 1.2.2 Nesstar) which contains inconsistent use of conceptual elements. See issue #17. For instance the following structure for anlyUnit:

    <anlyUnit>Description for analysis unit.
  <concept>concept.of.analysis.unit</concept>
</anlyUnit>
<anlyUnit>Description for another analysis unit.</anlyUnit>

    Will be interpreted as:

    [{'analysis_unit': 'concept.of.analysis.unit',
 'description': 'Description for analysis unit.',
 'language': 'en'}]

0.3.1 (2018-04-19)

  • Add optional keyword arguments to kuha_common.cli_setup.add_document_store_url() that are passed to parser.add().

  • Declare kuha_common.cli_setup.Settings.set() as public function.

  • kuha_common.document_store.field_types.Value.updates() now actually updates the value.

  • Support query filter for record’s _id.

  • Use document-store-url with full document store base url for kuha_common.document_store.query.Query. (Fixes #5)

  • Include request body for PUT and PATCH in kuha_common.server.log_request().

  • Add kuha_common.document_store.records.RecordBase.get_id().

  • Add kuha_common.document_store.records.RecordBase.updates().

  • Add updates method for each subclass of kuha_common.document_store.records.RecordBase.

  • Changes in kuha_common.document_store.mappings.ddi_c:

    • Fix default value for universe.attr_clusion.

    • Introduce new class kuha_common.document_store.mappings.ddi_c.RecordParser which stores default language; this way parsing can be done simultaneously to multiple XML-documents.

    • Default language is now a mandatory parameter for mapping methods:

      • kuha_common.document_store.mappings.ddi_c.Value.from_element()

      • kuha_common.document_store.mappings.ddi_c.ValueMap.__call__()

      • kuha_common.document_store.mappings.ddi_c.MultiValueMap.__call__()

    • Remove kuha_common.document_store.mappings.ddi_c.parse() to streamline the process.

    • Add global to store each parser and corresponding record collection.

    • Declare kuha_common.document_store.mappings.ddi_c.check_root() as public function.

    • Add kuha_common.document_store.mappings.ddi_c.get_root_language() function.

0.3.0 (2018-03-06)

  • Move ddi_c.py mapping module (DDI-C -> Document Store records) from kuha_document_store to kuha_common.document_store.mappings package.

  • Forward keyword arguments from Settings.load_parser to configargparse.get_arg_parser in cli_setup.py.

  • Make JSONStreamClient._get_request a public method JSONStreamClient.get_request

  • Forward keyword arguments from JSONStreamClient.get_request to DocumentStoreClient.streaming_query_request to support more options specifically more HTTP-methods than POST.

  • Assert _log_request() in server.py will not raise UnicodeDecodeError if request.body is not utf-8 encoded.

  • Add Study.document_uris

  • Add abbreviation-attribute to Study.publishers.

  • Add DDI-C mappings to Study.document_uris and Study.publishers.attr_abbreviation.

0.2.3 (2018-01-26)

  • Implement support for non-localizable containerized elements.

  • Add more fields to Study record.

  • Add more unit & integration tests.

0.2.2 (2017-11-10)

  • Update documentation

0.2.1 (2017-11-09)

  • Fix referring variables to questions. Variable may refer multiple questions.

  • Fix server.py log_request function. Call RequestHandler.CONTENT_TYPE_JSON, rather than handler-object.

  • Partial support for coroutine callbacks in QueryController

0.2.0 (2017-11-01)

  • Support referring variables to questions and vice versa.

  • Add tox.ini to support running tests with tox.

0.1.0 (2017-10-25)

  • Initial release