Kuha Common Changelog¶
0.15.1 (2021-09-06)¶
- Add missing mapping from DDI 2.5 and DDI 1.2.2 to Study.study_uris.
0.15.0 (2021-09-03)¶
This release ensures future compatibility with CESSDA Data Catalogue.
- Add following attributes to
kuha_common.document_store.records.Study
:document_titles
to contain metadata document titles.study_uris
to contain URIs linking to study.
- Add DDI mappings to new attributes in
kuha_common.document_store.mappings.ddi
:- DDI 2.5 and DDI 1.2.2 use codeBook/docDscr/citation/titlStmt/titl to populate Study.document_titles
- DDI 3.1 uses ddi:DDIInstance/r:Citation/r:Title to populate Study.document_titles
- DDI 2.5 and DDI 1.2.2 use codeBook/stdyDscr/citation/holdings/@URI to populate Study.study_uris.
- DDI 3.1 uses s:StudyUnit/a:Archive/a:ArchiveSpecific/a:Collection/a:URI to populate Study.study_uris. This XPATH was used to populate Study.document_uris, but it seems more appropriate to use it for Study.study_uris. There is currently no mapping to populate Study.document_uris from DDI 3.1.
- Jenkinsfile uses python-latest to build test environments. In FSD Jenkins python-latest always points to the latest installed python interpreter.
- Add py39 to test environments.
- Remove py35, py36 and py37 from tox test environments.
- Remove tasks running tests with py35, py36 and py37 interpreters from Jenkinsfile.
0.14.0 (2020-06-09)¶
Python 3.8 compatibility¶
kuha_common.testing.MockCoro
was not working properly with Python 3.8.2 AsyncMock. Switch to a closure function with pruned functionality. Introduce alias MockCoro to keep changes more backwards compatible. This is, however, backwards incompatible change.- MockCoro is now a closure function. Prior 0.14.0 it was a class.
- MockCoro now returns a coroutine function. Prior 0.14.0 it returned a MockCoro instance.
- MockCoro func does not get MockCoro instance as the first parameter. Instead the signature can now be exactly same as the mocked out function’s. Prior 0.14.0 MockCoro.func was called with MockCoro instance as the first argument.
- Since mock_coro (alias MockCoro) is no longer a class, it won’t hold any attributes. Code relying in mock_coro.func or mock_coro.dummy_rval needs to change.
- Use list(element_copy) instead of deprecated element_copy.getchildren() in
kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text()
- Add py38 to tox.ini environments. The code should now be Python 3.8. compatible.
Other changes to public APIs¶
These are backwards compatible changes.
- Add
kuha_common.testing.testcases.KuhaUnitTestCase.assert_mock_meth_has_calls()
to try to pinpoint the exact call that was missing from mocked method. - Add
kuha_common.cli_setup.add_server_process_count_configuration()
which adds tornado server process count to configuration options. - Change every add_* function signature in
kuha_common.cli_setup
to make parser an optional argument. If omitted the functions will use singleton stored inkuha_common.cli_setup.settings.parser
.
0.13.0 (2020-05-06)¶
- DDI 2.5 and DDI 1.2.2 mappings use relpubl/citation/holdings/@URI and relpubl/citation/distStmt/distDate elements that map into Study.related_publications attributes distribution_date and uri.
0.12.0 (2020-04-28)¶
- Add
related_publications
toStudy
- Add DDI 2.5 and DDI 1.2.2 mappings to
Study.related_publications
0.11.2 (2020-01-24)¶
- Fix logic in
kuha_common.document_store.field_types.ElementContainer._updates()
where secondary values were compared to other secondary values. This lead to faulty contained values, when a secondary value matched with another secondary value.
0.11.1 (2020-01-09)¶
- Reset settings in
kuha_common.testing.testcases.load_cli_args()
if tests get skipped, since skipping won’t call tearDownClass method
0.11.0 (2020-01-08)¶
- Initialize argumentparser using configargparse.ArgumentParser directly instead of
calling
configargparse.get_arg_parser()
inkuha_common.cli_setup
. This way the settings get stored only inkuha_common.cli_setup.settings
singleton. - Reset settings in
kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass()
. - Fix faulty call to parent’s setUpClass in
kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass()
.
0.10.0 (2019-10-22)¶
Warning
Breaks backwards compatibility in DDI 1.2.2. and DDI 2.5. mapping of attributes for
Study.study_groups.study_group
and StudyGroup.study_group_identifier
Change mappings for DDI 1.2.2. and DDI 2.5.¶
- Change mappings for Study.study_groups.study_group of DDI 2.5. and DDI 1.2.2.
The value for the contained element is now taken from
serStmt/@ID
instead ofserName/@ID
. Breaks backwards compatibility - Change mapping for StudyGroup.study_group_identifier of DDI 2.5. and DDI 1.2.2.
The value is now taken from
serStmt/@ID
instead ofserName/@ID
. Breaks backwards compatibility
Add new attributes to kuha_common.document_store.records
¶
Study.study_groups.attr_description
Study.study_groups.attr_uri
Study.data_kinds
Study.data_collection_copyrights
Study.citation_requirements
Study.deposit_requirements
Study.geographic_coverages
StudyGroup.descriptions
StudyGroup.uris
Add mappings for new attributes¶
DDI 1.2.2. and DDI 2.5.:
/codeBook/stdyDscr/citation/serStmt/serInfo
maps toStudy.study_groups.attr_description
/codeBook/stdyDscr/citation/serStmt/serInfo
maps toStudyGroups.descriptions
/codeBook/stdyDscr/citation/serStmt/@URI
maps toStudy.study_groups.attr_uri
/codeBook/stdyDscr/citation/serStmt/@URI
maps toStudyGroup.uris
/codeBook/stdyDscr/citation/prodStmt/copyright
maps toStudy.data_collection_copyrights
/codeBook/stdyDscr/dataAccs/useStmt/citReq
maps toStudy.citation_requirements
/codeBook/stdyDscr/dataAccs/useStmt/deposReq
maps toStudy.deposit_requirements
/codeBook/stdyDscr/stdyInfo/sumDscr/geogCover
maps toStudy.geographic_coverages
/codeBook/stdyDscr/stdyInfo/sumDscr/dataKind
maps toStudy.data_kinds
DDI 3.1.
g:Group/g:Abstract/r:Content
maps toStudyGroups.descriptions
g:Group/@externalReferenceDefaultURI
maps toStudyGroups.uris
0.9.1 (2019-04-03)¶
- DDI mappings:
kuha_common.document_store.mappings.xmlbase.MappedParams.has_arguments()
treat MappedParams objects which don’t have a value and all keyword arguments’ values are None as MappedParams objects that don’t have any arguments. This way all empty XML elements will be discarded in the mapping phase before they get turned into Document Store reocrds. (Fixes #26) - Treat attributes-dictionaries with only None values as invalid if the Element’s value is also None
in
kuha_common.document_store.field_types.Element.add_value()
. - Servers exception logging
kuha_common.server.log_exception()
will now log all traceback lines, instead of the last four.
0.9.0 (2019-03-14)¶
- Maintenance release with no new functionality besides bug fixes.
- Support tornado > 5
- Remove callbacks used with tornado client.
- requirements.txt declares tornado==6.0.1, which is currently the latest.
kuha_common.query.QueryController._query_multiple_with_limit()
now correctly handleslimit
andskip
query parameters.kuha_common.query.QueryController.query_single()
now correctly raiseskuha_common.document_store.query.QueryException
if passed alimit
parameter.- Add validation functions to
kuha_common.document_store.query
to validate query parameters. - Add common testcase functionality to
kuha_common.testing.testcases.KuhaUnitTestCase
. - Fix
kuha_common.document_store.client.JSONStreamClient.fetch()
so it won’t interfere with queue created bykuha_common.document_store.client.JSONStreamClient.queue_request()
. - Update copyright headers to 2019.
0.8.0 (2018-12-18)¶
- Refactor
kuha_common.server
.kuha_common.server.serve()
replaces run_server() function. It takes the web-application as a parameter and does not handle application settings. Setting up the application should be handled by the caller who instantiates the application and knows what settings the application needs in order to operate.
- Add
kuha_common.testing.MockCoro
to help mocking out coroutine functions.
0.7.1 (2018-11-20)¶
kuha_common.document_store.mappings.xmlbase.XMLParserBase._get_study_number_from_study_unit_element()
Change priority of elements when looking up Study.study_number:- a:Archive/a:ArchiveSpecific/a:Collection/a:CallNumber
- a:Archive/a:ArchiveSpecific/a:Item/a:CallNumber
- s:StudyUnit/r:UserID
0.7.0 (2018-11-19)¶
- Breaks backward compatibility Refactored DDI mapping profiles’ API.
Package structure of
kuha_common.document_store.mappings
changed.kuha_common.document_store.mappings.xmlbase.XMLParserBase
now mandates how subclasses should be built by providing the public API for callers. The implementation of the actual mapping of resources is an implementation detail of subclasses and does not need to be public.- Refactor all DDI mapping profiles to a single module
kuha_common.document_store.mappings.ddi
- Add more mapping exceptions to differentiate error conditions in
kuha_common.document_store.mappings.exceptions
- Add mapping profile for DDI 3.1. to a new class
kuha_common.document_store.mapping.ddi.DDI31RecordParser
(Implements #24)
0.6.0 (2018-07-18)¶
- Add methods to
kuha_common.testing.testcases
- Add support for parsing StudyGroups from DDI 1.2.2. in
kuha_common.document_store.mappings.ddi_122_nesstar.DDI122RecordParser.p_study_groups()
(Implements #20) - Omit logging of request body to request log if the body is larger than
kuha_common.server.REQUEST_LOG_BODY_MAXLEN
. (Implements #22) - Fix possible JSONDecodeError in
kuha_common.server.log_request()
(Fixes #23)
0.5.1 (2018-07-11)¶
- Declare testing.testcases.KuhaUnitTestCase.gen_id() as a classmethod, since it uses class’s method.
0.5.0 (2018-07-10)¶
- Package for common test functions and classes
kuha_common.testing
. (Implements #12)
0.4.1 (2018-07-04)¶
kuha_common.document_store.mappings.xmlbase.XMLMapper._values_from_parent()
resets the state of the attribute mapper if it needs to manipulate the mapper. (Fixes #19)
0.4.0 (2018-07-03)¶
- Relax record schema by decreasing the number of mandatory attributes.
It should be possible to populate an element with attributes but have
no value. (Implements #7)
kuha_common.document_store.field_types.Element.add_value()
can be called without a value-parameter.kuha_common.document_store.field_types.Value.export_dict()
returns an empty dict if value is None.kuha_common.document_store.field_types.ElementContainer._updates()
If sec_value_value is None and no match is found, create and append new sub-element.kuha_common.document_store.field_types.ElementContainer._updates()
If sec_value_value is None and a matching value and attributes is found, discard the sec_value completely. (Implements #15)- Fix possible KeyError in
kuha_common.document_store.field_types.FieldAttribute.value_from_dict()
This was regression introduced by #7. (Fixes #16) kuha_common.document_store.field_types.FieldAttribute.value_from_dict()
Now always returns None if nothing was found. Previously it was possible to get an empty list.
- Relax DDI-C mappings: allow record field’s value to be None, if there are attributes for the field. (Implements #8)
- Add support for DDI from Nesstar Publisher. (Implements #9)
- New mapping file ddi_122_nesstar.py to kuha_common.document_store.mappings package.
- Refactor document_store/mappings/ddi_c.py and separate common XML classes & functions to
a new module: xmlbase.py.
kuha_common.document_store.mappings.xmlbase.Value.from_element()
converts empty string values to None.
- Correctly handle mapping from DDI-C if Codebook instance contains multiple series separated by ID. (Fixes #10)
- Fix DDI-C mapping for localized codelists for variables. (Fixes #11)
- DDI-C mapping for Question now removes extra whitepace around research_instruments.
- Fix fetching multiple parent elements for mapped parameters which have no main element. (Fixes #18)
Known Issues¶
Mapping is unable to handle DDI-XML (DDI 2.5 and DDI 1.2.2 Nesstar) which contains inconsistent use of conceptual elements. See issue #17. For instance the following structure for anlyUnit:
<anlyUnit>Description for analysis unit. <concept>concept.of.analysis.unit</concept> </anlyUnit> <anlyUnit>Description for another analysis unit.</anlyUnit>
Will be interpreted as:
[{'analysis_unit': 'concept.of.analysis.unit', 'description': 'Description for analysis unit.', 'language': 'en'}]
0.3.1 (2018-04-19)¶
- Add optional keyword arguments to
kuha_common.cli_setup.add_document_store_url()
that are passed to parser.add(). - Declare
kuha_common.cli_setup.Settings.set()
as public function. kuha_common.document_store.field_types.Value.updates()
now actually updates the value.- Support query filter for record’s _id.
- Use document-store-url with full document store base url for
kuha_common.document_store.query.Query
. (Fixes #5) - Include request body for PUT and PATCH in
kuha_common.server.log_request()
. - Add
kuha_common.document_store.records.RecordBase.get_id()
. - Add
kuha_common.document_store.records.RecordBase.updates()
. - Add updates method for each subclass of
kuha_common.document_store.records.RecordBase
. - Changes in
kuha_common.document_store.mappings.ddi_c
:- Fix default value for universe.attr_clusion.
- Introduce new class
kuha_common.document_store.mappings.ddi_c.RecordParser
which stores default language; this way parsing can be done simultaneously to multiple XML-documents. - Default language is now a mandatory parameter for mapping methods:
kuha_common.document_store.mappings.ddi_c.Value.from_element()
kuha_common.document_store.mappings.ddi_c.ValueMap.__call__()
kuha_common.document_store.mappings.ddi_c.MultiValueMap.__call__()
- Remove
kuha_common.document_store.mappings.ddi_c.parse()
to streamline the process. - Add global to store each parser and corresponding record collection.
- Declare
kuha_common.document_store.mappings.ddi_c.check_root()
as public function. - Add
kuha_common.document_store.mappings.ddi_c.get_root_language()
function.
0.3.0 (2018-03-06)¶
- Move ddi_c.py mapping module (DDI-C -> Document Store records) from kuha_document_store to kuha_common.document_store.mappings package.
- Forward keyword arguments from Settings.load_parser to configargparse.get_arg_parser in cli_setup.py.
- Make JSONStreamClient._get_request a public method JSONStreamClient.get_request
- Forward keyword arguments from JSONStreamClient.get_request to DocumentStoreClient.streaming_query_request to support more options specifically more HTTP-methods than POST.
- Assert _log_request() in server.py will not raise UnicodeDecodeError if request.body is not utf-8 encoded.
- Add Study.document_uris
- Add abbreviation-attribute to Study.publishers.
- Add DDI-C mappings to Study.document_uris and Study.publishers.attr_abbreviation.
0.2.3 (2018-01-26)¶
- Implement support for non-localizable containerized elements.
- Add more fields to Study record.
- Add more unit & integration tests.
0.2.2 (2017-11-10)¶
- Update documentation
0.2.1 (2017-11-09)¶
- Fix referring variables to questions. Variable may refer multiple questions.
- Fix server.py log_request function. Call RequestHandler.CONTENT_TYPE_JSON, rather than handler-object.
- Partial support for coroutine callbacks in QueryController
0.2.0 (2017-11-01)¶
- Support referring variables to questions and vice versa.
- Add tox.ini to support running tests with tox.
0.1.0 (2017-10-25)¶
- Initial release