Kuha Common Changelog
2.6.0 (2024-11-29)
Added
Support more study-level DDI-C fields. (Implements #48)
Add
Study.distribution_dates
with description attribute.Add
Study.research_instruments
with system_name, uri and description attributes.Add
element_version
attribute toStudy.data_access_descriptions
.Add
description
attribute toStudy.collection_periods
.Map DDI-C
/codeBook/stdyDscr/citation/distStmt/distDate
toStudy.distribution_dates
.Map DDI-C
/codeBook/stdyDscr/method/dataColl/resInstru
toStudy.research_instruments
.Map DDI-C
/codeBook/stdyDscr/dataAccs/useStmt/conditions/@elementVersion
toStudy.data_access_descriptions.attr_element_version
Map DDI-C
/codeBook/stdyDscr/stdyInfo/sumDscr/collDate
CDATA toStudy.collection_periods.attr_description
.
Changed
Update dependecies in requirements.txt:
ConfigArgParse 1.7
Py12fLogging 0.7.0
2.5.0 (2024-08-22)
Added
Support Python versions 3.11 & 3.12.
Changed
Update Tornado to 6.4.1 in requirements.txt.
2.4.0 (2024-04-25)
Maintenance release due to move to https://gitlab.tuni.fi/fsd/kuha_common.
Changed
Hosting platform changed from bitbucket.org to gitlab.tuni.fi. Update documentation and setup.py accordingly.
2.3.0 (2024-03-08)
Added
Add
external_link
,external_link_role
,external_link_uri
andexternal_link_title
attributes toStudy.principal_investigators
.Map DDI-C
/codeBook/stdyDscr/citation/rspStmt/AuthEnty/ExtLink
values and attributes toStudy.principal_investigators
attributes.
2.2.0 (2023-05-19)
Changed
Do not raise MappingError when encountered a DDI-C file with relPubl/citation that has no titlStmt/titl child. Instead log a warning and continue iterating relPubl/citation elements. (Implements #47)
2.1.0 (2022-11-28)
Added
Add optional
tags_to_remove
list parameter tokuha_common.document_store.mappings.xmlbase.element_strip_descendant_text()
. If given, only the listed tags will be filtered out.Add
kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text_by_tag()
, which acts as a closure function that formats thetags_to_remove
parameter and proxies toelement_strip_descendant_text
.
Fixed
DDI-C mapping will filter out inner ExtLink contents from
/codeBook/stdyDscr/citation/rspStmt/AuthEnty
values stored to Study-object in DocStore. (Fixes #46)
2.0.1 (2022-11-18)
Fixed
2.0.0 (2022-10-24)
Added
Add localizable, multivalue-field
Study.funding_agencies
with role, agency, abbreviation, description and grant number attributes. (Implements #36)Add localizable, multivalue-field
Study.grant_numbers
with agency and role attributes. (Implements #36)Map DDI-C
stdyDscr/citation/prodStmt/fundAg
toStudy.funding_agencies
. (Implements #37)Map DDI-L (3.1. and 3.3.)
FundingInformation
structure toStudy.funding_agencies
. (Implements #38)Map DDI-C
stdyDscr/citation/prodStmt/grantNo
toStudy.grant_numbers
. (Implements #39)Map DDI-L (3.1. and 3.3.)
FundingInformation
structure toStudy.grant_numbers
. (Implements #40)Add
identifier
andidentifier_agency
attributes to Study.related_publications. (Implements #41)Map DDI-C
stdyDscr/othrStdyMat/relPubl/citation/titlStmt/IDNo
to Study.related_publication.attr_identifier andIDNo@agency
to Study.related_publication.attr_identifier_agency. (Implements #42)Map related publication identifier info from DDI-L to
identifier
andidentifier_agency
attributes of Study.related_publications. (Implements #43)
Changed
Do not use eventloop directly in testcases. This fixes a DeprecationWarning “There is no current event loop”, when running with Python 3.10.
KuhaUnitTestCase.run_until_complete()
replaces the practice of using the event loop directly withself._loop.run_until_complete
. This is reflected to existing test cases.
Require Tornado 6.2.0 in requirements.txt.
Change initialization signature of exception
kuha_common.document_store.mappings.exceptions.UnknownXMLRoot
. Both unknown and expected are mandatory positional arguments. The argument expected can be repeated if multiple elements can be used. This is a backwards incompatible API change.
1.1.0 (2022-06-29)
Added
Mapping study-level data from DDI3.3 to Kuha study record. (Implements #34)
Changed
Move kuha_common.document_store.mappings.ddi module contents to a package kuha_common.document_store.mappings.ddi by splitting up the original module to version spesific modules: codebook.py and ddi31.py. The original ddi.py module was already over 1000 lines long and became hard to manage.
Fixed
Invalid interpretation for content language in DDI Codebook concept-elements. (Fixes #35)
1.0.0 (2022-03-15)
Added
Add more metadata to records: deleted-timestamp, records-status, schema-version. This is a backwards incompatible change and requires migration of database data. Kuha does not support such migrations. Operators are required to roll their own migration solution or to rebuild the database after upgrade.
Add
kuha_common.document_store.records.RecordBase.is_deleted()
which returns True if the record has been logically deletedAdd
kuha_common.document_store.records.RecordBase.get_deleted()
which returns the deletion timestamps as a string.
Require py12flogging. Use it to provide JSON structured logging.
cli_setup.setup_logging()
sets up py12flogging configuration.cli_setup.setup_common_modules()
sets up a logging context populator by registeringserver.serverlog_ctx_populator()
when called with MOD_LOGGING and MOD_SERVER module setup constants.
Add
kuha_common.conf
to load, define and store application wide configuration.kuha_common.document_store.query._build_filter()
now supports logical operators for filter. Add constants for MongoDB logical operators. Add FilterKeyConstants.is_logical_operator().Support not_equal ($ne), in ($in) and elem_match ($elemMatch) MongoDB-operators.
Add
kuha_common.document_store.records.dig_and_set()
helper method, which digs into a dictionary and converts values in certain paths.Implement
kuha_common.document_store.field_types.Element.import_records()
Add
kuha_common.server.RequestHandler.get_valid_query_arguments()
which validates and returns query arguments for current request.Add
kuha_common.document_store.mappings.ddi.DDI122NesstarRecordParser
which uses an XML namespace for DDI1.2.2.
Changed
kuha_common.document_store.mappings.ddi
Document Store mappings modules will no longer be directly tied to using models in document_store/records.py. The mappings can be subclassed to change the records.kuha_common.document_store.records.RecordBase.export_metadata_dict()
now takes in an optional parameter as_datestamps, which controls if the datestamp values will be converted from python datetime to string datestamps.Testcase base classes in
kuha_common.testing.testcases
now isolate the global configuration store.kuha_common.testing.testcases.KuhaEndToEndTestCase.DELETE_to_document_store()
supports delete_type optional parameter.kuha_common.testing.testcases.KuhaEndToEndTestCase.DELETE_to_document_store()
returns the response body as python dict.Setup of common modules are now done via configure() functions
Update dependencies declared in requirements.txt.
Tornado 6.1.0
ConfigArgParse 1.5.3
Removed
Drop test support for python versions below Python 3.8. From now on 3.8. is the oldest supported Python version.
Remove base class
kuha_common.server.KuhaServerError
. Inherit Kuha HTTP exceptions straight from tornado.web.HTTPError.
Fixed
In case a DDI-C XML codeBook/stdyDscr/othrStdyMat/relPubl/citation does not have a titlStmt child, a
MappingError
is raised. Previously the situation lead to an AttributeError. (Fixes #29)
0.15.1 (2021-09-06)
Add missing mapping from DDI 2.5 and DDI 1.2.2 to Study.study_uris.
0.15.0 (2021-09-03)
This release ensures future compatibility with CESSDA Data Catalogue.
Add following attributes to
kuha_common.document_store.records.Study
:document_titles
to contain metadata document titles.study_uris
to contain URIs linking to study.
Add DDI mappings to new attributes in
kuha_common.document_store.mappings.ddi
:DDI 2.5 and DDI 1.2.2 use codeBook/docDscr/citation/titlStmt/titl to populate Study.document_titles
DDI 3.1 uses ddi:DDIInstance/r:Citation/r:Title to populate Study.document_titles
DDI 2.5 and DDI 1.2.2 use codeBook/stdyDscr/citation/holdings/@URI to populate Study.study_uris.
DDI 3.1 uses s:StudyUnit/a:Archive/a:ArchiveSpecific/a:Collection/a:URI to populate Study.study_uris. This XPATH was used to populate Study.document_uris, but it seems more appropriate to use it for Study.study_uris. There is currently no mapping to populate Study.document_uris from DDI 3.1.
Jenkinsfile uses python-latest to build test environments. In FSD Jenkins python-latest always points to the latest installed python interpreter.
Add py39 to test environments.
Remove py35, py36 and py37 from tox test environments.
Remove tasks running tests with py35, py36 and py37 interpreters from Jenkinsfile.
0.14.0 (2020-06-09)
Python 3.8 compatibility
kuha_common.testing.MockCoro
was not working properly with Python 3.8.2 AsyncMock. Switch to a closure function with pruned functionality. Introduce alias MockCoro to keep changes more backwards compatible. This is, however, backwards incompatible change.MockCoro is now a closure function. Prior 0.14.0 it was a class.
MockCoro now returns a coroutine function. Prior 0.14.0 it returned a MockCoro instance.
MockCoro func does not get MockCoro instance as the first parameter. Instead the signature can now be exactly same as the mocked out function’s. Prior 0.14.0 MockCoro.func was called with MockCoro instance as the first argument.
Since mock_coro (alias MockCoro) is no longer a class, it won’t hold any attributes. Code relying in mock_coro.func or mock_coro.dummy_rval needs to change.
Use list(element_copy) instead of deprecated element_copy.getchildren() in
kuha_common.document_store.mappings.xmlbase.element_strip_descendant_text()
Add py38 to tox.ini environments. The code should now be Python 3.8. compatible.
Other changes to public APIs
These are backwards compatible changes.
Add
kuha_common.testing.testcases.KuhaUnitTestCase.assert_mock_meth_has_calls()
to try to pinpoint the exact call that was missing from mocked method.Add
kuha_common.cli_setup.add_server_process_count_configuration()
which adds tornado server process count to configuration options.Change every add_* function signature in
kuha_common.cli_setup
to make parser an optional argument. If omitted the functions will use singleton stored inkuha_common.cli_setup.settings.parser
.
0.13.0 (2020-05-06)
DDI 2.5 and DDI 1.2.2 mappings use relpubl/citation/holdings/@URI and relpubl/citation/distStmt/distDate elements that map into Study.related_publications attributes distribution_date and uri.
0.12.0 (2020-04-28)
Add
related_publications
toStudy
Add DDI 2.5 and DDI 1.2.2 mappings to
Study.related_publications
0.11.2 (2020-01-24)
Fix logic in
kuha_common.document_store.field_types.ElementContainer._updates()
where secondary values were compared to other secondary values. This lead to faulty contained values, when a secondary value matched with another secondary value.
0.11.1 (2020-01-09)
Reset settings in
kuha_common.testing.testcases.load_cli_args()
if tests get skipped, since skipping won’t call tearDownClass method
0.11.0 (2020-01-08)
Initialize argumentparser using configargparse.ArgumentParser directly instead of calling
configargparse.get_arg_parser()
inkuha_common.cli_setup
. This way the settings get stored only inkuha_common.cli_setup.settings
singleton.Reset settings in
kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass()
.Fix faulty call to parent’s setUpClass in
kuha_common.testing.testcases.KuhaEndToEndTestCase.tearDownClass()
.
0.10.0 (2019-10-22)
Warning
Breaks backwards compatibility in DDI 1.2.2. and DDI 2.5. mapping of attributes for
Study.study_groups.study_group
and StudyGroup.study_group_identifier
Change mappings for DDI 1.2.2. and DDI 2.5.
Change mappings for Study.study_groups.study_group of DDI 2.5. and DDI 1.2.2. The value for the contained element is now taken from
serStmt/@ID
instead ofserName/@ID
. Breaks backwards compatibilityChange mapping for StudyGroup.study_group_identifier of DDI 2.5. and DDI 1.2.2. The value is now taken from
serStmt/@ID
instead ofserName/@ID
. Breaks backwards compatibility
Add new attributes to kuha_common.document_store.records
Study.study_groups.attr_description
Study.study_groups.attr_uri
Study.data_kinds
Study.data_collection_copyrights
Study.citation_requirements
Study.deposit_requirements
Study.geographic_coverages
StudyGroup.descriptions
StudyGroup.uris
Add mappings for new attributes
DDI 1.2.2. and DDI 2.5.:
/codeBook/stdyDscr/citation/serStmt/serInfo
maps toStudy.study_groups.attr_description
/codeBook/stdyDscr/citation/serStmt/serInfo
maps toStudyGroups.descriptions
/codeBook/stdyDscr/citation/serStmt/@URI
maps toStudy.study_groups.attr_uri
/codeBook/stdyDscr/citation/serStmt/@URI
maps toStudyGroup.uris
/codeBook/stdyDscr/citation/prodStmt/copyright
maps toStudy.data_collection_copyrights
/codeBook/stdyDscr/dataAccs/useStmt/citReq
maps toStudy.citation_requirements
/codeBook/stdyDscr/dataAccs/useStmt/deposReq
maps toStudy.deposit_requirements
/codeBook/stdyDscr/stdyInfo/sumDscr/geogCover
maps toStudy.geographic_coverages
/codeBook/stdyDscr/stdyInfo/sumDscr/dataKind
maps toStudy.data_kinds
DDI 3.1.
g:Group/g:Abstract/r:Content
maps toStudyGroups.descriptions
g:Group/@externalReferenceDefaultURI
maps toStudyGroups.uris
0.9.1 (2019-04-03)
DDI mappings:
kuha_common.document_store.mappings.xmlbase.MappedParams.has_arguments()
treat MappedParams objects which don’t have a value and all keyword arguments’ values are None as MappedParams objects that don’t have any arguments. This way all empty XML elements will be discarded in the mapping phase before they get turned into Document Store reocrds. (Fixes #26)Treat attributes-dictionaries with only None values as invalid if the Element’s value is also None in
kuha_common.document_store.field_types.Element.add_value()
.Servers exception logging
kuha_common.server.log_exception()
will now log all traceback lines, instead of the last four.
0.9.0 (2019-03-14)
Maintenance release with no new functionality besides bug fixes.
Support tornado > 5
Remove callbacks used with tornado client.
requirements.txt declares tornado==6.0.1, which is currently the latest.
kuha_common.query.QueryController._query_multiple_with_limit()
now correctly handleslimit
andskip
query parameters.kuha_common.query.QueryController.query_single()
now correctly raiseskuha_common.document_store.query.QueryException
if passed alimit
parameter.Add validation functions to
kuha_common.document_store.query
to validate query parameters.Add common testcase functionality to
kuha_common.testing.testcases.KuhaUnitTestCase
.Fix
kuha_common.document_store.client.JSONStreamClient.fetch()
so it won’t interfere with queue created bykuha_common.document_store.client.JSONStreamClient.queue_request()
.Update copyright headers to 2019.
0.8.0 (2018-12-18)
Refactor
kuha_common.server
.kuha_common.server.serve()
replaces run_server() function. It takes the web-application as a parameter and does not handle application settings. Setting up the application should be handled by the caller who instantiates the application and knows what settings the application needs in order to operate.
Add
kuha_common.testing.MockCoro
to help mocking out coroutine functions.
0.7.1 (2018-11-20)
kuha_common.document_store.mappings.xmlbase.XMLParserBase._get_study_number_from_study_unit_element()
Change priority of elements when looking up Study.study_number:a:Archive/a:ArchiveSpecific/a:Collection/a:CallNumber
a:Archive/a:ArchiveSpecific/a:Item/a:CallNumber
s:StudyUnit/r:UserID
0.7.0 (2018-11-19)
Breaks backward compatibility Refactored DDI mapping profiles’ API. Package structure of
kuha_common.document_store.mappings
changed.kuha_common.document_store.mappings.xmlbase.XMLParserBase
now mandates how subclasses should be built by providing the public API for callers. The implementation of the actual mapping of resources is an implementation detail of subclasses and does not need to be public.Refactor all DDI mapping profiles to a single module
kuha_common.document_store.mappings.ddi
Add more mapping exceptions to differentiate error conditions in
kuha_common.document_store.mappings.exceptions
Add mapping profile for DDI 3.1. to a new class
kuha_common.document_store.mapping.ddi.DDI31RecordParser
(Implements #24)
0.6.0 (2018-07-18)
Add methods to
kuha_common.testing.testcases
Add support for parsing StudyGroups from DDI 1.2.2. in
kuha_common.document_store.mappings.ddi_122_nesstar.DDI122RecordParser.p_study_groups()
(Implements #20)Omit logging of request body to request log if the body is larger than
kuha_common.server.REQUEST_LOG_BODY_MAXLEN
. (Implements #22)Fix possible JSONDecodeError in
kuha_common.server.log_request()
(Fixes #23)
0.5.1 (2018-07-11)
Declare testing.testcases.KuhaUnitTestCase.gen_id() as a classmethod, since it uses class’s method.
0.5.0 (2018-07-10)
Package for common test functions and classes
kuha_common.testing
. (Implements #12)
0.4.1 (2018-07-04)
kuha_common.document_store.mappings.xmlbase.XMLMapper._values_from_parent()
resets the state of the attribute mapper if it needs to manipulate the mapper. (Fixes #19)
0.4.0 (2018-07-03)
Relax record schema by decreasing the number of mandatory attributes. It should be possible to populate an element with attributes but have no value. (Implements #7)
kuha_common.document_store.field_types.Element.add_value()
can be called without a value-parameter.kuha_common.document_store.field_types.Value.export_dict()
returns an empty dict if value is None.kuha_common.document_store.field_types.ElementContainer._updates()
If sec_value_value is None and no match is found, create and append new sub-element.kuha_common.document_store.field_types.ElementContainer._updates()
If sec_value_value is None and a matching value and attributes is found, discard the sec_value completely. (Implements #15)Fix possible KeyError in
kuha_common.document_store.field_types.FieldAttribute.value_from_dict()
This was regression introduced by #7. (Fixes #16)kuha_common.document_store.field_types.FieldAttribute.value_from_dict()
Now always returns None if nothing was found. Previously it was possible to get an empty list.
Relax DDI-C mappings: allow record field’s value to be None, if there are attributes for the field. (Implements #8)
Add support for DDI from Nesstar Publisher. (Implements #9)
New mapping file ddi_122_nesstar.py to kuha_common.document_store.mappings package.
Refactor document_store/mappings/ddi_c.py and separate common XML classes & functions to a new module: xmlbase.py.
kuha_common.document_store.mappings.xmlbase.Value.from_element()
converts empty string values to None.
Correctly handle mapping from DDI-C if Codebook instance contains multiple series separated by ID. (Fixes #10)
Fix DDI-C mapping for localized codelists for variables. (Fixes #11)
DDI-C mapping for Question now removes extra whitepace around research_instruments.
Fix fetching multiple parent elements for mapped parameters which have no main element. (Fixes #18)
Known Issues
Mapping is unable to handle DDI-XML (DDI 2.5 and DDI 1.2.2 Nesstar) which contains inconsistent use of conceptual elements. See issue #17. For instance the following structure for anlyUnit:
<anlyUnit>Description for analysis unit. <concept>concept.of.analysis.unit</concept> </anlyUnit> <anlyUnit>Description for another analysis unit.</anlyUnit>
Will be interpreted as:
[{'analysis_unit': 'concept.of.analysis.unit', 'description': 'Description for analysis unit.', 'language': 'en'}]
0.3.1 (2018-04-19)
Add optional keyword arguments to
kuha_common.cli_setup.add_document_store_url()
that are passed to parser.add().Declare
kuha_common.cli_setup.Settings.set()
as public function.kuha_common.document_store.field_types.Value.updates()
now actually updates the value.Support query filter for record’s _id.
Use document-store-url with full document store base url for
kuha_common.document_store.query.Query
. (Fixes #5)Include request body for PUT and PATCH in
kuha_common.server.log_request()
.Add
kuha_common.document_store.records.RecordBase.updates()
.Add updates method for each subclass of
kuha_common.document_store.records.RecordBase
.Changes in
kuha_common.document_store.mappings.ddi_c
:Fix default value for universe.attr_clusion.
Introduce new class
kuha_common.document_store.mappings.ddi_c.RecordParser
which stores default language; this way parsing can be done simultaneously to multiple XML-documents.Default language is now a mandatory parameter for mapping methods:
kuha_common.document_store.mappings.ddi_c.Value.from_element()
kuha_common.document_store.mappings.ddi_c.ValueMap.__call__()
kuha_common.document_store.mappings.ddi_c.MultiValueMap.__call__()
Remove
kuha_common.document_store.mappings.ddi_c.parse()
to streamline the process.Add global to store each parser and corresponding record collection.
Declare
kuha_common.document_store.mappings.ddi_c.check_root()
as public function.Add
kuha_common.document_store.mappings.ddi_c.get_root_language()
function.
0.3.0 (2018-03-06)
Move ddi_c.py mapping module (DDI-C -> Document Store records) from kuha_document_store to kuha_common.document_store.mappings package.
Forward keyword arguments from Settings.load_parser to configargparse.get_arg_parser in cli_setup.py.
Make JSONStreamClient._get_request a public method JSONStreamClient.get_request
Forward keyword arguments from JSONStreamClient.get_request to DocumentStoreClient.streaming_query_request to support more options specifically more HTTP-methods than POST.
Assert _log_request() in server.py will not raise UnicodeDecodeError if request.body is not utf-8 encoded.
Add Study.document_uris
Add abbreviation-attribute to Study.publishers.
Add DDI-C mappings to Study.document_uris and Study.publishers.attr_abbreviation.
0.2.3 (2018-01-26)
Implement support for non-localizable containerized elements.
Add more fields to Study record.
Add more unit & integration tests.
0.2.2 (2017-11-10)
Update documentation
0.2.1 (2017-11-09)
Fix referring variables to questions. Variable may refer multiple questions.
Fix server.py log_request function. Call RequestHandler.CONTENT_TYPE_JSON, rather than handler-object.
Partial support for coroutine callbacks in QueryController
0.2.0 (2017-11-01)
Support referring variables to questions and vice versa.
Add tox.ini to support running tests with tox.
0.1.0 (2017-10-25)
Initial release