1.3. igsn_lib.oai¶
Methods
|
Create a Sickle instance |
|
Call the OAI-PMH Identify operation on the provided service URL. |
|
|
|
List the sets reported by an OAI-PMH service. |
|
|
|
Converts an OAI-PMH IGSN metadata record to a dict |
|
Determine the number of records that match an OAI-PMH ListRecords request. |
Detail
Methods in support of OAI-PMH harvesting of IGSN records.
-
class
igsn_lib.oai.DatetimeEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]¶ Bases:
json.encoder.JSONEncoder-
default(obj)[source]¶ Implement this method in a subclass such that it returns a serializable object for
o, or calls the base implementation (to raise aTypeError).For example, to support arbitrary iterators, you could implement default like this:
def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o)
-
-
igsn_lib.oai.IGSN_OAI_NAMESPACES_INV= {'dc': 'http://purl.org/dc/elements/1.1/', 'igsn': 'http://igsn.org/schema/kernel-v.1.0', 'igsn_desc': 'http://schema.igsn.org/description/1.0', 'oai': 'http://www.openarchives.org/OAI/2.0/', 'oai_dc': 'http://www.openarchives.org/OAI/2.0/oai_dc/', 'oai_identifier': 'http://www.openarchives.org/OAI/2.0/oai-identifier', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}¶ Metadata namespaces commonly seen in IGSN OAI-PMH responses
-
igsn_lib.oai.getSickle(url)[source]¶ Create a Sickle instance
- Parameters:
url – OAI-PMH service URL
- Returns:
sickle.Sickle instance
-
igsn_lib.oai.identify(url)[source]¶ Call the OAI-PMH Identify operation on the provided service URL.
- Parameters:
url (str) – OAI-PMH service URL
- Returns:
sickle.Identify object
Examples
import igsn_lib.oai import xmltodict import json url = "https://doidb.wdc-terra.org/igsnoaip/oai" res = igsn_lib.oai.identify(url) res_dict = xmltodict.parse( res.raw, process_namespaces=True, namespaces=igsn_lib.oai.IGSN_OAI_NAMESPACES ) print(json.dumps(res_dict, indent=2))
{ "oai:Identify": { "oai:repositoryName": "IGSN Registry Metadata Store", "oai:baseURL": "http://doidb.wdc-terra.org/igsnoaip/oai", "oai:protocolVersion": "2.0", "oai:adminEmail": "[email protected]", "oai:earliestDatestamp": "2011-01-01T00:00:00Z", "oai:deletedRecord": "persistent", "oai:granularity": "YYYY-MM-DDThh:mm:ssZ", "oai:compression": [ "gzip", "deflate" ], "oai:description": { "oai_identifier:oai-identifier": { "@xsi:schemaLocation": "http://www.openarchives.org/OAI/2.0/oai-identifier http://www.openarchives.org/OAI/2.0/oai-identifier.xsd", "@xmlns": { "": "http://www.openarchives.org/OAI/2.0/oai-identifier", "xsi": "http://www.w3.org/2001/XMLSchema-instance" }, "oai_identifier:scheme": "oai", "oai_identifier:repositoryIdentifier": "registry.igsn.org", "oai_identifier:delimiter": ":", "oai_identifier:sampleIdentifier": "oai:registry.igsn.org:12425" } } } }
-
igsn_lib.oai.listRecords(svc, metadata_prefix='igsn', ignore_deleted=False, set_spec=None, tfrom=None, tuntil=None)[source]¶
-
igsn_lib.oai.listSets(svc, get_counts=False)[source]¶ List the sets reported by an OAI-PMH service.
- Parameters:
svc (Sickle) – Initialized instance of Sickle
get_counts (boolean) – Return record counts with set
- Returns:
list of
{setSpec:, setName:, count:}- Return type:
list
Examples
import igsn_lib.oai svc_url = "https://doidb.wdc-terra.org/igsnoaip/oai" svc = igsn_lib.oai.getSickle(svc_url) sets = igsn_lib.oai.listSets(svc, get_counts=True) for s in sets: print(f"{s['count']:10} {s['setSpec']}: {s['setName']}")
0 REFQUALITY: Reference quality citations only. 0 ANDS.REFQUALITY: Australian National Data Service (reference quality citations only) 0 ANDS.AUSCOPE.REFQUALITY: AuScope (reference quality citations only) 0 CNRS.CNRS.REFQUALITY: Centre national de la recherche scientifique (reference quality citations only) 0 CSIRO.REFQUALITY: CSIRO (reference quality citations only) 0 CNRS.REFQUALITY: Centre national de la recherche scientifique (reference quality citations only) 0 CSIRO.CSIRO.REFQUALITY: CSIRO (reference quality citations only) 0 GEOAUS.AU.REFQUALITY: Geoscience Australia (reference quality citations only) 0 GEOAUS.REFQUALITY: Geoscience Australia (reference quality citations only) 0 GFZ.REFQUALITY: GFZ Allocator (reference quality citations only) 33248 CSIRO.CSIRO: CSIRO 33248 CSIRO: CSIRO 0 GFZ.GFZ.REFQUALITY: Deutsches GeoForschungsZentrum GFZ (reference quality citations only) 4124 CNRS.CNRS: Centre national de la recherche scientifique 4124 CNRS: Centre national de la recherche scientifique 3202 ANDS.AUSCOPE: AuScope 3202 ANDS: Australian National Data Service 0 IEDA.REFQUALITY: Integrated Earth Data Applications (reference quality citations only) 5264862 GEOAUS: Geoscience Australia 0 IEDA.SESAR.REFQUALITY: System for Earth Sample Registration (reference quality citations only) 0 IFREMER.REFQUALITY: Institut français de recherche pour l'exploitation de la mer (reference quality citations only) 0 IFREMER.IGSN.REFQUALITY: Institut français de recherche pour l'exploitation de la mer (reference quality citations only) 11304 GFZ: GFZ Allocator 0 KIGAM.REFQUALITY: Korea Institute of Geoscience and Mineral Resources (reference quality citations only) 0 KIGAM.DC.REFQUALITY: Korea Institute of Geoscience and Mineral Resources (reference quality citations only) 11304 GFZ.GFZ: Deutsches GeoForschungsZentrum GFZ 5264862 GEOAUS.AU: Geoscience Australia 42 LITHODAT: Lithodat Pty Ltd. 0 LITHODAT.REFQUALITY: Lithodat Pty Ltd. (reference quality citations only) 0 LITHODAT.AG.REFQUALITY: AuScope Geochemistry Network (reference quality citations only) 37 LITHODAT.AG: AuScope Geochemistry Network 5 LITHODAT.LD: LITHODAT PTY LTD 0 LITHODAT.LD.REFQUALITY: LITHODAT PTY LTD (reference quality citations only) 0 MARUM.REFQUALITY: MARUM Center for Marine Environmental Studies at University of Bremen (reference quality citations only) 0 MARUM.HB.REFQUALITY: MARUM Center for Marine Environmental Sciences (reference quality citations only) 0 UKI.REFQUALITY: Universität Kiel (reference quality citations only) 1 UKI.BOT: Botanisches Institut 0 UKI.BOT.REFQUALITY: Botanisches Institut (reference quality citations only) 411 UKI.GEOMAR: GEOMAR 0 UKI.GEOMAR.REFQUALITY: GEOMAR (reference quality citations only) 37480 IFREMER: Institut français de recherche pour l'exploitation de la mer 2884 KIGAM.DC: Korea Institute of Geoscience and Mineral Resources 0 UKI.RZ.REFQUALITY: Universität Kiel - Rechenzentrum (reference quality citations only) 2884 KIGAM: Korea Institute of Geoscience and Mineral Resources 37480 IFREMER.IGSN: Institut français de recherche pour l'exploitation de la mer 4709054 IEDA: Integrated Earth Data Applications 4709054 IEDA.SESAR: System for Earth Sample Registration 136587 MARUM: MARUM Center for Marine Environmental Studies at University of Bremen 136587 MARUM.HB: MARUM Center for Marine Environmental Sciences 412 UKI: Universität Kiel 0 UKI.RZ: Universität Kiel - Rechenzentrum
-
igsn_lib.oai.oaiRecordToDict(xml_string)[source]¶ Converts an OAI-PMH IGSN metadata record to a dict
The IGSN schema is at https://doidb.wdc-terra.org/igsn/schemas/igsn.org/schema/1.0/igsn.xsd
Times are returned as timezone aware python datetime, TZ=UTC.
- Parameters:
raw_record – OAI-PMH record XML in IGSN format
- Returns:
dict or None on failure
Example
import pprint import igsn_lib.oai xml = """<?xml version="1.0"?> <record xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <header> <identifier>oai:registry.igsn.org:6940929</identifier> <datestamp>2019-10-15T06:00:10Z</datestamp> <setSpec>IEDA</setSpec> <setSpec>IEDA.SESAR</setSpec> </header> <metadata> <sample xmlns="http://igsn.org/schema/kernel-v.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://igsn.org/schema/kernel-v.1.0 http://doidb.wdc-terra.org/igsn/schemas/igsn.org/schema/1.0/igsn.xsd"> <sampleNumber identifierType="igsn">10273/BSU0005JF</sampleNumber> <registrant> <registrantName>IEDA</registrantName> </registrant> <log> <logElement event="submitted" timeStamp="2019-10-15T04:00:09Z"/> </log> </sample> </metadata> </record> """ data = igsn_lib.oai.oaiRecordToDict(xml) pprint.pprint(data, indent=2)
{ '_source': OrderedDict([ ( 'oai:record', OrderedDict([ ( 'oai:header', OrderedDict([ ( 'oai:identifier', 'oai:registry.igsn.org:6940929'), ( 'oai:datestamp', '2019-10-15T06:00:10Z'), ( 'oai:setSpec', [ 'IEDA', 'IEDA.SESAR'])])), ( 'oai:metadata', OrderedDict([ ( 'igsn:sample', OrderedDict([ ( '@xsi:schemaLocation', 'http://igsn.org/schema/kernel-v.1.0 ' 'http://doidb.wdc-terra.org/igsn/schemas/igsn.org/schema/1.0/igsn.xsd'), ( '@xmlns', OrderedDict([ ( '', 'http://igsn.org/schema/kernel-v.1.0'), ( 'xsi', 'http://www.w3.org/2001/XMLSchema-instance')])), ( 'igsn:sampleNumber', OrderedDict([ ( '@identifierType', 'igsn'), ( '#text', '10273/BSU0005JF')])), ( 'igsn:registrant', OrderedDict([ ( 'igsn:registrantName', 'IEDA')])), ( 'igsn:log', OrderedDict([ ( 'igsn:logElement', OrderedDict([ ( '@event', 'submitted'), ( '@timeStamp', '2019-10-15T04:00:09Z')]))]))]))]))]))]), 'igsn_id': 'BSU0005JF', 'igsn_time': datetime.datetime(2019, 10, 15, 4, 0, 9, tzinfo=<StaticTzInfo 'UTC\+00:00'>), 'log': [{'event': 'submitted', 'time': '2019-10-15T04:00:09+0000'}], 'oai_id': 'oai:registry.igsn.org:6940929', 'oai_time': datetime.datetime(2019, 10, 15, 6, 0, 10, tzinfo=<StaticTzInfo 'UTC\+00:00'>), 'registrant': 'IEDA', 'related': [], 'set_spec': ['IEDA', 'IEDA.SESAR']}
-
igsn_lib.oai.recordCount(svc, metadata_prefix='igsn', ignore_deleted=False, set_spec=None, tfrom=None, tuntil=None)[source]¶ Determine the number of records that match an OAI-PMH ListRecords request.
- Parameters:
svc (Sickle) – Initialized instance of Sickle
metadata_prefix (str) – Metadata prefix to use (igsn)
ignore_deleted (bool) – Ignore deleted records in the request
setSpec (str) – Optional set name to limit records
tfrom (str or datetime) – Optional representation of time for the earliest record (inclusive)
tuntil (str or datetime) – Optional representation of time for the latest record (inclusive)
- Returns:
Number of records matching the specified subset.
- Return type:
int
Examples
import igsn_lib.oai svc_url = "https://doidb.wdc-terra.org/igsnoaip/oai" svc = igsn_lib.oai.getSickle(svc_url) count = igsn_lib.oai.recordCount( svc, set_spec='IEDA', tfrom='2020-01-01', tuntil='2020-02-01' ) print(f"Matching records = {count}")
Matching records = 2633