siibra.retrieval

File digestion and caching.

Subpackages

• siibra.retrieval.exceptions

Submodules

• siibra.retrieval.cache
• siibra.retrieval.datasets
• siibra.retrieval.repositories
• siibra.retrieval.requests

Package Contents

Classes

EbrainsHdgConnector	Download sensitive files from EBRAINS using
EbrainsPublicDatasetConnector	Access files from public EBRAINS datasets via the Knowledge Graph v3 API.
EbrainsRequest	Implements lazy loading of HTTP Knowledge graph queries.
GitlabConnector	Base class for repository connectors.
HttpRequest
LocalFileRepository	Base class for repository connectors.
OwncloudConnector	Base class for repository connectors.
ZipfileConnector	Base class for repository connectors.
ZipfileRequest

Attributes

CACHE

exception siibra.retrieval.NoSiibraConfigMirrorsAvailableException

Common base class for all non-exit exceptions.

exception siibra.retrieval.SiibraHttpRequestError(url: str, status_code: int, msg='Cannot execute http request.')

Common base class for all non-exit exceptions.

__str__()

Return str(self).

exception siibra.retrieval.TagNotFoundException

Common base class for all non-exit exceptions.

class siibra.retrieval.EbrainsHdgConnector(dataset_id)

Download sensitive files from EBRAINS using the Human Data Gateway (HDG) via the data proxy API. Service documentation can be found here https://data-proxy.ebrains.eu/api/docs

api_version = 'v1'

Base URL for the Dataset Endpoint of the Data-Proxy API https://data-proxy.ebrains.eu/api/docs#/datasets

Supported functions by the endpoint:

• POST: Request access to the dataset.

  This is required for the other functions.

• GET: Return list of all available objects in the dataset

base_url

Limit of returned objects Default value on API side is 50 objects

maxentries = 1000

get_loader(filename, folder='', decode_func=None)

Get a lazy loader for a file, for executing the query only once loader.data is accessed.

search_files(folder='', suffix=None, recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.EbrainsPublicDatasetConnector(dataset_id: str = None, version_id: str = None, title: str = None, in_progress=False)

Access files from public EBRAINS datasets via the Knowledge Graph v3 API.

property authors

property citation

property description

property name

QUERY_ID = 'bebbe365-a0d6-41ea-9ff8-2554c15f70b7'

base_url = 'https://core.kg.ebrains.eu/v3-beta/queries/'

maxentries = 1000

get_loader(filename, folder='', decode_func=None)

Get a lazy loader for a file, for executing the query only once loader.data is accessed.

search_files(folder='', suffix=None, recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.EbrainsRequest(url, decoder=None, params={}, msg_if_not_cached=None, post=False)

Implements lazy loading of HTTP Knowledge graph queries.

property auth_headers

property kg_token

keycloak_endpoint = 'https://iam.ebrains.eu/auth/realms/hbp/protocol/openid-connect/token'

classmethod device_flow(**kwargs)

classmethod fetch_token(**kwargs)

Fetch an EBRAINS token using commandline-supplied username/password using the data proxy endpoint.

Details on how to access EBRAINS are here.

get()

Evaluate KG Token is evaluated only on execution of the request.

classmethod init_oidc()

classmethod set_token(token)

class siibra.retrieval.GitlabConnector(server: str, project: int, reftag: str, skip_branchtest=False, *, archive_mode=False)

Base class for repository connectors.

property branches

property want_commit

__eq__(other)

Return self==value.

__str__()

Return str(self).

get_loader(filename, folder='', decode_func=None)

Get a lazy loader for a file, for executing the query only once loader.data is accessed.

search_files(folder='', suffix=None, recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.HttpRequest(url: str, func: Callable = None, msg_if_not_cached: str = None, refresh=False, post=False, **kwargs)

property cached

property data

get()

class siibra.retrieval.LocalFileRepository(folder: str)

Base class for repository connectors.

class FileLoader(file_url, decode_func)

Just a loads a local file, but mimics the behaviour of cached http requests used in other connectors.

property data

__eq__(other)

Return self==value.

__str__()

Return str(self).

get_loader(filename, folder='', decode_func=None)

Get a lazy loader for a file, for loading data only once loader.data is accessed.

search_files(folder='', suffix=None, recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.OwncloudConnector(server: str, share: int)

Base class for repository connectors.

abstract search_files(folder='', suffix=None, recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.ZipfileConnector(url: str)

Base class for repository connectors.

class ZipFileLoader(zipfile, filename, decode_func)

Loads a file from the zip archive, but mimics the behaviour of cached http requests used in other connectors.

property cached

property data

property zipfile

__eq__(other)

Return self==value.

__str__()

Return str(self).

clear_cache()

get_loader(filename, folder='', decode_func=None)

Get a lazy loader for a file, for loading data only once loader.data is accessed.

search_files(folder='', suffix='', recursive=False)

Get the files within the repository.

Parameters:

• folder (str) – folder or folders in the form ‘path/to/file’

• suffix (str) –

• recursive (bool, default: False) – If True, searches files in all subfolders

Returns:

List of file names.

Return type:

List[str]

class siibra.retrieval.ZipfileRequest(url, filename, func=None, refresh=False)

get()

siibra.retrieval.CACHE