lomas_server.dataset_store package

Submodules

lomas_server.dataset_store.basic_dataset_store module

class lomas_server.dataset_store.basic_dataset_store.BasicDatasetStore(admin_database: AdminDatabase)[source]

Bases: DatasetStore

Basic implementation of the QuerierManager interface.

The queriers are initialized lazily and put into a dict. There is no memory management => The manager will fail if the datasets are too large to fit in memory.

dp_queriers: Dict[str, Dict[str, DPQuerier]] = {}
get_querier(dataset_name: str, query_type: str) → DPQuerier[source]

Returns the querier for the given dataset and library

Parameters:

  • dataset_name (str) – The dataset name.

  • query_type (str) – The type of DP library. One of constants.DPLibraries

Returns:

The DPQuerier for the specified dataset and library.

Return type:

DPQuerier

lomas_server.dataset_store.dataset_store module

class lomas_server.dataset_store.dataset_store.DatasetStore(admin_database: AdminDatabase)[source]

Bases: ABC

Manages the DPQueriers for the different datasets and libraries

Holds a reference to the user database in order to get information about users.

We make the _add_dataset function private to enforce lazy loading of queriers.

admin_database: AdminDatabase
abstract get_querier(dataset_name: str, library: str) → DPQuerier[source]

Returns the querier for the given dataset and library

Parameters:

  • dataset_name (str) – The dataset name.

  • library (str) – The type of DP library. One of constants.DPLibraries

Returns:

The DPQuerier for the specified dataset and library.

Return type:

DPQuerier

lomas_server.dataset_store.lru_dataset_store module

class lomas_server.dataset_store.lru_dataset_store.LRUDatasetStore(admin_database: AdminDatabase, max_memory_usage: int = 1024)[source]

Bases: DatasetStore, PrivateDatasetObserver

Implementation of the DatasetStore interface, with an LRU cache.

Subscribes to the PrivateDatasets to get notified if their memory usage changes and then clears the cache accordingly in order stay below the maximum memory usage.

dataset_cache: OrderedDict[str, PrivateDataset]
get_querier(dataset_name: str, library: str) → DPQuerier[source]

Returns the querier for the given dataset and library

Parameters:

  • dataset_name (str) – The dataset name.

  • library (str) – The type of DP library. One of constants.DPLibraries

Returns:

The DPQuerier for the specified dataset and library.

Return type:

DPQuerier

update_memory_usage() → None[source]

Remove least recently used datasets until the cache is back to below or equal to its maximum size.

lomas_server.dataset_store.private_dataset_observer module

class lomas_server.dataset_store.private_dataset_observer.PrivateDatasetObserver[source]

Bases: ABC

We use this abstract class to “subscribe” to object instances (PrivateDataset) so that they can notify instances of this abstract class (LRUDatasetStore or other DatasetStore implementing caching) when their memory usage changes.

abstract update_memory_usage() → None[source]

Abstract method to update total memory used by datasets

lomas_server.dataset_store.utils module

lomas_server.dataset_store.utils.dataset_store_factory(config: DatasetStoreConfig, admin_database: AdminDatabase) → DatasetStore[source]

Instantiates and returns the correct DatasetStore based on the config.

Parameters:
Raises:

InternalServerException – If the dataset store type from the config does not exist.

Returns:

The correct DatasetStore instance.

Return type:

DatasetStore

Module contents