simpleml.save_patterns.libcloud

Module for cloud save pattern definitions Uses Apache Libcloud as a universal engine

Module Contents

Classes

CloudBase	Implementation class for extended base save patterns
CloudHDF5SavePattern	Save pattern implementation to save objects to Cloud in HDF5 format
CloudKerasHDF5SavePattern	Save pattern implementation to save objects to Cloud in Keras HDF5 format
CloudPickleSavePattern	Save pattern implementation to save objects to Cloud in pickled format
CloudSavePatternMixin	Mixin class to save/load objects via Apache Libcloud

simpleml.save_patterns.libcloud.__author__ = Elisha Yadgaran

class simpleml.save_patterns.libcloud.CloudBase

Bases: simpleml.save_patterns.base.BaseSavePattern, simpleml.save_patterns.libcloud.CloudSavePatternMixin

Implementation class for extended base save patterns to/from the cloud via Apache Libcloud

class simpleml.save_patterns.libcloud.CloudHDF5SavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in HDF5 format

SAVE_PATTERN = cloud_hdf5

classmethod load(cls, filename: str, **kwargs) → Any

Download HDF5 file from cloud to disk Then load files from disk in HDF5 format

classmethod save(cls, obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in HDF5 format Then upload HDF5 file from disk to cloud

class simpleml.save_patterns.libcloud.CloudKerasHDF5SavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in Keras HDF5 format

SAVE_PATTERN = cloud_keras_hdf5

classmethod load(cls, filename: str, **kwargs) → Any

Download HDF5 file from cloud to disk Then load files from disk in HDF5 format

classmethod save(cls, obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in Keras HDF5 format Then upload HDF5 file from disk to cloud

class simpleml.save_patterns.libcloud.CloudPickleSavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in pickled format

SAVE_PATTERN = cloud_pickled

classmethod load(cls, filename: str, **kwargs) → Any

Download pickled file from cloud to disk Then load files from disk in pickled format

classmethod save(cls, obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in pickled format Then upload pickled file from disk to cloud

class simpleml.save_patterns.libcloud.CloudSavePatternMixin

Bases: object

Mixin class to save/load objects via Apache Libcloud

Generic api for all cloud providers so naming convention is extremely important to follow in the config. Please reference libcloud documentation for supported input parameters

``` [cloud] section = name of the config section to use, ex: s3

[s3] param = value –> normal key:value syntax. match these to however they are referenced later, examples: key = abc123 secret = superSecure region = us-east-1 something_specific_to_s3 = s3_parameter — Below are internally referenced SimpleML params — driver = S3 –> this must be the Apache Libcloud provider (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/types.py) connection_params = key,secret,region,something_specific_to_s3 –> this determines the key: value params passed to the constructor (it can be different for each provider) path = simpleml/specific/root –> similar to disk based home directory, cloud home directory will start relative to here container = simpleml –> the cloud bucket or container name ```

How this gets used: ``` from libcloud.storage.types import Provider from libcloud.storage.providers import get_driver

cloud_section = CONFIG.get(CLOUD_SECTION, ‘section’) connection_params = CONFIG.getlist(cloud_section, ‘connection_params’) root_path = CONFIG.get(cloud_section, ‘path’, fallback=’’)

driver_cls = get_driver(getattr(Provider, CONFIG.get(cloud_section, ‘driver’))) driver = driver_cls(**{param: CONFIG.get(cloud_section, param) for param in connection_params}) container = driver.get_container(container_name=CONFIG.get(cloud_section, ‘container’)) extra = {‘content_type’: ‘application/octet-stream’}

obj = driver.upload_object(LOCAL_FILE_PATH,

container=container, object_name=root_path + simpleml_folder_path + filename, extra=extra)

obj = driver.download_object(CLOUD_OBJECT,

destination_path=LOCAL_FILE_PATH, overwrite_existing=True, delete_on_failure=True)

```

CLOUD_DRIVER

download_from_cloud(self, folder: str, filename: str) → None

Download any file from cloud to disk

property driver(self)

“classproperty” to return and optionally globally set the cloud provider

classmethod reset_driver(cls)

Convenience method to set parsed driver back to None. Forces a config reread on next invocation

upload_to_cloud(self, folder: str, filename: str) → None

Upload any file from disk to cloud