simpleml.save_patterns.libcloud module

Module for cloud save pattern definitions Uses Apache Libcloud as a universal engine

class simpleml.save_patterns.libcloud.CloudBase

Bases: simpleml.save_patterns.base.BaseSavePattern, simpleml.save_patterns.libcloud.CloudSavePatternMixin

Implementation class for extended base save patterns to/from the cloud via Apache Libcloud

class simpleml.save_patterns.libcloud.CloudHDF5SavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in HDF5 format

SAVE_PATTERN = 'cloud_hdf5'

classmethod load(filename: str, **kwargs) → Any

Download HDF5 file from cloud to disk Then load files from disk in HDF5 format

classmethod save(obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in HDF5 format Then upload HDF5 file from disk to cloud

class simpleml.save_patterns.libcloud.CloudKerasHDF5SavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in Keras HDF5 format

SAVE_PATTERN = 'cloud_keras_hdf5'

classmethod load(filename: str, **kwargs) → Any

Download HDF5 file from cloud to disk Then load files from disk in HDF5 format

classmethod save(obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in Keras HDF5 format Then upload HDF5 file from disk to cloud

class simpleml.save_patterns.libcloud.CloudPickleSavePattern

Bases: simpleml.save_patterns.libcloud.CloudBase

Save pattern implementation to save objects to Cloud in pickled format

SAVE_PATTERN = 'cloud_pickled'

classmethod load(filename: str, **kwargs) → Any

Download pickled file from cloud to disk Then load files from disk in pickled format

classmethod save(obj: Any, persistable_id: str, **kwargs) → str

Save method to save files to disk in pickled format Then upload pickled file from disk to cloud

class simpleml.save_patterns.libcloud.CloudSavePatternMixin

Bases: object

Mixin class to save/load objects via Apache Libcloud

Generic api for all cloud providers so naming convention is extremely important to follow in the config. Please reference libcloud documentation for supported input parameters

``` [cloud] section = name of the config section to use, ex: s3

[s3] param = value –> normal key:value syntax. match these to however they are referenced later, examples: key = abc123 secret = superSecure region = us-east-1 something_specific_to_s3 = s3_parameter — Below are internally referenced SimpleML params — driver = S3 –> this must be the Apache Libcloud provider (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/types.py) connection_params = key,secret,region,something_specific_to_s3 –> this determines the key: value params passed to the constructor (it can be different for each provider) path = simpleml/specific/root –> similar to disk based home directory, cloud home directory will start relative to here container = simpleml –> the cloud bucket or container name ```

How this gets used: ``` from libcloud.storage.types import Provider from libcloud.storage.providers import get_driver

cloud_section = CONFIG.get(CLOUD_SECTION, ‘section’) connection_params = CONFIG.getlist(cloud_section, ‘connection_params’) root_path = CONFIG.get(cloud_section, ‘path’, fallback=’’)

driver_cls = get_driver(getattr(Provider, CONFIG.get(cloud_section, ‘driver’))) driver = driver_cls(**{param: CONFIG.get(cloud_section, param) for param in connection_params}) container = driver.get_container(container_name=CONFIG.get(cloud_section, ‘container’)) extra = {‘content_type’: ‘application/octet-stream’}

obj = driver.upload_object(LOCAL_FILE_PATH,

container=container, object_name=root_path + simpleml_folder_path + filename, extra=extra)

obj = driver.download_object(CLOUD_OBJECT,

destination_path=LOCAL_FILE_PATH, overwrite_existing=True, delete_on_failure=True)

```

CLOUD_DRIVER = None

download_from_cloud(folder: str, filename: str) → None

Download any file from cloud to disk

driver

“classproperty” to return and optionally globally set the cloud provider

classmethod reset_driver()

Convenience method to set parsed driver back to None. Forces a config reread on next invocation

upload_to_cloud(folder: str, filename: str) → None

Upload any file from disk to cloud