simpleml.save_patterns.locations.libcloud

Module for cloud save pattern definitions Uses Apache Libcloud as a universal engine

Module Contents

Classes

LibcloudCopyFileLocation
LibcloudCopyFilesLocation	Libcloud transport for many individual files
LibcloudCopyFolderLocation	Libcloud doesnt have a notion of folder objects so iterate through filepaths
LibcloudMethods	Mixin class to save/load objects via Apache Libcloud

Attributes

__author__

simpleml.save_patterns.locations.libcloud.__author__ = Elisha Yadgaran

class simpleml.save_patterns.locations.libcloud.LibcloudCopyFileLocation

Bases: simpleml.save_patterns.base.BaseSerializer

static deserialize(filepath, source_directory='libcloud_root_path', destination_directory='system_temp', **kwargs)

Parameters

• filepath (str) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

static serialize(filepath, source_directory='system_temp', destination_directory='libcloud_root_path', **kwargs)

Parameters

• filepath (str) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

class simpleml.save_patterns.locations.libcloud.LibcloudCopyFilesLocation

Bases: simpleml.save_patterns.base.BaseSerializer

Libcloud transport for many individual files

classmethod deserialize(cls, filepaths, source_directory='libcloud_root_path', destination_directory='system_temp', **kwargs)

Parameters

• filepaths (List[str]) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

static serialize(filepaths, source_directory='system_temp', destination_directory='libcloud_root_path', **kwargs)

Parameters

• filepaths (List[str]) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

class simpleml.save_patterns.locations.libcloud.LibcloudCopyFolderLocation

Bases: simpleml.save_patterns.base.BaseSerializer

Libcloud doesnt have a notion of folder objects so iterate through filepaths individually

static common_path(paths)

Helper utility to return the common parent path for a bunch of filepaths

Parameters

paths (List[str]) –

Return type

str

classmethod deserialize(cls, filepaths, source_directory='libcloud_root_path', destination_directory='system_temp', **kwargs)

Parameters

• filepaths (List[str]) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

static serialize(filepath, source_directory='system_temp', destination_directory='libcloud_root_path', **kwargs)

Parameters

• filepath (str) –

• source_directory (str) –

• destination_directory (str) –

Return type

Dict[str, str]

class simpleml.save_patterns.locations.libcloud.LibcloudMethods

Bases: object

Mixin class to save/load objects via Apache Libcloud

Generic api for all cloud providers so naming convention is extremely important to follow in the config. Please reference libcloud documentation for supported input parameters

``` [cloud] section = name of the config section to use, ex: s3

[s3] param = value –> normal key:value syntax. match these to however they are referenced later, examples: key = abc123 secret = superSecure region = us-east-1 something_specific_to_s3 = s3_parameter — Below are internally referenced SimpleML params — driver = S3 –> this must be the Apache Libcloud provider (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/types.py) connection_params = key,secret,region,something_specific_to_s3 –> this determines the key: value params passed to the constructor (it can be different for each provider) path = simpleml/specific/root –> similar to disk based home directory, cloud home directory will start relative to here container = simpleml –> the cloud bucket or container name ```

How this gets used: ``` from libcloud.storage.types import Provider from libcloud.storage.providers import get_driver

cloud_section = CONFIG.get(CLOUD_SECTION, ‘section’) connection_params = CONFIG.getlist(cloud_section, ‘connection_params’) root_path = CONFIG.get(cloud_section, ‘path’, fallback=’’)

driver_cls = get_driver(getattr(Provider, CONFIG.get(cloud_section, ‘driver’))) driver = driver_cls(**{param: CONFIG.get(cloud_section, param) for param in connection_params}) container = driver.get_container(container_name=CONFIG.get(cloud_section, ‘container’)) extra = {‘content_type’: ‘application/octet-stream’}

obj = driver.upload_object(LOCAL_FILE_PATH,

container=container, object_name=root_path + simpleml_folder_path + filename, extra=extra)

obj = driver.download_object(CLOUD_OBJECT,

destination_path=LOCAL_FILE_PATH, overwrite_existing=True, delete_on_failure=True)

```

static download(driver, source_filepath, destination_filepath, container_name)

Download any file from cloud to disk

Parameters

• source_filepath (str) –

• destination_filepath (str) –

• container_name (str) –

Return type

None

static get_container_name(config_section=None, **kwargs)

Parameters

config_section (str) –

Return type

str

classmethod get_driver(cls, provider=None, **kwargs)

Parameters

provider (str) –

Return type

Any

static get_driver_config(config_section=None, **kwargs)

Parameters

config_section (str) –

Return type

Dict[str, str]

static upload(driver, source_filepath, destination_filepath, container_name)

Upload any file from disk to cloud

Parameters

• source_filepath (str) –

• destination_filepath (str) –

• container_name (str) –

Return type

None