simpleml.save_patterns

Package for the different save patterns available Patterns are loaded into global registry on import and more can be added externally by decorating

Nomenclature -> Save Location : Save Format

• Database Storage

  • database_table: Dataframe saving (as tables in dedicated schema)

  • database_pickled: In database as a binary blob

  • database_hdf5: In database as a binary blob

• Local Filesystem Storage

  • disk_pickled: Pickled file on local disk

  • disk_hdf5: HDF5 file on local disk

  • disk_keras_hdf5: Keras formatted HDF5 file on local disk

• Cloud Storage

  • cloud_pickled: Pickled file on cloud backend

  • cloud_hdf5: HDF5 file on cloud backend

  • cloud_keras_hdf5: Keras formatted HDF5 file on cloud backend

  Supported Backends:

  • Amazon S3

  • Google Cloud Platform

  • Microsoft Azure

  • Microsoft Onedrive

  • Aurora

  • Backblaze B2

  • DigitalOcean Spaces

  • OpenStack Swift

  Backend is determined by cloud_section in the configuration file

• Remote filestore saving

  • SCP to remote server

Submodules

• simpleml.save_patterns.base
• simpleml.save_patterns.database
• simpleml.save_patterns.decorators
• simpleml.save_patterns.libcloud
• simpleml.save_patterns.local
• simpleml.save_patterns.onedrive

Package Contents

Classes

BaseSavePattern	Abstract base class for save patterns
SavePatternDecorators	Decorators that can be used for registering methods for loading
SavePatternMixin	Mixin class with methods for different save operations

Functions

deregister_save_pattern(cls: Optional[Type] = None, save_pattern: Optional[str] = None, save: Optional[bool] = True, load: Optional[bool] = True) → None	Deregister the class to use for saving and
register_save_pattern(cls: Type, save_pattern: Optional[str] = None, save: Optional[bool] = True, load: Optional[bool] = True, overwrite: Optional[bool] = False) → None	Register the class to use for saving and

simpleml.save_patterns.__author__ = Elisha Yadgaran

class simpleml.save_patterns.BaseSavePattern

Bases: simpleml.save_patterns.base.SavePatternMixin

Abstract base class for save patterns

abstract load(self)

The load method invoked

abstract save(self)

The save method invoked

class simpleml.save_patterns.SavePatternDecorators

Bases: object

Decorators that can be used for registering methods for loading and saving.

static deregister_save_pattern(cls_or_save_pattern: Optional[str] = None, save: Optional[bool] = True, load: Optional[bool] = True) → Callable

Class level decorator to deregister allowed save patterns. Doesnt actually make use of the class but included for completeness. Recommended to use importable deregister_save_pattern function directly

Parameters

• cls_or_save_pattern – the optional string or class denoting the pattern this class implements (e.g. disk_pickled). Checks class attribute cls.SAVE_PATTERN if null cls is automatically passed when calling decorator without parameters (@SavePatternDecorators.deregister_save_pattern)

• save – optional bool; default true; whether to drop the decorated class as the save method for the registered save pattern

• load – optional bool; default true; whether to drop the decorated class as the load method for the registered save pattern

static register_save_pattern(cls_or_save_pattern: Optional[Union[str, Type]] = None, save: Optional[bool] = True, load: Optional[bool] = True, overwrite: Optional[bool] = False) → Callable

Decorates a class to register the method(s) to use for saving and/or loading for the particular pattern

IT IS ALLOWABLE TO HAVE DIFFERENT CLASSES HANDLE SAVING AND LOADING FOR THE SAME REGISTERED PATTERN

Parameters

• cls_or_save_pattern – the optional string or class denoting the pattern this class implements (e.g. disk_pickled). Checks class attribute cls.SAVE_PATTERN if null cls is automatically passed when calling decorator without parameters (@SavePatternDecorators.register_save_pattern)

• save – optional bool; default true; whether to use the decorated class as the save method for the registered save pattern

• load – optional bool; default true; whether to use the decorated class as the load method for the registered save pattern

• overwrite – optional bool; default false; whether to overwrite the the registered class for the save pattern, if it exists. Otherwise throw an error

class simpleml.save_patterns.SavePatternMixin

Bases: object

Mixin class with methods for different save operations

static df_to_sql(engine, df: pandas.DataFrame, table: str, dtype: Optional[Dict[str, str]] = None, schema: str = 'public', if_exists: str = 'replace', sep: str = '|', encoding: str = 'utf8', index: bool = False) → None

Utility to bulk insert pandas dataframe via copy from

Parameters

• df – dataframe to insert

• table – destination table

• dtype – column schema of destination table

• schema – destination schema

• if_exists – what to do if destination table exists; valid inputs are:

[replace, append, fail] :param sep: separator key between cells :param encoding: character encoding to use :param index: whether to output index with data

static hickle_object(obj: Any, filepath: str, overwrite: bool = True, root_directory: str = HDF5_FILESTORE_DIRECTORY) → None

Serializes an object to the filesystem in HDF5 format.

Prepends path to SimpleML HDF5 directory before saving. ONLY pass in a relative filepath from that location

Parameters

overwrite – Boolean indicating whether to first check if HDF5 object is already serialized. Defaults to not checking, but can be leverage by implementations that want the same artifact in multiple places

static load_hickled_object(filepath: str, root_directory: str = HDF5_FILESTORE_DIRECTORY) → Any

Loads an object from the filesystem.

Prepends path to SimpleML HDF5 directory before loading. ONLY pass in a relative filepath from that location

static load_keras_object(filepath: str, root_directory: str = HDF5_FILESTORE_DIRECTORY) → Any

Loads a Keras object from the filesystem.

Prepends path to SimpleML HDF5 directory before loading. ONLY pass in a relative filepath from that location

static load_pickled_object(filepath: str, stream: bool = False, root_directory: str = PICKLED_FILESTORE_DIRECTORY) → Any

Loads an object from a serialized string or filesystem. When stream is True, it tries to load the file directly from the string.

Prepends path to SimpleML Pickle directory before loading. ONLY pass in a relative filepath from that location

static load_sql(query: str, connection, **kwargs) → pandas.DataFrame

Helper method to read in sql data

static pickle_object(obj: Any, filepath: Optional[str] = None, overwrite: bool = True, root_directory: str = PICKLED_FILESTORE_DIRECTORY) → Union[str, None]

Pickles an object to a string or to the filesystem. Assumes that a NULL filepath expects a serialized string returned

Prepends path to SimpleML Pickle directory before saving. ONLY pass in a relative filepath from that location

Parameters

overwrite – Boolean indicating whether to first check if pickled object is already serialized. Defaults to not checking, but can be leverage by implementations that want the same artifact in multiple places

static save_keras_object(obj: Any, filepath: str, overwrite: bool = True, root_directory: str = HDF5_FILESTORE_DIRECTORY) → None

Serializes an object to the filesystem in Keras HDF5 format.

Prepends path to SimpleML HDF5 directory before saving. ONLY pass in a relative filepath from that location

Parameters

overwrite – Boolean indicating whether to first check if HDF5 object is already serialized. Defaults to not checking, but can be leverage by implementations that want the same artifact in multiple places

simpleml.save_patterns.deregister_save_pattern(cls: Optional[Type] = None, save_pattern: Optional[str] = None, save: Optional[bool] = True, load: Optional[bool] = True) → None

Deregister the class to use for saving and loading for the particular pattern

Parameters

• save_pattern – the optional string denoting the pattern this class implements (e.g. disk_pickled). Checks class attribute cls.SAVE_PATTERN if null

• save – optional bool; default true; whether to remove the class as the save method for the registered save pattern

• load – optional bool; default true; whether to remove the class as the load method for the registered save pattern

simpleml.save_patterns.register_save_pattern(cls: Type, save_pattern: Optional[str] = None, save: Optional[bool] = True, load: Optional[bool] = True, overwrite: Optional[bool] = False) → None

Register the class to use for saving and loading for the particular pattern

IT IS ALLOWABLE TO HAVE DIFFERENT CLASSES HANDLE SAVING AND LOADING FOR THE SAME REGISTERED PATTERN

Parameters

• save_pattern – the optional string denoting the pattern this class implements (e.g. disk_pickled). Checks class attribute cls.SAVE_PATTERN if null

• save – optional bool; default true; whether to use the decorated class as the save method for the registered save pattern

• load – optional bool; default true; whether to use the decorated class as the load method for the registered save pattern

• overwrite – optional bool; default false; whether to overwrite the the registered class for the save pattern, if it exists. Otherwise throw an error