simpleml.orm.initialization

Util module to initialize SimpleML and configure database management

Module Contents

Classes

AlembicDatabase

Base database class to manage dbs with schema tracking. Includes alembic

BaseDatabase

Base Database class to configure db connection

BinaryStorageDatabase

Hardcoded database mapped to binary storage metadata

Database

SimpleML specific configuration to interact with the database

DatasetDatabase

Hardcoded database mapped to dataset storage metadata

Attributes

DATABASE_CONF

DATABASE_DRIVERNAME

DATABASE_HOST

DATABASE_NAME

DATABASE_PASSWORD

DATABASE_PORT

DATABASE_QUERY

DATABASE_URI

DATABASE_USERNAME

LOGGER

__author__

simpleml.orm.initialization.DATABASE_CONF[source]
simpleml.orm.initialization.DATABASE_DRIVERNAME[source]
simpleml.orm.initialization.DATABASE_HOST[source]
simpleml.orm.initialization.DATABASE_NAME[source]
simpleml.orm.initialization.DATABASE_PASSWORD[source]
simpleml.orm.initialization.DATABASE_PORT[source]
simpleml.orm.initialization.DATABASE_QUERY[source]
simpleml.orm.initialization.DATABASE_URI[source]
simpleml.orm.initialization.DATABASE_USERNAME[source]
simpleml.orm.initialization.LOGGER[source]
simpleml.orm.initialization.__author__ = Elisha Yadgaran[source]
class simpleml.orm.initialization.AlembicDatabase(alembic_filepath, script_location='migrations', *args, **kwargs)[source]

Bases: BaseDatabase

Base database class to manage dbs with schema tracking. Includes alembic config references

Parameters

  • use_ssh_tunnel – boolean - default false. Whether to tunnel sqlalchemy connection through an ssh tunnel or not

  • sshtunnel_params – Dict of ssh params - specify according to sshtunnel project https://github.com/pahaz/sshtunnel/ - direct passthrough

property alembic_config(self)[source]
create_tables(self, base, drop_tables=False, ignore_errors=False)[source]

Creates database tables (and potentially drops existing ones). Assumes to be running under a sufficiently privileged user

Parameters

drop_tables – Whether or not to drop the existing tables first.

Returns

None

downgrade(self, revision)[source]

Proxy Method to invoke alembic downgrade command to specified revision Indirectly runs the alembic env.py code

initialize(self, base_list, upgrade=False, validate=True, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Raises a SimpleML error if database schema is not up to date

Parameters

  • drop_tables – Bool, whether to drop existing tables in database

  • upgrade – Bool, whether to run an upgrade migration after establishing a connection

Returns

None

upgrade(self, revision='head')[source]

Proxy Method to invoke alembic upgrade command to specified revision Indirectly runs the alembic env.py code

validate_schema_version(self, base_list)[source]

Check that the newly initialized database is up-to-date Raises an error otherwise (ahead of any table model mismatches later)

class simpleml.orm.initialization.BaseDatabase(config=None, configuration_section=None, uri=None, use_ssh_tunnel=False, sshtunnel_params=None, **credentials)[source]

Bases: object

Base Database class to configure db connection Does not assume schema tracking or any other validation

Starting in sqlalchemy 1.4.2, the signature of sqlalchemy.engine.url.URL has changed to an immutable object without an __init__

Parameters

  • use_ssh_tunnel (bool) – boolean - default false. Whether to tunnel sqlalchemy connection through an ssh tunnel or not

  • sshtunnel_params (Optional[Dict[str, Any]]) – Dict of ssh params - specify according to sshtunnel project https://github.com/pahaz/sshtunnel/ - direct passthrough

  • config (Optional[Dict[str, Any]]) –

  • configuration_section (Optional[str]) –

  • uri (Optional[str]) –

__repr__(self)[source]

Return repr(self).

__str__(self)[source]

Return str(self).

_initialize(self, base, create_tables=False, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Parameters

  • create_tables – Bool, whether to create tables in database

  • drop_tables – Bool, whether to drop existing tables in database

Returns

None

close_tunnel(self)[source]
Return type

None

configure_ssh_tunnel(self, credentials, ssh_config)[source]
Parameters

  • credentials (Dict[str, Any]) –

  • ssh_config (Dict[str, Any]) –

Return type

Tuple[Dict[str, Any], Dict[str, Any]]

create_tables(self, base, drop_tables=False, ignore_errors=False)[source]

Creates database tables (and potentially drops existing ones). Assumes to be running under a sufficiently privileged user

Parameters

  • drop_tables (bool) – Whether or not to drop the existing tables first.

  • ignore_errors (bool) –

Returns

None

Return type

None

property engine(self)[source]
Return type

Any

initialize(self, base_list, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Raises a SimpleML error if database schema is not up to date

Parameters

  • drop_tables – Bool, whether to drop existing tables in database

  • upgrade – Bool, whether to run an upgrade migration after establishing a connection

Returns

None

open_tunnel(self)[source]
Return type

None

property ssh_tunnel(self)[source]
Return type

simpleml.imports.SSHTunnelForwarder

class simpleml.orm.initialization.BinaryStorageDatabase(config=None, configuration_section=None, uri=None, use_ssh_tunnel=False, sshtunnel_params=None, **credentials)[source]

Bases: BaseDatabase

Hardcoded database mapped to binary storage metadata

Parameters

  • use_ssh_tunnel (bool) – boolean - default false. Whether to tunnel sqlalchemy connection through an ssh tunnel or not

  • sshtunnel_params (Optional[Dict[str, Any]]) – Dict of ssh params - specify according to sshtunnel project https://github.com/pahaz/sshtunnel/ - direct passthrough

  • config (Optional[Dict[str, Any]]) –

  • configuration_section (Optional[str]) –

  • uri (Optional[str]) –

initialize(self, base_list=None, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Raises a SimpleML error if database schema is not up to date

Parameters

  • drop_tables – Bool, whether to drop existing tables in database

  • upgrade – Bool, whether to run an upgrade migration after establishing a connection

Returns

None

class simpleml.orm.initialization.Database(configuration_section=None, uri=None, database=None, username=None, password=None, drivername=None, host=None, port=None, query=None, *args, **kwargs)[source]

Bases: AlembicDatabase

SimpleML specific configuration to interact with the database Defaults to sqlite db in filestore directory

Parameters

  • use_ssh_tunnel – boolean - default false. Whether to tunnel sqlalchemy connection through an ssh tunnel or not

  • sshtunnel_params – Dict of ssh params - specify according to sshtunnel project https://github.com/pahaz/sshtunnel/ - direct passthrough

initialize(self, base_list=None, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Raises a SimpleML error if database schema is not up to date

Parameters

  • drop_tables – Bool, whether to drop existing tables in database

  • upgrade – Bool, whether to run an upgrade migration after establishing a connection

Returns

None

class simpleml.orm.initialization.DatasetDatabase(config=None, configuration_section=None, uri=None, use_ssh_tunnel=False, sshtunnel_params=None, **credentials)[source]

Bases: BaseDatabase

Hardcoded database mapped to dataset storage metadata

Parameters

  • use_ssh_tunnel (bool) – boolean - default false. Whether to tunnel sqlalchemy connection through an ssh tunnel or not

  • sshtunnel_params (Optional[Dict[str, Any]]) – Dict of ssh params - specify according to sshtunnel project https://github.com/pahaz/sshtunnel/ - direct passthrough

  • config (Optional[Dict[str, Any]]) –

  • configuration_section (Optional[str]) –

  • uri (Optional[str]) –

initialize(self, base_list=None, **kwargs)[source]

Initialization method to set up database connection and inject session manager

Raises a SimpleML error if database schema is not up to date

Parameters

  • drop_tables – Bool, whether to drop existing tables in database

  • upgrade – Bool, whether to run an upgrade migration after establishing a connection

Returns

None