simpleml.metrics.base_metric

Module Contents

Classes

AbstractMetric	Abstract Base class for all Metric objects
Metric	Base class for all Metric objects

simpleml.metrics.base_metric.LOGGER

simpleml.metrics.base_metric.__author__ = Elisha Yadgaran

class simpleml.metrics.base_metric.AbstractMetric(name=None, has_external_files=False, author=None, project=None, version_description=None, save_patterns=None, **kwargs)

Bases: future.utils.with_metaclass()

Abstract Base class for all Metric objects

name: the metric name values: JSON object with key: value pairs for performance on test dataset

(ex: FPR: TPR to create ROC Curve) Singular value metrics take the form - {‘agg’: value}

__abstract__ = True

object_type = METRIC

values

_get_dataset_split(self, **kwargs)

Default accessor for dataset data. REFERS TO RAW DATASETS not the pipelines superimposed. That means that datasets that do not define explicit splits will have no notion of downstream splits (e.g. RandomSplitPipeline)

_get_latest_version(self)

Versions should be autoincrementing for each object (constrained over friendly name and model). Executes a database lookup and increments..

_get_pipeline_split(self, column: str, split: str, **kwargs)

For special case where dataset is the same as the model’s dataset, the dataset splits can refer to the pipeline imposed splits, not the inherent dataset’s splits. Use the pipeline split then ex: RandomSplitPipeline on NoSplitDataset evaluating “in_sample” performance

_hash(self)

Hash is the combination of the: 1) Model 2) Dataset (optional) 3) Metric 4) Config

add_dataset(self, dataset)

Setter method for dataset used

add_model(self, model)

Setter method for model used

load(self, **kwargs)

Extend main load routine to load relationship class

save(self, **kwargs)

Extend parent function with a few additional save routines

abstract score(self, **kwargs)

Abstract method for each metric to define

Should set self.values

class simpleml.metrics.base_metric.Metric(name=None, has_external_files=False, author=None, project=None, version_description=None, save_patterns=None, **kwargs)

Bases: simpleml.metrics.base_metric.AbstractMetric

Base class for all Metric objects

model_id: foreign key to the model that was used to generate predictions

TODO: Should join criteria be composite of model and dataset for multiple

duplicate metric objects computed over different test datasets?

__table_args__

__tablename__ = metrics

dataset

dataset_id

model

model_id