simpleml.metrics.classification module¶
Module for classification metrics https://en.wikipedia.org/wiki/Confusion_matrix
- Includes base class and derived metrics following the nomenclature:
- ConstraintValueMetric
- Where:
- Constraint is the lookup criteria (ex FPR in ROC curve) Value is desired value (ex TPR in ROC curve)
- This module is organized by metric and prediction dependencies:
- Base classes with methods and utilities
- Aggregate Metrics (single value output)
- 2a) Single values computed via Predict method (operating points) 2b) Single values computed via proba method (agg over curve)
- Curve Metrics (constraint: value)
- 3a) Threshold: confusion matrix metrics 3b) confusion matrix metrics: threshold or other metrics
-
class
simpleml.metrics.classification.
AccuracyMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
BinaryClassificationMetric
(dataset_split='TRAIN', **kwargs)[source]¶ Bases:
simpleml.metrics.classification.ClassificationMetric
-
confusion_matrix
¶ Property method to return (or generate) dataframe of confusion matrix at each threshold
-
created_timestamp
¶
-
static
dedupe_curve
(keys, values, maximize=True, round_places=3)[source]¶ Method to deduplicate multiple values for the same key on a curve (ex multiple thresholds with the same fpr and different tpr for roc)
Parameters: maximize – Boolean, whether to choose the maximum value for each unique key or the minimum
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
predictions
¶
-
probabilities
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ClassificationMetric
(name=None, has_external_files=False, author=None, project=None, version_description=None, save_method='disk_pickled', **kwargs)[source]¶ Bases:
simpleml.metrics.base_metric.BaseMetric
TODO: Figure out multiclass generalizations
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
labels
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
predictions
¶
-
probabilities
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
F1ScoreMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
FprMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
FprTprMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
RocAucMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdAccuracyMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdF1ScoreMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdFdrMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdFnrMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdForMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdFprMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdInformednessMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdMarkednessMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdMccMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdNpvMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdPpvMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdTnrMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
ThresholdTprMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-
-
class
simpleml.metrics.classification.
TprMetric
(**kwargs)[source]¶ Bases:
simpleml.metrics.classification.BinaryClassificationMetric
-
created_timestamp
¶
-
filepaths
¶
-
has_external_files
¶
-
hash_
¶
-
id
¶
-
metadata_
¶
-
model
¶
-
model_id
¶
-
modified_timestamp
¶
-
name
¶
-
project
¶
-
registered_name
¶
-
values
¶
-
version
¶
-
version_description
¶
-