Skip to main content

metrics

Defines functions which compute metrics.

Module​

Functions​

def sum_fpr_curve(    test_target: List[float],    test_pred: List[float],    test_var_to_sum: List[float],    test_var_margin: Optional[List[float]] = None,    granularity: int = 100,) ‑> Tuple[List[float], List[float]]:

Returns the false positive rate to sums of a given variable.

Returns this sum as a tuple of arrays (fpr, sums).

This is calculated by moving a threshold that is used for deciding to accept/reject a given entry and then summing the associated variable. This can be used, for example, to calculate the total profit that would be achieved according to different false positive rates.

Granularity is the number of regions to have that split up the threshold space.

Classes​

ClassificationMetric​

class ClassificationMetric(    func: Callable[[np.ndarray, np.ndarray], float], probabilities: bool,):

A classification metric used for assessing ML model performance.

Arguments

  • probabilities: Whether y_pred needs to be classes or probabilities.

Ancestors​

Variables​

  • static probabilities : bool

Metric​

class Metric(func: Callable[[np.ndarray, np.ndarray], float]):

A metric used for assessing ML model performance.

Attributes

  • func: A function which computes the metric. Must take two arguments: y_true and y_pred and return a metric as a float.

Subclasses​

Variables​

MetricCollection​

class MetricCollection(    problem: MetricsProblem, metrics: Optional[MutableMapping[str, Metric]] = None,):

Container class for metrics to calculate.

Static methods​

def create_from_model(    model: Union[_BaseModel, DistributedModelProtocol],    metrics: Optional[MutableMapping[str, Metric]] = None,) ‑> MetricCollection:

Creates a MetricCollection object from a _BaseModel.

Arguments

  • model: A _BaseModel.
  • metrics: The metrics dictionary. Defaults to None.

Returns Instance of MetricCollection

Methods​

def compute(    self,    test_target: np.ndarray,    test_preds: np.ndarray,    metric_to_optimise: str = 'F1',    threshold: Optional[float] = None,) ‑> Dict[str, float]:

Compute list of metrics and save results in self.results.

Arguments

  • test_target: A list of targets.
  • test_preds: A list of predictions.
  • metric_to_optimise: What metric to optimize in order to compute the optimal threshold. This will have no effect if there aren't any metrics to which a threshold is applied. Must be present in 'self.metrics'.
  • threshold: If this argument is provided, this threshold will be used instead of optimising the threshold as per 'optimise'
thresholds do not apply to multiclass problems.
def get_results_df(self) ‑> pandas.core.frame.DataFrame:

Calls get_results and returns results as a dataframe.

def view_thresholds(self) ‑> matplotlib.figure.Figure:

Displays chart of how thresholds affect relevant metrics.

MetricsProblem​

class MetricsProblem(    value, names=None, *, module=None, qualname=None, type=None, start=1,):

Simple wrapper for different problem types for MetricCollection.

Ancestors​

Variables​

  • static BINARY_CLASSIFICATION
  • static MULTICLASS_CLASSIFICATION
  • static MULTILABEL_CLASSIFICATION
  • static REGRESSION
  • static SEGMENTATION