Skip to content

Metrics Utilities

The metrics utilities module provides functions for evaluating the performance of normative models. This includes functions for computing various error metrics, such as mean squared error (MSE), mean absolute error (MAE), and R-squared (R²) scores, among others. These metrics are essential for assessing the fit of the model to the data and for comparing different models.

spectranorm.utils.metrics

utils/metrics.py

Utility functions for computing various model metrics in the Spectranorm package.

compute_bic(model_log_likelihoods: npt.NDArray[np.floating[Any]], n_params: int, n_samples: int) -> npt.NDArray[np.floating[Any]]

Compute Bayesian Information Criterion (BIC) for model evaluation.

Parameters:

Name Type Description Default
model_log_likelihoods NDArray[floating[Any]]

np.ndarray Log likelihoods from the model. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
n_params int

int Number of parameters in the model.

 required
n_samples int

int Number of samples used in the model.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Bayesian Information Criterion. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
def compute_bic(
    model_log_likelihoods: npt.NDArray[np.floating[Any]],
    n_params: int,
    n_samples: int,
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Bayesian Information Criterion (BIC) for model evaluation.

    Args:
        model_log_likelihoods: np.ndarray
            Log likelihoods from the model. (n_samples) or (n_samples, n_outputs)
            if multiple outputs are being assessed.
        n_params: int
            Number of parameters in the model.
        n_samples: int
            Number of samples used in the model.

    Returns:
        np.ndarray
            Bayesian Information Criterion. (n_outputs,) if multiple outputs are
            assessed, otherwise a scalar.
    """
    return np.asarray(
        -2 * np.mean(model_log_likelihoods, axis=0) + n_params * np.log(n_samples),
    )

compute_expv(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Explained Variance (EXPV) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Explained Variance. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
def compute_expv(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Explained Variance (EXPV) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            Explained Variance. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    return np.asarray(1 - np.var(y - y_pred, axis=0) / np.var(y, axis=0))

compute_mae(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Mean Absolute Error (MAE) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Mean Absolute Error. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
def compute_mae(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Mean Absolute Error (MAE) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            Mean Absolute Error. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    return np.asarray(np.mean(np.abs(y - y_pred), axis=0))

compute_mape(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Mean Absolute Percentage Error (MAPE) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Mean Absolute Percentage Error. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def compute_mape(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Mean Absolute Percentage Error (MAPE) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            Mean Absolute Percentage Error. (n_outputs,) if multiple outputs are
            assessed, otherwise a scalar.
    """
    return np.asarray(np.mean(np.abs((y - y_pred) / y), axis=0) * 100)

compute_mse(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Mean Squared Error (MSE) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Mean Squared Error. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def compute_mse(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Mean Squared Error (MSE) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            Mean Squared Error. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    return np.asarray(np.mean((y - y_pred) ** 2, axis=0))

compute_msll(model_log_likelihoods: npt.NDArray[np.floating[Any]], baseline_log_likelihoods: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Mean Standardized Log Loss (MSLL) based on model and baseline log likelihoods.

Parameters:

Name Type Description Default
model_log_likelihoods NDArray[floating[Any]]

np.ndarray Log likelihoods from the model. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
baseline_log_likelihoods NDArray[floating[Any]]

np.ndarray Log likelihoods from the baseline model (a trivial model). Same shape as model_log_likelihoods.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Mean Standardized Log Loss. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
def compute_msll(
    model_log_likelihoods: npt.NDArray[np.floating[Any]],
    baseline_log_likelihoods: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Mean Standardized Log Loss (MSLL) based on model and baseline log
    likelihoods.

    Args:
        model_log_likelihoods: np.ndarray
            Log likelihoods from the model. (n_samples) or (n_samples, n_outputs)
            if multiple outputs are being assessed.
        baseline_log_likelihoods: np.ndarray
            Log likelihoods from the baseline model (a trivial model). Same shape as
            model_log_likelihoods.

    Returns:
        np.ndarray
            Mean Standardized Log Loss. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    return np.asarray(
        -(
            np.mean(model_log_likelihoods, axis=0)
            - np.mean(baseline_log_likelihoods, axis=0)
        ),
    )

compute_r2(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute R-squared (coefficient of determination) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray R-squared value. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def compute_r2(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute R-squared (coefficient of determination) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            R-squared value. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    ss_res = np.sum((y - y_pred) ** 2, axis=0)
    ss_tot = np.sum((y - np.mean(y, axis=0)) ** 2, axis=0)
    return np.asarray(1 - (ss_res / ss_tot))

compute_rmse(y: npt.NDArray[np.floating[Any]], y_pred: npt.NDArray[np.floating[Any]]) -> npt.NDArray[np.floating[Any]]

Compute Root Mean Squared Error (RMSE) between true and predicted values.

Parameters:

Name Type Description Default
y NDArray[floating[Any]]

np.ndarray True values. (n_samples) or (n_samples, n_outputs) if multiple outputs are being assessed.

 required
y_pred NDArray[floating[Any]]

np.ndarray Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

 required

Returns:

Type Description
NDArray[floating[Any]]

np.ndarray Root Mean Squared Error. (n_outputs,) if multiple outputs are assessed, otherwise a scalar.
Source code in src/spectranorm/utils/metrics.py 
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
def compute_rmse(
    y: npt.NDArray[np.floating[Any]],
    y_pred: npt.NDArray[np.floating[Any]],
) -> npt.NDArray[np.floating[Any]]:
    """
    Compute Root Mean Squared Error (RMSE) between true and predicted values.

    Args:
        y: np.ndarray
            True values. (n_samples) or (n_samples, n_outputs) if multiple outputs
            are being assessed.
        y_pred: np.ndarray
            Predicted values. (n_samples) or (n_samples, n_outputs) same shape as y.

    Returns:
        np.ndarray
            Root Mean Squared Error. (n_outputs,) if multiple outputs are assessed,
            otherwise a scalar.
    """
    return np.asarray(np.sqrt(compute_mse(y, y_pred)))