Accuracy

class mmpretrain.evaluation.Accuracy(topk=(1,), thrs=0.0, collect_device='cpu', prefix=None)

Accuracy evaluation metric.

For either binary classification or multi-class classification, the accuracy is the fraction of correct predictions in all predictions:

\[\text{Accuracy} = \frac{N_{\text{correct}}}{N_{\text{all}}}\]

Parameters:

• topk (int | Sequence[int]) – If the ground truth label matches one of the best k predictions, the sample will be regard as a positive prediction. If the parameter is a tuple, all of top-k accuracy will be calculated and outputted together. Defaults to 1.

• thrs (Sequence[float | None] | float | None) – If a float, predictions with score lower than the threshold will be regard as the negative prediction. If None, not apply threshold. If the parameter is a tuple, accuracy based on all thresholds will be calculated and outputted together. Defaults to 0.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

Examples

>>> import torch
>>> from mmpretrain.evaluation import Accuracy
>>> # -------------------- The Basic Usage --------------------
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> Accuracy.calculate(y_pred, y_true)
tensor([50.])
>>> # Calculate the top1 and top5 accuracy.
>>> y_score = torch.rand((1000, 10))
>>> y_true = torch.zeros((1000, ))
>>> Accuracy.calculate(y_score, y_true, topk=(1, 5))
[[tensor([9.9000])], [tensor([51.5000])]]
>>>
>>> # ------------------- Use with Evalutor -------------------
>>> from mmpretrain.structures import DataSample
>>> from mmengine.evaluator import Evaluator
>>> data_samples = [
...     DataSample().set_gt_label(0).set_pred_score(torch.rand(10))
...     for i in range(1000)
... ]
>>> evaluator = Evaluator(metrics=Accuracy(topk=(1, 5)))
>>> evaluator.process(data_samples)
>>> evaluator.evaluate(1000)
{
    'accuracy/top1': 9.300000190734863,
    'accuracy/top5': 51.20000076293945
}

static calculate(pred, target, topk=(1,), thrs=(0.0,))

Calculate the accuracy.

Parameters:

• pred (torch.Tensor | np.ndarray | Sequence) – The prediction results. It can be labels (N, ), or scores of every class (N, C).

• target (torch.Tensor | np.ndarray | Sequence) – The target of each prediction with shape (N, ).

• thrs (Sequence[float]) – Predictions with scores under the thresholds are considered negative. It’s only used when pred is scores. None means no thresholds. Defaults to (0., ).

• thrs – Predictions with scores under the thresholds are considered negative. It’s only used when pred is scores. Defaults to (0., ).

Returns:

Accuracy.

• torch.Tensor: If the pred is a sequence of label instead of score (number of dimensions is 1). Only return a top-1 accuracy tensor, and ignore the argument topk` and ``thrs.

• List[List[torch.Tensor]]: If the pred is a sequence of score (number of dimensions is 2). Return the accuracy on each topk and thrs. And the first dim is topk, the second dim is thrs.

Return type:

torch.Tensor | List[List[torch.Tensor]]

compute_metrics(results)

Compute the metrics from processed results.

Parameters:

results (dict) – The processed results of each batch.

Returns:

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type:

Dict

process(data_batch, data_samples)

Process one batch of data samples.

The processed results should be stored in self.results, which will be used to computed the metrics when all batches have been processed.

Parameters:

• data_batch – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.