Add BenchmarkEvaluator with basic precision/recall computation by Muhammedswalihu · Pull Request #1870 · roboflow/supervision

Muhammedswalihu · 2025-07-06T17:07:55Z

Summary

This PR introduces a utility class BenchmarkEvaluator in supervision/metrics/benchmark.py to support benchmarking object detection results across different datasets or models.

Features

Computes basic precision and recall
Accepts Detections objects for ground truth and prediction
Optional support for class mapping and IoU thresholding (future extensions)
Includes a unit test at tests/metrics/test_benchmark.py

Motivation

Addresses Issue #1778: Improving object detection benchmarking process for unrelated datasets.

Let me know if you'd like me to extend this in future PRs with:

mAP, F1, or per-class metrics
Confusion matrix visualization
Colab notebook example

Thanks for the opportunity to contribute!

CLAassistant · 2025-07-06T17:08:01Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ Borda
❌ Muhammed Swalihu

Muhammed Swalihu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Muhammedswalihu · 2025-07-06T17:16:02Z

Hi @SkalskiP @onuralpszr — I've submitted this PR for the BenchmarkEvaluator (Issue #1778 ). Let me know if you'd like me to fix the pre-commit error or extend this further. Thanks for reviewing!

soumik12345 · 2025-07-08T10:03:13Z

Hi @Muhammedswalihu, this seems like a really valuable feature!
Can you please replace the placeholder logic with a working one, provide a working example and testcases; and we can review the PR.

Muhammedswalihu · 2025-07-08T21:06:54Z

Hi @soumik12345 , thanks for the review!

I’ll go ahead and:

Replace the placeholder logic in BenchmarkEvaluator with full precision/recall/mAP computation,

Add a working demo example (maybe in a Colab notebook for clarity), and

Improve the test coverage with more edge cases and per-class evaluation.

Let me know if there’s anything specific you’d like to see included. Appreciate the opportunity — excited to take this further!

review-notebook-app · 2025-07-08T21:26:12Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Muhammedswalihu · 2025-07-08T21:30:34Z

Hi @soumik12345, I've added a Colab-style demo notebook BenchmarkEvaluator_Demo.ipynb!

It includes:

How to import and use the BenchmarkEvaluator
Per-class precision and recall visualization
A visual example comparing predicted and ground truth bounding boxes

This should help users understand and adopt the module more easily.

Let me know if you'd like me to polish or extend this notebook further!

soumik12345

Hi @Muhammedswalihu, thanks for providing the PoC!
Please feel free to proceed with the actual implementation.
Also, there's no need to commit the notebook to supervision, you can just attach a colab notebook in a comment when the PR is ready for review with the complete logic.

soumik12345 · 2025-07-09T14:03:12Z

+        # TODO: Add class alignment, matching using IoU
+        tp = len(self.predictions.xyxy)  # Placeholder
+        fp = 0
+        fn = len(self.ground_truth.xyxy) - tp


The logic here is incomplete, please add the correct logic to compute precision and recall.

soumik12345 · 2025-07-09T14:04:18Z

+from supervision.metrics.benchmark import BenchmarkEvaluator
+
+
+def test_basic_precision_recall():


This too seems like a placeholder test; please proceed with the implementation and add comprehensive unit tests.

galafis · 2025-09-27T14:39:29Z

Great initiative on the BenchmarkEvaluator! This addresses a crucial need for standardized evaluation metrics. I'd like to offer some technical guidance to help you complete the implementation effectively.

Key Implementation Recommendations:

IoU-based Matching Algorithm: For proper TP/FP/FN computation, you'll need Hungarian assignment or greedy matching based on IoU thresholds:

def compute_matches(pred_boxes, gt_boxes, iou_threshold=0.5):
    # Compute IoU matrix
    # Apply optimal assignment (e.g., scipy.optimize.linear_sum_assignment)
    # Return matched pairs, unmatched predictions (FP), unmatched ground truth (FN)

Multi-class Support: Consider class-aware matching for per-class metrics:
- Group detections by class_id
- Compute metrics separately for each class
- Aggregate for overall performance
Confidence Thresholding: Implement confidence-based filtering for realistic evaluation scenarios
Standard Metrics: Beyond precision/recall, consider adding:
- F1-score
- Average Precision (AP) at different IoU thresholds
- Mean Average Precision (mAP)

Performance Considerations:

Vectorized IoU computation using numpy/supervision utilities
Batch processing for large evaluation sets
Memory-efficient handling of detection arrays

This evaluator will be invaluable for the community's benchmarking needs. Happy to provide more specific implementation details if needed!

Best regards,
Gabriel

Copilot

Pull request overview

This PR adds a new BenchmarkEvaluator utility intended to compute basic object-detection precision/recall for Detections pairs, alongside a small unit test and a demo notebook.

Changes:

Introduces BenchmarkEvaluator with a compute_precision_recall() API and a summary() helper.
Adds a unit test for the perfect-match case.
Adds a Jupyter notebook demonstrating intended usage.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
supervision/metrics/benchmark.py	Adds `BenchmarkEvaluator` (currently placed outside the `src/` package discovery root and includes placeholder metric logic).
tests/metrics/test_benchmark.py	Adds an initial unit test for precision/recall (currently too narrow to validate FP/FN, class mismatch, or IoU threshold behavior).
BenchmarkEvaluator_Demo.ipynb	Adds a demo notebook (currently calls a non-existent per-class method).

Comments suppressed due to low confidence (1)

supervision/metrics/benchmark.py:40

summary() prints directly to stdout, which is an unexpected side effect for a library utility and makes the method hard to use in applications/tests. Prefer returning a string (or the metrics dict) and letting callers decide how to log/print it.

        metrics = self.compute_precision_recall()
        print("Benchmark Summary:")
        for k, v in metrics.items():
            print(f"{k}: {v:.4f}")

+# supervision/metrics/benchmark.py
+
+from typing import Dict, Optional
+
+from supervision.detection.core import Detections


+    def compute_precision_recall(self) -> Dict[str, float]:
+        """
+        Compute basic precision and recall metrics.
+        For demo purposes — you will expand this.
+        """
+        # TODO: Add class alignment, matching using IoU
+        tp = len(self.predictions.xyxy)  # Placeholder
+        fp = 0
+        fn = len(self.ground_truth.xyxy) - tp
+
+        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
+        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
+
+        return {"precision": precision, "recall": recall}


+        ground_truth: Detections,
+        predictions: Detections,
+        class_map: Optional[Dict[str, str]] = None,
+        iou_threshold: float = 0.5,
+    ):


+import numpy as np
+
+from supervision.detection.core import Detections
+from supervision.metrics.benchmark import BenchmarkEvaluator
+
+
+def test_basic_precision_recall():
+    gt = Detections(xyxy=np.array([[0, 0, 100, 100]]), class_id=np.array([0]))
+    pred = Detections(xyxy=np.array([[0, 0, 100, 100]]), class_id=np.array([0]))
+
+    evaluator = BenchmarkEvaluator(ground_truth=gt, predictions=pred)
+    metrics = evaluator.compute_precision_recall()
+
+    assert metrics["precision"] == 1.0
+    assert metrics["recall"] == 1.0


+    "per_class = evaluator.compute_precision_recall_per_class()\n",
+    "for cls, metric in per_class.items():\n",
+    "    print(\n",
+    "        f\"Class {cls} - Precision: {metric['precision']:.2f}, Recall: {metric['recall']:.2f}\"\n",
+    "    )"
+   ]


Add BenchmarkEvaluator with unit test for precision and recall

9cf68c7

Muhammedswalihu requested review from SkalskiP and onuralpszr as code owners July 6, 2025 17:07

fix(pre_commit): 🎨 auto format pre-commit hooks

3a748e6

Add demo notebook for BenchmarkEvaluator

bf7cd7a

fix(pre_commit): 🎨 auto format pre-commit hooks

d03470b

soumik12345 suggested changes Jul 9, 2025

View reviewed changes

Borda force-pushed the develop branch from 2413a08 to 840226a Compare April 24, 2026 16:08

Borda requested review from Copilot and removed request for onuralpszr July 17, 2026 16:55

Merge branch 'develop' into benchmark-evaluator

ce9b70e

Copilot started reviewing on behalf of Borda July 17, 2026 16:56 View session

Borda requested a review from soumik12345 July 17, 2026 16:56

fix(pre_commit): 🎨 auto format pre-commit hooks

2089bb1

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add BenchmarkEvaluator with basic precision/recall computation - #1870

Add BenchmarkEvaluator with basic precision/recall computation#1870
Muhammedswalihu wants to merge 6 commits into
roboflow:developfrom
Muhammedswalihu:benchmark-evaluator

Muhammedswalihu commented Jul 6, 2025

Uh oh!

CLAassistant commented Jul 6, 2025 •

edited

Loading

Uh oh!

Muhammedswalihu commented Jul 6, 2025

Uh oh!

soumik12345 commented Jul 8, 2025

Uh oh!

Muhammedswalihu commented Jul 8, 2025

Uh oh!

review-notebook-app Bot commented Jul 8, 2025

Uh oh!

Muhammedswalihu commented Jul 8, 2025

Uh oh!

soumik12345 left a comment

Uh oh!

soumik12345 Jul 9, 2025

Uh oh!

soumik12345 Jul 9, 2025

Uh oh!

galafis commented Sep 27, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		from supervision.metrics.benchmark import BenchmarkEvaluator


		def test_basic_precision_recall():

Uh oh!

Conversation

Muhammedswalihu commented Jul 6, 2025

Summary

Features

Motivation

Uh oh!

CLAassistant commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Muhammedswalihu commented Jul 6, 2025

Uh oh!

soumik12345 commented Jul 8, 2025

Uh oh!

Muhammedswalihu commented Jul 8, 2025

Uh oh!

review-notebook-app Bot commented Jul 8, 2025

Uh oh!

Muhammedswalihu commented Jul 8, 2025

Uh oh!

soumik12345 left a comment

Choose a reason for hiding this comment

Uh oh!

soumik12345 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

soumik12345 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

galafis commented Sep 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CLAassistant commented Jul 6, 2025 •

edited

Loading