Localization recall precision (LRP): A new performance metric for object detection

Download

index.pdf

Date

2018-09-14

Author

Öksüz, Kemal
Çam, Baris Can
Akbaş, Emre
Kalkan, Sinan

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

104
views

35
downloads

Average precision (AP), the area under the recall-precision (RP) curve, is the standard performance measure for object detection. Despite its wide acceptance, it has a number of shortcomings, the most important of which are (i) the inability to distinguish very different RP curves, and (ii) the lack of directly measuring bounding box localization accuracy. In this paper, we propose “Localization Recall Precision (LRP) Error”, a new metric specifically designed for object detection. LRP Error is composed of three components related to localization, false negative (FN) rate and false positive (FP) rate. Based on LRP, we introduce the “Optimal LRP” (oLRP), the minimum achievable LRP error representing the best achievable configuration of the detector in terms of recall-precision and the tightness of the boxes. In contrast to AP, which considers precisions over the entire recall domain, oLRP determines the “best” confidence score threshold for a class, which balances the trade-off between localization and recall-precision. In our experiments, we show that oLRP provides richer and more discriminative information than AP. We also demonstrate that the best confidence score thresholds vary significantly among classes and detectors. Moreover, we present LRP results of a simple online video object detector and show that the class-specific optimized thresholds increase the accuracy against the common approach of using a general threshold for all classes. Our experiments demonstrate that LRP is more competent than AP in capturing the performance of detectors. Our source code for PASCAL VOC AND MSCOCO datasets are provided at https://github.com/cancam/LRP.

Subject Keywords

Average precision, Object detection, Performance metric, Optimal threshold, Recall-precision

URI

https://hdl.handle.net/11511/41609

DOI

https://doi.org/10.1007/978-3-030-01234-2_31

Collections

Department of Computer Engineering, Conference / Seminar

Suggestions

OpenMETU
Core

One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks Oksuz, Kemal; Cam, Baris Can; Kalkan, Sinan; Akbaş, Emre (2021-1-01) Despite being widely used as a performance measure for visual detection tasks, Average Precision (AP) is limited in (i) reflecting localisation quality, (ii) interpretability and (iii) robustness to the design choices regarding its computation, and its applicability to outputs without confidence scores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic segmentation (Kirillov et al., 2019), does not suffer from these limitations but is limited to panoptic segmentation. In this paper, we propo...
Applications of simulated annealing for the design of special digital filters - Comments Çiloğlu, Tolga; Ünver, Zafer (1996-04-01) The way of measuring the performance of a discrete coefficient filter which is designed by scaling optimization is discussed.
Region of Interest Detection Based Fast and Robust Geo-Spatial Object Recognition Gürbüz, Yeti Ziya; Alatan, Abdullah Aydın (2013-01-01) In this paper a novel computationally efficient algorithm to detect objects automatically from high definition satellite imagery with high performance is presented. The proposed algorithm has three main steps supporting each other: Filtering, shape based and appearance based object detection. A region of interest indicating the possible regions that may have the objects to be detected is determined in a very short time via filtering step. In the remaining steps, the objects are extracted from that region an...
Multitarget tracking performance metric: deficiency aware subpattern assignment Oksuz, Kemal; CEMGİL, ALİ TAYLAN (2018-03-01) Multitarget tracking is a sequential estimation problem where conditioned on noisy sensor measurements, state variables of several targets need to be estimated recursively. In this study, the authors propose a novel performance measure for multitarget tracking named as Deficiency Aware Subpattern Assignment (DASA), that can be used to consistently compare algorithms in a broad spectrum of formulations ranging from conventional data association methods to random finite set based multitarget tracking algorith...
Multi-perspective analysis and systematic benchmarking for binary-classification performance evaluation instruments Canbek, Gürol; Taşkaya Temizel, Tuğba; Department of Information Systems (2019) This thesis proposes novel methods to analyze and benchmark binary-classification performance evaluation instruments. It addresses critical problems found in the literature, clarifies terminology and distinguishes instruments as measure, metric, and as a new category indicator for the first time. The multi-perspective analysis introduces novel concepts such as canonical form, geometry, duality, complementation, dependency, and leveling with formal definitions as well as two new basic instruments. An indicat...

Citation Formats

K. Öksüz, B. C. Çam, E. Akbaş, and S. Kalkan, “Localization recall precision (LRP): A new performance metric for object detection,” 2018, Accessed: 00, 2020. [Online]. Available: https://hdl.handle.net/11511/41609.