One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks

Download

index.pdf

Date

2021-1-01

Author

Oksuz, Kemal
Cam, Baris Can
Kalkan, Sinan
Akbaş, Emre

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

88
views

0
downloads

Despite being widely used as a performance measure for visual detection tasks, Average Precision (AP) is limited in (i) reflecting localisation quality, (ii) interpretability and (iii) robustness to the design choices regarding its computation, and its applicability to outputs without confidence scores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic segmentation (Kirillov et al., 2019), does not suffer from these limitations but is limited to panoptic segmentation. In this paper, we propose Localisation Recall Precision (LRP) Error as the performance measure for all visual detection tasks. LRP Error, initially proposed only for object detection by Oksuz et al. (2018), does not suffer from the aforementioned limitations and is applicable to all visual detection tasks. We also introduce Optimal LRP (oLRP) Error as the minimum LRP error obtained over confidence scores to evaluate visual detectors and obtain optimal thresholds for deployment. We provide a detailed comparative analysis of LRP with AP and PQ, and use nearly 100 state-of-the-art visual detectors from seven visual detection tasks (i.e. object detection, keypoint detection, instance segmentation, panoptic segmentation, visual relationship detection, zero-shot detection and generalised zero-shot detection) using ten datasets to empirically show that LRP provides richer and more discriminative information than its counterparts. Code available at: https://github.com/kemaloksuz/LRP-Error

Subject Keywords

Localisation Recall Precision Average Precision Panoptic Quality Object Detection Keypoint Detection Instance Segmentation Panoptic Segmentation Performance Metric Threshold

URI

https://hdl.handle.net/11511/99673

Journal

IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI

https://doi.org/10.1109/tpami.2021.3130188

Collections

Department of Computer Engineering, Article

Suggestions

OpenMETU
Core

Localization recall precision (LRP): A new performance metric for object detection Öksüz, Kemal; Çam, Baris Can; Akbaş, Emre; Kalkan, Sinan (2018-09-14) Average precision (AP), the area under the recall-precision (RP) curve, is the standard performance measure for object detection. Despite its wide acceptance, it has a number of shortcomings, the most important of which are (i) the inability to distinguish very different RP curves, and (ii) the lack of directly measuring bounding box localization accuracy. In this paper, we propose “Localization Recall Precision (LRP) Error”, a new metric specifically designed for object detection. LRP Error is composed of ...
A software for analysis and design optimization of switched reluctance motor Yalçıner, Levent Burak; Ertan, Hulusi Bülent; Department of Electrical and Electronics Engineering (2004) In this study, development of software, which can analyze and optimize an SRM by accurately calculating its performance, is aimed. Existing methods in the literature are investigated. Some studies for the calculation of performance use 2D field solutions and are known to be accurate; however, using field solutions is not feasible for the optimization purpose. So, a method based on a set of normalized permeance and force data are chosen for prediction of magnetizing characteristics. Selected methods are prog...
Segmentation and deciısion fusion for building detection Karadag, Ozge Oztimur; Senaras, Caglar; Yarman Vural, Fatoş Tunay (2014-07-18) Segment based classification is one of the popular approaches for object detection, where the performance of the classification task is sensitive to the accuracy of the output of the initial segmentation. Most of these studies includes generic segmentation methods and it is assumed that the segmentation output is compatible with the subsequent classification method. However, depending on the problem domain the properties of the regions such as size, shape etc. which are suitable for classification may vary....
Recursive shortest spanning tree algorithms for image segmentation Bayramoglu, NY; Bazlamaçcı, Cüneyt Fehmi (2005-11-24) Image segmentation has an important role in image processing and the speed of the segmentation algorithm may become a drawback for some applications. This study analyzes the run time performances of some variations of the Recursive Shortest Spanning Tree Algorithm (RSST) and proposes simple but effective modifications on these algorithms to improve their speeds. In addition, the effect of link weight cost function on the run time performance and the segmentation quality is examined. For further improvement ...
Evaluation of UAS Camera Operator Interfaces in a Simulated Task Environment An Optical Brain Imaging Approach Çakır, Murat Perit; Akay, Daryal; Ayaz, Hasan; İşler, Veysi (null; 2012-07-11) In this paper we focus on the effect of different interface designs on the performance and cognitive workload of sensor operators (SO) during a target detection task in a simulated environment. Functional near-infrared (fNIR) spectroscopy is used to investigate whether there is a relationship between target detection performance across three SO interfaces and brain activation data obtained from the subjects’ prefrontal cortices that are associated with relevant higher-order cognitive functions such as atten...

Citation Formats

K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbaş, “One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 0–0, 2021, Accessed: 00, 2022. [Online]. Available: https://hdl.handle.net/11511/99673.