One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks

Download
2021-1-01
Oksuz, Kemal
Cam, Baris Can
Kalkan, Sinan
Akbaş, Emre
IEEEDespite being widely used as a performance measure for visual detection tasks, Average Precision (AP) is limited in (i) reflecting localisation quality, (ii) interpretability and (iii) robustness to the design choices regarding its computation, and its applicability to outputs without confidence scores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic segmentation (Kirillov et al., 2019), does not suffer from these limitations but is limited to panoptic segmentation. In this paper, we propose Localisation Recall Precision (LRP) Error as the performance measure for all visual detection tasks. LRP Error, initially proposed only for object detection by Oksuz et al. (2018), does not suffer from the aforementioned limitations and is applicable to all visual detection tasks. We also introduce Optimal LRP (oLRP) Error as the minimum LRP error obtained over confidence scores to evaluate visual detectors and obtain optimal thresholds for deployment. We provide a detailed comparative analysis of LRP with AP and PQ, and use nearly 100 state-of-the-art visual detectors from seven visual detection tasks (i.e. object detection, keypoint detection, instance segmentation, panoptic segmentation, visual relationship detection, zero-shot detection and generalised zero-shot detection) using ten datasets to empirically show that LRP provides richer and more discriminative information than its counterparts. Code available at: https://github.com/kemaloksuz/LRP-Error
IEEE Transactions on Pattern Analysis and Machine Intelligence

Suggestions

Localization recall precision (LRP): A new performance metric for object detection
Öksüz, Kemal; Çam, Baris Can; Akbaş, Emre; Kalkan, Sinan (2018-09-14)
Average precision (AP), the area under the recall-precision (RP) curve, is the standard performance measure for object detection. Despite its wide acceptance, it has a number of shortcomings, the most important of which are (i) the inability to distinguish very different RP curves, and (ii) the lack of directly measuring bounding box localization accuracy. In this paper, we propose “Localization Recall Precision (LRP) Error”, a new metric specifically designed for object detection. LRP Error is composed of ...
Segmentation and deciısion fusion for building detection
Karadag, Ozge Oztimur; Senaras, Caglar; Yarman Vural, Fatoş Tunay (2014-07-18)
Segment based classification is one of the popular approaches for object detection, where the performance of the classification task is sensitive to the accuracy of the output of the initial segmentation. Most of these studies includes generic segmentation methods and it is assumed that the segmentation output is compatible with the subsequent classification method. However, depending on the problem domain the properties of the regions such as size, shape etc. which are suitable for classification may vary....
Evaluation of UAS Camera Operator Interfaces in a Simulated Task Environment An Optical Brain Imaging Approach
Çakır, Murat Perit; Akay, Daryal; Ayaz, Hasan; İşler, Veysi (null; 2012-07-11)
In this paper we focus on the effect of different interface designs on the performance and cognitive workload of sensor operators (SO) during a target detection task in a simulated environment. Functional near-infrared (fNIR) spectroscopy is used to investigate whether there is a relationship between target detection performance across three SO interfaces and brain activation data obtained from the subjects’ prefrontal cortices that are associated with relevant higher-order cognitive functions such as atten...
Recursive shortest spanning tree algorithms for image segmentation
Bayramoglu, NY; Bazlamaçcı, Cüneyt Fehmi (2005-11-24)
Image segmentation has an important role in image processing and the speed of the segmentation algorithm may become a drawback for some applications. This study analyzes the run time performances of some variations of the Recursive Shortest Spanning Tree Algorithm (RSST) and proposes simple but effective modifications on these algorithms to improve their speeds. In addition, the effect of link weight cost function on the run time performance and the segmentation quality is examined. For further improvement ...
A software for analysis and design optimization of switched reluctance motor
Yalçıner, Levent Burak; Ertan, Hulusi Bülent; Department of Electrical and Electronics Engineering (2004)
In this study, development of software, which can analyze and optimize an SRM by accurately calculating its performance, is aimed. Existing methods in the literature are investigated. Some studies for the calculation of performance use 2D field solutions and are known to be accurate; however, using field solutions is not feasible for the optimization purpose. So, a method based on a set of normalized permeance and force data are chosen for prediction of magnetizing characteristics. Selected methods are prog...
Citation Formats
K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbaş, “One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 0–0, 2021, Accessed: 00, 2022. [Online]. Available: https://hdl.handle.net/11511/99673.