Accuracy Barrier (ACCBAR): A novel performance indicator for binary classification

Date

2022-01-01

Author

Canbek, Gurol
Taşkaya Temizel, Tuğba
SAĞIROĞLU, ŞEREF

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

116
views

0
downloads

Although several binary classification performance metrics have been defined, a few of them are used for performance evaluation of classifiers and performance comparison/reporting in the literature. Specifically, F1 and Accuracy (ACC) are the most known and conventionally used metrics. Despite their popularity and easy-to-understand characteristics, those metrics exhibit critical robustness issues. This paper suggests a new instrument category named 'performance indicators' and proposes a novel indicator named accuracy barrier (ACCBAR for short) that works to uncover confounding problems in performance reporting of ACC metric. The given case study in mobile malware classification, which is a domain of cyber security, has shown that the indicator gives an accurate interpretation of the results presented in terms of ACC. This study also recommends that researchers should use ACCBAR to eliminate potential publication or confirmation bias in classification performance evaluation.

Subject Keywords

confirmation bias, performance evaluation, performance indicators, performance measures, publication bias

URI

https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85142796888&origin=inward
https://hdl.handle.net/11511/101495

DOI

https://doi.org/10.1109/iscturkey56345.2022.9931888

Conference Name

15th International Conference on Information Security and Cryptography, ISCTURKEY 2022

Collections

Graduate School of Informatics, Conference / Seminar

Suggestions

OpenMETU
Core

PToPI: A Comprehensive Review, Analysis, and Knowledge Representation of Binary Classification Performance Measures/Metrics Canbek, Gürol; Taşkaya Temizel, Tuğba; SAĞIROĞLU, ŞEREF (2023-1-01) Although few performance evaluation instruments have been used conventionally in different machine learning-based classification problem domains, there are numerous ones defined in the literature. This study reviews and describes performance instruments via formally defined novel concepts and clarifies the terminology. The study first highlights the issues in performance evaluation via a survey of 78 mobile-malware classification studies and reviews terminology. Based on three research questions, it propose...
Multi-perspective analysis and systematic benchmarking for binary-classification performance evaluation instruments Canbek, Gürol; Taşkaya Temizel, Tuğba; Department of Information Systems (2019) This thesis proposes novel methods to analyze and benchmark binary-classification performance evaluation instruments. It addresses critical problems found in the literature, clarifies terminology and distinguishes instruments as measure, metric, and as a new category indicator for the first time. The multi-perspective analysis introduces novel concepts such as canonical form, geometry, duality, complementation, dependency, and leveling with formal definitions as well as two new basic instruments. An indicat...
Power-delay optimized VLSI threshold detection circuits and their use in parallel integer multiplication Ercan, Furkan; Muhtaroğlu, Ali; Sustainable Environment and Energy Systems (2015-6) Threshold detection is a fundamental logic function that has broad use in arithmetic processors, and other digital applications. Thus, any improvement in threshold detection in terms of power and/or delay contributes significantly to the field of digital circuit design. A recently reported parallel integer multiplier architecture, ABACUS, uses column compression networks to compress partial products through the final addition network. Architecture of column compression network of ABACUS is suitable for thre...
Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling Gupta, Hoshin V.; Kling, Harald; Yılmaz, Koray Kamil; Martinez, Guillermo F. (2009-10-20) The mean squared error (MSE) and the related normalization, the Nash-Sutcliffe efficiency (NSE), are the two criteria most widely used for calibration and evaluation of hydrological models with observed data. Here, we present a diagnostically interesting decomposition of NSE (and hence MSE), which facilitates analysis of the relative importance of its different components in the context of hydrological modelling, and show how model calibration problems can arise due to interactions among these components. T...
Identification and prioritization of stage-level KPIs for BOT projects - evidence from Turkey BUDAYAN, Cenk; OKUDAN, Ozan; Dikmen Toker, İrem (2020-09-01) Purpose The purpose of this paper is to identify and prioritize key performance indicators (KPIs) that can be used for stage-based performance assessment of build-operate-transfer (BOT) projects. Design/methodology/approach This research was conducted through focus group discussions and face-to-face questionnaires. Firstly, stage-level KPIs for BOT projects were identified by conducting a literature survey. The list of KPIs that can be used for measuring performance at different stages of a BOT project was ...

Citation Formats

G. Canbek, T. Taşkaya Temizel, and Ş. SAĞIROĞLU, “Accuracy Barrier (ACCBAR): A novel performance indicator for binary classification,” presented at the 15th International Conference on Information Security and Cryptography, ISCTURKEY 2022, Ankara, Türkiye, 2022, Accessed: 00, 2023. [Online]. Available: https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85142796888&origin=inward.