Comparison of Subjective and Objective Evaluation Methods for Audio Source Separation

Josef, Kornycky
Günel Kılıç, Banu
Ahmet, Kondoz
The evaluation of audio separation algorithms can either be performed objectively by calculation of numerical measures, or subjectively through listening tests. Although objective evaluation is inherently more straightforward, subjective listening tests are still essential in determining the perceived quality of separation. This paper aims to find relationships between objective and subjective results so that numerical values can be translated into perceptual criteria. A generic audio source separation system was modelled which provided varying levels of interference, noise and artifacts. This enabled a full spread of objective measurement values to be obtained. Extensive tests were performed utilising the output synthesised by this separation model. The relationships found were presented and the factors of prime importance were determined.
Proceedings of Meetings on Acoustics


Multi-perspective analysis and systematic benchmarking for binary-classification performance evaluation instruments
Canbek, Gürol; Taşkaya Temizel, Tuğba; Department of Information Systems (2019)
This thesis proposes novel methods to analyze and benchmark binary-classification performance evaluation instruments. It addresses critical problems found in the literature, clarifies terminology and distinguishes instruments as measure, metric, and as a new category indicator for the first time. The multi-perspective analysis introduces novel concepts such as canonical form, geometry, duality, complementation, dependency, and leveling with formal definitions as well as two new basic instruments. An indicat...
Alternative Approach to Maurer's Universal Statistical Test
Tezcan, Cihangir; Doğanaksoy, Ali (null; 2008-12-01)
Statistical tests for randomness play an important role in cryptography since many cryptographic applications require random or pseudorandom numbers. In this study, we introduce an alternative approach to Maurer’s Universal Test. This approach allows us to test short binary sequences as small as 66 bits and to choose slightly larger block sizes. Moreover, it does not have an initialization part and requires less time to test a binary sequence.
Quantitative measure of observability for linear stochastic systems
Subasi, Yuksel; Demirekler, Mübeccel (Elsevier BV, 2014-06-01)
In this study we define a new observability measure for stochastic systems: the mutual information between the state sequence and the corresponding measurement sequence for a given time horizon. Although the definition is given for a general system representation, the paper focuses on the linear time invariant Gaussian case. Some basic analytical results are derived for this special case. The measure is extended to the observability of a subspace of the state space, specifically an individual state and/or t...
Implementation of an enhanced fixed point Variable Bit-Rate MELP vocoder on TMS320C549
Ertan, AE; Aksu, EB; Ilk, HG; Karci, H; Karpat, O; Kolcak, T; Sendur, L; Demirekler, Mübeccel; Cetin, AE (1999-03-19)
In this paper, a fixed point Variable Bit-Rate (VBR) Mixed Excitation Linear Predictive Coding (MELP(TM)) vocoder is presented. The VBR-MELP vocoder is also implemented on a TMS320C54x and it achieves virtually indistinguishable federal standard MELP quality at bit-rates between 1.0 to 1.6 kb/s. The backbone of VBR-MELP vocoder is similar to that of federal standard MELP. It utilizes a novel sub-band based voice activity detector in the back-end of encoder to discriminate background noise from speech activi...
Evaluation of visual quality metrics
Ölgün, Ramazan Ferhat; Akar, Gözde; Department of Electrical and Electronics Engineering (2011)
The aim of this study is to work on the visual quality metrics that are widely accepted in literature, to evaluate them on different distortion types and to give a comparison of overall performances in terms of prediction accuracy, monotonicity, consistency and complexity. The algorithms behind the quality metrics in literature and parameters used for quality metric performance evaluations are studied. This thesis also includes the explanation of Human Visual System, classification of visual quality metrics...
Citation Formats
K. Josef, B. Günel Kılıç, and K. Ahmet, “Comparison of Subjective and Objective Evaluation Methods for Audio Source Separation,” Proceedings of Meetings on Acoustics, pp. 0–0, 2008, Accessed: 00, 2020. [Online]. Available: