An inter-annotator agreement measurement methodology for the Turkish Discourse Bank (TDB)

Download

index.pdf

Date

2010

Author

Yalçınkaya, Şaban İhsan

Metadata

Show full item record

Item Usage Stats

280
views

110
downloads

In the TDB[1]-like corpora annotation efforts, which are constructed by the intuitions of the annotators, the reliability of the corpus can only be determined via correct interannotator agreement measurement methodology (Artstein, & Poesio, 2008). In this thesis, a methodology was defined to measure the inter-annotator agreement among the TDB annotators. The statistical tests and the agreement coefficients that are widely used in scientific communities, including Cochran’s Q test (1950), Fleiss’ Kappa (1971), and Krippendorff’s Alpha (1995), were examined in detail. The inter-annotator agreement measurement approaches of the various corpus annotation efforts were scrutinized in terms of the reported statistical results. It was seen that none of the reported interannotator agreement approaches were statistically appropriate for the TDB. Therefore, a comprehensive inter-annotator agreement measurement methodology was designed from scratch. A computer program, the Rater Agreement Tool (RAT), was developed in order to perform statistical measurements on the TDB with different corpus parameters and data handling approaches. It was concluded that Krippendorff’s Alpha is the most appropriate statistical method for the TDB. It was seen that the measurements are affected with data handling approach preferences, as well as the used agreement statistic methods. It was also seen that there is not only one correct approach but several approaches valid for different research considerations. For the TDB, the major data handling suggestions that emerged are: (1) considering the words as building blocks of the annotations and (2) using the interval approach when it is preferred to weigh the partial disagreements, and using the boundary approach when it is preferred to evaluate all disagreements in same way.

Subject Keywords

Linguistics.

URI

http://etd.lib.metu.edu.tr/upload/12612643/index.pdf
https://hdl.handle.net/11511/19899

Collections

Graduate School of Informatics, Thesis

Suggestions

OpenMETU
Core

An investigation of incidental vocabulary acquisition in relation to learner proficiency level and word frequency Tekmen, E. Anne Ferrell; Daloğlu, Ayşegül (Wiley, 2006-06-01) This study examined the relationship between learners' incidental vocabulary acquisition and their level of proficiency, and between acquisition and word frequency in a text. Participants were Turkish learners of English at three proficiency levels. One reading text and four vocabulary tests were administered over a two-week period. Analyses of the data revealed that lexical gains from reading were significant for each group (p < .05). The higher proficiency groups were able to acquire more words than lower...
The syntax of relative clauses in Croatian Gracanın Yüksek, Martına (Walter de Gruyter GmbH, 2013-01-01) In this paper, I propose that Croatian relative clauses (RCs) introduced by the complementizer to 'what/that' do not form a homogeneous class with respect to their derivation: some are derived by movement, and some are derived by a non-movement strategy. Unless the relativized element is the subject, sto-RCs normally require a resumptive pronoun to appear in the site of relativization. However, this requirement is removed under morphological case matching between the head of the RC and the resumptive pronou...
A tune-based account of Turkish information structure Özge, Umut; Bozşahin, Hüseyin Cem; Zeyrek, Deniz; Department of Cognitive Sciences (2003) Languages differ in the means they avail themselves of for the structural realization of information structure, where available options are word order,prosody and morphology. Turkish has long been characterized as predominantly using word order and its variation in realizing information structure, where certain positions in a sentence are associated with certain pragmatic functions related to information structure. Prosody has been proposed to play only a secondary role interacting with word order. Contrar...
A Graph-Based Concept Discovery Method for n-Ary Relations Abay, Nazmiye Ceren; MUTLU, ALEV; Karagöz, Pınar (2015-09-04) Concept discovery is a multi-relational data mining task for inducing definitions of a specific relation in terms of other relations in the data set. Such learning tasks usually have to deal with large search spaces and hence have efficiency and scalability issues. In this paper, we present a hybrid approach that combines association rule mining methods and graph-based approaches to cope with these issues. The proposed method inputs the data in relational format, converts it into a graph representation, and...
Investigation of semantic effects in oddball paradigm through event related potentials Dumlu, Seda Nilgün; Gökçay, Didem; Öniz, Adile; Department of Medical Informatics (2012) In this study, the effect of semantic information processing was investigated by the oddball paradigm, by presenting consecutive Turkish words or word-like non-words while EEG signals are recorded. In an oddball paradigm, a series of events are presented of which one class is rarer than the other. Subjects are asked to respond to the infrequent stimuli (e.g. press a button, or count the number). The event related potential (ERP) component P300 obtained from EEG is considered as the marker of this attention ...

Citation Formats

Ş. İ. Yalçınkaya, “An inter-annotator agreement measurement methodology for the Turkish Discourse Bank (TDB),” M.S. - Master of Science, Middle East Technical University, 2010.