ASSESSING THE EFFECTIVENESS OF CHATGPT IN GENERATING MULTIPLE-CHOICE QUESTIONS FOR ENGLISH LANGUAGE TEACHING: A MIXED-METHODS APPROACH

Date

2026-6-25

Author

Mutlu, Barış

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

70
views

0
downloads

Cite This

This study examined ChatGPT-generated multiple-choice English as a Foreign Language (EFL) reading comprehension questions produced via meta-prompting in a university preparatory school context. Adopting a convergent mixed methods design, data were collected from 116 intermediate-level EFL students through an AI-generated reading test and from 10 Subject Matter Experts (SMEs) through a researcher-developed evaluation questionnaire. Expert data from Likert-scale ratings were analyzed using descriptive statistics, while data from open-ended questions were analyzed using thematic analysis. Student response data were subjected to CTT-based psychometric analysis. Expert evaluations demonstrated acceptable interrater reliability (ICC = .79) and revealed favorable ratings for wording clarity and level-appropriateness, while distractor-related aspects consistently received lower scores. Qualitative findings indicated that implausible distractors and text-independence, defined as the tendency for items to be answerable without engaging with the reading text, were pervasive problems. Psychometric analysis of student responses revealed a severe ceiling effect, with 12 of the 20 items answered correctly by all participants, while discrimination indices were negligible and nearly all distractors were non-functional. The convergent findings indicate that ChatGPT can generate items meeting surface-level language standards. However, distractor construction remains a persistent limitation that meta-prompting alone does not resolve. These findings underscore the need for expert review and psychometric validation prior to operational use, with implications for EFL teachers, test developers, and teacher educators.

Subject Keywords

EFL reading comprehension, automated item generation, multiple-choice questions, Classical Test Theory, ChatGPT

URI

https://hdl.handle.net/11511/119692

Collections

Graduate School of Social Sciences, Thesis