Show/Hide Menu
Hide/Show Apps
anonymousUser
Logout
Türkçe
Türkçe
Search
Search
Login
Login
OpenMETU
OpenMETU
About
About
Açık Bilim Politikası
Açık Bilim Politikası
Frequently Asked Questions
Frequently Asked Questions
Browse
Browse
By Issue Date
By Issue Date
Authors
Authors
Titles
Titles
Subjects
Subjects
Communities & Collections
Communities & Collections
A gaze-centered multimodal approach to face-to-face interaction
Download
index.pdf
Date
2020
Author
Arslan Aydın, Ülkü
Metadata
Show full item record
This work is licensed under a
Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
.
Item Usage Stats
0
views
0
downloads
Face-to-face conversation implies that interaction should be characterized as an inherently multimodal phenomenon involving both verbal and nonverbal signals. Gaze is a nonverbal cue that plays a key role in achieving social goals during the course of conversation. The purpose of this study is twofold: (i) to examine gaze behavior (i.e., aversion and gaze on face) and relations between gaze and speech in face to face interaction, (ii) to construct computational models to predict gaze behavior using high-level speech features. We employed a job interview setting, where pairs (a professional interviewer and an interviewee) conducted mock job interviews. Twenty-eight pairs of native speakers took part in the experiment. Two eye-tracking glasses recorded the scene video, the audio and the eye gaze position of the participants. To achieve the first purpose, we developed an open-source framework, named MAGiC (A Multimodal Framework for Analyzing Gaze in Communication), for the analyses of multimodal data including video recording data for face tracking, gaze data from the eye trackers, and the audio data for speech segmentation. We annotated speech with two methods: (i) ISO 24617-2 Standard for Dialogue Act Annotation and, (ii) using tags employed by the previous studies that examined gaze behavior in a social context. We then trained simplified versions of two CNN architectures (VGGNet and ResNet) by using both speech annotation methods.
Subject Keywords
Interpersonal communication.
,
Mobile Eye Tracking
,
face-to-face interaction
,
gaze analysis
,
ISO 24617-2 standard
,
CNN for time series
URI
http://etd.lib.metu.edu.tr/upload/12625103/index.pdf
https://hdl.handle.net/11511/45474
Collections
Graduate School of Informatics, Thesis