NEURAL AND OCULAR CORRELATES OF CONCEPTUAL GROUNDING IN VERBAL INTERACTION: A MULTIMODAL HYPERSCANNING APPROACH

Date

2024-9-28

Author

Yilmaz, Efecan

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

53
views

62
downloads

Cite This

Effective communication depends on interlocutors agreeing on how to resolve ambiguities and constructing conceptual representations into shared meanings. To achieve this, continuous adaptation by all participants to establish and to maintain alignments is required, which is known as grounding in communication. Previous work on grounding has largely explored linguistic and ocular phenomena, whereas neural correlates have not been explicitly studied. In this thesis, hyperscanning and verbal interaction analysis approaches are combined to investigate the process of establishing a common ground – a social aspect of communication – using a multi-modal approach that includes EEG, eye tracking, and fNIRS devices. Eye-tracking results indicated that with establishment of common ground, shared gaze becomes less central with explicit verbal communication taking precedence. Gaze entropy reduces as interlocutors move from high workload visual feature extraction to utilizing stable common meanings. EEG results showed that increased inter-brain synchrony in the frontal, midline, temporal, parietal, and occipital electrode sites in delta, theta, and gamma bands, which aligns with sustained attention, memory processes, and conscious integration of perceptions into actions that all correlate with shared linguistic contexts in communication. The synchrony observed over different frequency bands highlights the collaborative nature of communication and the role of pragmatic alignment in facilitating effective communication. fNIRS hyperscanning results were inconclusive due to device malfunctions. However, single device hemodynamic response analysis revealed significant effects of repeated trials and experiment conditions on PFC oxygenation, supporting the role of this region in interaction and communication, particularly in relation to working memory and communicative intent. The findings reveal that dyadic communication involves multiplex synchrony, relying on both verbal and non-verbal cues as they established a common ground, with both neural and ocular underpinnings.

Subject Keywords

Conceptual alignment, hyperscanning, common ground, eye-tracking, fNIRS, electroencephalography

URI

https://hdl.handle.net/11511/111441

Collections

Graduate School of Informatics, Thesis