Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows

Download

index.pdf

Date

2021-12-01

Author

Acartürk, Cengiz
Emil, Serap

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

128
views

31
downloads

In daily communication and instructional settings, arrows and spontaneous gestures usually convey similar spatial information to interlocutors. They can visualize spatial information in 2D or 3D space based on shared conceptual and spatial representations. The present study investigates the relationship between spontaneous gestures and arrows from the perspective of generation and comprehension in instructional settings. We introduce a corpus of arrows in written multimodal text and a corpus of gestures generated during communication through multimodal instructional material. We also report an experimental investigation that used corpus material to study the impact of gestures on comprehension learning. A comparison of the corpora reveals functional similarities and differences between arrows and gestures and their complementary role in multimodal communication. The findings suggest that arrows and gestures may share a common conceptual space during communication, having a significant impact on comprehension and learning.

Subject Keywords

Modality, Deixis, Arrows, Gestures, Multimodality

URI

https://hdl.handle.net/11511/95212

Journal

Revista Signos

DOI

https://doi.org/10.4067/s0718-09342021000300867

Collections

Graduate School of Informatics, Article

Suggestions

OpenMETU
Core

Gestures in communication through line graphs Özge, Alaçam; Christopher, Habel; Acartürk, Cengiz (null; 2012-08-31) Line graphs are widely used in communication settings, for conveying information about states and processes that unfold in time. The communication is achieved by the contribution of other modalities than graphs, such as language and gestures. In a set of experimental investigations, we analyzed the production and comprehension of gestures during communication through line graphs. The findings reveal a systematic use of gestures as well as the limitations of cognitive resources due to the split of attention ...
Multimodal comprehension of graphics with textual annotations The role of graphical means relating annotations and graph lines Acartürk, Cengiz; Çağıltay, Kürşat (2008-09-30) Graphs are often accompanied by text, i.e. linguistically coded information. augmenting information presented diagrammatically. Thus graph comprehension by human Often Constitutes comprehension and integration of information provided by different representational modalities, namely graphical elements and Verbal Constituents. In this Study We focus on text annotations to line graphs providing information about events, processes and their temporal properties its well it temporal relations about the events and...
Bimodal visualisation: A financial trading case study Taşkaya Temizel, Tuğba (2003-01-01) A visualisation system that deals with two modalities of information - numerical and textual - is presented. The current application domain is that of prediction in financial trading. The system synthesises news stories and time series data to generate buy/sell signals for financial instruments.
Multichannel Audio Coding Based on Analysis by Synthesis Elfitri, Ikhwana; Günel Kılıç, Banu; AHMET, KONDOZ (2011-04-01) Spatial hearing enables translation of an auditory scene into a perceived 3-D image by interpreting the acoustic cues related to the sounding objects, their locations, and the physical characteristics of the space. Spatial audio production requires multichannel audio signals in order to convey this information and increase the realism of a real or virtual environment for applications such as the home entertainment, virtual reality, and remote collaboration. As demand to spatial audio continues to expand, ef...
Multi-modal stereo-vision using infrared / visible camera pairs Yaman, Mustafa; Kalkan, Sinan; Department of Computer Engineering (2014) In this thesis, a novel method for computing disparity maps from a multi-modal stereo-vision system composed of an infrared-visible camera pair is introduced. The method uses mutual information as the basic similarity measure where a segmentation based adaptive windowing mechanism is proposed along with a novel mutual information computation surface for greatly enhancing the results. Besides, the method incorporates joint prior probabilities when computing the cost matrix in addition to negative mutual info...

Citation Formats

C. Acartürk and S. Emil, “Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows,” Revista Signos, vol. 54, no. 107, pp. 867–892, 2021, Accessed: 00, 2022. [Online]. Available: https://hdl.handle.net/11511/95212.