MULTIMODAL SPATIAL DEIXIS IN SPEECH AND GESTURES: FRAMES OF REFERENCE IN EXPLANATORY DISCOURSE

MULTIMODAL SPATIAL DEIXIS IN SPEECH AND GESTURES: FRAMES OF REFERENCE IN EXPLANATORY DISCOURSE

Author:  O.K. Iriskhanova1, 2, M.I. Kiose1, 2, A.V. Leonteva1, 2, O.V. Agafonova2

Affiliation:  1Institute of Linguistics RAS (Moscow, Russia) 2Moscow State Linguistic University (Moscow, Russia)

Abstract:  The study explores the distribution of deictic words and gestures in multimodal explanatory discourse. To proceed, we compile the corpus of speech and gesture in which the experiment participants are explaining the difference between close synonyms.
The study develops the method of modelling deixis as a discourse frame of reference which allows to identify objects and situations via pointing at them.
We show that two frames of reference – egocentric and allocentric – help explore the discourse semantics of deictic words and gestures on the same grounds using the coordinates of two frames which are the distance from the speaker (closer to the speaker / farther from the speaker) and the type of communicative situation in discourse (the internal communicative situation / the external communicative situation).
The results reveal that the differences between the two frames of reference may not always be explicit in the semantics of deictic words, still they may appear in the use of gesture types.

Keywords:  deixis, multimodal behavior, deictic word, gesture, egocentric frame of reference, allocentric frame of reference.

References:  Апресян Ю.Д. Дейксис в лексике и грамматике и наивная модель мира // Семиотика и информатика, 1986. 28. С. 5-33. [Apresyan Yu.D. Deyksis v leksike i grammatike i naivnaya model′ mira // Semiotika i informatika, 1986. 28. S. 5-33.]
Апресян В.Ю. Тут, здесь и сейчас. О временных значениях пространственных дейктических слов // Русский язык в научном освещении, 2014. 27(1). С. 9-41. [Apresyan V.Yu. Tut, zdes′ i seychas. O vremennykh znacheniyakh prostranstvennykh deykticheskikh slov // Russkiy yazyk v nauchnom osveshchenii, 2014. 27(1). S. 9-41.]
Борисова Е.Г., Овчинникова Т.Е. Параметр близости в метафорическом пространстве // Компьютерная лингвистика и интеллектуальные технологии. 2011. 10. С. 153-158. [Borisova E.G., Ovchinnikova T.E. Parametr blizosti v metaforicheskom prostranstve // Komp′yuternaya lingvistika i intellektual′nyye tekhnologii. 2011. 10. S. 153-158.]
Ирисханова О.К. К вопросу о коммуникативном мышлении: полимодальное измерение аспектуальности // Когнитивные исследования языка. 2017. Вып. 30. C. 57-60. [Iriskhanova O.K. K voprosu o kommunikativnom myshlenii: polimodal′noye izmereniye aspektual′nosti // Kognitivnyye issledovaniya yazyka. 2017. Vyp. 30. S. 57-60.]
Ирисханова О.К., Абрамова Ю.С. Нечеткая референция при решении метаязыковых задач как полимодальный феномен // Когнитивные исследования языка. 2021. Вып. 4 (47). С. 233-244. [Iriskhanova O.K., Abramova YU.S. Nechetkaya referentsiya pri reshenii metayazykovykh zadach kak polimodal′nyy fenomen // Kognitivnyye issledovaniya yazyka. 2021. Vyp. 4 (47). S. 233-244.]
Падучева Е.В. Режим интерпретации как контекст, снимающий неоднозначность // Компьютерная лингвистика и интеллектуальные технологии. 2008. 7(14). С. 412-419. [Paducheva E.V. Rezhim interpretatsii kak kontekst, snimayushchiy neodnoznachnost′ // Komp′yuternaya lingvistika i intellektual′nyye tekhnologii. 2008. 7(14). S. 412-419.]
Степанов Ю.С. Язык и Метод. К современной философии языка. М.: Языки русской культуры, 1998. [Stepanov Yu.S. Yazyk i Metod. K sovremennoy filosofii yazyka. M.: Yazyki russkoy kul′tury, 1998.]
Ожегов С.И., Шведова Н.Ю. Толковый словарь русского языка. URL: https://slovarozhegova.ru/word.php?wordid [Ozhegov S.I., Shvedova N.YU. Tolkovyy slovar′ russkogo yazyka. URL: https://slovarozhegova.ru/word.php?wordid]
Ушаков Д.Н. Толковый словарь русского языка. URL: https://slovar.cc/rus/ushakov [Ushakov D.N. Tolkovyy slovar′ russkogo yazyka. URL: https://slovar.cc/rus/ushakov]
Alibali M.W., Kita S. Gesture highlights perceptually present information for speakers // Gesture, 2010. 10. P. 3-28.
Bangerter A. Using pointing and describing to achieve joint focus of attention in dialogue // Psychological Science, 2004. 15. P. 415-419.
Berman R.A., Nir-Sagiv B. Comparing narrative and expository text construction across adolescence: A developmental paradox // Discourse Processes, 2007. 43(2). P. 79-120.
Cienki A. Ten lectures on Spoken language and Gesture from Perspective of Cognitive Linguistics. Issues of Dynamicity and Multimodality. Leiden, Boston, Brill, 2017.
Clark H.H. Depicting as a method of communication // Psychological Review, 2016. 123. P. 324-347.
Cooperrider K. Reference in action: links between pointing and language (Ph.D. thesis), University of California at San Diego, San Diego, CA, United States, 2011.
Cooperrider K. The co-organization of demonstratives and pointing gestures // Discourse Processes, 2015. 53. P. 632-656.
Hassemer J., McCleary L. The multidimensionality of pointing // Gesture, 2018. 17. P. 417-463.
Iriskhanova O., Cienki A. The semiotics of gestures in cognitive linguistics: Contributions and challenges // Issues in Cognitive Linguistics, 4, 2018. P. 25-36.
Iriskhanova O., Kiose M., Leonteva A., Agafonova O. Staged and natural gesturing in argumentation and description // Компьютерная лингвистика и интеллектуальные технологии, 2022. 21(2). P. 1055-1067.
Kendon A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
Kita S. How representational gestures help speaking // Language and gesture / D. McNeill (ed.). Cambridge: Cambridge University Press, 2000. P. 162-185.
Kita S. (ed.). Pointing: Where language, culture, and cognition meet. Mahwah, NJ: Lawrence Erlbaum Associates, 2003.
Kok K.I., Cienki A. Cognitive Grammar and gesture: Points of convergence, advances and challenges // Cognitive Linguistics, 2016. 27 (1). P. 67-100.
Krahmer E., van Erk S., Verleg A. Graphbased Generation of Referring Expressions // Computational Linguistics, 2003. 29(1). P. 53-72.
Le Guen O. Modes of pointing to existing spaces and the use of frames of reference // Gesture, 2011. 11 (3). P. 271-307.
Levinson S.C. Frames of reference and Molyneaux’s question // Language and Space / eds. by Bloom P, Peterson M.A., Nadel L., Garrett M.F. Cambridge MA: MIT Press, 1996. P. 109-170.
Levinson S.C. Space in Language and Cognition: Explorations in Cognitive Diversity, Cambridge University Press, Language, Culture and Cognition 2003. 5.
Li P., Gleitman L. Turning the tables: Language and spatial reasoning // Cognition, 2002. 83. P. 265-294.
Longacre R.E. The grammar of discourse. N.Y.: Plenum, 1983.
McNeill D. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press, 1992.
Majid A., Bowerman M., Kita S., Haun D.B.M., Levinson S.C. Can language restructure cognition? The case for space // Trends in Cognitive Science, 2004. 8. P. 108-114.
Muller-Tomfelde C. Dwell-Based Pointing in Applications of Human Computer Interaction // Human-Computer Interaction – INTERACT 2007 (LNCS 4662), 2009. P. 560-573.
Pagán Cánovas C., Valenzuela J., Alcaraz Carrión D., Olza I., Ramscar M. Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions // PLOS ONE, 2020. 15(6): e0233892.
Shusterman A., Li P. A Framework for Work on Frames of Reference // Core Knowledge and Conceptual Change / D. Barner, A.S. Baron (eds.). Oxford: Oxford University Press, 2016. P. 191-206.

Acknowledgments:  This research is part of the project “Kinesic and vocal aspects of communication: parameters of variance”
(№ FMNE-2022-0015) which is carried out at the Institute of Linguistics RAS.

For citation:  Iriskhanova, O. K., Kiose, M. I., Leonteva, A. V., & Agafonova, O. V. (2022). Multimodal spatial deixis in speech and gestures: frames of reference in explanatory discourse. Voprosy Kognitivnoy Lingvistiki, 4, 17-31 (In Russ.).

Pages:  17-31