MULTIMODAL CONCEPTUALIZATION OF IMMERSIVE VISUAL ART: DISCOURSE SCHEMAS IN SPEECH, GESTURE AND GAZE

MULTIMODAL CONCEPTUALIZATION OF IMMERSIVE VISUAL ART: DISCOURSE SCHEMAS IN SPEECH, GESTURE AND GAZE

Author:  V.O. Potekhin 1, M.I. Kiose 2

Affiliation:  1 Diplomatic Academy of the Russian Ministry of Foreign Affairs, 2 Moscow State Linguistic University

Abstract:  This study addresses the problem of multimodal thinking of an individual exposed to immersive visual art that stimulates one’s transformation from an observer into a participant in the stimulus situation. To proceed, we conduct an experiment which tests a subject’s multimodal behavior at the stage with stimulus exposure (via one’s speech, gesture, and gaze) and immediately after the viewing experience (via one’s speech and gesture).
This paper aims to establish the contingency of immersive environment upon the subject’s multimodal behavior at both stages. We hypothesize that the exposure regime of the immersive art stimulus (at the while and after viewing stages) modulates discourse schemas distribution in the subject’s speech synchronized with functional gestures. Additionally, we assume that discourse schemas distribution modulates gaze behavior at the while viewing stage. 44 individual probes are subjected to analysis and further examined in terms of discourse schemas in speech, gesture, and gaze measures.
The results attest to significant variance in gestural activity with representational, deictic, and pragmatic gesture types prevailing at the after viewing stage; this stage was also found to warrant more descriptive discourse schemas in speech. We also established the contingency of discourse schemas use on the subject’s gaze measures which is manifest in more focal gaze movements when relating narration and description and in more ambient gaze movements when relating exposition.
The study offers a contribution to advancing cognitive theories of multimodal thinking as modulated by communicative environment.

Keywords:  immersive visual art, multimodal behavior, discourse schema, speech, gaze behavior, gesture

References:  Apostolopoulos J.G., Chou P.A., Culbertson B., Kalker T., Trott M.D. & Wee S. The road to immersive communication // Proceedings of IEEE, 100(4), Apr. 2012 https://doi:10.1109/JPROC.2011.2182069
Berman R.A. & Nir-Sagiv B. Comparing narrative and expository text construction across adolescence: A developmental paradox // Discourse Processes, 43(2), 2007. P. 79-120. https://doi.org/ 10.1080/01638530709336894
Brône G., Oben B., Jehoul A., Vranjes J. & Feyaerts K. Eye gaze and viewpoint in multimodal interaction management // Cognitive Linguistics, 28(3), 2017. P. 449-483.
Cajar A., Engbert R. & Laubrock J. Potsdam Eye-Movement Corpus for scene memorization and search with color and spatial-frequency filtering // Frontiers in Psychology, 13, 2022. URL: https://www.frontiersin.org/articles/10.3389/fpsyg.2022.850482
Caselli T., Sprugnoli R. & Moretti G. Identifying communicative functions in discourse with content types // Language Resources and Evaluation, 56(2), 2022. P. 417-450. https://doi:10.1007/s10579-021-09554-4.
Chafe W.L. Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing. Chicago: University of Chicago Press, 1994.
Chan D.M. & Kelly S. Construing events first-hand: Gesture viewpoints interact with speech to shape the attribution and memory of agency // Memory and Cognition, 49, 2021. P. 884-894. https://doi.org/10.3758/s13421-020-01135-0
Cienki A. Cognitive Linguistics, gesture studies, and multimodal communication // Cognitive Linguistics, 27(4), 2016. P. 603-618. https://doi.org/10.1515/cog-2016-0063
Cienki A. Schemas in Cognitive Linguistic Theory and in Gesture Studies // Ten Lectures on Spoken Language and Gesture from the Perspective of Cognitive Linguistics. Leiden: Brill, 2017. P. 49-68. https://doi.org/10.1163/9789004336230_004
Fuchs I., Ansorge U., Redies C. & Leder H. Salience in paintings: bottom-up influences on eye fixations // Cognitive Computing. Mar 1; 3(1), 2011. P. 25-36.
Goldin-Meadow S. Hearing gesture: How our hands help us think. Harvard: Harvard University Press, 2005.
Grady J. Image schemas and perception: Refining a definition // From Perception to Meaning. Image Schemas in Cognitive Linguistics / ed. by B. Hampe. Berlin: Mouton de Gruyter, 2005. P. 35-56.
Grigorovich S., Blinnikova I. & Izmalkova A. Strategies of space scan in the process of visual semantic search. In Soloviev V.D., Poliakov V.N. and Masalova S. I. (eds.) // Cognitive Modeling. Rostov-on-Don: Southern Federal University Press, 2014. P. 49-53.
Henderson J.M., Shinkareva S. V., Wang J., Luke S. G., Olejarczyk J. Predicting Cognitive State from Eye Movements // PLoS ONE, 8(5), e64937, 2013. https://doi.org/10.1371/journal.pone.0064937
Holler J. Speakers’ Use of Interactive Gestures as Markers of Common Ground, in: Kopp, S. and Wachsmuth, I. (eds.) // GW 2009, Lecture Notes in Computer Science 5934, 2010. P. 11-22.
Hostetter A.B. & Alibali M.W. Gesture as simulated action: Revisiting the framework // Psychonomic Bulletin & Review, 26(3), 2019. P. 721-752. https://doi.org/10.3758/s13423-018-1548-0
Iriskhanova O.K. & Cienki A. The semiotics of gestures in cognitive linguistics: contribution and challenges // Issues of Cognitive Linguistics, 4 (57), 2018. P. 25-36. https://doi.org/10.20916/1812-3228-2018-4-25-36
Iriskhanova O.K., Kiose M.I., Leonteva A.V., Agafonova O.V. & Petrov A.A. Multimodal collaboration in expository discourse: verbal and nonverbal moves alignment // Lecture Notes in Artificial Intelligence, 14338, 2023. P. 1-14. https://doi.org/10.1007/978-3-031-48309-7_29
Johnson M. The meaning of the body: Aesthetics of human understanding. Chicago: Chicago University Press, 2007. DOI: 10.7208/chicago/9780226026992.001.0001
Kokkos A. Exploring Art for Perspective Transformation. Series: International Issues in Adult Education, v. 32. Leiden: Brill, 2021.
MacSaveny T. Towards a Description of Descriptive Discourse // GIALens, 3, 2010. P. 1-5.
Mandler J.M. & Johnson N.S. Remembrance of things parsed: Story structure and recall // Cognitive Psychology, 9(1), 1977. P. 111-151.
McNeill D. Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press, 1992.
McNeill D. Gesture, gaze, and ground, in Renals, S. & Bengio, S (eds.) // Proceedings of Machine learning for multimodal interaction: Second international workshop 2005, Berlin/Heidelberg: Springer Verlag, 2006. P. 1-14.
Mittelberg I. Gestures as image schemas and force gestalts: A dynamic systems approach augmented with motion-capture data analyses // Cognitive Semiotics, 11(1), 2018. Article number: 20180002 https://doi.org/10.1515/cogsem-2018-0002
Nippold M.A. & Scott C. M. Expository discourse in children, adolescents, and adults: Development and disorders. N.Y.: Psychology Press, 2010.
Parrill F., & Stec K. Seeing first person changes gesture but saying first person does not // Gesture, 17(1), 2018. P. 158-175. https://doi.org/10.1075/gest.00014.par
Pustejovsky J. The Generative Lexicon. Cambridge, Massachusetts, USA: MIT Press, 1995.
Rubio-Tamayo J.L., Gertrudix B.M. & García G.F. Immersive Environments and Virtual Reality: Systematic Review and Advances in Communication, Interaction and Simulation // Multimodal Technol. Interact., 1, 2017. P. 21. https://doi.org/10.3390/mti1040021
Schraube E. First-Person Perspective. In: T. Teo (ed.) Encyclopedia of Critical Psychology. Springer, New York, NY, 2014. https://doi.org/10.1007/978-1-4614-5583-7_113
Scott C. M. & Windsor J. General language performance measures in spoken and written narrative and expository discourse of school-age children with language learning disabilities // Journal of Speech, Language, and Hearing Research, 43(2), 2000. P. 324-339.
Shiffrin D. Discourse markers: Language, meaning, and context, in Shiffrin, D., Tannen, D., and Hamilton, H.E. (eds.), The Handbook of Discourse Analysis. Oxford: Blackwell Publishers, 2001. P. 54-75.
Shilina M.G. & Wirth J. Immersive technologies in media: Towards the concept of generative mediatization? // RUDN Journal of Studies in Literature and Journalism, 26(4), 2021. P. 672-680.
van Eemeren F.H. Strategic maneuvering. Extending the pragma-dialectical theory of argumentation. Argumentation in context 2. Amsterdam-Philadelphia: John Benjamins, 2010. https://doi.org/10.1075/aic.2 (in English)
Velichkovsky B.M., Joos M., Helmert J.R. & Pannasch S. Two visual systems and their eye movements: evidence from static and dynamic scene perception. In: Bara B.G., Barsalou L., Bucciarelli M. (eds.). Proceedings of the XXVII annual conference of the Cognitive Science Society. Mahwah: Lawrence Erlbaum 2005. P. 2283-2288.
Walker F., Bucker B., Anderson N.C., Schreij D. & Theeuwes J. Looking at paintings in the Vincent Van Gogh Museum: Eye movement patterns of children and adults. PLoS ONE 12(6): e0178912, 2017. https://doi.org/10.1371/journal.pone.0178912
Wolfe M.B.W. & Woodwyk J. Processing and memory of information presented in narrative or expository texts // The British Journal of Educational Psychology, 80(3), 2010. P. 341-362. https://doi.org/10.1348/000709910X485700
Wu Y.C. & Coulson S. How iconic gestures enhance communication: An ERP study. Brain and Language, 101(3), 2007. P. 234-245. https://doi.org/10.1016/j.bandl.2006.12.003
Ирисханова O.K., Киосе M.И., Леонтьева А.В., Агафонова О.В. Полимодальный пространственный дейксис в речи и жестах: системы координат в экспланаторном дискурсе // Вопросы когнитивной лингвистики. 2022. № 4. С. 17-31 [Irishanova O.K., Kiose M.I., Leont'eva A.V., Agafonova O.V. Polimodal'nyj prostranstvennyj dejksis v rechi i zhestah: sistemy koordinat v jeksplanatornom diskurse // Voprosy kognitivnoj lingvistiki. 2022. № 4. S. 17-31]. https://doi.org/10.20916/1812-3228-2022-4-17-31
Кибрик А.А. Модус, жанр и другие параметры классификации дискурсов // Вопросы языкознания. 2009. № 2. P. 3-21. [Kibrik A.A. Modus, zhanr i drugie parametry klassifikacii diskursov // Voprosy Jazykoznanija. 2009. № 2. S. 3-21].
Потехин В.О. Синхронизация речи-рассуждения и жеста в иммерсивной коммуникации у испытуемых с разным уровнем интро- / экстраверсии // Вестник Московского государственного лингвистического университета. Гуманитарные науки. 2023. № 2(870). С. 100-107. [Potekhin V.O. Sinhronizaciya rechi-rassuzhdeniya i zhesta v immersivnoj kommunikacii u ispytuemyh s raznym urovnem intro- / ekstraversii // Vestnik Moskovskogo gosudarstvennogo lingvisticheskogo universiteta. Gumanitarnye nauki. 2023. № 2(870). S. 100-107].
Прокофьева О.Н., Ирисханова О.К., Киосе М.И. Речь и жесты в описательном дискурсе // Полимодальные измерения дискурса / О.К. Ирисханова (ред.). М.: ЯСК, 2021. С. 63-109 [Prokof'eva O.N., Iriskhanova O.K., Kiose M.I. Rech' i zhesty v opisatel'nom diskurse // Polimodal'nye izmereniya diskursa / O.K. Iriskhanova (red.). M.: YASK, 2021. S. 63-109].

For citation:  Potekhin, V. O., & Kiose, M. I. (2024). Multimodal Conceptualization of Immersive Visual Art: Discourse Schemas in Speech, Gesture and Gaze. Voprosy Kognitivnoy Lingvistiki, 3, 20-32.

Pages:  20-32