Multimodal Interaction Mechanism of Speech Videos Based on ELANA Case Study of TED Educational Speech Video
-
摘要: 近年来,多模态语篇分析已成为了语言学研究的热点。演讲视频是典型的动态多模态语篇。文章运用Norris提出的多模态互动分析理论,采用“模态密度前景−背景连续统一体(modal density foreground-background continuum)”的分析框架,以ELAN多模态分析软件作为模态标注工具,结合Norris提出的交流模态,对一则TED教育类动态多模态演讲视频中各交流模态在构建演讲信息中所起的作用以及这些模态之间的互动机制进行分析。研究发现在演讲视频的问题导入、问题提出、问题分析和问题解决这四个部分中各模态使用的强度和密度存在差异,言语交流模态与其他交流模态之间呈现强化、互补和并列关系从而共同构建多模态互动意义。
-
关键词:
- Ted教育类演讲视频 /
- 多模态互动 /
- 交流模态 /
- ELAN
Abstract: In recent years, multimodal discourse analysis has become a hot topic in the study of linguistics. Speech video is a typical dynamic multimodal discourse. This paper adopts modal density foreground-background continuum as methodological framework and ELAN multimodal analysis software as the modal annotation tool. According to multimodal interaction analysis theory proposed by Norris, this paper analyzes the role of each communicative mode in the construction of this speech information and the interactive mechanism between these modes in a TED educational speech video. It is found that modal intensity and modal density of each communicative mode in the problem introduction part, problem posing part, problem analysis part and problem solving part of this speech video are different. Spoken language and other communicative modes show the relationship of reinforcement, complementarity and juxtaposition, thereby jointly constructing the meaning of multimodal interaction.-
Key words:
- TED educational speech video /
- multimodal interaction /
- communicative modes /
- ELAN
-
图 1 ELAN标注界面
表 1 语料特征
语料特征 TED教育类演讲 问题导入时长 00:10—01:14 问题提出时长 01:14—02:41 问题分析时长 02:41—04:12 问题解决时长 04:13—05:59 话题类型 教育类:Grit: The power of passion and perseverance 场景布局 TED 圆形讲台 表 2 ELAN附码表
附码 含义 附码 含义 1. Spoken language /S-/ 2. Prosody /P-/ Intonation /PI-/ [PIF] Intonation fall [PIR] Intonation rise [PIRF] Intonation rise & fall [PIS] Intonation stress 3.Hand movement [HN-] [HNI] Hand iconic [HNM] Hand metaphoric [HNDe] Hand deictic [HNB] Hand beats 4.Head movement [HD-] [HDU] Head up [HDD] Head down [HDN] Head nod [HDMA] Head move around 5.Facial expression [F-] [FFS] Face smile [FFSe] Face serious 6.Eye movement (Gaze) [EGA] Gaze at the audience [EGAr] Gaze around 7. Posture [PO-] [POO] Posture open [POC] Posture closed 表 3 ELAN对整个演讲视频的各交流模态标注数据汇总表
层级 标注
次数平均 总标注
时长/s标注时长
百分比/%言语(spoken language) 19 13.378947 254.2 68.714 手势(gesture) 23 10.023913 230.55 62.321 头部动作(head movement) 16 15.78125 252.5 68.254 脸部表情(facial expression) 12 19.908335 238.9 64.578 眼神(gaze) 15 17.563334 263.45 71.214 身体姿势(posture) 20 13.99 279.8 75.634 音韵(prosody) 24 13.14375 315.45 85.271 表 4 ELAN对演讲视频四个部分各交流模态的标注数据汇总表
标注层 标注内容 标注次数 持续时长/s 标注时长百分比/% 问题
导入问题
提出问题
分析问题
解决问题
导入问题
提出问题
分析问题
解决问题
导入问题
提出问题
分析问题
解决问题
导入问题
提出问题
分析问题
解决手势 HNDe HNDe HNDe HNDe 2 3 2 2 00:17 00:28 00:28 00:35 16.9 28.2 25.4 35.4 HNB HNB HNB 1 1 0 1 00:09 00:06 0 00:06 8.6 6 0 5.55 HNI HNI HNI HNI 2 3 2 2 00:07 00:23 00:21 00:34 7 22.5 21.1 34.2 HNM 0 2 0 0 0 00:07 0 0 0 7.6 0 0 头部动作 HDN HDN 1 0 0 1 00:09 0 0 00:05 9.1 0 0 4.9 HDMA HDMA HDMA HDMA 2 4 2 4 00:18 00:55 00:27 00:85 28.4 52.9 27 84.8 HDD HDD HDD 1 1 1 0 00:18 00:08 00:20 0 17.7 7.7 20 0 脸部表情 FFS FFS FFS FFS 1 2 1 1 00:09 00:29 00:42 00:04 8.7 28.6 42.5 4.4 FFSe FFSe FFSe FFSe 2 2 1 2 00:23 00:33 00:25 00:74 22.4 32.2 25.4 74.7 眼神 EGAr EGAr EGAr EGAr 2 3 2 4 00:25 00:47 00:31 00:39 25.3 46.1 31.3 39.6 EGA EGA EGA EGA 2 1 1 2 00:31 00:20 00:22 00:47 30.7 20.3 22.4 47.45 身体姿势 POO POO POO POO 2 3 2 3 00:26 00:48 00:48 00:66 26.4 57.9 47.7 66.3 POC POC POC POC 2 2 1 3 00:10 00:16 00:07 00:19 8.0 16.1 7.3 19 音韵 PIF PIF PIF PIF 3 3 1 2 00:25 00:67 00:23 00:25 25.5 66.6 22.8 24.5 PIS PIS PIS PIS 3 1 2 1 00:26 00:06 00:13 00:03 25.95 5.55 12.85 2.5 PIRF PIRF PIRF PIRF 1 1 2 4 00:04 00:11 00:35 00:79 4.3 11.3 24.6 79 表 5 演讲的问题导入部分言语模态与其他交流模态的例证
言语模态 头部动作 手势 脸部表情 眼神 身体姿势 音韵 互动关系 I left a very demanding job in management consulting for a job that was even more demanding: teaching. HDN / FFS EGAr POO PIF/PIS 强化 I.Q. was not the only difference between my best and my worst students. HDMA HNB FFSe EGAr POC PIS 强化 I was firmly convinced that every one of my students could learn the material if they worked hard and long enough. HDMA HNI FFSe EGA POC PIS 强化 表 6 演讲的问题提出部分言语模态与其他交流模态的例证
言语模态 头部动作 手势 脸部表情 眼神 身体姿势 音韵 互动关系 What we need in education is a much better understanding of students and learning from a motivational perspective, from a psychological perspective. HDN HNI FFSe EGAr POC PIF 互补 I started studying kids and adults in all kinds of super challenging settings, and in every study my question was who is successful here and why. HDMA HNDe FFS EGA POO PIF 互补 My research team and I went to West Point Military Academy...the National Spelling Bee... rookie teachers ...We partnered with private companies... HDMA HNDe FFSe EGAr POC PIF 互补 表 7 演讲的问题分析部分言语模态与其他交流模态的例证
言语模态 头部动作 手势 脸部表情 眼神 身体姿势 音韵 互动关系 Grit is having stamina. Grit is sticking with your future, day in, day out,... Grit is living life like it's a marathon, not a sprint. HDD HNI FFSe EGAr POC PIF 互补 I asked thousands of high school juniors to take grit questionnaires,
...Turns out that grittier kids were significantly more likely to graduate, ...HDMA HNDe FFS EGA POO PIF 互补 ... the most shocking thing about grit is how little we know, how little science knows... HDMA HNDe FFSe EGA POC PIRF 互补 表 8 演讲的问题解决部分言语模态与其他交流模态的例证
言语模态 头部动作 手势 脸部表情 眼神 身体姿势 音韵 互动关系 Parents and teachers ask me“how do I build grit in kids?...” The honest answer is, I don't know. HDMA HNDe FFS EGAr POO PIRF 并列 Our data show... many talented individuals ...do not follow through on their commitments. ...grit is usually unrelated... HDMA HNI FFSe EGAr POO PIS/
PIRF并列 Growth mindset is a great idea for building grit...we need more, ...we need to... .and we need to.... we need to be gritty about getting our kids grittier. HDN
HDMAHNDe FFSe EGA POC PIS/
PIRF并列 -
[1] KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design [M]. London: Routledge, 1996. [2] NORRIS S. Analyzing Multimodal Interaction: A Methodological Framework [M]. New York: Routledge, 2004. [3] 张佐成, 刑建玉, 王彦, 等. 多模态即席话语研究 [M]. 广州: 世界图书出版公司, 2014. [4] KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design (2nd ed. )[M]. London: Routledge, 2006. [5] GU Y. Multimodal text analysis: a corpus linguistic approach to situated discourse[J]. Text & Talk, 2006, 26(2):127-167. [6] O’HALLORAN K L. Multimodal Discourse Analysis: Systemic-Functional Perspectives [M]. London: Continuum, 2004. [7] BALDRY A & THIBAULT P. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Course Book [M]. London: Equinox, 2006. [8] 李战子. 多模式话语的社会符号学分析[J]. 外语研究,2003(5):1-8. doi: 10.3969/j.issn.1005-7242.2003.05.001 [9] 李战子,陆丹云. 多模态符号学: 理论基础, 研究途径与发展前景[J]. 外语研究,2012(2):1-8. doi: 10.3969/j.issn.1005-7242.2012.02.001 [10] 胡壮麟. 社会符号学研究中的多模态化[J]. 语言教学与研究,2007(1):1-10. [11] 张德禄. 多模态话语分析综合理论框架探索[J]. 中国外语,2009,6(1):24-30. doi: 10.3969/j.issn.1672-9382.2009.01.006 [12] 张德禄,郭恩华. 多模态话语分析的双重视角−社会符号观与概念隐喻观的连接与互补[J]. 外国语,2013,36(3):20-28. [13] 张德禄. 系统功能理论视阈下的多模态话语分析综合框架[J]. 现代外语,2018,41(6):731-743. [14] 梁兵,蒋平. 旅游语篇多模态话语分析与中国文化对外传播[J]. 外语学刊,2015(2):155-158. [15] 国防. 多模态话语分析研究热点及趋势分析−基于文献计量学方法[J]. 外语与外语教学,2016(3):58-66,146. [16] 管乐. 动态演讲视频的多模态话语分析[J]. 浙江理工大学学报(社会科学版),2017,38(1):25-32. [17] RENDLE-SHORT J. Managing the transition between talk and silence in the academic monologue[J]. Research on Language and Social Interaction, 2005(38):179-218. [18] MORTON J. Genre and disciplinary competence: a case study of contextualization in an academic speech genre[J]. English for Specific Purposes, 2009(28):217-229. [19] TARDY C M. Building Genre Knowledge [M]. West Lafayette, Indiana: Parlor Press, 2009. [20] 赵燕飞. 运用TED演讲辅助英语视听说教学[J]. 教学与管理,2016(21):103-106. [21] 李颖. 虚拟现实技术在英语演讲中的实训原理分析[J]. 中国外语,2018,15(6):73-78. [22] 刘森,吴鸿皓. “同一”理论对英语语篇教学的启示−基于美国总统就职演讲的修辞分析[J]. 外语电化教学,2019(4):67-72, 80. [23] 林岩. 英语演讲教学中的思辨能力培养−“说服策略”单元课例研究[J]. 中国外语,2020,17(3):63-69. [24] NORRIS S. Modal density and modal configurations: multimodal actions [C] // Jewitt C. The Routledge Handbook of Multimodal Analysis. London: Routledge, 2009: 78-90. [25] NORRIS S. Identity In (Inter)action——Introducing Multimodal (Inter)action Analysis [M]. Berlin: De Gruyter Mouton, 2011. [26] NORRIS S. The micropolitics of personal national and ethnicity identity[J]. Discourse and Society, 2007, 18(5):653-674. doi: 10.1177/0957926507079633 [27] 王立非,文艳. 应用语言学研究的多模态分析方法[J]. 外语电化教学,2008(3):8-12. doi: 10.3969/j.issn.1001-5795.2008.03.002 [28] 管乐. 多模态话语分析的新发展−《多模态互动分析》述评[J]. 语言教育,2017,5(1):85-90.