基于ELAN的演讲视频多模态互动机制研究

管乐

基于ELAN的演讲视频多模态互动机制研究—以一则TED教育类演讲视频为例

管乐

广州南方学院大学英语教学中心，广州510970

基金项目: 广东省哲学社会科学“十四五”规划2021年度项目“粤港澳大湾区城市国际形象的多模态认知建构”（编号：GD21CWY08）；广州市哲学社科规划2020年度课题“广州地铁公益广告的多模态认知机制研究”（编号：2020GZGJ323）。

详细信息

作者简介:
管乐（1980−），女，湖南湘潭人，广州南方学院大学英语教学中心副教授

中图分类号: G642.0; H0-05
计量
- 文章访问数: 699
- HTML全文浏览量: 657
- PDF下载量: 80
- 被引次数: 0
出版历程
- 收稿日期: 2020-09-01
- 网络出版日期: 2021-09-13
- 刊出日期: 2021-10-25

Multimodal Interaction Mechanism of Speech Videos Based on ELANA Case Study of TED Educational Speech Video

GUAN Yue

College English Education Center, Nanfang College. Guangzhou, Guangzhou 510970, China

摘要

摘要: 近年来，多模态语篇分析已成为了语言学研究的热点。演讲视频是典型的动态多模态语篇。文章运用Norris提出的多模态互动分析理论，采用“模态密度前景−背景连续统一体（modal density foreground-background continuum）”的分析框架，以ELAN多模态分析软件作为模态标注工具，结合Norris提出的交流模态，对一则TED教育类动态多模态演讲视频中各交流模态在构建演讲信息中所起的作用以及这些模态之间的互动机制进行分析。研究发现在演讲视频的问题导入、问题提出、问题分析和问题解决这四个部分中各模态使用的强度和密度存在差异，言语交流模态与其他交流模态之间呈现强化、互补和并列关系从而共同构建多模态互动意义。
- Ted教育类演讲视频 /
- 多模态互动 /
- 交流模态 /
- ELAN
Abstract: In recent years, multimodal discourse analysis has become a hot topic in the study of linguistics. Speech video is a typical dynamic multimodal discourse. This paper adopts modal density foreground-background continuum as methodological framework and ELAN multimodal analysis software as the modal annotation tool. According to multimodal interaction analysis theory proposed by Norris, this paper analyzes the role of each communicative mode in the construction of this speech information and the interactive mechanism between these modes in a TED educational speech video. It is found that modal intensity and modal density of each communicative mode in the problem introduction part, problem posing part, problem analysis part and problem solving part of this speech video are different. Spoken language and other communicative modes show the relationship of reinforcement, complementarity and juxtaposition, thereby jointly constructing the meaning of multimodal interaction.
- TED educational speech video /
- multimodal interaction /
- communicative modes /
- ELAN
注释:

1) 截至2021年4月3日笔者登录TED该视频的网站时，该视频的点击量已达23 070 759次；该视频的网址为：https://www.ted.com/talks/angela_lee_duckworth_grit_the_power_of_passion_and_perseverance。

HTML全文

图 1 ELAN标注界面

下载: 全尺寸图片幻灯片

表 1 语料特征

语料特征	TED教育类演讲
问题导入时长	00:10—01:14
问题提出时长	01:14—02:41
问题分析时长	02:41—04:12
问题解决时长	04:13—05:59
话题类型	教育类：Grit: The power of passion and perseverance
场景布局	TED 圆形讲台

下载: 导出CSV

表 2 ELAN附码表

附码	含义	附码	含义
1. Spoken language /S-/
2. Prosody /P-/
Intonation /PI-/
[PIF]	Intonation fall	[PIR]	Intonation rise
[PIRF]	Intonation rise & fall	[PIS]	Intonation stress
3.Hand movement [HN-]
[HNI]	Hand iconic	[HNM]	Hand metaphoric
[HNDe]	Hand deictic	[HNB]	Hand beats
4.Head movement [HD-]
[HDU]	Head up	[HDD]	Head down
[HDN]	Head nod	[HDMA]	Head move around
5.Facial expression [F-]
[FFS]	Face smile	[FFSe]	Face serious
6.Eye movement (Gaze)
[EGA]	Gaze at the audience	[EGAr]	Gaze around
7. Posture [PO-]
[POO]	Posture open	[POC]	Posture closed

下载: 导出CSV

表 3 ELAN对整个演讲视频的各交流模态标注数据汇总表

层级	标注次数	平均	总标注时长/s	标注时长百分比/%
言语（spoken language）	19	13.378947	254.2	68.714
手势（gesture）	23	10.023913	230.55	62.321
头部动作（head movement）	16	15.78125	252.5	68.254
脸部表情（facial expression）	12	19.908335	238.9	64.578
眼神（gaze）	15	17.563334	263.45	71.214
身体姿势（posture）	20	13.99	279.8	75.634
音韵（prosody）	24	13.14375	315.45	85.271

下载: 导出CSV

表 4 ELAN对演讲视频四个部分各交流模态的标注数据汇总表

标注层	标注内容				标注次数				持续时长/s				标注时长百分比/%
标注层	问题导入	问题提出	问题分析	问题解决	问题导入	问题提出	问题分析	问题解决	问题导入	问题提出	问题分析	问题解决	问题导入	问题提出	问题分析	问题解决
手势	HNDe	HNDe	HNDe	HNDe	2	3	2	2	00:17	00:28	00:28	00:35	16.9	28.2	25.4	35.4
	HNB	HNB		HNB	1	1	0	1	00:09	00:06	0	00:06	8.6	6	0	5.55
	HNI	HNI	HNI	HNI	2	3	2	2	00:07	00:23	00:21	00:34	7	22.5	21.1	34.2
		HNM			0	2	0	0	0	00:07	0	0	0	7.6	0	0
头部动作	HDN			HDN	1	0	0	1	00:09	0	0	00:05	9.1	0	0	4.9
	HDMA	HDMA	HDMA	HDMA	2	4	2	4	00:18	00:55	00:27	00:85	28.4	52.9	27	84.8
	HDD	HDD	HDD		1	1	1	0	00:18	00:08	00:20	0	17.7	7.7	20	0
脸部表情	FFS	FFS	FFS	FFS	1	2	1	1	00:09	00:29	00:42	00:04	8.7	28.6	42.5	4.4
脸部表情	FFSe	FFSe	FFSe	FFSe	2	2	1	2	00:23	00:33	00:25	00:74	22.4	32.2	25.4	74.7
眼神	EGAr	EGAr	EGAr	EGAr	2	3	2	4	00:25	00:47	00:31	00:39	25.3	46.1	31.3	39.6
眼神	EGA	EGA	EGA	EGA	2	1	1	2	00:31	00:20	00:22	00:47	30.7	20.3	22.4	47.45
身体姿势	POO	POO	POO	POO	2	3	2	3	00:26	00:48	00:48	00:66	26.4	57.9	47.7	66.3
身体姿势	POC	POC	POC	POC	2	2	1	3	00:10	00:16	00:07	00:19	8.0	16.1	7.3	19
音韵	PIF	PIF	PIF	PIF	3	3	1	2	00:25	00:67	00:23	00:25	25.5	66.6	22.8	24.5
	PIS	PIS	PIS	PIS	3	1	2	1	00:26	00:06	00:13	00:03	25.95	5.55	12.85	2.5
	PIRF	PIRF	PIRF	PIRF	1	1	2	4	00:04	00:11	00:35	00:79	4.3	11.3	24.6	79

下载: 导出CSV

表 5 演讲的问题导入部分言语模态与其他交流模态的例证

言语模态	头部动作	手势	脸部表情	眼神	身体姿势	音韵	互动关系
I left a very demanding job in management consulting for a job that was even more demanding: teaching.	HDN	/	FFS	EGAr	POO	PIF/PIS	强化
I.Q. was not the only difference between my best and my worst students.	HDMA	HNB	FFSe	EGAr	POC	PIS	强化
I was firmly convinced that every one of my students could learn the material if they worked hard and long enough.	HDMA	HNI	FFSe	EGA	POC	PIS	强化

下载: 导出CSV

表 6 演讲的问题提出部分言语模态与其他交流模态的例证

言语模态	头部动作	手势	脸部表情	眼神	身体姿势	音韵	互动关系
What we need in education is a much better understanding of students and learning from a motivational perspective, from a psychological perspective.	HDN	HNI	FFSe	EGAr	POC	PIF	互补
I started studying kids and adults in all kinds of super challenging settings, and in every study my question was who is successful here and why.	HDMA	HNDe	FFS	EGA	POO	PIF	互补
My research team and I went to West Point Military Academy...the National Spelling Bee... rookie teachers ...We partnered with private companies...	HDMA	HNDe	FFSe	EGAr	POC	PIF	互补

下载: 导出CSV

表 7 演讲的问题分析部分言语模态与其他交流模态的例证

言语模态	头部动作	手势	脸部表情	眼神	身体姿势	音韵	互动关系
Grit is having stamina. Grit is sticking with your future, day in, day out,... Grit is living life like it's a marathon, not a sprint.	HDD	HNI	FFSe	EGAr	POC	PIF	互补
I asked thousands of high school juniors to take grit questionnaires, ...Turns out that grittier kids were significantly more likely to graduate, ...	HDMA	HNDe	FFS	EGA	POO	PIF	互补
... the most shocking thing about grit is how little we know, how little science knows...	HDMA	HNDe	FFSe	EGA	POC	PIRF	互补

下载: 导出CSV

表 8 演讲的问题解决部分言语模态与其他交流模态的例证

言语模态	头部动作	手势	脸部表情	眼神	身体姿势	音韵	互动关系
Parents and teachers ask me“how do I build grit in kids?...” The honest answer is, I don't know.	HDMA	HNDe	FFS	EGAr	POO	PIRF	并列
Our data show... many talented individuals ...do not follow through on their commitments. ...grit is usually unrelated...	HDMA	HNI	FFSe	EGAr	POO	PIS/ PIRF	并列
Growth mindset is a great idea for building grit...we need more, ...we need to... .and we need to.... we need to be gritty about getting our kids grittier.	HDN HDMA	HNDe	FFSe	EGA	POC	PIS/ PIRF	并列

下载: 导出CSV

参考文献(28)

[1]	KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design [M]. London: Routledge, 1996.
[2]	NORRIS S. Analyzing Multimodal Interaction: A Methodological Framework [M]. New York: Routledge, 2004.
[3]	张佐成, 刑建玉, 王彦, 等. 多模态即席话语研究 [M]. 广州: 世界图书出版公司, 2014.
[4]	KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design (2nd ed. )[M]. London: Routledge, 2006.
[5]	GU Y. Multimodal text analysis: a corpus linguistic approach to situated discourse[J]. Text & Talk, 2006, 26(2):127-167.
[6]	O’HALLORAN K L. Multimodal Discourse Analysis: Systemic-Functional Perspectives [M]. London: Continuum, 2004.
[7]	BALDRY A & THIBAULT P. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Course Book [M]. London: Equinox, 2006.
[8]	李战子. 多模式话语的社会符号学分析[J]. 外语研究,2003(5):1-8. doi: 10.3969/j.issn.1005-7242.2003.05.001
[9]	李战子,陆丹云. 多模态符号学: 理论基础, 研究途径与发展前景[J]. 外语研究,2012(2):1-8. doi: 10.3969/j.issn.1005-7242.2012.02.001
[10]	胡壮麟. 社会符号学研究中的多模态化[J]. 语言教学与研究,2007(1):1-10.
[11]	张德禄. 多模态话语分析综合理论框架探索[J]. 中国外语,2009,6(1):24-30. doi: 10.3969/j.issn.1672-9382.2009.01.006
[12]	张德禄,郭恩华. 多模态话语分析的双重视角−社会符号观与概念隐喻观的连接与互补[J]. 外国语,2013,36(3):20-28.
[13]	张德禄. 系统功能理论视阈下的多模态话语分析综合框架[J]. 现代外语,2018,41(6):731-743.
[14]	梁兵,蒋平. 旅游语篇多模态话语分析与中国文化对外传播[J]. 外语学刊,2015(2):155-158.
[15]	国防. 多模态话语分析研究热点及趋势分析−基于文献计量学方法[J]. 外语与外语教学,2016(3):58-66,146.
[16]	管乐. 动态演讲视频的多模态话语分析[J]. 浙江理工大学学报(社会科学版),2017,38(1):25-32.
[17]	RENDLE-SHORT J. Managing the transition between talk and silence in the academic monologue[J]. Research on Language and Social Interaction, 2005(38):179-218.
[18]	MORTON J. Genre and disciplinary competence: a case study of contextualization in an academic speech genre[J]. English for Specific Purposes, 2009(28):217-229.
[19]	TARDY C M. Building Genre Knowledge [M]. West Lafayette, Indiana: Parlor Press, 2009.
[20]	赵燕飞. 运用TED演讲辅助英语视听说教学[J]. 教学与管理,2016(21):103-106.
[21]	李颖. 虚拟现实技术在英语演讲中的实训原理分析[J]. 中国外语,2018,15(6):73-78.
[22]	刘森,吴鸿皓. “同一”理论对英语语篇教学的启示−基于美国总统就职演讲的修辞分析[J]. 外语电化教学,2019(4):67-72, 80.
[23]	林岩. 英语演讲教学中的思辨能力培养−“说服策略”单元课例研究[J]. 中国外语,2020,17(3):63-69.
[24]	NORRIS S. Modal density and modal configurations: multimodal actions [C] // Jewitt C. The Routledge Handbook of Multimodal Analysis. London: Routledge, 2009: 78-90.
[25]	NORRIS S. Identity In (Inter)action——Introducing Multimodal (Inter)action Analysis [M]. Berlin: De Gruyter Mouton, 2011.
[26]	NORRIS S. The micropolitics of personal national and ethnicity identity[J]. Discourse and Society, 2007, 18(5):653-674. doi: 10.1177/0957926507079633
[27]	王立非,文艳. 应用语言学研究的多模态分析方法[J]. 外语电化教学,2008(3):8-12. doi: 10.3969/j.issn.1001-5795.2008.03.002
[28]	管乐. 多模态话语分析的新发展−《多模态互动分析》述评[J]. 语言教育,2017,5(1):85-90.