留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于ELAN的演讲视频多模态互动机制研究

管乐

管乐. 基于ELAN的演讲视频多模态互动机制研究[J]. 北京科技大学学报(社会科学版), 2021, 37(5): 499-510.
引用本文: 管乐. 基于ELAN的演讲视频多模态互动机制研究[J]. 北京科技大学学报(社会科学版), 2021, 37(5): 499-510.
GUAN Yue. Multimodal Interaction Mechanism of Speech Videos Based on ELAN[J]. Journal of University of Science and Technology Beijing ( Social Sciences Edition), 2021, 37(5): 499-510.
Citation: GUAN Yue. Multimodal Interaction Mechanism of Speech Videos Based on ELAN[J]. Journal of University of Science and Technology Beijing ( Social Sciences Edition), 2021, 37(5): 499-510.

基于ELAN的演讲视频多模态互动机制研究—以一则TED教育类演讲视频为例

基金项目: 广东省哲学社会科学“十四五”规划2021年度项目“粤港澳大湾区城市国际形象的多模态认知建构”(编号:GD21CWY08);广州市哲学社科规划2020年度课题“广州地铁公益广告的多模态认知机制研究”(编号:2020GZGJ323)。
详细信息
    作者简介:

    管乐(1980−),女,湖南湘潭人,广州南方学院大学英语教学中心副教授

  • 中图分类号: G642.0; H0-05

Multimodal Interaction Mechanism of Speech Videos Based on ELANA Case Study of TED Educational Speech Video

  • 摘要: 近年来,多模态语篇分析已成为了语言学研究的热点。演讲视频是典型的动态多模态语篇。文章运用Norris提出的多模态互动分析理论,采用“模态密度前景−背景连续统一体(modal density foreground-background continuum)”的分析框架,以ELAN多模态分析软件作为模态标注工具,结合Norris提出的交流模态,对一则TED教育类动态多模态演讲视频中各交流模态在构建演讲信息中所起的作用以及这些模态之间的互动机制进行分析。研究发现在演讲视频的问题导入、问题提出、问题分析和问题解决这四个部分中各模态使用的强度和密度存在差异,言语交流模态与其他交流模态之间呈现强化、互补和并列关系从而共同构建多模态互动意义。

     

  • 图  1  ELAN标注界面

    表  1  语料特征

    语料特征TED教育类演讲
    问题导入时长00:10—01:14
    问题提出时长01:14—02:41
    问题分析时长02:41—04:12
    问题解决时长04:13—05:59
    话题类型教育类:Grit: The power of passion and perseverance
    场景布局TED 圆形讲台
    下载: 导出CSV

    表  2  ELAN附码表

    附码含义附码含义
    1. Spoken language /S-/
    2. Prosody /P-/
    Intonation /PI-/
    [PIF]Intonation fall[PIR]Intonation rise
    [PIRF]Intonation rise & fall[PIS]Intonation stress
    3.Hand movement [HN-]
    [HNI]Hand iconic[HNM]Hand metaphoric
    [HNDe]Hand deictic[HNB]Hand beats
    4.Head movement [HD-]
    [HDU]Head up[HDD]Head down
    [HDN]Head nod[HDMA]Head move around
    5.Facial expression [F-]
    [FFS]Face smile[FFSe]Face serious
    6.Eye movement (Gaze)
    [EGA]Gaze at the audience[EGAr]Gaze around
    7. Posture [PO-]
    [POO]Posture open[POC]Posture closed
    下载: 导出CSV

    表  3  ELAN对整个演讲视频的各交流模态标注数据汇总表

    层级标注
    次数
    平均总标注
    时长/s
    标注时长
    百分比/%
    言语(spoken language)1913.378947254.268.714
    手势(gesture)2310.023913230.5562.321
    头部动作(head movement)1615.78125252.568.254
    脸部表情(facial expression)1219.908335238.964.578
    眼神(gaze)1517.563334263.4571.214
    身体姿势(posture)2013.99279.875.634
    音韵(prosody)2413.14375315.4585.271
    下载: 导出CSV

    表  4  ELAN对演讲视频四个部分各交流模态的标注数据汇总表

    标注层标注内容 标注次数 持续时长/s 标注时长百分比/%
    问题
    导入
    问题
    提出
    问题
    分析
    问题
    解决
    问题
    导入
    问题
    提出
    问题
    分析
    问题
    解决
    问题
    导入
    问题
    提出
    问题
    分析
    问题
    解决
    问题
    导入
    问题
    提出
    问题
    分析
    问题
    解决
    手势HNDeHNDeHNDeHNDe 232200:1700:2800:2800:3516.928.225.435.4
    HNBHNBHNB110100:0900:06000:068.6605.55
    HNIHNIHNIHNI232200:0700:2300:2100:34722.521.134.2
    HNM0200000:070007.600
    头部动作HDN HDN 1001 00:090000:05 9.1004.9
    HDMAHDMAHDMAHDMA 2424 00:1800:5500:2700:85 28.452.92784.8
    HDDHDDHDD111000:1800:0800:20017.77.7200
    脸部表情FFSFFSFFSFFS121100:0900:2900:4200:048.728.642.54.4
    FFSeFFSeFFSeFFSe221200:2300:3300:2500:7422.432.225.474.7
    眼神EGArEGArEGArEGAr232400:2500:4700:3100:3925.346.131.339.6
    EGAEGAEGAEGA211200:3100:2000:2200:4730.720.322.447.45
    身体姿势POOPOOPOOPOO232300:2600:4800:4800:6626.457.947.766.3
    POCPOCPOCPOC221300:1000:1600:0700:198.016.17.319
    音韵PIFPIFPIFPIF331200:2500:6700:2300:2525.566.622.824.5
    PISPISPISPIS312100:2600:0600:1300:0325.955.5512.852.5
    PIRFPIRFPIRFPIRF112400:0400:1100:3500:794.311.324.679
    下载: 导出CSV

    表  5  演讲的问题导入部分言语模态与其他交流模态的例证

    言语模态头部动作手势脸部表情眼神身体姿势音韵互动关系
    I left a very demanding job in management consulting for a job that was even more demanding: teaching.HDN/FFS
    EGArPOO
    PIF/PIS强化
    I.Q. was not the only difference between my best and my worst students.HDMAHNB
    FFSe
    EGArPOC
    PIS强化
    I was firmly convinced that every one of my students could learn the material if they worked hard and long enough.HDMAHNI
    FFSe


    EGAPOC
    PIS强化
    下载: 导出CSV

    表  6  演讲的问题提出部分言语模态与其他交流模态的例证

    言语模态头部动作手势脸部表情眼神身体姿势音韵互动关系
    What we need in education is a much better understanding of students and learning from a motivational perspective, from a psychological perspective.HDNHNI
    FFSe
    EGArPOC
    PIF互补
    I started studying kids and adults in all kinds of super challenging settings, and in every study my question was who is successful here and why.HDMAHNDe
    FFS
    EGAPOO
    PIF互补
    My research team and I went to West Point Military Academy...the National Spelling Bee... rookie teachers ...We partnered with private companies...HDMAHNDe
    FFSe
    EGArPOC
    PIF互补
    下载: 导出CSV

    表  7  演讲的问题分析部分言语模态与其他交流模态的例证

    言语模态头部动作手势脸部表情眼神身体姿势音韵互动关系
    Grit is having stamina. Grit is sticking with your future, day in, day out,... Grit is living life like it's a marathon, not a sprint.
    HDDHNI
    FFSe
    EGArPOC
    PIF互补
    I asked thousands of high school juniors to take grit questionnaires,
    ...Turns out that grittier kids were significantly more likely to graduate, ...
    HDMAHNDe
    FFS
    EGAPOO
    PIF互补
    ... the most shocking thing about grit is how little we know, how little science knows...HDMAHNDe
    FFSe
    EGAPOC
    PIRF互补
    下载: 导出CSV

    表  8  演讲的问题解决部分言语模态与其他交流模态的例证

    言语模态头部动作手势脸部表情眼神身体姿势音韵互动关系
    Parents and teachers ask me“how do I build grit in kids?...” The honest answer is, I don't know.HDMAHNDe
    FFS
    EGArPOO
    PIRF并列
    Our data show... many talented individuals ...do not follow through on their commitments. ...grit is usually unrelated...HDMAHNI
    FFSe
    EGArPOO
    PIS/
    PIRF
    并列
    Growth mindset is a great idea for building grit...we need more, ...we need to... .and we need to.... we need to be gritty about getting our kids grittier.HDN
    HDMA
    HNDe
    FFSe
    EGAPOC
    PIS/
    PIRF
    并列
    下载: 导出CSV
  • [1] KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design [M]. London: Routledge, 1996.
    [2] NORRIS S. Analyzing Multimodal Interaction: A Methodological Framework [M]. New York: Routledge, 2004.
    [3] 张佐成, 刑建玉, 王彦, 等. 多模态即席话语研究 [M]. 广州: 世界图书出版公司, 2014.
    [4] KRESS G & VAN LEEUWEN T. Reading Images: The Grammar of Visual Design (2nd ed. )[M]. London: Routledge, 2006.
    [5] GU Y. Multimodal text analysis: a corpus linguistic approach to situated discourse[J]. Text & Talk, 2006, 26(2):127-167.
    [6] O’HALLORAN K L. Multimodal Discourse Analysis: Systemic-Functional Perspectives [M]. London: Continuum, 2004.
    [7] BALDRY A & THIBAULT P. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Course Book [M]. London: Equinox, 2006.
    [8] 李战子. 多模式话语的社会符号学分析[J]. 外语研究,2003(5):1-8. doi: 10.3969/j.issn.1005-7242.2003.05.001
    [9] 李战子,陆丹云. 多模态符号学: 理论基础, 研究途径与发展前景[J]. 外语研究,2012(2):1-8. doi: 10.3969/j.issn.1005-7242.2012.02.001
    [10] 胡壮麟. 社会符号学研究中的多模态化[J]. 语言教学与研究,2007(1):1-10.
    [11] 张德禄. 多模态话语分析综合理论框架探索[J]. 中国外语,2009,6(1):24-30. doi: 10.3969/j.issn.1672-9382.2009.01.006
    [12] 张德禄,郭恩华. 多模态话语分析的双重视角−社会符号观与概念隐喻观的连接与互补[J]. 外国语,2013,36(3):20-28.
    [13] 张德禄. 系统功能理论视阈下的多模态话语分析综合框架[J]. 现代外语,2018,41(6):731-743.
    [14] 梁兵,蒋平. 旅游语篇多模态话语分析与中国文化对外传播[J]. 外语学刊,2015(2):155-158.
    [15] 国防. 多模态话语分析研究热点及趋势分析−基于文献计量学方法[J]. 外语与外语教学,2016(3):58-66,146.
    [16] 管乐. 动态演讲视频的多模态话语分析[J]. 浙江理工大学学报(社会科学版),2017,38(1):25-32.
    [17] RENDLE-SHORT J. Managing the transition between talk and silence in the academic monologue[J]. Research on Language and Social Interaction, 2005(38):179-218.
    [18] MORTON J. Genre and disciplinary competence: a case study of contextualization in an academic speech genre[J]. English for Specific Purposes, 2009(28):217-229.
    [19] TARDY C M. Building Genre Knowledge [M]. West Lafayette, Indiana: Parlor Press, 2009.
    [20] 赵燕飞. 运用TED演讲辅助英语视听说教学[J]. 教学与管理,2016(21):103-106.
    [21] 李颖. 虚拟现实技术在英语演讲中的实训原理分析[J]. 中国外语,2018,15(6):73-78.
    [22] 刘森,吴鸿皓. “同一”理论对英语语篇教学的启示−基于美国总统就职演讲的修辞分析[J]. 外语电化教学,2019(4):67-72, 80.
    [23] 林岩. 英语演讲教学中的思辨能力培养−“说服策略”单元课例研究[J]. 中国外语,2020,17(3):63-69.
    [24] NORRIS S. Modal density and modal configurations: multimodal actions [C] // Jewitt C. The Routledge Handbook of Multimodal Analysis. London: Routledge, 2009: 78-90.
    [25] NORRIS S. Identity In (Inter)action——Introducing Multimodal (Inter)action Analysis [M]. Berlin: De Gruyter Mouton, 2011.
    [26] NORRIS S. The micropolitics of personal national and ethnicity identity[J]. Discourse and Society, 2007, 18(5):653-674. doi: 10.1177/0957926507079633
    [27] 王立非,文艳. 应用语言学研究的多模态分析方法[J]. 外语电化教学,2008(3):8-12. doi: 10.3969/j.issn.1001-5795.2008.03.002
    [28] 管乐. 多模态话语分析的新发展−《多模态互动分析》述评[J]. 语言教育,2017,5(1):85-90.
  • 加载中
图(1) / 表(8)
计量
  • 文章访问数:  956
  • HTML全文浏览量:  955
  • PDF下载量:  90
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-09-01
  • 网络出版日期:  2021-09-13
  • 刊出日期:  2021-10-25

目录

    /

    返回文章
    返回