Accessibility statement

Multimodality: Language & the Body - LAN00066H

« Back to module search

  • Department: Language and Linguistic Science
  • Module co-ordinator: Dr. Marina Cantarutti
  • Credit value: 20 credits
  • Credit level: H
  • Academic year of delivery: 2024-25

Module summary

Most of our language use and acquisition happens through face-to-face interaction, which gives us access not only to what we and others say, but also how we use our bodies in interacting with others. In this module, we explore language as a multimodal phenomenon, grounding our work in everyday interactions, and using the methodological tools of multimodal conversation analysis, gesture studies, and the phonetics of talk-in-interaction.

Related modules

Pre-requisite modules

Co-requisite modules

  • None

Prohibited combinations

  • None

Module will run

Occurrence Teaching period
A Semester 2 2024-25

Module aims

The module aims at introducing students to a multimodal perspective to language and language use in interaction, as well as to the methods and tools used to study different kinds of vocal and visual behaviour that are communicative. Using methods from conversation analysis, gesture studies, and the phonetics of talk-in-interaction, this module allows students to register, describe, and transcribe visual behaviour (gaze, hand gestures, body posture, among others) alongside speech and how they are used by participants in interaction to get things done.

In addition, this module aims to develop students’ skills in synthesis and application of data analysis and communicate findings and conclusions in speech and writing using appropriate conventions. This module involves applying the analysis techniques learned in other modules to a new method of analysing everyday talk; being able to explicitly relate knowledge gained from different spheres of experience and knowledge (and thus confidently tackle unfamiliar problems you may meet in a job). Students will also develop the ability to analyse and talk about the nonverbal behaviour of participants in social interaction, a skill which has practical applications in numerous professions.

Module learning outcomes

At the end of this module students will be able to:

  • recognise the different modalities that are relevant for language use in spoken face-to-face interaction;

  • use appropriate software to study visible bodily actions, including gesture and speech;

  • be able to annotate and transcribe spoken and visible/gestural behaviour using standard transcription methods;

  • write about a problem in face-to-face interaction, drawing on technical skills, and showing an appreciation of some of the debates around the topic.

At the end of the module, students will know:

  • various ways in which speakers use language and the body together as resources for meaning and action in face-to-face interaction;

  • major theories developed to account for specific modalities such as manual gesture and eye gaze;

  • important empirical studies conducted on specific modalities and their coordination with language production.

Module content

The module will cover a range of topics related to multimodality, including:

  • The multimodal view of language

  • Multimodal transcription and annotation

  • The definition, description, and analysis of gesture

  • Temporal alignment of speech and gesture, including visual prosody

  • Gaze and body posture in interaction

  • The development and analysis of multimodal collections of data extracts

Assessment

Task Length % of module mark
Essay/coursework
Project report 3000 words		 N/A 75
Essay/coursework
Transcription and analysis 750 words		 N/A 25

Special assessment rules

None

Reassessment

Task Length % of module mark
Essay/coursework
Project report 3000 words		 N/A 75
Essay/coursework
Transcription and analysis 750 words		 N/A 25

Module feedback

Students will receive feedback within 25 working days. Feedback on formative assessment submitted on W4 will be returned on Week 6. Feedback on Summative 1 due Week 7 will be returned on Week 11, and on Summative 2 due RA3 will be returned 25 working days after.

Indicative reading

This list illustrates some general readings assigned for the module. Depending on the year’s theme, other readings specific to the practice that will be studied will be added.

Abner, N., Cooperrider, K., & Goldin-Meadow, S. (2015). Gesture for Linguists: A Handy Primer. Language and Linguistics Compass, 9(11), 437–451. https://doi.org/10.1111/lnc3.12168

Cantarutti. (2021). Co-animation and the Multimodal Management of Contextualisation Problems when Jointly “Doing Being” Others. Social Interaction. Video-Based Studies of Human Sociality, 4(4). https://doi.org/10.7146/si.v4i4.128166

Hoey, & Kendrick, K. (2017). Conversation Analysis. In A. M. B. de Groot, P. Hagoort, A. M. B. de Groot, & P. Hagoort (Eds.), Research methods in psycholinguistics and the neurobiology of language : a practical guide (p. 1 online resource). John Wiley & Sons Inc

Levinson, S. C. (2013). Action formation and ascription. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp. 101–130). Malden: Blackwell Publishing Ltd.

Lilja, N., & Piirainen-Marsh, A. (2019). How Hand Gestures Contribute to Action Ascription. Research on Language and Social Interaction, 52(4), 343–364.

Loehr. (2012). Temporal, structural, and pragmatic synchrony between intonation and gesture. Laboratory Phonology. https://doi.org/https://doi.org/10.1515/lp-2012-0006

McNeill. (1992). 3. Guide to Gesture Classification, Transcription, and Distribution. In McNeill, Hand and Mind. What Gestures Reveal about Thought. (pp. 75–104). https://pages.ucsd.edu/~bkbergen/cogs200/McNeill_CH3_PS.pdf

Mondada, L. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20(3), 336–366. https://doi.org/10.1111/josl.1_12177

Mondada, L. (2019a). Contemporary issues in conversation analysis: Embodiment and materiality, multimodality and multisensoriality in social interaction. Journal of Pragmatics, 145, 47–62. https://doi.org/10.1016/j.pragma.2019.01.016

Mondada, L. (2019b). Conventions for multimodal transcription. Available from https://www.lorenzamondada.net/multimodal-transcription.

Perniss, P. (2018). Why We Should Study Multimodal Language. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.01109

Rossano. (2012). Gaze in Conversation. In Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp. 308–329). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118325001.ch15

Sidnell. (2013). Basic Conversation Analytic Methods. In Sidnell, T. Stivers, J. Sidnell, & T. Stivers (Eds.), The handbook of conversation analysis (pp. 77–99). Wiley-Blackwell. https://doi.org/10.1002/9781118325001

Stivers, T., & Sidnell, J. (2005). Introduction: Multimodal interaction. Semiotica, 2005(156), 1–20. https://doi.org/10.1515/semi.2005.2005.156.1

Streeck, J., Goodwin, C., & LaBaron, C. (Eds.). (2011). Embodied Interaction: Language and Body in the Material World. Cambridge University Press.


The information on this page is indicative of the module that is currently on offer. The University is constantly exploring ways to enhance and improve its degree programmes and therefore reserves the right to make variations to the content and method of delivery of modules, and to discontinue modules, if such action is reasonably considered to be necessary by the University. Where appropriate, the University will notify and consult with affected students in advance about any changes that are required in line with the University's policy on the Approval of Modifications to Existing Taught Programmes of Study.