Introducing the Freiburg Multimodal Interaction Corpus (FreMIC)

Speakers: Dr Christoph Rühlemann, Albert-Ludwigs-Universität Freiburg

About Dr Christoph Rühlemann: Christoph Rühlemann is a researcher at Albert-Ludwigs-UniversityFreiburg. Beside journal articles on conversational language, storytelling interaction and conversational structure he has published monographs including Visual Linguistics with R: A practical introduction to quantitative Interactional Linguistics (2020) published with Benjamins, Corpus linguistics for pragmatics (2018) published with Routledge, Narrative in English conversation: A corpus analysis of storytelling (2013) published with CUP, and Conversation in context. A corpus-driven approach (2007) with Continuum and is the co-editor of Corpus pragmatics. A handbook (2015; with Karin Aijmer) published with CUP. Following his role as a Project Director of the DFG-funded research project on “Multimodal Storytelling Interaction” (2019-2022), he is now Project Director of the DFG-funded research project “Human conversational turntaking: the role of multimodal turn-completion cues in facilitating precision-timed turn transition” (2022-2025; together with Peter Auer, Stefan Th. Gries, Marion Schulte, and Judith Holler as Mercator Fellow).

Abstract: Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017:2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson & Holler 2014).

This paper introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 25 hrs of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data.

All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g., overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g., intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside & Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of reenactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech).

The multimodal data are derived from psychophysiological measurements and eyetracking. The psychophysiological measurements include, inter alia, heart beat and electrodermal activity (EDA, or GSR), etc., which are indicative of emotional arousal (e.g., Peräkylä et al. 2015). Eyetracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the Area-of-Interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g., Auer 2021) and reenactments (e.g., Pfeiffer & Weiss 2021), while changes in pupil size provide a window onto cognitive intensity (e.g., Barthel & Sauppe 2019).

To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data opens up for research in Interactional (Corpus) Linguistics, this paper reports on interim results derived from a number of case studies.