Speaker: Petra Wagner (Bielefeld University)
Abstract: While gestures and speech are known to align, and to synchronize tighter than speech and co-speech actions, the mechanisms underlying this co-ordination are not yet well understood. Our study investigates the rigidity of this synchronization, depending on whether or not it may fulfill a communicative function. Our study analyzes the temporal co-ordination of speech and co-speech actions in a dyadic game interaction, in which players perform manual game moves while verbalizing their actions. We particularly investigate whether the degree of cross-modal alignment is determined by mutual visibility and as a function of informativity (predictability, importance). As previous research has identified prosody as a strong cross-modal anchor, we analyze cross-modal alignment as the delay between co-speech actions and corresponding (1) pitch peaks as well as (2) word onsets.
For (1), we find no effect of mutual visibility on cross-modal synchronization. However, pitch accents and co-speech actions have a tendency to be tightly aligned when the game moves are both unpredictable and important for the outcome of the game. This result is in line with a view of tight cross-modal coupling, where prominence-lending modifications in the prosodic structure attract corresponding manual actions.
For (2), we do find an effect of mutual visibility, under which co-speech actions are synchronized stronger with word onsets. This result is in line with a view in which co-speech actions are communicative affordances, and can be realized as co-speech gestures in cases where they fulfill a communicative function. On this level of analysis, prosodic prominence modulation did not go hand in hand with cross-modal synchronization, as prominences were produced weaker under mutual visibility, probably due to their communicative redundancy.
Our results are in agreement with a view of a strong low-level cross-modal alignment of prosody (pitch accents) and co-speech actions as 'beat gestures'. However, they also support the idea that speakers can deliberately and selectively synchronize co-speech actions with co-occurring speech, thereby realizing them as co-speech gestures. These can then take over communicative functions by themselves, in the form of ‚representational gestures‘, and reduce the communicative burden of the verbal channel.
About Petra Wagner: Petra Wagner (M.A. linguistics, Bielefeld; Ph.D. phonetics and communication sciences, Bonn) is professor for phonetics at the Center for Cognitive Interaction Technology at Bielefeld University, Germany, where she leads the Bielefeld Phonetics Workgroup. She was the recipient of the 2018 Swedish Rijksbanken’s Humboldt-award for outstanding German researchers in the humanities. Prof. Wagner's expertise spans phonetics, prosody, multimodal communication, speech synthesis and evaluation, and conversational speech analysis. Currently, she is particularly interested in (a) applying speech technology for tackling fundamental questions within speech science and (b) using speech science to better evaluate existing speech technological applications.
