Towards the identification of joint meaning construal: language and co-speech gestures on focus

Speakers

Mr Nickil Maveli and Dr Tiago Torrent, Federal University of Juiz de Fora, Brazil

Abstract

In this talk, we report on a project aimed at developing a prototype that is capable of identifying joint meaning construal using multi-modal channels of communication. Specifically, for a co-speech gestures dataset, we devise a mechanism to disambiguate meaning considering the influence of all the different modalities involved in a particular frame. To circumvent the complexity arising from various gesticulations possible, we limit our analyses to hand gesture types. While developing the prototype, we were confronted with a lack of annotated datasets that could be representative of the joint meaning construal phenomenon. Hence, we leverage semi-supervised learning techniques to annotate additional unlabeled data after identifying potential lexical triggers and their broader lexical context. Furthermore, since each frame could have multiple interpretations, we use human-in-the-loop approaches to annotate a subset of our validation set, for instance, tagging a frame with a predefined binary choice questionnaire to assess construal dimensions. We report our performance on that set.