The interplay of speech and gesture has communicative effects [1]. We explore the function of disambiguating an ambiguous expression, “Next Wednesday’s meeting has been moved forward two days. On what day is it now?”, by means of gestures and speech. We run a study where participants (n=274) watch a video of a speaker saying the sentences above accompanied by a directional gesture, and answer the question (cf. [2]). We manipulate Gesture Direction (forward/backward) and Gesture Position (adverb/verb/sentence-final). In the latter two cases, the adverb “forward” is not said, leading to another factor Adverb Presence (adverb/none). We run Bayesian logistic regression models over “Monday”/ ”Friday” responses.
Results show a Gesture Direction effect: the forward gesture is associated with “Friday”, the backward gesture more with “Monday”. We find a Gesture Position effect with the adverb position being associated with “Monday”, while verb and sentence-final positions are associated with “Friday”. However, this effect is likely rather an Adverb Presence effect, due to the adverb being present in the first Gesture Position, but not in the latter two. We find disambiguation effects of gesture and speech, that influence the response to the stated question in specific directions. Gesture and speech may work together to disambiguate but are able to cancel out each other’s effect.
[1] C. Ebert, S. Evert, & K. Wilmes, “Focus marking via gestures,” Proc. Sinn und Bedeutung, 2011.
[2] B. Winter & S. E. Duffy, “Can co-speech gestures alone carry the mental time line?,” J. Exp. Psychol. Learn. Mem. Cogn., 2020.