Auditory emotion recognition: Insight from affective speech, music and vocalisations

The communication of emotion is an essential part of our daily interaction, and recognising emotions can elicit a reaction in the observer, such as the well-known fight-or-flight response. An index of this response can be observed in pupil dilations, which have been demonstrated to dilate more for emotional stimuli in comparison to neutral across several stimulus types, including pictures, environmental sounds and even music. However, the pupillary response has also been shown to reflect mental effort, attention, and other cognitive processes. In this study, we aimed to understand further if the pupil reflects an automatic response to emotions, or if it is indicative of the cognitive processes involved in the labelling of emotions.

We recorded pupil dilations as participants listened to affective speech, as well as other auditory stimuli in the form of music and vocalisations. Participants were instructed to provide behavioural responses to the perceived valence, arousal and emotion category of the stimulus. Our three stimulus types differ in their emotional salience, for instance, crying is a clear indicator of sadness whereas a short minor musical piece may require the listener to use their cognitive abilities to recognise the emotion. We expect our semantically neutral speech to elicit a response somewhere in-between these two types. Therefore, we can trigger different emotional responses in the pupil that may be indicative of autonomic arousal in response to crying and cognitive processing in response to music. By investigating the pupil’s response in this way, we explored the existence of a signature in the pupil to emotional stimuli, and this was shown to be somewhat generalisable across stimulus types, but the response also contains unique features to each stimulus set; speech, music and vocalisations.