The Effect Of Sensory Modalities Of Naturalistic Stimuli On Emotion Recognition

초록

Emotion recognition is a critical skill for individuals to interpret others’ emotions and adapt to social contexts. Although emotions are conveyed through multiple sensory modalities in real life, prior studies have largely relied on limited sensory modalities for emotional stimuli. It is unclear how sensory modalities play roles in recognizing emotions or whether there is any dominance across modalities. We hypothesized that visual modality would be more dominant in emotion recognition over audio modality, particularly in positive emotions. In the current study, participants were randomly presented with seven naturalistic emotional stimuli (‘angry, ’calm,’ ‘disgusted,’ ‘fearful,’ ‘happy,’ ‘sad,’ and ‘surprised’) in three modalities (audio-only, visual-only, and audio-visual) and rated each stimulus on valence and arousal. Data analysis using two-way repeated-measures ANOVA revealed significant differences across modalities. We found that emotions presented in audio-only conditions showed lower valence and arousal scores compared to visual-only and audio-visual conditions. Specifically, ‘angry’ and ‘surprised’ were perceived less negatively and ‘calm’ more positively in audio-only conditions. Moreover, ‘disgusted’ and ‘happy’ were recognized less accurately in audio-only compared to other modalities. In visual-only conditions, ‘surprised’ was evaluated to be more negative. Furthermore, in visual-only conditions, ‘calm’ was perceived to be stronger than other conditions. However, ‘fearful’ and ‘sad’ did not significantly differ in valence and arousal ratings across modalities. These findings indicate different impacts of sensory cues on emotion recognition, highlighting the predominant role of visual cues in interpreting emotions.