Toward Real-World Emotion Recognition: An fNIRS Study with Multisensory Stimuli

초록

We constantly attend to various emotional cues in our daily lives to facilitate social interaction. Specifically, multisensory approaches are essential for capturing genuine emotional responses. However, previous neuroscientific studies have limitations in integrating modalities and often lack naturalistic experimental settings. To address these gaps, we conducted an emotion recognition task using audio-video movies and functional near-infrared spectroscopy (fNIRS). We hypothesized that individuals would be more engaged in recognizing emotions when provided with more sensory information. In total, 63 participants (N = 63, 32 males and 31 females, mean age = 25.4 years, standard deviation = 3.2) wore an fNIRS cap with 18 channels covering the bilateral dorsolateral prefrontal cortex (dlPFC) and posterior parietal cortex (PPC) while rating valence and arousal in response to an actor portraying seven emotions: angry, calm, disgusted, fearful, happy, sad, and surprised. Stimuli were presented in three sensory modality conditions—audio-only, visual-only, and audiovisual—in a pseudo-randomized order. We performed repeated measures analysis of variances on hemodynamic data. Changes in oxyhemoglobin concentration (HbO) in the dlPFC were significantly greater in the audiovisual condition compared to the audio-only and visual-only conditions. However, HbO changes in the PPC were significantly higher in both the audiovisual and visual-only conditions than in the audio-only condition. These results suggest that processing visual emotional cues, whether or not accompanied by sound, requires greater cognitive effort. Participants appeared to engage in more active processing in the fully multimodal condition and focused more on sensory integration when visual cues were present. Using more naturalistic and multisensory stimuli, this study highlights that visual emotional cues outweigh audio-only cues in engaging higher-order cognitive processing.