<ul data-eligibleForWebStory="true"><li>Human emotional expression involves coordinated vocal, facial, and gestural signals.</li><li>Understanding the dynamics between speech and facial/hand gestures is crucial for real interaction understanding.</li><li>Sequential turn-taking in conversations creates stable temporal windows for synchrony.</li><li>Simultaneous speech disrupts alignment and impacts emotional clarity.</li><li>Analyzing multimodal emotion coupling in dyadic interactions using the IEMOCAP corpus.</li><li>Speech features like prosody, MFCCs, arousal, valence, and emotions were aligned with facial and hand movements.</li><li>Expressive activeness measured through displacement magnitudes, showing more in the lower face during non-overlapping speech.</li><li>Sadness exhibited increased expressivity during non-overlapping speech, while anger suppressed gestures during overlaps.</li><li>Predictive mapping from speech features to gestures showed greater accuracy for prosody and MFCCs compared to arousal and valence.</li><li>Hand speech synchrony was enhanced under low arousal and overlapping speech.</li></ul>

Multimodal Emotion Coupling via Speech-to-Facial and Bodily Gestures in Dyadic Interaction

Discover more