Speaker
Description
Analyzing facial expressions through muscle activity, captured via surface electromyography (sEMG), offers rich insights for psychology, medicine, and animation. However, sEMG electrodes introduce significant occlusion, hindering accurate facial expression analysis, particularly for monocular 3D face reconstruction. Our method, EIFER: Electromyography-Informed Facial Expression Reconstruction via Unpaired Image-to-Image Translation, addresses this challenge. EIFER decouples facial geometry and appearance by leveraging unpaired image-to-image translation within a CycleGAN-like adversarial framework [3, 6]. We combine this with 3D Morphable Models (3DMMs), specifically FLAME [5] and BFM [2], to represent facial shape and expression parameters. EIFER performs monocular 3D face reconstruction using a neural renderer and learns to remove sEMG electrode occlusions via adversarial training against unpaired, occlusion-free reference images. Critically, EIFER establishes a bidirectional mapping between 3DMM expression parameters and sEMG muscle activity, enabling both physiologically-based expression synthesis and electrode-free facial electromyography. This approach overcomes the limitations of occlusion-sensitive photometric methods and integrates physiological information into 3DMMs [1, 4]. We validate EIFER on a novel dataset of synchronized sEMG and facial expressions, demonstrating faithful geometry and appearance reconstruction, expression synthesis from muscle activity, and muscle activity prediction from facial expressions. EIFER introduces a new paradigm for facial electromyography with potential extensions to other multi-modal face recordings.
[1] Balconi, M., Bortolotti, A., & Gonzaga, L. (2011). Emotional face recognition, EMG response, and medial pre-frontal activity in empathic behaviour. Neuroscience Research, 71(3), 251-259.
[2] Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 187-194.
[3] Büchner, T., Guntinas-Lichius, O., & Denzler, J. (2023). Improved obstructed facial feature reconstruction for emotion recognition with minimal change cyclegans. International Conference on Advanced Concepts for Intelligent Vision Systems, 262-274. Springer.
[4] Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System: A technique for the measurement of facial movement. Consulting Psychologists Press.
[5] Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017). Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics (TOG), 36(6), 1-17.
[6] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, 2223-2232.