Dynamic animatronic robot facial expressions generated from speech. The figure shows the system's capability to produce diverse and lifelike facial expressions in real-time, synchronized with the corresponding audio speech input. The waveform at the top represents the audio input, while the series of images below showcase the robot's facial responses at different time points.
The proposed approach for creating a speech-driven animatronic robot face using LBS. The approach comprises three major components: (1) skinning-oriented robot development designs and constructs the animatronic face paired with a kinematics simulator based on the target skinning appearance, (2) skinning motion imitation learning involves training an LBS-based model from 3D human demonstrations to generate facial expressions from speech input, and (3) speech-driven robot orchestration generates animatronic facial expressions during inference by utilizing the developed platform, simulator, and learned model. The diagram highlights key development steps, outputs, and inference processes, demonstrating the end-to-end workflow from concept to final animatable robot face.
The proposed skinning-oriented robot design. The figure comprises two primary components: (1) LBS-oriented kinematics design, which showcases the facial mesh model with strategically placed control points for various facial features to achieve actuation topology for the facial muscular system that matches the designed LBS motion space and references facial anatomy, and (2) electro-mechanical design and development accounting for physical constraints of the embodiment, including key mechanical components of the skin, skeleton and muscular system, as well as the electrical control system. This comprehensive view demonstrates how the theoretical LBS model is translated into a functional, physically embodied animatronic face.
The proposed speech-driven facial skinning motion imitation learning method. The model architecture (blue section) comprises three key components: (1) a frame-level speech encoder that processes audio input and generates phoneme logits, (2) a speaking style encoder that captures individual speaking styles, and (3) an LBS encoder that generates blendshape coefficients. During training (red section), the model learns to imitate human facial skinning motions by minimizing the difference between generated and target expressions. In the inference branch (orange section), the trained model generates blendshape coefficients for the robot LBS decoder, producing robot-specific facial skinning motions as reference signals for the downstream kinematics simulator.
Motion Space Validation. Actuated blendshape error for different facial regions (left figure): Color-coded skinning landmarks represent different facial regions for evaluation. The 3D face model shows color-coded landmarks for different facial areas. Error distributions between simulated and physically actuated blendshapes are visualized using violin plots, box plots, and scattered points. Each point represents a single blendshape, evaluated using region-specific landmarks. Median errors (in mm) are provided for each facial region, ranging from 1.76mm (nose) to 8.63mm (jaw). Qualitative comparison (right figure): Eight comparisons between simulated and actuated blendshapes are shown. Blendshapes (1)-(6) demonstrate high accuracy, while (7) mouth close and (8) jaw open highlight limitations in the current design, exhibiting maximum errors for their respective regions.
