Nearly half of our attention in face-to-face conversation is drawn to lip movements. Yet robots still have difficulty reproducing them accurately. Even the most sophisticated humanoids manage little more than puppet-like mouth motions—assuming they have a face at all.
We place disproportionate weight on facial expressions overall, and on lip movement especially. An odd stride or clumsy hand gesture might go unnoticed, but even a minor facial misstep is hard to ignore. This sensitivity gives rise to the so-called “Uncanny Valley.” Robots often appear dull—or even unsettling—because their lips fail to move convincingly. That, however, is about to change.
Columbia Engineers Create Robot That Learns Lip Movements
A research team at Columbia Engineering announced today that they have developed a robot capable, for the first time, of learning facial lip movements for activities like speaking and singing. In a new study published in Science Robotics, the researchers show the robot using these skills to pronounce words across multiple languages and even perform a song from its AI-generated debut album, Hello World.
The robot developed this capability through observation rather than explicit programming. It started by controlling its 26 facial motors while watching itself, then learned to mimic human lip movements from hours of YouTube videos.
“The more it interacts with people, the better it will become,” said Hod Lipson, James and Sally Scapa Professor of Innovation in the Department of Mechanical Engineering and director of Columbia’s Creative Machines Lab, where the research was conducted.
The Robot Observes its own Speech Movements
Producing lifelike lip movements in robots is difficult for two main reasons. First, it requires specialized hardware: flexible facial skin powered by numerous fast, quiet, precisely coordinated motors. Second, lip motion follows highly complex patterns shaped by sequences of sounds and phonemes in speech.
Human faces are powered by dozens of muscles beneath a soft, compliant skin, naturally synchronized with the voice and mouth movements. Humanoid robots, in contrast, typically have rigid faces with limited freedom of motion, and their lip movements are often controlled by fixed, rule-based scripts. The result is motion that feels stiff, artificial, and unsettling.
The researchers tackled these challenges by creating a highly actuated, flexible robotic face that learns facial control through observation. They first placed a 26-motor robotic face before a mirror to study how its movements matched motor activations. Much like a child experimenting with expressions in a mirror, the robot generated thousands of random facial and lip movements. Over time, it learned how to drive its motors to produce specific expressions—an approach known as a “vision-to-action” language model (VLA).
Robot Learns Lip Sync from Humans
Next, the researchers exposed the robot to recorded videos of people speaking and singing, allowing the AI controlling the robot to learn how human mouths move in relation to the sounds they produce. With both learning stages complete, the robot’s AI could translate audio signals directly into coordinated lip motor movements.
The team evaluated this capability across a range of sounds, languages, contexts, and even songs. Despite having no understanding of the meaning of the audio, the robot was able to synchronize its lip movements with the speech and music.
The researchers note that the results are still imperfect. “We struggled especially with hard sounds like ‘B’ and with lip-rounding sounds such as ‘W,’” said Lipson. “But with time and continued practice, these skills are likely to improve.”
More significantly, lip synchronization should be viewed as one element of a broader, more integrated approach to robotic communication.
“When lip-syncing is paired with conversational AI systems like ChatGPT or Gemini, it adds an entirely new layer of depth to the relationship a robot can build with a human,” said Yuhang Hu, who led the study as part of his Ph.D. research. “As the robot observes more human conversations, it becomes increasingly adept at reproducing the subtle facial cues that people naturally respond to emotionally.”
He added, “As conversations unfold over longer time spans, these gestures will also grow more sensitive to context.”
The long-Missing Piece in Robotic Capability
The researchers argue that facial expression is the “missing link” in robotics.
“Today, much of humanoid robotics centers on leg and hand movements for tasks like walking and grasping,” said Lipson. “Yet the ability to convey emotion through the face is just as crucial for any robot that interacts with humans.”
Lipson and Hu anticipate that expressive, lifelike faces will become increasingly vital as humanoid robots are used in fields like entertainment, education, healthcare, and elder care. Some economists even forecast that over a billion humanoid robots could be produced within the next ten years.
“There’s no future in which all these humanoid robots won’t have a face,” Lipson said. “And once they do, their eyes and lips will need to move convincingly—or they will always feel uncanny.”
Hu added, “It’s just how humans are wired—we can’t help it. We’re on the verge of fully crossing the uncanny valley.”
Potential Challenges and Limitations
This research is part of Lipson’s decade-long pursuit to help robots interact more naturally with humans by mastering facial expressions like smiling, eye movement, and speech. He emphasizes that these skills must be learned, not rigidly programmed with preset rules.
“Something magical occurs when a robot learns to smile or speak simply by observing and listening to humans,” he said. “Even as a seasoned roboticist, I can’t help but smile back at a robot that smiles at me spontaneously.”
Hu added that the human face is the ultimate communication interface, and we are only beginning to uncover its complexities.
“Robots that can do this will naturally connect with humans far more effectively, since such a large part of our communication relies on facial expressions—a channel that has largely gone untapped,” Hu said.
The researchers also acknowledge the potential risks and ethical debates tied to giving robots enhanced capabilities for human connection.
“This technology has enormous potential. We need to proceed gradually and cautiously to maximize its benefits while keeping the risks under control,” Lipson said.
Read the original article on: Tech Xplore