A director's job is to get the best performance out of his/her actors, which often requires several takes. After shooting has wrapped, filmmakers choose the best takes and stitch them together to form a movie. Sometimes even the best takes aren't quite perfect enough, and that's where Disney's new FaceDirector software comes in handy. Disney's Research Hub recently posted a video to YouTube showing how the software can blend different takes together to produce a range of emotional performances.
In addition to mixing different emotions together, the software synchronizes audio and video so that the change in emotion feels natural. In a paper about FaceDirector posted alongside the video [PDF], the Disney team explains that there are challenges to synchronizing takes because of the many variables at play during a human performance. "Difference of head pose, emotion, expression intensity, as well as pitch, accentuation and potentially even wording of the speech are just a few of the many difficulties. We present an automatic, joint audio-visual synchronization approach that first analyzes both facial expression and audio cues and then robustly determines a dense set of frame correspondences between takes using a graph-based framework."
While the report dives deep into exactly how the software works with various equations, the video does a great job of illustrating the magic of it all and puts things into terms that everyone can understand.
Header image via YouTube.
[h/t The Verge]