The whole “Deep Fake” thing is something I have been interested in for some time. Each year it gets better and better, and as AI / Machine learning and technology advances being able to discern what is real, and what isn’t is getting harder. Where will this be in 10 years is hard to say, but the implications are pretty obvious. What does a person do when they can no longer tell if a video of someone is the real thing or not?
OK, enough of what could lead down a long and disheartening rabbit hole of despair about the future and how technology wreaks havoc on humanity.
To showcase the 2022 Spring Balenciaga fashion collection, creative director Marcus Dryden and the talented crew at MDC combined AI/machine learning, a real-time game engine, and hands-on VFX work to create a deepfake of American artist Eliza Douglas wrapped in every look from Balenciaga’s Spring ’22 collection. There is some live-action footage blended with CG and some solid post-work. They don’t say what the game engine is that they used, but I bet it was probably something like “Unreal”, or “Unity”. Below the video is a statement from Dryden breaking down the production.
“In Pre-production, we were able to plan the whole show. We used a games engine to previsualize which looks could be body doubles vs. which one needed to be the real Eliza. Also, the pre-viz defined the scale of the set for the art department and allowed production to choose the best lens, angles, and positions needed to run the multiple cameras in sync whilst on location.
During the shoot, MPC on-set supervisors Carsten Keller and Damien Canameras captured photogrammetry of Eliza’s face and oversaw a variety of in-situ plates to extract her face and transpose it onto the body doubles shot on the catwalk.
We also used a CG scan of Eliza’s head and an on-set photo reference to build a proxy Eliza head to help visualize the face replacements. This allowed our compositing team to study and analyze each shot, each face to define the best process to achieve the highest-quality clone.
The team then applied the best technique to create the face replacements: Planar tracking, roto animation, Keen Tools (a 3D tracking and modeling tool inside Nuke), and Machine Learning (AI/deep fake).
Once we began attaching Eliza’s faces, we matched light, textures, and motion artifacts using compositing. Using the references and the scan of the head, we made sure each clone’s face was as pixel-accurate to Eliza’s face as possible while still retaining the nuance of the specific Balenciaga design aesthetic.
The final film shows all the clones with Eliza’s photogrammetry-captured and CG-scanned face as they march down a minimalist runway to a sci-fi-inspired soundtrack composed by BFRND, which includes an AI voice narrating the lyrics of La Vie En Rose.”
Modular 4 