The EBU and several member organizations are collaborating on an IBC Accelerator project with a final POC showcase to be demonstrated as part of IBC 2021.
3D animation is an enticing option in many media content creation scenarios such as children’s programming or science programmes. It eliminates constraints of conventional production methods and offers great creative potential, but also requires highly specialized tools and knowledge, putting it just out of reach for many productions.
We think this situation can be improved. Public media organizations RAI, RTÉ, VRT, Yle, as well as technology specialists ETC, Unreal/Epic and Digital Domain, and the EBU, are among the champions of the project, and organized as an IBC Media Innovation Accelerator. The results of the project, which is also supported by Nvidia, will be revealed on Saturday 4 December at 14:00 CET as part of the IBC 2021 online event.
Our proposed solution explores and establishes a standardized data pipeline that connects smartphones, webcams, affordable AI-powered toolkits, collaborative platforms for graphics artists and real-time render engines, allowing content makers to create 3D animated content quickly and collaboratively.
Together with other champions and participating partners involved in the project, such as Respeecher and RADiCAL, we are testing pipelines based on Nvidia's Omniverse as well as Unreal in order to:
- enable live collaboration between graphics artists
- animate the faces of the CG characters in real-time using the voice of the performers
- animate the body of the CG characters in real-time using the skeleton information coming from a 2D cameras
- compose the scenes in real time using a game-engine environment
We expect a number of benefits coming out this collaborative development. One is to reduce the time spent on keyframe animation by using an audio-visual input driving the CG performance instead. Another is a better understanding of options for collaborative workflows, their advantages and limitations, and the integration of game engines in traditional broadcast workflows. And of course, to help compress typical post-production timelines.
The project was proposed in March 2021 as part of IBC Kickstart Day, followed by an onboarding phase for the consortium members, and a process to set up the challenge.
In detail
The project is broken up into two phases. Phase 1 focuses on the implementation of Nvidia’s ‘Omniverse pipeline’, while phase 2 then looks at the ‘Unreal only’ pipeline.
AI-based tools play an integral part in the proposed solution, and the list of components that are being considered is long:
- Respeecher: A voice cloning tool powered by AI and capable of creating pitch-perfect voice-to-voice swapping models
- Audio2Face (Nvidia): An AI tool that can generate facial expressions and lip sync based on an audio source
- RADiCAL: A cloud-based and markerless 3D motion capture system that can create an FBX animated skeleton based on a simple video recording. The skeleton’s movements can then easily be transferred to a custom character
- Live Link Face (Unreal): A tool that uses the depth camera feature on certain iPhones to deliver compelling facial performances
Project partners
Among the champions and partners contributing to the project are innovative public broadcasters and media organisations seeking to create a win-win situation for their audiences and media creators alike.
Irish broadcaster RTÉ, who co-leads the consortium of broadcasters, has also taken on the task of overseeing the final proof of concept. The organization hopes to be able to 'develop a story visually', to create a more efficient ‘smart remote workflow’, and to establish best practice for recording voice-talent in the new pipeline.
“As a public broadcaster, RTÉ is keen to embrace new tools and platforms that can empower the next generation of content-makers to tell stories in ever more imaginative ways”, says Ultan Courtney, Motion Graphics Designer at RTÉ. “This IBC Accelerator project has enabled RTÉ to build working relationships within the broadcast family whilst exploring unique opportunities by partnering with internationally renowned companies at the forefront of virtual production.
VRT in Belgium is supporting the IBC Accelerator project with 3D modelling, designing and worldbuilding for the Unreal pipeline.
"VRT is already using game engine technology in virtual production pipelines. The knowhow built upon in the project using real-time animations will be another step towards what we see as a highly effective future production capability", says Gregg Young, Project Manager at VRT.
Finnish public media organization Yle has created visual assets, brought in a voice actor and introduced new partners to complete the pipeline. Yle is also benchmarking the motion capture tools.
“Working with the team of public media champions, vendors and the IBC has been fun! We have learnt a lot and extended our professional network”, says Jouni Frilander, Innovation Lead at Yle. “It is very important for Yle to learn this kind of technology as it’s needed in our productions and digital services. Also, people need to create content live and work in different locations, so being able to collaborate remotely is a must for us in the modern era.“
Italy’s public media organization RAI performed the technical tests and recordings for AI-driven facial and body performances in Nvidia’s Omniverse pipeline.
“At RAI, we are working to apply new techniques to TV animation, especially in the field of motion capture", says Roberto Iacoviello, Lead Research Engineer at RAI. "Our wish is to enable our distributed production teams to collaborate in real-time in a shared virtual space using low budget equipment. And we think that we can take our production center to the next level of virtual production by leveraging the latest technological advances in Artificial Intelligence.“
The EBU is co-leading the IBC Accelerator project together with RTÉ, provides some of the equipment, runs the infrastructure to host the Omniverse and Unreal collaborative environment, and provides the access to RADiCAL services.
“This project is just the beginning of what we think is a journey to further explore collaborative CG animation pipelines using AI tools and off-the-shelf equipment for video and audio capture. We are still learning but we’ll soon share this great experience and future plans with the EBU community”, the EBU’s project lead Paola Sunna says.