How new AI models are compressing videos without reducing quality

Advancements in video compression technology: LivePortrait, a recent model in the field of 2D avatar/portrait animation, demonstrates significant potential for revolutionizing video compression, particularly for talking head videos.

• The model can animate still images, avoiding the need for rendering complex 3D models that struggle with small facial details.
• This technology has implications for social media, where it could become ubiquitous, but also raises concerns about trust in online content.

The core concept of compression: By leveraging predictive algorithms, LivePortrait can compress frame information into a sparse set of cues for reconstruction, building upon Nvidia’s facevid2vid paper.

• The compression method relies on transmitting only changes in expression, pose, and facial keypoints, using a shared source image between sender and receiver.
• This approach achieves high perceptual quality at extremely low bitrates, outperforming traditional video codecs in terms of visual artifacts at comparable levels.

Advantages and limitations: While offering impressive compression ratios, the technology comes with certain trade-offs and challenges.

• There is no straightforward way to adjust the quality-bitrate balance, unlike traditional codecs.
• As a generative model, there’s potential for significant reconstruction errors in worst-case scenarios.
• The high compression rate comes at the cost of increased computational requirements, with LivePortrait needing an RTX 4090 for real-time processing.

Performance analysis: Experiments with LivePortrait show promising results in reconstructing video frames, particularly in scenarios closely matching its training conditions.

• The model performs best when animating frames from the same video, simulating a video call scenario.
• Some discrepancies are noticeable, such as slight head shakiness and occasional issues with eye gaze and teeth rendering.
• Performance degrades when dealing with significant shoulder movement or difficult head angles.

Bitrate efficiency: LivePortrait achieves remarkably low bitrates for video transmission, potentially as low as 22kbit/s with additional optimizations.

• The model transmits only transformation parameters for facial keypoints, resulting in a base bitrate of 36kbit/s for 30FPS video.
• Further compression is possible through entropy coding and temporal priors, potentially reducing the bitrate to around 22kbit/s.
• This bitrate is significantly lower than the 50kbit/s challenge in CLIC 2024, while maintaining better subjective quality.

Technical underpinnings: The success of LivePortrait builds on key innovations in 3D facial modeling and dataset improvements.

• The model uses 3D rotation of abstract tensors to learn facial keypoints without explicit labeling.
• LivePortrait’s improvements over facevid2vid include a significantly larger training dataset and region-specific GAN losses.
• The approach allows for direct controllability of avatars, a advantage over many generative models.

Potential applications: While currently computationally intensive, the technology shows promise for various use cases in video conferencing and digital avatar creation.

• Possible applications include creating formal avatars for casual settings, enhancing low-quality webcam feeds, and enabling more immersive virtual meeting spaces.
• The technology could potentially allow for programmatic control of photorealistic video avatars, opening up possibilities for digital twins in meetings or messaging.

Looking ahead: As video compression technology continues to evolve, models like LivePortrait represent a significant step forward in achieving high-quality, low-bitrate video transmission.

• Future improvements in model efficiency and hardware capabilities could make this technology more accessible for edge devices.
• The balance between compression efficiency, computational requirements, and practical applications will likely shape the adoption and evolution of these technologies in video communication platforms.