×
How new AI models are compressing videos without reducing quality
Written by
Published on
Join our daily newsletter for breaking news, product launches and deals, research breakdowns, and other industry-leading AI coverage
Join Now

Advancements in video compression technology: LivePortrait, a recent model in the field of 2D avatar/portrait animation, demonstrates significant potential for revolutionizing video compression, particularly for talking head videos.

  • The model can animate still images, avoiding the need for rendering complex 3D models that struggle with small facial details.
  • This technology has implications for social media, where it could become ubiquitous, but also raises concerns about trust in online content.

The core concept of compression: By leveraging predictive algorithms, LivePortrait can compress frame information into a sparse set of cues for reconstruction, building upon Nvidia’s facevid2vid paper.

  • The compression method relies on transmitting only changes in expression, pose, and facial keypoints, using a shared source image between sender and receiver.
  • This approach achieves high perceptual quality at extremely low bitrates, outperforming traditional video codecs in terms of visual artifacts at comparable levels.

Advantages and limitations: While offering impressive compression ratios, the technology comes with certain trade-offs and challenges.

  • There is no straightforward way to adjust the quality-bitrate balance, unlike traditional codecs.
  • As a generative model, there’s potential for significant reconstruction errors in worst-case scenarios.
  • The high compression rate comes at the cost of increased computational requirements, with LivePortrait needing an RTX 4090 for real-time processing.

Performance analysis: Experiments with LivePortrait show promising results in reconstructing video frames, particularly in scenarios closely matching its training conditions.

  • The model performs best when animating frames from the same video, simulating a video call scenario.
  • Some discrepancies are noticeable, such as slight head shakiness and occasional issues with eye gaze and teeth rendering.
  • Performance degrades when dealing with significant shoulder movement or difficult head angles.

Bitrate efficiency: LivePortrait achieves remarkably low bitrates for video transmission, potentially as low as 22kbit/s with additional optimizations.

  • The model transmits only transformation parameters for facial keypoints, resulting in a base bitrate of 36kbit/s for 30FPS video.
  • Further compression is possible through entropy coding and temporal priors, potentially reducing the bitrate to around 22kbit/s.
  • This bitrate is significantly lower than the 50kbit/s challenge in CLIC 2024, while maintaining better subjective quality.

Technical underpinnings: The success of LivePortrait builds on key innovations in 3D facial modeling and dataset improvements.

  • The model uses 3D rotation of abstract tensors to learn facial keypoints without explicit labeling.
  • LivePortrait’s improvements over facevid2vid include a significantly larger training dataset and region-specific GAN losses.
  • The approach allows for direct controllability of avatars, a advantage over many generative models.

Potential applications: While currently computationally intensive, the technology shows promise for various use cases in video conferencing and digital avatar creation.

  • Possible applications include creating formal avatars for casual settings, enhancing low-quality webcam feeds, and enabling more immersive virtual meeting spaces.
  • The technology could potentially allow for programmatic control of photorealistic video avatars, opening up possibilities for digital twins in meetings or messaging.

Looking ahead: As video compression technology continues to evolve, models like LivePortrait represent a significant step forward in achieving high-quality, low-bitrate video transmission.

  • Future improvements in model efficiency and hardware capabilities could make this technology more accessible for edge devices.
  • The balance between compression efficiency, computational requirements, and practical applications will likely shape the adoption and evolution of these technologies in video communication platforms.
Perceptually lossless (talking head) video compression at 22kbit/s

Recent News

Google’s Anthropic deal faces Justice Department scrutiny

U.S. regulators seek to restrict Google's ability to invest in AI startups, marking the first major government intervention in big tech's artificial intelligence deals.

AI will drive major scientific advances, NVIDIA CEO tells SC24

NVIDIA's latest computing tools reduce scientific simulation times from weeks to minutes, making advanced research more accessible to labs and companies.

More than a name change: Digital transformation is now AI transformation

CEOs are shifting resources from broad digital initiatives to focused AI projects, while technology leaders wrestle with practical implementation challenges and risk management.