Hugging Face boosts file transfers for AI repos with new chunking system

The Hugging Face Xet team is developing a new system to optimize file transfers for AI model repositories through an innovative approach to content-defined chunking (CDC). This technology aims to dramatically improve upload and download speeds for large AI models and datasets while maintaining efficient storage through smart deduplication.

Core innovation: Content-defined chunking enables efficient deduplication of data by breaking files into smaller pieces, but implementing this at scale requires careful optimization to balance performance and infrastructure demands.

• The team has open-sourced xet-core and hf_xet, tools that integrate with huggingface_hub to enable chunk-based file transfers
• Initial testing shows 2-3x faster transfer speeds in some cases
• The system aims to support rapid experimentation and collaboration for AI development teams

Technical challenges: Managing large-scale repositories through pure chunk-based approaches presents significant infrastructure and performance hurdles.

• A 200GB repository can generate approximately 3 million chunks at 64KB per chunk
• With 45PB of data across 2 million repositories, a purely chunk-based approach could result in 690 billion chunks
• Individual chunk management creates unsustainable network overhead and database strain

Solution architecture: The team implemented a multi-tiered aggregation strategy to optimize performance while maintaining deduplication benefits.

• Blocks bundle multiple chunks together in 64MB units, reducing storage entries by 1000x
• Shards track the relationship between files and chunks, enabling efficient identification of changed content
• Key chunks, representing 0.1% of total chunks, serve as reference points for local deduplication

Real-world implementation: Testing with quantized AI models demonstrates significant performance improvements.

• A repository containing 29 quantizations of a Gemma model (191GB total) required only 97GB of storage
• Upload times were reduced from 509 minutes to 258 minutes at 50MB/s
• Local chunk caching enables faster downloads by only retrieving changed content

Technical impact and roadmap: The new approach represents a fundamental shift in how large AI files are transferred and stored on the Hugging Face Hub.

• The system particularly benefits quantized models, which naturally contain repeated data patterns
• Initial deployment of Xet-backed repositories is planned for the coming weeks and months
• The technology will eventually be available to all builders on the Hub

Looking ahead: While the initial results are promising, the true test will come as the system scales across the Hub’s entire user base, potentially transforming how AI developers collaborate and iterate on their models.