The Hugging Face Xet team is developing a new system to optimize file transfers for AI model repositories through an innovative approach to content-defined chunking (CDC). This technology aims to dramatically improve upload and download speeds for large AI models and datasets while maintaining efficient storage through smart deduplication.
Core innovation: Content-defined chunking enables efficient deduplication of data by breaking files into smaller pieces, but implementing this at scale requires careful optimization to balance performance and infrastructure demands.
- The team has open-sourced xet-core and hf_xet, tools that integrate with huggingface_hub to enable chunk-based file transfers
- Initial testing shows 2-3x faster transfer speeds in some cases
- The system aims to support rapid experimentation and collaboration for AI development teams
Technical challenges: Managing large-scale repositories through pure chunk-based approaches presents significant infrastructure and performance hurdles.
- A 200GB repository can generate approximately 3 million chunks at 64KB per chunk
- With 45PB of data across 2 million repositories, a purely chunk-based approach could result in 690 billion chunks
- Individual chunk management creates unsustainable network overhead and database strain
Solution architecture: The team implemented a multi-tiered aggregation strategy to optimize performance while maintaining deduplication benefits.
- Blocks bundle multiple chunks together in 64MB units, reducing storage entries by 1000x
- Shards track the relationship between files and chunks, enabling efficient identification of changed content
- Key chunks, representing 0.1% of total chunks, serve as reference points for local deduplication
Real-world implementation: Testing with quantized AI models demonstrates significant performance improvements.
- A repository containing 29 quantizations of a Gemma model (191GB total) required only 97GB of storage
- Upload times were reduced from 509 minutes to 258 minutes at 50MB/s
- Local chunk caching enables faster downloads by only retrieving changed content
Technical impact and roadmap: The new approach represents a fundamental shift in how large AI files are transferred and stored on the Hugging Face Hub.
- The system particularly benefits quantized models, which naturally contain repeated data patterns
- Initial deployment of Xet-backed repositories is planned for the coming weeks and months
- The technology will eventually be available to all builders on the Hub
Looking ahead: While the initial results are promising, the true test will come as the system scales across the Hub’s entire user base, potentially transforming how AI developers collaborate and iterate on their models.
Recent Stories
DOE fusion roadmap targets 2030s commercial deployment as AI drives $9B investment
The Department of Energy has released a new roadmap targeting commercial-scale fusion power deployment by the mid-2030s, though the plan lacks specific funding commitments and relies on scientific breakthroughs that have eluded researchers for decades. The strategy emphasizes public-private partnerships and positions AI as both a research tool and motivation for developing fusion energy to meet data centers' growing electricity demands. The big picture: The DOE's roadmap aims to "deliver the public infrastructure that supports the fusion private sector scale up in the 2030s," but acknowledges it cannot commit to specific funding levels and remains subject to Congressional appropriations. Why...
Oct 17, 2025Tying it all together: Credo’s purple cables power the $4B AI data center boom
Credo, a Silicon Valley semiconductor company specializing in data center cables and chips, has seen its stock price more than double this year to $143.61, following a 245% surge in 2024. The company's signature purple cables, which cost between $300-$500 each, have become essential infrastructure for AI data centers, positioning Credo to capitalize on the trillion-dollar AI infrastructure expansion as hyperscalers like Amazon, Microsoft, and Elon Musk's xAI rapidly build out massive computing facilities. What you should know: Credo's active electrical cables (AECs) are becoming indispensable for connecting the massive GPU clusters required for AI training and inference. The company...
Oct 17, 2025Vatican launches Latin American AI network for human development
The Vatican hosted a two-day conference bringing together 50 global experts to explore how artificial intelligence can advance peace, social justice, and human development. The event launched the Latin American AI Network for Integral Human Development and established principles for ethical AI governance that prioritize human dignity over technological advancement. What you should know: The Pontifical Academy of Social Sciences, the Vatican's research body for social issues, organized the "Digital Rerum Novarum" conference on October 16-17, combining academic research with practical AI applications. Participants included leading experts from MIT, Microsoft, Columbia University, the UN, and major European institutions. The conference...