Not RAG but CAG: How cache-augmented generation improves AI for businesses

Large language model systems are adopting cache-augmented generation (CAG) as a simpler alternative to retrieval-augmented generation (RAG) for handling specialized information, according to new research from National Chengchi University in Taiwan.

Key innovation: Cache-augmented generation enables organizations to input their entire knowledge base directly into the prompt while leveraging advanced caching techniques to maintain performance.

• CAG eliminates the need for complex retrieval systems by placing all relevant documents directly in the prompt
• The approach works particularly well for organizations with smaller, static knowledge bases that fit within an LLM’s context window
• Advanced caching techniques from providers like OpenAI and Anthropic can reduce costs by up to 90% and latency by 85% on cached prompt components

Technical advantages: The emergence of long-context language models and improved training methods has made CAG increasingly viable for enterprise applications.

• Modern LLMs can handle significantly larger contexts: Claude 3.5 Sonnet (200,000 tokens), GPT-4 (128,000 tokens), and Gemini (2 million tokens)
• New benchmarks like BABILong and RULER demonstrate improved capabilities in processing and reasoning with longer sequences
• Pre-computing attention values for knowledge documents reduces processing time for each request

Performance comparison: Research experiments using a Llama-3.1-8B model demonstrated CAG’s advantages over traditional RAG approaches.

• CAG outperformed RAG systems using both basic BM25 and OpenAI embeddings on standard question-answering benchmarks
• The approach showed particular strength in scenarios requiring reasoning across multiple documents
• Response generation time was significantly reduced, especially with longer reference texts

Implementation considerations: While promising, CAG has specific limitations and use cases where it performs best.

• Best suited for organizations with relatively small, stable knowledge bases
• May not be appropriate when documents contain conflicting information
• Simple implementation makes it worth testing before investing in more complex RAG solutions

Looking ahead: The continued evolution of language models suggests broader applications for CAG in enterprise settings, though careful evaluation remains essential for determining the best approach for specific use cases. Organizations should consider starting with CAG experiments before committing to more complex retrieval systems, particularly when working with smaller document collections.