What exactly are ‘foundation models’

The concept of foundation models emerged in 2021 as researchers identified a new category of AI neural networks capable of handling diverse tasks after being trained on massive unlabeled datasets. These models represent a significant shift from earlier AI systems that were narrowly focused on specific tasks, as they can be adapted for various applications ranging from language processing to image analysis.

Key characteristics and capabilities: Foundation models represent a breakthrough in AI architecture by combining massive-scale training with adaptability across multiple domains.

• These AI systems learn from unlabeled datasets, eliminating the need for time-consuming manual data labeling
• Through fine-tuning, they can perform diverse tasks from text translation to medical image analysis
• The number of published foundation models more than doubled from 2022 to 2023, with 149 models released in 2023 alone

Technical evolution: The development of foundation models traces back to the breakthrough transformer architecture introduced in 2017, which sparked rapid advancement in the field.

• Google’s BERT model became open-source software, initiating a race to build increasingly powerful language models
• GPT-3, released in 2020, demonstrated unprecedented capabilities with 175 billion parameters
• Google’s 2024 Gemini Ultra represents the current state-of-the-art, requiring 50 billion petaflops of computing power

Multimodal capabilities: Foundation models have expanded beyond text to handle multiple types of data simultaneously.

• Vision Language Models (VLMs) can process and generate text, images, audio, and video
• The Cosmos Nemotron 34B model, trained on 355,000 videos and 2.8 million images, exemplifies advanced multimodal capabilities
• Diffusion models have enabled text-to-image generation, attracting millions of users to services like Midjourney

Real-world applications: These models are finding practical applications across various industries and use cases.

• Businesses are customizing pre-trained foundation models rather than building from scratch
• Digital twins powered by foundation models help optimize factory and warehouse operations
• The NVIDIA NeMo framework enables companies to create custom chatbots and AI assistants

Challenges and concerns: The widespread adoption of foundation models has raised important considerations about their responsible development and use.

• Models can amplify biases present in training data
• There are risks of generating inaccurate or misleading information
• Intellectual property rights violations remain a concern
• Researchers are developing safeguards including prompt filtering and dataset scrubbing

Looking ahead: While foundation models have demonstrated remarkable capabilities, researchers believe we’ve only scratched the surface of their potential applications and impact on various industries.