×
MiniMax releases new open-source LLM with 4M token context window
Written by
Published on
Join our daily newsletter for breaking news, product launches and deals, research breakdowns, and other industry-leading AI coverage
Join Now

MiniMax, a Singaporean AI company, has released and open-sourced a new family of AI models featuring an unprecedented 4-million token context window, doubling the previous industry record.

Key innovation: MiniMax’s new language model series introduces groundbreaking context handling capabilities that allow it to process the equivalent of a small library’s worth of text in a single exchange.

  • The MiniMax-01 series includes both a foundation large language model (MiniMax-Text-01) and a visual multi-modal model (MiniMax-VL-01)
  • The models are now available through Hugging Face, Github, Hailuo AI Chat, and MiniMax’s API
  • Pricing is highly competitive at $0.2 per million input tokens and $1.1 per million output tokens, significantly undercutting competitors like OpenAI

Technical architecture: The models employ a novel Lightning Attention mechanism that represents a significant departure from traditional transformer architecture.

  • Features 456 billion parameters with 45.9 billion activated during inference
  • Utilizes a mixture of experts (MoE) framework with 32 experts for improved scalability
  • Combines linear and traditional SoftMax layers to achieve near-linear complexity for long inputs
  • Implements specialized optimizations including MoE all-to-all communication and varlen ring attention

Performance metrics: The new models demonstrate competitive capabilities against industry leaders while excelling in specific areas.

  • Achieves performance comparable to GPT-4 and Claude-3.5 on mainstream benchmarks
  • Scored 100% accuracy on the Needle-In-A-Haystack task with 4-million-token context
  • Shows minimal degradation in performance as input length increases
  • Maintains efficiency through optimized CUDA kernel implementations

Accessibility and collaboration: MiniMax has emphasized openness and developer engagement with this release.

  • The models are available under a custom MiniMax license
  • Developers can access the technology through multiple platforms and integration options
  • The company welcomes technical suggestions and collaboration through [email protected]
  • Regular updates are planned to expand capabilities, including code and multi-modal enhancements

Market implications: As AI agents become increasingly sophisticated, MiniMax’s breakthrough in context handling positions it as a significant player in the evolving landscape.

  • The extended context window addresses growing demands for sustained memory in AI applications
  • Competitive pricing could disrupt the market, with rates significantly lower than established providers
  • The open-source approach may accelerate innovation in long-context AI applications

Looking ahead: While MiniMax has achieved a technical milestone with its 4-million token context window, the real test will be how developers leverage this expanded capability to create practical applications that take advantage of the model’s enhanced memory and processing abilities.

MiniMax unveils its own open source LLM with industry-leading 4M token context

Recent News

New framework prevents AI agents from taking unsafe actions in enterprise settings

The framework provides runtime guardrails that intercept unsafe AI agent actions while preserving core functionality, addressing a key barrier to enterprise adoption.

Leaked database reveals China’s AI-powered censorship system targeting political content

The leaked database exposes how China is using advanced language models to automatically identify and censor indirect references to politically sensitive topics beyond traditional keyword filtering.

Study: Anthropic uncovers neural circuits behind AI hallucinations

Anthropic researchers have identified specific neural pathways that determine when AI models fabricate information versus admitting uncertainty, offering new insights into the mechanics behind artificial intelligence hallucinations.