×
MiniMax releases new open-source LLM with 4M token context window
Written by
Published on
Join our daily newsletter for breaking news, product launches and deals, research breakdowns, and other industry-leading AI coverage
Join Now

MiniMax, a Singaporean AI company, has released and open-sourced a new family of AI models featuring an unprecedented 4-million token context window, doubling the previous industry record.

Key innovation: MiniMax’s new language model series introduces groundbreaking context handling capabilities that allow it to process the equivalent of a small library’s worth of text in a single exchange.

  • The MiniMax-01 series includes both a foundation large language model (MiniMax-Text-01) and a visual multi-modal model (MiniMax-VL-01)
  • The models are now available through Hugging Face, Github, Hailuo AI Chat, and MiniMax’s API
  • Pricing is highly competitive at $0.2 per million input tokens and $1.1 per million output tokens, significantly undercutting competitors like OpenAI

Technical architecture: The models employ a novel Lightning Attention mechanism that represents a significant departure from traditional transformer architecture.

  • Features 456 billion parameters with 45.9 billion activated during inference
  • Utilizes a mixture of experts (MoE) framework with 32 experts for improved scalability
  • Combines linear and traditional SoftMax layers to achieve near-linear complexity for long inputs
  • Implements specialized optimizations including MoE all-to-all communication and varlen ring attention

Performance metrics: The new models demonstrate competitive capabilities against industry leaders while excelling in specific areas.

  • Achieves performance comparable to GPT-4 and Claude-3.5 on mainstream benchmarks
  • Scored 100% accuracy on the Needle-In-A-Haystack task with 4-million-token context
  • Shows minimal degradation in performance as input length increases
  • Maintains efficiency through optimized CUDA kernel implementations

Accessibility and collaboration: MiniMax has emphasized openness and developer engagement with this release.

  • The models are available under a custom MiniMax license
  • Developers can access the technology through multiple platforms and integration options
  • The company welcomes technical suggestions and collaboration through [email protected]
  • Regular updates are planned to expand capabilities, including code and multi-modal enhancements

Market implications: As AI agents become increasingly sophisticated, MiniMax’s breakthrough in context handling positions it as a significant player in the evolving landscape.

  • The extended context window addresses growing demands for sustained memory in AI applications
  • Competitive pricing could disrupt the market, with rates significantly lower than established providers
  • The open-source approach may accelerate innovation in long-context AI applications

Looking ahead: While MiniMax has achieved a technical milestone with its 4-million token context window, the real test will be how developers leverage this expanded capability to create practical applications that take advantage of the model’s enhanced memory and processing abilities.

MiniMax unveils its own open source LLM with industry-leading 4M token context

Recent News

Could automated journalism replace human journalism?

This experimental AI news site combines automation with journalistic principles, producing over 20 daily articles at just 30 cents each while maintaining factual accuracy and source credibility.

Biosecurity concerns mount as AI outperforms virus experts

AI systems demonstrate superior practical problem-solving in virology laboratories, posing a concerning dual-use scenario where the same capabilities accelerating medical breakthroughs could provide step-by-step guidance for harmful applications to those without scientific expertise.

How AI is transforming smartphone communication

AI capabilities are now being embedded directly into existing messaging platforms, eliminating the need for separate apps while maintaining conversational context for more efficient communication.