×
Anthropic just gave its AI the ability to control your computer
Written by
Published on
Join our daily newsletter for breaking news, product launches and deals, research breakdowns, and other industry-leading AI coverage
Join Now

AI-powered computer control: Anthropic introduces groundbreaking feature: Anthropic’s latest Claude 3.5 Sonnet AI model now includes a revolutionary “computer use” feature in public beta, allowing the AI to control a computer by visually interpreting the screen and interacting with it through cursor movements, clicks, and text input.

  • The new capability is available through Anthropic’s API, enabling developers to integrate AI-powered computer control into their applications.
  • A demonstration video showcases Claude operating a Mac computer, highlighting the potential for AI to perform human-like interactions with digital interfaces.

Competitive landscape and industry context: While other tech giants have explored similar concepts, Anthropic’s implementation represents a significant leap forward in AI-computer interaction capabilities.

  • Microsoft’s Copilot Vision and OpenAI’s ChatGPT desktop app offer screen analysis features but lack the direct computer control demonstrated by Claude.
  • Google’s Gemini app for Android provides comparable functionalities on mobile devices.
  • Rabbit, a startup, has promised similar capabilities for its R1 device but has yet to deliver on this commitment.

Current limitations and future outlook: Anthropic acknowledges that the computer use feature is still in its early stages and has room for improvement.

  • The company describes the current implementation as potentially “cumbersome and error-prone.”
  • Anthropic is actively seeking developer feedback to refine and enhance the capability over time.
  • Certain complex computer actions, such as dragging and zooming, are not yet supported by Claude.
  • The AI’s perception of the screen is limited to a series of screenshots rather than a continuous video stream, potentially causing it to miss brief actions or notifications.

Ethical considerations and safeguards: Anthropic has implemented measures to ensure responsible use of the new feature.

  • Claude has been programmed to avoid engaging with social media platforms.
  • The AI includes systems to monitor and prevent election-related activities.
  • Safeguards are in place to discourage Claude from generating and posting content on social media, registering web domains, or interacting with government websites.

Claude 3.5 Sonnet: Performance improvements and pricing: Alongside the computer use feature, Anthropic has released an updated version of its Claude 3.5 Sonnet model.

  • The new model demonstrates significant improvements across various industry benchmarks.
  • Notable advancements have been made in agentic coding and tool use tasks.
  • On the SWE-bench Verified coding benchmark, Claude 3.5 Sonnet achieved a score of 49.0%, surpassing all publicly available models.
  • Performance on the TAU-bench agentic tool use task improved to 69.2% in the retail domain and 46.0% in the more challenging airline domain.
  • Despite these enhancements, Anthropic is offering the updated model to customers at the same price and speed as its predecessor.

Implications for AI development and human-computer interaction: Anthropic’s computer use feature represents a significant milestone in the evolution of AI capabilities and their potential impact on how we interact with technology.

  • This advancement could pave the way for more intuitive and efficient human-computer interaction, potentially reducing the learning curve for complex software applications.
  • The development raises questions about the future role of AI in automating digital tasks and its potential impact on workforce dynamics in various industries.
  • As AI systems become more capable of directly interacting with computer interfaces, it may lead to new paradigms in software design and user experience optimization.
Anthopic’s latest AI update can use a computer on its own

Recent News

Nvidia’s new AI agents can search and summarize huge quantities of visual data

NVIDIA's new AI Blueprint combines computer vision and generative AI to enable efficient analysis of video and image content, with potential applications across industries and smart city initiatives.

How Boulder schools balance AI innovation with student data protection

Colorado school districts embrace AI in classrooms, focusing on ethical use and data privacy while preparing students for a tech-driven future.

Microsoft Copilot Vision nears launch — here’s what we know right now

Microsoft's new AI feature can analyze on-screen content, offering contextual assistance without the need for additional searches or explanations.