Anthropic just gave its AI the ability to control your computer

AI-powered computer control: Anthropic introduces groundbreaking feature: Anthropic’s latest Claude 3.5 Sonnet AI model now includes a revolutionary “computer use” feature in public beta, allowing the AI to control a computer by visually interpreting the screen and interacting with it through cursor movements, clicks, and text input.

• The new capability is available through Anthropic’s API, enabling developers to integrate AI-powered computer control into their applications.
• A demonstration video showcases Claude operating a Mac computer, highlighting the potential for AI to perform human-like interactions with digital interfaces.

Competitive landscape and industry context: While other tech giants have explored similar concepts, Anthropic’s implementation represents a significant leap forward in AI-computer interaction capabilities.

• Microsoft’s Copilot Vision and OpenAI’s ChatGPT desktop app offer screen analysis features but lack the direct computer control demonstrated by Claude.
• Google’s Gemini app for Android provides comparable functionalities on mobile devices.
• Rabbit, a startup, has promised similar capabilities for its R1 device but has yet to deliver on this commitment.

Current limitations and future outlook: Anthropic acknowledges that the computer use feature is still in its early stages and has room for improvement.

• The company describes the current implementation as potentially “cumbersome and error-prone.”
• Anthropic is actively seeking developer feedback to refine and enhance the capability over time.
• Certain complex computer actions, such as dragging and zooming, are not yet supported by Claude.
• The AI’s perception of the screen is limited to a series of screenshots rather than a continuous video stream, potentially causing it to miss brief actions or notifications.

Ethical considerations and safeguards: Anthropic has implemented measures to ensure responsible use of the new feature.

• Claude has been programmed to avoid engaging with social media platforms.
• The AI includes systems to monitor and prevent election-related activities.
• Safeguards are in place to discourage Claude from generating and posting content on social media, registering web domains, or interacting with government websites.

Claude 3.5 Sonnet: Performance improvements and pricing: Alongside the computer use feature, Anthropic has released an updated version of its Claude 3.5 Sonnet model.

• The new model demonstrates significant improvements across various industry benchmarks.
• Notable advancements have been made in agentic coding and tool use tasks.
• On the SWE-bench Verified coding benchmark, Claude 3.5 Sonnet achieved a score of 49.0%, surpassing all publicly available models.
• Performance on the TAU-bench agentic tool use task improved to 69.2% in the retail domain and 46.0% in the more challenging airline domain.
• Despite these enhancements, Anthropic is offering the updated model to customers at the same price and speed as its predecessor.

Implications for AI development and human-computer interaction: Anthropic’s computer use feature represents a significant milestone in the evolution of AI capabilities and their potential impact on how we interact with technology.

• This advancement could pave the way for more intuitive and efficient human-computer interaction, potentially reducing the learning curve for complex software applications.
• The development raises questions about the future role of AI in automating digital tasks and its potential impact on workforce dynamics in various industries.
• As AI systems become more capable of directly interacting with computer interfaces, it may lead to new paradigms in software design and user experience optimization.