Anthropic’s Claude can now control computers like people do

the claude computer control logoAnthropic

Anthropic’s already impressive Claude 3.5 Sonnet gains a significant performance boost on Tuesday as the generative AI startup rolls out an enhanced and updated version of the model alongside the new, lightweight Claude 3.5 Haiku. The Sonnet update includes a public beta feature that gives the AI basic control over the computer it’s running on.

Claude 3.5 Sonnet was already a performance leader when it comes to coding tasks, but the new version shows significant across-the-board improvements over its predecessor and steadily outperforms both Gemini 1.5 and GPT-4o on a variety of industry benchmarks. Gemini 1.5 Pro was the only model to best the new 3.5 Sonnet on any test, and did so on the MATH benchmark.

Recommended Videos

The new 3.5 Haiku is no slouch, either, despite its small size. Scheduled to be released later this month, 3.5 Haiku outperforms Claude 3.0 Opus, the company’s largest last generation model. Like its larger version, the new Haiku is exceedingly proficient at coding tasks, scoring 40.6% on the SWE-bench Verified — higher than both GPT-40 and the original 3.5 Sonnet.

Anthropic

Even more impressive, the new Claude 3.5 Sonnet can now interact with desktop apps via the “Computer Use” API. The AI can generate the necessary keystrokes, mouse clicks, and movements needed to emulate the human user. The company is quick to point out that the system is currently quite experimental and prone to errors. The underlying purpose of the public beta release is to elicit feedback from developers to rapidly improve the API’s performance.

“We trained Claude to see what’s happening on a screen and then use the software tools available to carry out tasks,” Anthropic wrote in a blog post. “When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place.”

It’s an AI agent, essentially. That is, its an AI that can automate other software processes, whether that’s generating and qualifying marketing leads, uncovering patterns and trends in medical data, or simply navigating to a specific website and filling out a form you need. Think of them as a more advanced version of existing Robotic Process Automation systems.

The company cites Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company as early adopters of the new feature. Replit, for example, is using Computer Control to “develop a key feature that evaluates apps as they’re being built for their Replit Agent product,” per the announcement.

There’s no need to worry about the AI going all Skynet on us (yet), as Anthropic explains. “Humans remain in control by providing specific prompts that direct Claude’s actions, like ‘use data from my computer and online to fill out this form,’” an Anthropic spokesperson told TechCrunch. “People enable access and limit access as needed. Claude breaks down the user’s prompts into computer commands (e.g., moving the cursor, clicking, typing) to accomplish that specific task.”

Anthropic also concedes that Computer Control could be misused to generate spam, spread misinformation, or commit fraud. In response, the company has developed new classifiers that identify when the API is being used and whether that use is “causing harm.”

Editors’ Recommendations

  • ChatGPT’s new Canvas feature sure looks a lot like Claude’s Artifacts

  • Here’s how Claude 3.5 Sonnet and GPT-4o stack up in a direct comparison




Related posts

Latest posts

2025 is going to be another big year for commercial moon missions

As soon as late February, a lunar lander will depart from NASA’s Kennedy Space Center on its way to the

2025 is going to be another big year for commercial moon missions

As soon as late February, a lunar lander will depart from NASA’s Kennedy Space Center on its way to the

You can officially download the TikTok app again on Android phones

TikTok is still absent from the Google Play Store. But the company is now letting users download the app officially on their Android phones.

I like the Galaxy S25 Ultra far more than I expected to

Samsung’s newest flagship has finally landed in stores and it would be easy to look at the as nothing more than an iterative upgrade that brings a few small upgrades to the table. However, as Andy covered in our , to do so would be to do a disservice to the overall experience. While reviewing […]

This adorable Noctua cooler completely transformed my gaming PC

I invested $60 in this tiny Noctua cooler, and it's made my small form factor gaming PC so much better.

Samsung’s ultra-thin Galaxy S25 Edge might get a monster camera

As per fresh leak, Galaxy S25 Edge will match the Galaxy S25 Ultra by serving a 200-megapixel camera, despite being the thinnest phone in Samsung's portfolio.

Spigen just accidentally leaked iPhone SE 4 renders

Case manufacturer Spigen has uploaded renders of the iPhone 4 SE (inside a case) to its website, and these renders line up with earlier leaks.

Fiio’s BTR13 is a budget DAC that lets you easily upgrade your phone audio

The BTR13 brings upgraded audio while still retaining the same focus on value.

The Galaxy S24 Ultra just broke new ground for Android flagships

A Samsung Galaxy flagship is back in the best-sellers club after six long years.

Fiio’s FT1 is the new king of budget headphones

With the FT1, Fiio is showing that it knows how to make incredible headphones on a budget.