ConsoleX AI provides the Computer Use feature, which allows users to grant AI the ability to control the local computer for tasks like taking screenshots, mouse and keyboard operations, and file and terminal operations during conversations. It's like giving AI its own computer to help you complete more complex tasks.

There is a video demonstrates how it works:

How to Use the Computer Use Tool?

Preparation: Deploy Computer Use Desktop Environment and API

Before using the Computer Use tool, you need to deploy the Computer Use desktop environment and enable LLM to access your computer through API. The simplest way is through ConsoleY, ConsoleX's paired computer use environment, though you can also implement your own computer operation API interface.

Enable Computer Use Feature in ConsoleX AI

On ConsoleX, after selecting the Claude 3.5 Sonnet model from the model dropdown list, click the sidebar icon in the top right corner, then toggle the "Enable Computer Use" switch to the on position. First-time users need to enter the Computer Use Base URL, then click the "Save" button.

Use Computer Use Tool in Conversations

When you enter instructions that require the Computer Use tool in a conversation, the large language model will automatically call the Computer Use tool to complete the response.

For example, you can try these instructions

Describe what you see on the computer desktop

Open the browser and visit ConsoleX.ai

Important Notes:

The Computer Use feature is currently limited to use with the latest Claude 3.5 Sonnet model. The Computer Use option won't appear when other models are selected.

Using Computer Use will generate multiple interactions with the model, thus consuming more Tokens. It's recommended to enable "Auto Caching" to cache your prompts during chat and reduce costs.

— The ConsoleX Team