OpenAI's computer tool on ZenithAI

On the Genius tier, ZenithAI’s /v1/responses takes OpenAI’s built-in computer tool. The model looks at your screenshots and answers with computer_call items whose coordinates are already screen pixels. Your code performs each action and sends back a new screenshot. Code written for OpenAI’s computer tool works after you change the base URL and the model name.

Which tool shape should I send?

Send one computer tool in tools, with model: "Genius". There are two forms, as on OpenAI:

Tool	Shape	Coordinates you receive
Preview	`{"type": "computer_use_preview", "display_width": 1920, "display_height": 1080, "environment": "windows"}`. Sizes are integers from 1 to 10000. `environment` is `windows`, `mac`, `linux`, `ubuntu` or `browser`	Pixels of the declared display
GA	`{"type": "computer"}`, with no other fields. The OpenAI Agents SDK sends this form	Pixels of the latest screenshot in `input`

Declare the size of the space your automation library clicks in as display_width and display_height. You can then send smaller screenshots, for example 1280 pixels wide, and still click the returned coordinates as they are. The server converts the model’s positions for you: do not convert them again.

A first request, with the task and a screenshot (base64 shortened):

curl --fail-with-body "https://dummydomain/v1/responses" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "model": "Genius",
    "truncation": "auto",
    "tools": [{"type": "computer_use_preview", "display_width": 1920, "display_height": 1080,
               "environment": "windows"}],
    "input": [{"role": "user", "content": [
      {"type": "input_text", "text": "Click Sign in."},
      {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgo..."}]}]
  }'

output holds one computer_call (shortened; ids are examples):

{"id": "cu_7f3a...", "type": "computer_call", "call_id": "call_1a2b3c4d5e6f7a8b", "status": "completed",
 "pending_safety_checks": [], "action": {"type": "click", "button": "left", "x": 1748, "y": 58}}

Click at (1748, 58) on your screen, take a screenshot, and send the next request with the whole input again, followed by this item and a computer_call_output:

{"type": "computer_call_output", "call_id": "call_1a2b3c4d5e6f7a8b",
 "output": {"type": "computer_screenshot", "image_url": "data:image/png;base64,iVBORw0KGgo..."}}

Stop when output has no computer_call: the model’s text in output_text is its answer.

Which actions can the model send?

With the preview tool a computer_call carries exactly one action, in action. With the GA tool it carries actions, a list of 1 to 5 actions. Perform them in order, with no screenshot in between, then send one screenshot. They are all in pixels of the same screenshot. The model is asked to group only steps that need no new look at the screen, such as a click on a field followed by typing.

Action	Fields	What your code does
`click`	`x`, `y`, `button` (`left`, `right`, `wheel`, `back`, `forward`)	Click there
`double_click`	`x`, `y`	Double-click there
`move`	`x`, `y`	Move the pointer, for example to open a hover menu
`drag`	`path`, 2 to 16 points `{"x", "y"}`	Press at the first point, move through the rest, release
`keypress`	`keys`, 1 to 8 key names such as `CTRL`, `S` or `ENTER`	Press them together
`type`	`text`	Type into the focused field
`scroll`	`x`, `y`, `scroll_x`, `scroll_y`	Scroll at that point. Pixels; positive `scroll_y` is down, positive `scroll_x` is right
`wait`	none	Wait a moment, then send a screenshot
`screenshot`	none	Send a screenshot without changing anything

Held keys. click, double_click, drag, move and scroll carry keys, up to 4 modifier keys such as ["CTRL"] or ["SHIFT"], when the model wants them held during the action, for example a ctrl-click. Press them down, perform the action, then release them. When no key is held the field is left out. On keypress, keys are the keys to press together, as before.

With the GA tool and no screenshot in input yet, the first call is a single screenshot or wait action, because there is no screen to measure yet.

A complete agent in Python

This agent uses the OpenAI SDK, and, as an example, mss and Pillow to capture the screen and pyautogui to act. Install with pip install openai mss pillow pyautogui and run it in a test environment (see safety).

import base64, io, time
import mss
import pyautogui
from PIL import Image
from openai import OpenAI

client = OpenAI(base_url="https://dummydomain/v1", api_key="YOUR_API_KEY")
MAX_STEPS = 20
SHOT_WIDTH = 1280            # screenshots are scaled to this width before they are sent
CONFIRM_EACH = True          # ask before every action; set False only inside a sandbox

SCREEN_W, SCREEN_H = pyautogui.size()      # the space pyautogui clicks in
TOOLS = [{"type": "computer_use_preview", "display_width": SCREEN_W, "display_height": SCREEN_H,
          "environment": "windows"}]       # or "mac", "linux", "ubuntu", "browser"
KEYS = {"cmd": "command", "super": "win", "meta": "win", "arrowup": "up", "arrowdown": "down",
        "arrowleft": "left", "arrowright": "right", "escape": "esc", "return": "enter"}
pyautogui.FAILSAFE = True    # slam the mouse into a screen corner to abort


def screenshot():
    """The primary monitor as a PNG data: URL, scaled to SHOT_WIDTH."""
    with mss.mss() as sct:
        raw = sct.grab(sct.monitors[1])
    image = Image.frombytes("RGB", raw.size, raw.bgra, "raw", "BGRX")
    if image.width > SHOT_WIDTH:
        image = image.resize((SHOT_WIDTH, round(image.height * SHOT_WIDTH / image.width)), Image.LANCZOS)
    buffer = io.BytesIO()
    image.save(buffer, "PNG")
    return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode()


def perform(a):
    """Run one action, holding its modifier keys (for example CTRL for a ctrl-click) while it runs."""
    held = [] if a["type"] == "keypress" else [KEYS.get(k.lower(), k.lower()) for k in a.get("keys") or []]
    for key in held:
        pyautogui.keyDown(key)
    try:
        act(a)
    finally:
        for key in reversed(held):
            pyautogui.keyUp(key)


def act(a):
    """Run one OpenAI computer action. x and y are already pixels of the declared display."""
    kind = a["type"]
    if kind == "click":
        button = {"wheel": "middle"}.get(a["button"], a["button"])
        if button in ("left", "right", "middle"):
            pyautogui.click(a["x"], a["y"], button=button)
    elif kind == "double_click":
        pyautogui.doubleClick(a["x"], a["y"])
    elif kind == "move":
        pyautogui.moveTo(a["x"], a["y"])
    elif kind == "drag":
        first, *rest = a["path"]
        pyautogui.moveTo(first["x"], first["y"])
        pyautogui.mouseDown()
        for point in rest:
            pyautogui.moveTo(point["x"], point["y"], duration=0.2)
        pyautogui.mouseUp()
    elif kind == "keypress":
        pyautogui.hotkey(*[KEYS.get(k.lower(), k.lower()) for k in a["keys"]])
    elif kind == "type":
        pyautogui.write(a["text"], interval=0.02)        # ASCII only; use the clipboard for other text
    elif kind == "scroll":
        pyautogui.moveTo(a["x"], a["y"])
        if a["scroll_y"]:                                # positive = down; pyautogui: positive = up
            pyautogui.scroll(-round(a["scroll_y"] / 100 * 120))   # about 100 px per wheel notch; tune it
        if a["scroll_x"]:
            pyautogui.hscroll(round(a["scroll_x"] / 100 * 120))
    elif kind == "wait":
        time.sleep(1)
    # "screenshot": nothing to do, a new screenshot is sent below


def run(task):
    items = [{"role": "user", "content": [{"type": "input_text", "text": task},
                                          {"type": "input_image", "image_url": screenshot()}]}]
    for step in range(1, MAX_STEPS + 1):
        response = client.responses.create(model="Genius", tools=TOOLS, input=items, truncation="auto")
        items += [item.model_dump(exclude_none=True) for item in response.output]
        calls = [item for item in response.output if item.type == "computer_call"]
        if not calls:                                    # text only: the model says it is finished
            print("Answer:", response.output_text)
            return
        for call in calls:
            data = call.model_dump(exclude_none=True)
            actions = data.get("actions") or [data["action"]]   # GA tool: 1 to 5 actions; preview: one
            print(f"step {step}: {actions}")
            if not CONFIRM_EACH or input("  run them all? [y/N] ").lower() == "y":
                for action in actions:                   # in order, no screenshot in between
                    perform(action)
                    time.sleep(0.2)
            time.sleep(0.6)                              # let the screen settle
            items.append({"type": "computer_call_output", "call_id": call.call_id,
                          "output": {"type": "computer_screenshot", "image_url": screenshot()}})

    # Step limit reached: tool_choice "none" allows a text answer only.
    items.append({"role": "user", "content": "Step limit reached. Say what you did and what is still open."})
    final = client.responses.create(model="Genius", tools=TOOLS, input=items, truncation="auto",
                                    tool_choice="none")
    print("Answer after step limit:", final.output_text)


if __name__ == "__main__":
    run("Open Notepad, type the words Hello from ZenithAI, and tell me what the window shows.")

The loop replays the whole input each turn. truncation: "auto" keeps it inside the limits: when the history no longer fits, the oldest items, and with them the oldest screenshots, are dropped. A call and its screenshot are always kept or dropped together. Without it, the request that replays a 9th screenshot is refused with HTTP 400 too_many_images.

You can keep the history on the server instead. Send store: true on every request, and from the second request on send only the new computer_call_output items with previous_response_id:

response = client.responses.create(model="Genius", tools=TOOLS, input=items, truncation="auto", store=True)
# ... perform the call, then send only its screenshot:
response = client.responses.create(model="Genius", tools=TOOLS, truncation="auto", store=True,
                                   previous_response_id=response.id, input=[screenshot_output])

The stored history still counts toward the image limits and the context window, so keep truncation: "auto". Unlike OpenAI, ZenithAI keeps a response only when the request sends store: true. The screenshots are then kept on your server for up to 30 days, so delete the responses when the run is done. See stored responses.

A turn can also hold a short message item next to the computer_call, one sentence on what the model is doing. It is not the final answer: perform the call and continue, as the sample does.

Can I use the OpenAI Agents SDK?

Yes. Give the agent a ComputerTool with your own computer implementation, as the SDK describes, use OpenAIResponsesModel(model="Genius", openai_client=client) with a client pointed at your server, and set ModelSettings(truncation="auto"). The SDK sends the GA tool, {"type": "computer"}, and the coordinates it passes to your computer are pixels of your latest screenshot. To chain on the server instead of replaying, add store=True to ModelSettings and pass auto_previous_response_id=True to Runner.run. Call set_tracing_disabled(True), or the SDK uploads traces to OpenAI.

The SDK’s on_safety_check is never called, because ZenithAI has no safety classifier and pending_safety_checks is always empty. Put your confirmation step in your computer implementation instead.

What else should I know about the computer tool?

One computer tool per request. It may sit next to your function tools if they are all strict: true. A non-strict function next to it is refused, and so is a function named computer.
Batches and confirmation. A GA call can hold up to 5 actions that run before you see the screen again. If your code confirms actions, check every action in the list, not only the first.
Forcing it. tool_choice: {"type": "computer_use_preview"} or {"type": "computer"}, matching the tool, makes the turn an action.
Screenshots in a computer_call_output must be data: URLs; file_id and remote addresses are refused. acknowledged_safety_checks and current_url are accepted and ignored.
Streaming. With stream: true, a computer call arrives whole in response.output_item.added and response.output_item.done, with no argument delta events.
Accuracy. The shape of every call is guaranteed. Whether it is the right action is the model’s judgement, so a click can still land on the wrong target. In our own test on Genius, 15 of 15 clicks landed within 6 pixels of the target. This is a small test of ours, not a benchmark.
Not on Chat Completions. /v1/chat/completions takes function tools only. Use your own function tools there.

Computer use overview: how the loop works and which way to choose
With your own function tools: strict tools, the 0 to 1000 grid, Python and Node
Errors, safety and limits
Computer tool reference and truncation auto
Function calling and tools: tool_choice, strict mode and the Agents SDK
Models: which tiers have vision and tools, and their limits

Common questions

Does ZenithAI support OpenAI's computer_use_preview tool?

Yes, on /v1/responses with the Genius tier. Send a computer_use_preview tool with your display size, or the GA computer tool, and the model answers with computer_call items whose coordinates are already screen pixels. Your code performs each action and sends back a screenshot as a computer_call_output.

Do I need to convert the coordinates of a computer_call?

No. The server converts the model's positions to pixels of the display size you declared with computer_use_preview, or of your latest screenshot with the GA computer tool. Click the returned x and y as they are.

Does ZenithAI run safety checks on computer actions?

No. pending_safety_checks is always empty and the Agents SDK's on_safety_check is never called. Your code must ask a person to confirm sensitive actions such as payments, sending messages, deleting data or entering credentials.

How do I keep a long screenshot loop inside the request limits?

Send truncation auto on /v1/responses. When the history no longer fits, the oldest items and their screenshots are dropped, and each call stays paired with its screenshot. Without it, the request that replays a 9th screenshot is refused with HTTP 400 too_many_images.

Can one computer_call hold several actions on ZenithAI?

With the GA computer tool, yes. Its actions list holds 1 to 5 actions, such as a click on a field and then the text to type, all in pixels of the same screenshot. Perform them in order, then send one screenshot. The computer_use_preview tool always returns exactly one action.

Does the computer tool support ctrl-click and shift-click?

Yes. click, double_click, drag, move and scroll carry keys, the modifier keys to hold during the action, such as CTRL for a ctrl-click. Press them down, perform the action, then release them. When no key is held the field is left out.