Computer use with your own function tools

Use this way on /v1/chat/completions, on Smart, or when you want actions of your own. You describe the actions as strict function tools, send screenshots as images, and map the model’s coordinates to your screen. On /v1/responses with Genius you can use OpenAI’s computer tool instead, which returns pixels.

Which function tools should I define?

OpenAI’s computer actions map one to one onto function tools you define. This set covers them:

OpenAI action	Function tool here	Notes
click	click	x, y on the 0 to 1000 grid and a button
double_click	double_click	x, y
type	type_text	text
keypress	press_keys	An array of keys from a fixed list, pressed together
scroll	scroll	x, y, a direction and an amount
drag	drag	Start and end points
move	move_mouse	x, y
wait	wait	seconds
screenshot	none	Not needed: every turn already returns a fresh screenshot

Mark every tool strict: true, set additionalProperties: false, and list every property in required. With strict tools, tool_choice auto and a single call per turn, the model’s output is generated to match these schemas, so a click always has both x and y. See how strict mode works. Save this as tools.json, in Chat Completions format. Both code samples below read it.

[
  {"type": "function", "function": {
    "name": "click", "strict": true,
    "description": "Click a point on the screen. x and y are on a 0 to 1000 grid of the latest screenshot: (0, 0) is the top left corner and (1000, 1000) the bottom right.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y", "button"],
      "properties": {
        "x": {"type": "integer", "description": "Horizontal position on a 0 to 1000 grid"},
        "y": {"type": "integer", "description": "Vertical position on a 0 to 1000 grid"},
        "button": {"type": "string", "enum": ["left", "right", "middle"]}}}}},
  {"type": "function", "function": {
    "name": "double_click", "strict": true,
    "description": "Double-click a point on the screen, for example to open a file. x and y are on a 0 to 1000 grid of the latest screenshot.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y"],
      "properties": {
        "x": {"type": "integer", "description": "Horizontal position on a 0 to 1000 grid"},
        "y": {"type": "integer", "description": "Vertical position on a 0 to 1000 grid"}}}}},
  {"type": "function", "function": {
    "name": "move_mouse", "strict": true,
    "description": "Move the pointer without clicking, for example to open a hover menu. x and y are on a 0 to 1000 grid of the latest screenshot.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y"],
      "properties": {
        "x": {"type": "integer", "description": "Horizontal position on a 0 to 1000 grid"},
        "y": {"type": "integer", "description": "Vertical position on a 0 to 1000 grid"}}}}},
  {"type": "function", "function": {
    "name": "type_text", "strict": true,
    "description": "Type text into the focused field. Click the field first. To press Enter or Tab use press_keys.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["text"],
      "properties": {"text": {"type": "string"}}}}},
  {"type": "function", "function": {
    "name": "press_keys", "strict": true,
    "description": "Press keys together as one shortcut, for example [\"ctrl\", \"s\"], or press a single key such as [\"enter\"].",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["keys"],
      "properties": {"keys": {"type": "array", "minItems": 1, "maxItems": 4,
        "items": {"type": "string", "enum": ["ctrl", "shift", "alt", "win", "enter", "tab", "esc", "backspace", "delete", "space", "up", "down", "left", "right", "home", "end", "pageup", "pagedown", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]}}}}}},
  {"type": "function", "function": {
    "name": "scroll", "strict": true,
    "description": "Scroll at a point. x and y are on a 0 to 1000 grid of the latest screenshot. amount is the number of wheel steps, from 1 to 10.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y", "direction", "amount"],
      "properties": {
        "x": {"type": "integer", "description": "Horizontal position on a 0 to 1000 grid"},
        "y": {"type": "integer", "description": "Vertical position on a 0 to 1000 grid"},
        "direction": {"type": "string", "enum": ["up", "down", "left", "right"]},
        "amount": {"type": "integer", "description": "Wheel steps, 1 to 10"}}}}},
  {"type": "function", "function": {
    "name": "drag", "strict": true,
    "description": "Press at the start point, move to the end point and release. All four values are on a 0 to 1000 grid of the latest screenshot.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["start_x", "start_y", "end_x", "end_y"],
      "properties": {
        "start_x": {"type": "integer", "description": "Start, horizontal, 0 to 1000 grid"},
        "start_y": {"type": "integer", "description": "Start, vertical, 0 to 1000 grid"},
        "end_x": {"type": "integer", "description": "End, horizontal, 0 to 1000 grid"},
        "end_y": {"type": "integer", "description": "End, vertical, 0 to 1000 grid"}}}}},
  {"type": "function", "function": {
    "name": "wait", "strict": true,
    "description": "Wait for the screen to change, for example while a page loads. seconds is from 1 to 10.",
    "parameters": {"type": "object", "additionalProperties": false, "required": ["seconds"],
      "properties": {"seconds": {"type": "integer", "description": "Seconds to wait, 1 to 10"}}}}}
]

Three rules make these tools work well:

• Describe the grid in every coordinate. Say “0 to 1000 grid” in each description, even though the models use that grid regardless.
• One parameter per value. Use x and y, never a list such as [x, y].
• No finish tool is needed. With tool_choice auto the model ends the run by answering in text. If you prefer an explicit signal, add a done tool with one summary string.

JSON Schema ranges such as minimum and maximum are not enforced while the call is generated, so state the ranges in the descriptions and clamp the values in your code, as the samples do.

How do I send a screenshot?

Send each screenshot as a data: URL in an image part. Remote https addresses work only if your administrator has allowed the host, so use data: URLs.

• Format: PNG, JPEG, WebP, GIF or BMP. PNG keeps small text sharp. JPEG is smaller.
• Detail: detail may be left out or set to "auto", "low", "high" or "original". It is a hint only: every image is read at the tier’s own resolution.
• Where: on Chat Completions, images go in user messages only, because tool messages are text-only. So send the tool result as text, then a user message with the new screenshot, as the Python sample does. On Responses, a function_call_output can carry text and input_image parts together.

Scale screenshots down. A large screenshot adds little accuracy and a lot of context. The server reserves context for every image when it checks a request, from about 1,200 tokens on some tiers, up to about 4,100 tokens for a very large image on others (about one token per 32 by 32 pixels). A width of 1280 pixels is a good default. Make sure text is still readable at the size you send, and always map coordinates using the size of the image you sent (next section).

Keep only the latest screenshots. Every request carries the whole history, and the endpoint stores nothing. On /v1/responses, truncation: "auto" drops the oldest items for you when a request no longer fits. A request holds at most 8 images, 10 MiB each and 20 MiB decoded in total, and tool requests are admitted against the tier’s tool_context_window (24,576 tokens by default), which also holds your tool schemas and the answer reserve. Keep the last two or three screenshots. Replace each older one with a short text note, such as “(older screenshot removed)”, and keep the text of the steps so the model still knows what it did. Both samples below do this. Read tool_context_window, max_output_tokens_with_tools and max_images_per_request from /v1/models instead of hard-coding them.

How do I turn the model’s coordinates into screen pixels?

The vision models answer screen positions on a 0 to 1000 grid of the screenshot they were sent, with (0, 0) at the top left, whatever the tool description says. Your client maps them:

pixel_x = x * screenshot_width  / 1000
pixel_y = y * screenshot_height / 1000

If you resized the screenshot before sending it, scale from the screenshot to the screen next:

screen_x = pixel_x * screen_width  / screenshot_width
screen_y = pixel_y * screen_height / screenshot_height

For example, a 1920 by 1080 screen sent as a 1280 by 720 screenshot. The model returns x = 500, y = 250:

Step	x	y
On the grid	500	250
Pixel in the screenshot	500 * 1280 / 1000 = 640	250 * 720 / 1000 = 180
Pixel on the screen	640 * 1920 / 1280 = 960	180 * 1080 / 720 = 270

def to_screen(x, y, shot_w, shot_h, screen_w, screen_h):
    """Grid (0 to 1000) on the screenshot you sent -> a point in your automation library's pixel space."""
    x, y = min(max(x, 0), 1000), min(max(y, 0), 1000)
    px, py = x * shot_w / 1000, y * shot_h / 1000          # pixel in the screenshot
    return round(px * screen_w / shot_w), round(py * screen_h / shot_h)

In our test of six clicks on a real screenshot with Genius, mapping the grid this way landed 4 to 23 px from the target. Taking the same values as pixels missed by 168 to 237 px. This is a small test of ours, not a benchmark.

High-DPI screens

Use the size of the space your automation library clicks in, not the size of the capture. On a display with scaling (for example 200 percent) a capture can be 3840 by 2160 pixels while the mouse library works in 1920 by 1080 points. Use the second size as screen_w and screen_h.

Can I see a complete function-tool agent in Python?

This agent uses Chat Completions and the OpenAI SDK for the model and, as an example, mss and Pillow to capture the screen and pyautogui to act. The screen library is your choice: use any code that can capture the screen and perform the actions, or a browser driver or a virtual machine API. Install with pip install openai mss pillow pyautogui, put tools.json next to the script, and run it in a test environment (see safety).

agent.py

import base64, io, json, time
import mss
import pyautogui
from PIL import Image
from openai import OpenAI, BadRequestError, InternalServerError

client = OpenAI(base_url="https://dummydomain/v1", api_key="YOUR_API_KEY")
MODEL = "Genius"           # a tier with capabilities.vision and capabilities.tools
MAX_STEPS = 20
KEEP_SCREENSHOTS = 2
SHOT_WIDTH = 1280          # screenshots are scaled to this width before they are sent
CONFIRM_EACH = True        # ask before every action; set False only inside a sandbox

TOOLS = json.load(open("tools.json"))
SYSTEM = (
    "You control a computer by looking at screenshots. Take one action at a time with the tools. "
    "Positions are on a 0 to 1000 grid of the latest screenshot: (0, 0) is the top left, (1000, 1000) the "
    "bottom right. Look at the new screenshot after every action. When the task is done, or cannot be done, "
    "reply in plain text and call no tool."
)

pyautogui.FAILSAFE = True  # slam the mouse into a screen corner to abort


def screenshot():
    """Capture the primary monitor. Returns (data URL, width, height) of the image that will be SENT."""
    with mss.mss() as sct:
        raw = sct.grab(sct.monitors[1])
    image = Image.frombytes("RGB", raw.size, raw.bgra, "raw", "BGRX")
    if image.width > SHOT_WIDTH:
        image = image.resize((SHOT_WIDTH, round(image.height * SHOT_WIDTH / image.width)), Image.LANCZOS)
    buffer = io.BytesIO()
    image.save(buffer, "PNG")
    data = base64.b64encode(buffer.getvalue()).decode()
    return "data:image/png;base64," + data, image.width, image.height


def to_screen(x, y, shot_w, shot_h):
    screen_w, screen_h = pyautogui.size()                  # the space pyautogui clicks in
    x, y = min(max(x, 0), 1000), min(max(y, 0), 1000)
    px, py = x * shot_w / 1000, y * shot_h / 1000
    return round(px * screen_w / shot_w), round(py * screen_h / shot_h)


def act(name, a, size):
    """Perform one action. Returns a short text result for the model."""
    if name in ("click", "double_click", "move_mouse"):
        sx, sy = to_screen(a["x"], a["y"], *size)
        if name == "click":
            pyautogui.click(sx, sy, button=a["button"])
        elif name == "double_click":
            pyautogui.doubleClick(sx, sy)
        else:
            pyautogui.moveTo(sx, sy)
    elif name == "type_text":
        pyautogui.write(a["text"], interval=0.02)          # ASCII only; use a clipboard tool for other text
    elif name == "press_keys":
        pyautogui.hotkey(*a["keys"])
    elif name == "scroll":
        sx, sy = to_screen(a["x"], a["y"], *size)
        pyautogui.moveTo(sx, sy)
        steps = min(max(a["amount"], 1), 10) * 120
        if a["direction"] in ("up", "down"):
            pyautogui.scroll(steps if a["direction"] == "up" else -steps)
        else:
            pyautogui.hscroll(steps if a["direction"] == "right" else -steps)
    elif name == "drag":
        sx, sy = to_screen(a["start_x"], a["start_y"], *size)
        ex, ey = to_screen(a["end_x"], a["end_y"], *size)
        pyautogui.moveTo(sx, sy)
        pyautogui.dragTo(ex, ey, duration=0.4, button="left")
    elif name == "wait":
        time.sleep(min(max(a["seconds"], 1), 10))
    else:
        return f"Unknown action {name}."
    return f"Done: {name}."


def image_part(shot):
    return {"type": "image_url", "image_url": {"url": shot[0]}}


def drop_old_screenshots(messages, keep):
    """Replace all but the last `keep` screenshots with text, so the request stays inside the image and context limits."""
    holders = [m for m in messages if isinstance(m, dict) and m["role"] == "user" and isinstance(m["content"], list)
               and any(p["type"] == "image_url" for p in m["content"])]
    for m in holders[:-keep] if keep else holders:
        m["content"] = [p for p in m["content"] if p["type"] == "text"] + [
            {"type": "text", "text": "(older screenshot removed)"}]


def ask(messages, **extra):
    """One model turn with the error handling this loop needs."""
    keep = KEEP_SCREENSHOTS
    for attempt in range(3):
        try:
            reply = client.chat.completions.create(
                model=MODEL, messages=messages, tools=TOOLS, parallel_tool_calls=False, **extra)
            return reply.choices[0].message
        except InternalServerError as err:                 # 5xx, after the SDK's own retries
            if err.code == "invalid_tool_call":            # never a partial call; ask again
                continue
            raise                                          # e.g. tool_choice_violation: change the request
        except BadRequestError as err:
            if err.code == "context_length_exceeded" and keep > 1:
                keep -= 1                                  # too much context: keep fewer screenshots
                drop_old_screenshots(messages, keep)
                continue
            raise                                          # other 400s mean the request itself must change
    raise RuntimeError("the model did not return a valid call after 3 attempts")


def run(task):
    shot = screenshot()
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": [{"type": "text", "text": task}, image_part(shot)]},
    ]
    for step in range(1, MAX_STEPS + 1):
        message = ask(messages)
        messages.append(message)
        if message.content and message.tool_calls:         # a short note next to the call
            print("  note:", message.content)
        if not message.tool_calls:                         # plain text: the model says it is finished
            print("Answer:", message.content)
            return
        call = message.tool_calls[0]                       # strict mode returns one call per turn
        args = json.loads(call.function.arguments)
        print(f"step {step}: {call.function.name} {args}")
        if CONFIRM_EACH and input("  run it? [y/N] ").lower() != "y":
            result = "The user refused this action. Choose another way or stop."
        else:
            try:
                result = act(call.function.name, args, shot[1:])
            except Exception as err:                       # tell the model what happened
                result = f"The action failed: {err}"
        time.sleep(0.6)                                    # let the screen settle
        shot = screenshot()
        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
        messages.append({"role": "user", "content": [{"type": "text", "text": "Screen after the action:"},
                                                     image_part(shot)]})
        drop_old_screenshots(messages, KEEP_SCREENSHOTS)

    # Step limit reached: ask for the final answer. tool_choice "none" allows text only.
    messages.append({"role": "user", "content": "Step limit reached. Say what you did and what is still open."})
    final = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS, tool_choice="none")
    print("Answer after step limit:", final.choices[0].message.content)


if __name__ == "__main__":
    run("Open Notepad, type the words Hello from ZenithAI, and tell me what the window shows.")

Can I do the same in Node.js?

This version uses the Responses API, where the screenshot travels inside the function_call_output. Screen capture and input control depend on your platform, so they are stubs here: fill them in with the desktop automation library, browser driver or virtual machine API you use. It needs Node 18 or later, npm install openai and the same tools.json. Save it as agent.mjs.

agent.mjs

import OpenAI from "openai";
import fs from "node:fs";

const client = new OpenAI({ baseURL: "https://dummydomain/v1", apiKey: process.env.API_KEY });
const MODEL = "Genius";     // a tier with capabilities.vision and capabilities.tools
const MAX_STEPS = 20;
const KEEP_SCREENSHOTS = 2;
const SCREEN = { width: 1920, height: 1080 };   // the pixel space your input functions use

// Responses tools are flat: {type, name, description, strict, parameters}.
const tools = JSON.parse(fs.readFileSync("tools.json", "utf8")).map((t) => ({ type: "function", ...t.function }));
const SYSTEM =
  "You control a computer by looking at screenshots. Take one action at a time with the tools. " +
  "Positions are on a 0 to 1000 grid of the latest screenshot: (0, 0) is the top left, (1000, 1000) the " +
  "bottom right. Look at the new screenshot after every action. When the task is done, or cannot be done, " +
  "reply in plain text and call no tool.";

// ---- Your code: replace these two stubs ------------------------------------------------------
// Return the screen as PNG bytes with the size of the image you will send (scale it to ~1280 px wide).
async function captureScreen() { throw new Error("implement captureScreen() -> { png: Buffer, width, height }"); }
// Perform one action at screen pixel coordinates and return a short text result.
async function perform(name, args, point) { throw new Error("implement perform()"); }
// -----------------------------------------------------------------------------------------------

const clamp = (v, lo, hi) => Math.min(Math.max(v, lo), hi);
// Grid (0 to 1000) on the screenshot you sent -> a pixel on your screen.
function toScreen(x, y, shot) {
  const px = (clamp(x, 0, 1000) * shot.width) / 1000;    // pixel in the screenshot
  const py = (clamp(y, 0, 1000) * shot.height) / 1000;
  return { x: Math.round((px * SCREEN.width) / shot.width), y: Math.round((py * SCREEN.height) / shot.height) };
}
const imagePart = (shot) => ({ type: "input_image", image_url: `data:image/png;base64,${shot.png.toString("base64")}` });

// Replace all but the last `keep` screenshots with text so the request stays inside the image and context limits.
function dropOldScreenshots(input, keep) {
  const holders = [];
  for (const item of input) {
    const parts = item.type === "function_call_output" ? item.output : item.role === "user" ? item.content : null;
    if (Array.isArray(parts) && parts.some((p) => p.type === "input_image")) holders.push(item);
  }
  for (const item of holders.slice(0, Math.max(holders.length - keep, 0))) {
    const key = item.type === "function_call_output" ? "output" : "content";
    item[key] = [...item[key].filter((p) => p.type === "input_text"), { type: "input_text", text: "(older screenshot removed)" }];
  }
}

function pointsFor(name, a, shot) {
  if ("x" in a) return toScreen(a.x, a.y, shot);
  if ("start_x" in a) return { start: toScreen(a.start_x, a.start_y, shot), end: toScreen(a.end_x, a.end_y, shot) };
  return null;
}

async function run(task) {
  let shot = await captureScreen();
  const input = [{ role: "user", content: [{ type: "input_text", text: task }, imagePart(shot)] }];

  for (let step = 1; step <= MAX_STEPS; step++) {
    const response = await client.responses.create({
      model: MODEL, instructions: SYSTEM, input, tools, parallel_tool_calls: false,
    });
    const calls = response.output.filter((item) => item.type === "function_call");
    if (calls.length === 0) { console.log("Answer:", response.output_text); return; }

    input.push(...response.output);              // output items can be sent back unchanged
    const call = calls[0];                       // strict mode returns one call per turn
    const args = JSON.parse(call.arguments);
    console.log(`step ${step}: ${call.name}`, args);

    let result;
    try { result = await perform(call.name, args, pointsFor(call.name, args, shot)); }
    catch (err) { result = `The action failed: ${err.message}`; }

    shot = await captureScreen();
    input.push({
      type: "function_call_output",
      call_id: call.call_id,
      output: [{ type: "input_text", text: result }, imagePart(shot)],
    });
    dropOldScreenshots(input, KEEP_SCREENSHOTS);
  }

  // Step limit reached: tool_choice "none" allows a text answer only.
  input.push({ role: "user", content: "Step limit reached. Say what you did and what is still open." });
  const final = await client.responses.create({ model: MODEL, instructions: SYSTEM, input, tools, tool_choice: "none" });
  console.log("Answer after step limit:", final.output_text);
}

await run("Open Notepad, type the words Hello from ZenithAI, and tell me what the window shows.");

What does one turn look like in curl?

A request with a task and a screenshot, on Chat Completions. The base64 data is shortened here. The tools are the entries of tools.json, shortened to the first two:

curl --fail-with-body "https://dummydomain/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "model": "Genius",
    "parallel_tool_calls": false,
    "messages": [
      {"role": "system", "content": "You control a computer by looking at screenshots. Take one action at a time with the tools. Positions are on a 0 to 1000 grid of the latest screenshot. When the task is done, reply in plain text and call no tool."},
      {"role": "user", "content": [
        {"type": "text", "text": "Click the Save button."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
      ]}
    ],
    "tools": [
      {"type": "function", "function": {"name": "click", "strict": true,
        "description": "Click a point on the screen. x and y are on a 0 to 1000 grid of the latest screenshot.",
        "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y", "button"],
          "properties": {"x": {"type": "integer"}, "y": {"type": "integer"},
                         "button": {"type": "string", "enum": ["left", "right", "middle"]}}}}},
      {"type": "function", "function": {"name": "type_text", "strict": true,
        "description": "Type text into the focused field.",
        "parameters": {"type": "object", "additionalProperties": false, "required": ["text"],
          "properties": {"text": {"type": "string"}}}}}
    ]
  }'

The reply is one tool call, with finish_reason set to tool_calls (shortened, ids are examples):

{
  "object": "chat.completion",
  "model": "Genius",
  "choices": [{
    "index": 0,
    "finish_reason": "tool_calls",
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1a2b3c4d5e6f7a8b",
        "type": "function",
        "function": {"name": "click", "arguments": "{\"x\":845,\"y\":112,\"button\":\"left\"}"}
      }]
    }
  }]
}

Your code maps (845, 112) to screen pixels, clicks, and sends the next request with the earlier messages, the assistant message above, a tool message and a user message holding the new screenshot:

[
  {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1a2b3c4d5e6f7a8b", "type": "function",
    "function": {"name": "click", "arguments": "{\"x\":845,\"y\":112,\"button\":\"left\"}"}}]},
  {"role": "tool", "tool_call_id": "call_1a2b3c4d5e6f7a8b", "content": "Done: click."},
  {"role": "user", "content": [
    {"type": "text", "text": "Screen after the action:"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
  ]}
]

What is the Responses version of the same turn?

On /v1/responses the tool is flat and the screenshot comes back inside the function_call_output. The model’s turn is a function_call item, and the next request sends the whole conversation again. With store: true it can send only the new items and previous_response_id instead (stored responses):

{
  "model": "Genius",
  "instructions": "You control a computer by looking at screenshots. ...",
  "parallel_tool_calls": false,
  "tools": [
    {"type": "function", "name": "click", "strict": true,
     "description": "Click a point on the screen. x and y are on a 0 to 1000 grid of the latest screenshot.",
     "parameters": {"type": "object", "additionalProperties": false, "required": ["x", "y", "button"],
       "properties": {"x": {"type": "integer"}, "y": {"type": "integer"},
                      "button": {"type": "string", "enum": ["left", "right", "middle"]}}}}
  ],
  "input": [
    {"role": "user", "content": [
      {"type": "input_text", "text": "Click the Save button."},
      {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgo..."}]},
    {"type": "function_call", "call_id": "call_1a2b3c4d5e6f7a8b", "name": "click",
     "arguments": "{\"x\":845,\"y\":112,\"button\":\"left\"}"},
    {"type": "function_call_output", "call_id": "call_1a2b3c4d5e6f7a8b", "output": [
      {"type": "input_text", "text": "Done: click."},
      {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgo..."}]}
  ]
}

Send each function_call followed immediately by its function_call_output. An image output without a matching call in that block is refused with unpaired_tool_output_image. See Return images from your functions.

What does a whole run look like?

Task: “Save this note as report.txt on the Desktop.” The screenshot is 1280 by 720 and the screen 1920 by 1080. Each row is one turn of the loop.

Turn	The model sends	Your code does	Next screenshot shows
1	press_keys ["ctrl", "s"]	Presses Ctrl+S	The Save As dialog
2	click x: 500, y: 430, button: "left"	Clicks the file name box at screen pixel (960, 464)	The box focused
3	type_text text: "report.txt"	Types the name	The name in the box
4	click x: 780, y: 610, button: "left"	Clicks Save	The title bar reads report.txt
5	Text: “I saved the note as report.txt.”	Prints the answer, ends the loop

Each turn is one request. Turns 2 to 5 carry the earlier steps as text and only the latest two screenshots.

How do I end the run and get a final answer?

Count the turns in your loop. Two things end a run:

• The model answers in text. With tool_choice auto, a text reply with no tool call means it considers the task done or impossible. Check the screen yourself before you trust it.
• Your step limit. Stop after a fixed number of turns (20 in the samples) so a confused run cannot go on forever. Then make one more request with tool_choice: "none" and the whole conversation, as the samples do.

A request with tool_choice: "none" succeeds even though the history is full of tool calls. ZenithAI shows the earlier calls and results to the model as plain text and asks it to answer in prose from what the steps showed, without calling tools and without inventing links or details. Your own message list is not changed. Send the tools again or leave them out, both work. The answer is a normal message. Treat it as a summary to check, not as proof: a small model can still get a detail wrong. See what tool_choice none does.

Related

• Computer use overview: how the loop works and which way to choose
• OpenAI’s computer tool: computer_call items in pixels, on Genius
• Errors, safety and limits
• Computer tool reference and truncation auto
• Function calling and tools: tool_choice, strict mode and the Agents SDK
• Models: which tiers have vision and tools, and their limits
• Responses API guide and Chat Completions images

Common questions

Why build a computer-use agent with function tools instead of the computer tool?

Function tools work on /v1/chat/completions and /v1/responses, on Smart and Genius, and let you define actions of your own. The built-in computer tool is on /v1/responses and Genius only.

What coordinates does the model return in my own click tool?

Positions on a 0 to 1000 grid of the screenshot it was sent, whatever the tool description says. Convert with pixel_x = x * screenshot_width / 1000 and pixel_y = y * screenshot_height / 1000, then scale to your screen if you resized the screenshot.

How many screenshots can I send in one request?

At most 8 images per request, 10 MiB each and 20 MiB decoded in total, and every image also uses context. Keep the last two or three screenshots, or send truncation auto on /v1/responses, and scale screenshots to about 1280 pixels wide.