Agent example: match bills to the ledger

The job: your data-entry team types bills into the accounts system and attaches the scan. Before payment, the agent opens each scan, reads the bill number and total, and checks them against what was typed. Matches go ahead, mismatches and unreadable scans go to a person.

Tools used: a function that returns an image on /v1/responses, with a JSON schema for the final answer. The comparison is done by your own code.

The code

import base64, json
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="https://dummydomain/v1", api_key="YOUR_API_KEY")

# Your accounts system: bills entered by the data-entry team, with a link to each scan.
LEDGER = {
    "B-17": {"vendor": "Sharma Traders", "bill_number": "INV-2026-0412", "amount_inr": 118000,
             "scan": "inbox/bill-001.png"},
    "B-18": {"vendor": "Sharma Traders", "bill_number": "INV-2026-0412", "amount_inr": 118000,
             "scan": "inbox/bill-002-blurred.png"},
}

TOOLS = [{
    "type": "function",
    "name": "get_bill_scan",
    "description": "Return the scanned paper bill for a ledger entry, as an image",
    "parameters": {"type": "object", "properties": {"entry_id": {"type": "string"}},
                   "required": ["entry_id"], "additionalProperties": False},
    "strict": True,
}]

VERDICT = {
    "type": "object",
    "properties": {
        "scan_readable": {"type": "boolean"},
        "bill_number_on_scan": {"type": ["string", "null"]},
        "total_on_scan_inr": {"type": ["number", "null"]},
    },
    "required": ["scan_readable", "bill_number_on_scan", "total_on_scan_inr"],
    "additionalProperties": False,
}

def get_bill_scan(entry_id):
    data = base64.b64encode(Path(LEDGER[entry_id]["scan"]).read_bytes()).decode()
    return [{"type": "input_text", "text": f"Scan for ledger entry {entry_id}:"},
            {"type": "input_image", "image_url": f"data:image/png;base64,{data}"}]

def read_scan(entry_id):
    items = [{"role": "user", "content": (
        f"Fetch the scan for ledger entry {entry_id} and read the bill number and the total payable "
        "exactly as printed. If the scan is blurred or cut off, set scan_readable to false and use null.")}]
    fmt = {"format": {"type": "json_schema", "name": "scan", "schema": VERDICT, "strict": True}}
    response = client.responses.create(model="Fast", input=items, tools=TOOLS, text=fmt)
    for _ in range(3):
        calls = [item for item in response.output if item.type == "function_call"]
        if not calls:
            break
        items += response.output
        for call in calls:
            items.append({"type": "function_call_output", "call_id": call.call_id,
                          "output": get_bill_scan(**json.loads(call.arguments))})
        response = client.responses.create(model="Fast", input=items, tools=TOOLS, text=fmt)
    return json.loads(response.output_text)

for entry_id, entry in LEDGER.items():
    scan = read_scan(entry_id)
    if not scan["scan_readable"]:
        print("REVIEW", entry_id, "scan not readable, ask for a clearer copy")
    elif scan["bill_number_on_scan"] != entry["bill_number"] or scan["total_on_scan_inr"] != entry["amount_inr"]:
        print("MISMATCH", entry_id, "ledger:", entry["bill_number"], entry["amount_inr"], "scan:", scan)
    else:
        print("MATCH ", entry_id, entry["bill_number"], entry["amount_inr"])

What it prints

MATCH  B-17 INV-2026-0412 118000
REVIEW B-18 scan not readable, ask for a clearer copy

How it works

The model asks for the scan. It calls get_bill_scan with the entry id. Your code fetches the image from wherever it lives (file share, document system, object store) and returns it as an input_image part.
The image goes back right after the call. The loop sends the function_call items and their outputs together, which is the order the API expects.
The answer is JSON. text.format holds a schema, so output_text always has scan_readable, bill_number_on_scan and total_on_scan_inr.
Your code decides. The model only reads the paper. Matching against the ledger, and the choice to pay or hold, stay in your program.

Where else this works

Delivery challans against goods received notes, signed agreements against the CRM, meter photos against billed units, ID cards against KYC records, site photos against a work order. Any check where the truth is on paper and the record is in a system.

Common questions

Can a ZenithAI agent look at images from our own systems?

Yes. On /v1/responses your function can return images as well as text. The agent asks for a scan, a photo or a chart, your code fetches it from your system, and the model reads it before it answers.

Why does the agent return JSON instead of a sentence?

The request sets text.format to a JSON schema, so the final answer always has the fields your code compares. Tools and a schema work together in one request.