Skip to content

Agent example: check site videos

The job: a supervisor records a short walk-through of a site, a warehouse aisle or a shop floor. The agent watches it, notes what it sees, and returns a checklist: helmets, blocked exits, spills, loose cables. Your code then raises a ticket for each problem it finds.

Tools used: video reading on POST /chat, then structured JSON on /v1/chat/completions.

  • It watches the whole video. Up to a minute long, it takes one frame every second. Longer videos, up to 5 minutes, get one frame every two seconds, from start to end.
  • Every frame carries its time, so you can ask when something happens, such as “When does the forklift first come in?” or “What happens at the end?”.
  • It turns speech in the soundtrack into text, when there is a soundtrack.
  • It reads MP4, MOV, M4V, MKV, AVI and WebM files. Inside them, H.264 and H.265 (HEVC) work, so phone and most CCTV videos go in as they are. VP8, VP9, MPEG-4 and Motion JPEG work too.
  • A video can be up to 100 MB and 5 minutes long.
  • Send one video per message. A message with a video can also carry up to 8 images.

The agent works in two steps. First it asks ZenithAI to describe the video in plain words. Then it turns that description into checklist JSON your code can trust.

import base64, os
from pathlib import Path
from typing import Literal
import requests
from openai import OpenAI
from pydantic import BaseModel, Field
BASE = "https://dummydomain"
KEY = os.environ["API_KEY"]
client = OpenAI(base_url=BASE + "/v1", api_key=KEY)
def watch(clip: Path) -> str:
r = requests.post(f"{BASE}/chat", headers={"Authorization": "Bearer " + KEY}, timeout=900, json={
"message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you "
"note. Note people and whether they wear helmets and safety vests, exits and whether they "
"are clear, spills, loose cables, and anything said about safety. Say clearly when "
"something is not visible.",
"attachments": [{"name": clip.name, "data": base64.b64encode(clip.read_bytes()).decode()}],
"stream": False,
})
r.raise_for_status()
return r.json()["reply"]
class Check(BaseModel):
result: Literal["safe", "problem", "not_visible"]
evidence: str
at: str = Field(description="mm:ss when the problem is first seen, or 'none'")
class Checklist(BaseModel):
helmets_worn: Check
safety_vests_worn: Check
exits_clear: Check
floor_free_of_spills: Check
cables_secured: Check
summary: str
def to_checklist(description: str) -> Checklist:
reply = client.chat.completions.create(
model="Fast",
messages=[{"role": "system", "content": "Fill the safety checklist only from the description. "
"Use problem when the description shows something unsafe, and not_visible when it "
"does not show it."},
{"role": "user", "content": description}],
response_format={"type": "json_schema", "json_schema": {
"name": "Checklist", "schema": Checklist.model_json_schema(), "strict": True}},
)
return Checklist.model_validate_json(reply.choices[0].message.content)
checklist = to_checklist(watch(Path("site-walk.mp4")))
for name in ["helmets_worn", "safety_vests_worn", "exits_clear", "floor_free_of_spills", "cables_secured"]:
check = getattr(checklist, name)
if check.result == "problem":
print("Raise a ticket:", name, "at", check.at, "|", check.evidence)
print(checklist.summary)

A real run of the code above, on made-up sample files. Press Replay to watch the steps in order.

Recorded on 07-10-2026 against a ZenithAI test server, running the code above. The times are real.

What went in

site-walk.mp4, a drawn warehouse aisle with a spoken note, saved the way an Android phone saves video (H.264)

What the agent did

  1. POST /chat5.1 s

    Sent site-walk.mp4 to ZenithAI, which watched and listened to it and answered.

    Request and response
    Request
    {
      "message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you note. Note people and whether they wear helmets and safety vests, exits and whether they are clear, spills, loose cables, and anything said about safety. Say clearly when something is not visible.",
      "attachments": [
        {
          "name": "site-walk.mp4",
          "data": "<file, 186,234 bytes>"
        }
      ],
      "stream": false
    }
    Response 200
    {
      "attempted_route": "general",
      "current_route": "general",
      "finish_reason": "stop",
      "max_tokens": 4096,
      "num_predict": 4096,
      "reply": "Based on the video transcript and visual walk-through, here is the timeline of observations:\n\n*   **00:00 – 00:12**: A worker is visible standing near a rack. They are wearing an orange **safety vest** but are **not wearing a helmet**.\n*   **00:00 – 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by several cardboard boxes.\n*   **00:00 – 00:12**: The floor is dry and clean. No spills or loose cables are visible.\n*   **Audio Note**: The speaker identifies the location as **aisle 3** of the warehouse and explicitly mentions that the fire exit is blocked by cardboard boxes and the worker is not wearing a helmet.",
      "route_disagreement": false,
      "session_id": "2680bf41-f0cf-48bc-8146-a20d2284bab7",
      "shadow_confidence": 0.7,
      "shadow_evidence_status": "n/a",
      "shadow_reason": "no tool route necessary; general knowledge",
      "shadow_route": "general"
    }
  2. POST /v1/chat/completionsFast1.9 s

    Fast returned JSON that matches the Checklist schema.

    Request and response
    Request
    {
      "messages": [
        {
          "role": "system",
          "content": "Fill the safety checklist only from the description. Use problem when the description shows something unsafe, and not_visible when it does not show it."
        },
        {
          "role": "user",
          "content": "Based on the video transcript and visual walk-through, here is the timeline of observations:\n\n*   **00:00 – 00:12**: A worker is visible standing near a rack. They are wearing an orange **safety vest** but are **not wearing a helmet**.\n*   **00:00 – 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by several cardboard boxes.\n*   **00:00 – 00:12**: The floor is dry and clean. No spills or loose cables are visible.\n*   **Audio Note**: The speaker identifies the location as **aisle 3** of the warehouse and explicitly mentions that the fire exit is blocked by cardboard boxes and the worker is not wearing a helmet."
        }
      ],
      "response_format": {
        "type": "json_schema",
        "json_schema": {
          "name": "Checklist",
          "schema": {
            "$defs": {
              "Check": {
                "properties": {
                  "result": {
                    "enum": [
                      "safe",
                      "problem",
                      "not_visible"
                    ],
                    "title": "Result",
                    "type": "string"
                  },
                  "evidence": {
                    "title": "Evidence",
                    "type": "string"
                  },
                  "at": {
                    "description": "mm:ss when the problem is first seen, or 'none'",
                    "title": "At",
                    "type": "string"
                  }
                },
                "required": [
                  "result",
                  "evidence",
                  "at"
                ],
                "title": "Check",
                "type": "object"
              }
            },
            "properties": {
              "helmets_worn": {
                "$ref": "#/$defs/Check"
              },
              "safety_vests_worn": {
                "$ref": "#/$defs/Check"
              },
              "exits_clear": {
                "$ref": "#/$defs/Check"
              },
              "floor_free_of_spills": {
                "$ref": "#/$defs/Check"
              },
              "cables_secured": {
                "$ref": "#/$defs/Check"
              },
              "summary": {
                "title": "Summary",
                "type": "string"
              }
            },
            "required": [
              "helmets_worn",
              "safety_vests_worn",
              "exits_clear",
              "floor_free_of_spills",
              "cables_secured",
              "summary"
            ],
            "title": "Checklist",
            "type": "object"
          },
          "strict": true
        }
      }
    }
    Response 200
    {
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
            "content": "{\"helmets_worn\":{\"result\":\"problem\",\"evidence\":\"The worker is not wearing a helmet.\",\"at\":\"00:00\"},\"safety_vests_worn\":{\"result\":\"safe\",\"evidence\":\"The worker is wearing an orange safety vest.\",\"at\":\"none\"},\"exits_clear\":{\"result\":\"problem\",\"evidence\":\"A Fire Exit is blocked by several cardboard boxes.\",\"at\":\"00:00\"},\"floor_free_of_spills\":{\"result\":\"safe\",\"evidence\":\"The floor is dry and clean.\",\"at\":\"none\"},\"cables_secured\":{\"result\":\"not_visible\",\"evidence\":\"No cables are mentioned or seen in the description.\",\"at\":\"none\"},\"summary\":\"The worker is not wearing a helmet and the fire exit is blocked by cardboard boxes.\"}",
            "role": "assistant"
          }
        }
      ],
      "object": "chat.completion",
      "usage": {
        "completion_tokens": 167,
        "prompt_tokens": 485,
        "total_tokens": 652
      }
    }

What the program printed

Raise a ticket: helmets_worn at 00:00 | The worker is not wearing a helmet.
Raise a ticket: exits_clear at 00:00 | A Fire Exit is blocked by several cardboard boxes.
The worker is not wearing a helmet and the fire exit is blocked by cardboard boxes.
  1. See, then sort. /chat watches and listens. A second, cheaper call on Fast turns the description into fixed checklist values.
  2. Every finding has a time. The ticket says when the problem first shows, such as at 01:36, so the person fixing it can jump straight to that moment in the video.
  3. “Not visible” is an allowed answer. The agent doesn’t have to guess about things the camera missed.
  4. Your code raises the tickets. The model reports; your program decides who is told.

POST /chat answers 400 with a fixed error_code, so your code can tell the person what to do:

error_code What it means What to do
video_too_large The file is over 100 MB. Cut it into shorter clips, or record at a lower quality.
video_too_long The video is over 5 minutes. Cut it into clips of a minute or two.
video_unreadable The file is damaged or not really a video. Check that it plays on your computer, then send it again.
video_unavailable This server can’t read videos at the moment. Ask your ZenithAI administrator to check video support.

Checking that a product demo video shows the right steps, reviewing a short training clip, describing CCTV clips for an incident report, or checking that a delivery video shows the seal intact.

Common questions

How long a video can ZenithAI watch?

A video can be up to 5 minutes and 100 MB, and ZenithAI watches all of it. It looks at one frame every second in a video up to a minute long, and one frame every two seconds in a longer one. For a recording over 5 minutes, cut it into clips in your own code and send them one by one.

Can ZenithAI tell me when something happens in a video?

Yes. Every frame ZenithAI looks at carries its time, so you can ask when something first appears or what happens at the end, and get an answer like 01:36. In a video over a minute the frames are two seconds apart, so a time can be off by a second or two.

Can ZenithAI read a video recorded on a phone?

Yes. Send the file just as the phone saved it. Android phones usually save H.264 MP4 and iPhones save H.265 MOV, and ZenithAI reads both. There is no need to convert the video first.

Does ZenithAI hear what people say in a video?

Yes, when the video has a soundtrack. The speech is turned into text and used with the frames, so an answer can cover both what was shown and what was said.