Agent example: check site videos
The job: a supervisor records a short walk-through of a site, a warehouse aisle or a shop floor. The agent watches it, notes what it sees, and returns a checklist: helmets, blocked exits, spills, loose cables. Your code then raises a ticket for each problem it finds.
Tools used: video reading on POST /chat, then structured JSON on /v1/chat/completions.
How ZenithAI watches a video
Section titled “How ZenithAI watches a video”- It watches the whole video. Up to a minute long, it takes one frame every second. Longer videos, up to 5 minutes, get one frame every two seconds, from start to end.
- Every frame carries its time, so you can ask when something happens, such as “When does the forklift first come in?” or “What happens at the end?”.
- It turns speech in the soundtrack into text, when there is a soundtrack.
- It reads MP4, MOV, M4V, MKV, AVI and WebM files. Inside them, H.264 and H.265 (HEVC) work, so phone and most CCTV videos go in as they are. VP8, VP9, MPEG-4 and Motion JPEG work too.
- A video can be up to 100 MB and 5 minutes long.
- Send one video per message. A message with a video can also carry up to 8 images.
The code
Section titled “The code”The agent works in two steps. First it asks ZenithAI to describe the video in plain words. Then it turns that description into checklist JSON your code can trust.
import base64, osfrom pathlib import Pathfrom typing import Literalimport requestsfrom openai import OpenAIfrom pydantic import BaseModel, Field
BASE = "https://dummydomain"KEY = os.environ["API_KEY"]client = OpenAI(base_url=BASE + "/v1", api_key=KEY)
def watch(clip: Path) -> str: r = requests.post(f"{BASE}/chat", headers={"Authorization": "Bearer " + KEY}, timeout=900, json={ "message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you " "note. Note people and whether they wear helmets and safety vests, exits and whether they " "are clear, spills, loose cables, and anything said about safety. Say clearly when " "something is not visible.", "attachments": [{"name": clip.name, "data": base64.b64encode(clip.read_bytes()).decode()}], "stream": False, }) r.raise_for_status() return r.json()["reply"]
class Check(BaseModel): result: Literal["safe", "problem", "not_visible"] evidence: str at: str = Field(description="mm:ss when the problem is first seen, or 'none'")
class Checklist(BaseModel): helmets_worn: Check safety_vests_worn: Check exits_clear: Check floor_free_of_spills: Check cables_secured: Check summary: str
def to_checklist(description: str) -> Checklist: reply = client.chat.completions.create( model="Fast", messages=[{"role": "system", "content": "Fill the safety checklist only from the description. " "Use problem when the description shows something unsafe, and not_visible when it " "does not show it."}, {"role": "user", "content": description}], response_format={"type": "json_schema", "json_schema": { "name": "Checklist", "schema": Checklist.model_json_schema(), "strict": True}}, ) return Checklist.model_validate_json(reply.choices[0].message.content)
checklist = to_checklist(watch(Path("site-walk.mp4")))for name in ["helmets_worn", "safety_vests_worn", "exits_clear", "floor_free_of_spills", "cables_secured"]: check = getattr(checklist, name) if check.result == "problem": print("Raise a ticket:", name, "at", check.at, "|", check.evidence)print(checklist.summary)See it run
Section titled “See it run”A real run of the code above, on made-up sample files. Press Replay to watch the steps in order.
What went in
What the agent did
POST /chat5.1 sSent site-walk.mp4 to ZenithAI, which watched and listened to it and answered.
Request and response
Request { "message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you note. Note people and whether they wear helmets and safety vests, exits and whether they are clear, spills, loose cables, and anything said about safety. Say clearly when something is not visible.", "attachments": [ { "name": "site-walk.mp4", "data": "<file, 186,234 bytes>" } ], "stream": false }Response 200 { "attempted_route": "general", "current_route": "general", "finish_reason": "stop", "max_tokens": 4096, "num_predict": 4096, "reply": "Based on the video transcript and visual walk-through, here is the timeline of observations:\n\n* **00:00 – 00:12**: A worker is visible standing near a rack. They are wearing an orange **safety vest** but are **not wearing a helmet**.\n* **00:00 – 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by several cardboard boxes.\n* **00:00 – 00:12**: The floor is dry and clean. No spills or loose cables are visible.\n* **Audio Note**: The speaker identifies the location as **aisle 3** of the warehouse and explicitly mentions that the fire exit is blocked by cardboard boxes and the worker is not wearing a helmet.", "route_disagreement": false, "session_id": "2680bf41-f0cf-48bc-8146-a20d2284bab7", "shadow_confidence": 0.7, "shadow_evidence_status": "n/a", "shadow_reason": "no tool route necessary; general knowledge", "shadow_route": "general" }POST /v1/chat/completionsFast1.9 sFast returned JSON that matches the Checklist schema.
Request and response
Request { "messages": [ { "role": "system", "content": "Fill the safety checklist only from the description. Use problem when the description shows something unsafe, and not_visible when it does not show it." }, { "role": "user", "content": "Based on the video transcript and visual walk-through, here is the timeline of observations:\n\n* **00:00 – 00:12**: A worker is visible standing near a rack. They are wearing an orange **safety vest** but are **not wearing a helmet**.\n* **00:00 – 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by several cardboard boxes.\n* **00:00 – 00:12**: The floor is dry and clean. No spills or loose cables are visible.\n* **Audio Note**: The speaker identifies the location as **aisle 3** of the warehouse and explicitly mentions that the fire exit is blocked by cardboard boxes and the worker is not wearing a helmet." } ], "response_format": { "type": "json_schema", "json_schema": { "name": "Checklist", "schema": { "$defs": { "Check": { "properties": { "result": { "enum": [ "safe", "problem", "not_visible" ], "title": "Result", "type": "string" }, "evidence": { "title": "Evidence", "type": "string" }, "at": { "description": "mm:ss when the problem is first seen, or 'none'", "title": "At", "type": "string" } }, "required": [ "result", "evidence", "at" ], "title": "Check", "type": "object" } }, "properties": { "helmets_worn": { "$ref": "#/$defs/Check" }, "safety_vests_worn": { "$ref": "#/$defs/Check" }, "exits_clear": { "$ref": "#/$defs/Check" }, "floor_free_of_spills": { "$ref": "#/$defs/Check" }, "cables_secured": { "$ref": "#/$defs/Check" }, "summary": { "title": "Summary", "type": "string" } }, "required": [ "helmets_worn", "safety_vests_worn", "exits_clear", "floor_free_of_spills", "cables_secured", "summary" ], "title": "Checklist", "type": "object" }, "strict": true } } }Response 200 { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "{\"helmets_worn\":{\"result\":\"problem\",\"evidence\":\"The worker is not wearing a helmet.\",\"at\":\"00:00\"},\"safety_vests_worn\":{\"result\":\"safe\",\"evidence\":\"The worker is wearing an orange safety vest.\",\"at\":\"none\"},\"exits_clear\":{\"result\":\"problem\",\"evidence\":\"A Fire Exit is blocked by several cardboard boxes.\",\"at\":\"00:00\"},\"floor_free_of_spills\":{\"result\":\"safe\",\"evidence\":\"The floor is dry and clean.\",\"at\":\"none\"},\"cables_secured\":{\"result\":\"not_visible\",\"evidence\":\"No cables are mentioned or seen in the description.\",\"at\":\"none\"},\"summary\":\"The worker is not wearing a helmet and the fire exit is blocked by cardboard boxes.\"}", "role": "assistant" } } ], "object": "chat.completion", "usage": { "completion_tokens": 167, "prompt_tokens": 485, "total_tokens": 652 } }
What the program printed
Raise a ticket: helmets_worn at 00:00 | The worker is not wearing a helmet. Raise a ticket: exits_clear at 00:00 | A Fire Exit is blocked by several cardboard boxes. The worker is not wearing a helmet and the fire exit is blocked by cardboard boxes.
What went in
What the agent did
POST /chat5.4 sSent site-walk.mov to ZenithAI, which watched and listened to it and answered.
Request and response
Request { "message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you note. Note people and whether they wear helmets and safety vests, exits and whether they are clear, spills, loose cables, and anything said about safety. Say clearly when something is not visible.", "attachments": [ { "name": "site-walk.mov", "data": "<file, 186,903 bytes>" } ], "stream": false }Response 200 { "attempted_route": "general", "current_route": "general", "finish_reason": "stop", "max_tokens": 4096, "num_predict": 4096, "reply": "Based on the video and transcript provided, here is the chronological description of the site walk-through:\n\n* **00:00 - 00:12**: A worker is visible standing near a storage rack. The worker is wearing an **orange safety vest** but is **not wearing a helmet**.\n* **00:00 - 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by a stack of cardboard boxes.\n* **00:00 - 00:12**: The floor appears **dry and clean**.\n* **00:00 - 00:12**: There are **no loose cables** visible in the frame.\n* **Audio Transcript (Safety notes)**: The narrator identifies the location as **aisle 3** of the warehouse and explicitly mentions that the **fire exit is blocked** by cardboa ...", "route_disagreement": false, "session_id": "a5d7a765-c1b0-4c8d-b289-116754f4806a", "shadow_confidence": 0.7, "shadow_evidence_status": "n/a", "shadow_reason": "no tool route necessary; general knowledge", "shadow_route": "general" }POST /v1/chat/completionsFast3.3 sFast returned JSON that matches the Checklist schema.
Request and response
Request { "messages": [ { "role": "system", "content": "Fill the safety checklist only from the description. Use problem when the description shows something unsafe, and not_visible when it does not show it." }, { "role": "user", "content": "Based on the video and transcript provided, here is the chronological description of the site walk-through:\n\n* **00:00 - 00:12**: A worker is visible standing near a storage rack. The worker is wearing an **orange safety vest** but is **not wearing a helmet**.\n* **00:00 - 00:12**: A **Fire Exit** is visible at the end of the aisle. It is **blocked** by a stack of cardboard boxes.\n* **00:00 - 00:12**: The floor appears **dry and clean**.\n* **00:00 - 00:12**: There are **no loose cables** visible in the frame.\n* **Audio Transcript (Safety notes)**: The narrator identifies the location as **aisle 3** of the warehouse and explicitly mentions that the **fire exit is blocked** by cardboa ..." } ], "response_format": { "type": "json_schema", "json_schema": { "name": "Checklist", "schema": { "$defs": { "Check": { "properties": { "result": { "enum": [ "safe", "problem", "not_visible" ], "title": "Result", "type": "string" }, "evidence": { "title": "Evidence", "type": "string" }, "at": { "description": "mm:ss when the problem is first seen, or 'none'", "title": "At", "type": "string" } }, "required": [ "result", "evidence", "at" ], "title": "Check", "type": "object" } }, "properties": { "helmets_worn": { "$ref": "#/$defs/Check" }, "safety_vests_worn": { "$ref": "#/$defs/Check" }, "exits_clear": { "$ref": "#/$defs/Check" }, "floor_free_of_spills": { "$ref": "#/$defs/Check" }, "cables_secured": { "$ref": "#/$defs/Check" }, "summary": { "title": "Summary", "type": "string" } }, "required": [ "helmets_worn", "safety_vests_worn", "exits_clear", "floor_free_of_spills", "cables_secured", "summary" ], "title": "Checklist", "type": "object" }, "strict": true } } }Response 200 { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "{\n \"helmets_worn\": {\n \"result\": \"problem\",\n \"evidence\": \"The worker is visible standing near a storage rack but is not wearing a helmet.\",\n \"at\": \"00:00\"\n },\n \"safety_vests_worn\": {\n \"result\": \"safe\",\n \"evidence\": \"The worker is wearing an orange safety vest.\",\n \"at\": \"none\"\n },\n \"exits_clear\": {\n \"result\": \"problem\",\n \"evidence\": \"A Fire Exit is visible at the end of the aisle. It is blocked by a stack of cardboard boxes.\",\n \"at\": \"00:00\"\n },\n \"floor_free_of_spills\": {\n \"result\": \"safe\",\n \"evidence\": \"The floor appears dry and clean.\",\n \"at\": \"none\"\n },\n \"cables_secured\": {\n \"result\": \"not_visible\",\n \"evidence\": \"There are no loose cables v ...", "role": "assistant" } } ], "object": "chat.completion", "usage": { "completion_tokens": 301, "prompt_tokens": 520, "total_tokens": 821 } }
What the program printed
Raise a ticket: helmets_worn at 00:00 | The worker is visible standing near a storage rack but is not wearing a helmet. Raise a ticket: exits_clear at 00:00 | A Fire Exit is visible at the end of the aisle. It is blocked by a stack of cardboard boxes. The site walk-through identified two primary safety issues: a worker is not wearing a helmet, and a fire exit is obstructed by cardboard boxes. The floor is clean, and no cable hazards were observed.
What went in
What the agent did
POST /chat5.1 sSent site-walk.webm to ZenithAI, which watched and listened to it and answered.
Request and response
Request { "message": "Describe this site walk-through in time order, with the time (mm:ss) of each thing you note. Note people and whether they wear helmets and safety vests, exits and whether they are clear, spills, loose cables, and anything said about safety. Say clearly when something is not visible.", "attachments": [ { "name": "site-walk.webm", "data": "<file, 180,981 bytes>" } ], "stream": false }Response 200 { "attempted_route": "general", "current_route": "general", "finish_reason": "stop", "max_tokens": 4096, "num_predict": 4096, "reply": "Based on the site walk-through video provided, here is the sequence of observations:\n\n* **00:00 - 00:12**: A worker is visible wearing an **orange safety vest** but **no helmet**. \n* **00:00 - 00:12**: A **fire exit** is visible at the end of the aisle, but it is **blocked** by several cardboard boxes.\n* **00:00 - 00:12**: No spills or loose cables are visible. The floor appears dry and clean.\n* **00:00 - 00:12 (Audio)**: The speaker notes that this is aisle 3 of the warehouse, identifies the blocked fire exit, mentions the worker is not wearing a helmet, and confirms the floor is dry and clean.", "route_disagreement": false, "session_id": "bb7ada45-e4a4-465a-8ce2-2e9ee919faac", "shadow_confidence": 0.7, "shadow_evidence_status": "n/a", "shadow_reason": "no tool route necessary; general knowledge", "shadow_route": "general" }POST /v1/chat/completionsFast1.9 sFast returned JSON that matches the Checklist schema.
Request and response
Request { "messages": [ { "role": "system", "content": "Fill the safety checklist only from the description. Use problem when the description shows something unsafe, and not_visible when it does not show it." }, { "role": "user", "content": "Based on the site walk-through video provided, here is the sequence of observations:\n\n* **00:00 - 00:12**: A worker is visible wearing an **orange safety vest** but **no helmet**. \n* **00:00 - 00:12**: A **fire exit** is visible at the end of the aisle, but it is **blocked** by several cardboard boxes.\n* **00:00 - 00:12**: No spills or loose cables are visible. The floor appears dry and clean.\n* **00:00 - 00:12 (Audio)**: The speaker notes that this is aisle 3 of the warehouse, identifies the blocked fire exit, mentions the worker is not wearing a helmet, and confirms the floor is dry and clean." } ], "response_format": { "type": "json_schema", "json_schema": { "name": "Checklist", "schema": { "$defs": { "Check": { "properties": { "result": { "enum": [ "safe", "problem", "not_visible" ], "title": "Result", "type": "string" }, "evidence": { "title": "Evidence", "type": "string" }, "at": { "description": "mm:ss when the problem is first seen, or 'none'", "title": "At", "type": "string" } }, "required": [ "result", "evidence", "at" ], "title": "Check", "type": "object" } }, "properties": { "helmets_worn": { "$ref": "#/$defs/Check" }, "safety_vests_worn": { "$ref": "#/$defs/Check" }, "exits_clear": { "$ref": "#/$defs/Check" }, "floor_free_of_spills": { "$ref": "#/$defs/Check" }, "cables_secured": { "$ref": "#/$defs/Check" }, "summary": { "title": "Summary", "type": "string" } }, "required": [ "helmets_worn", "safety_vests_worn", "exits_clear", "floor_free_of_spills", "cables_secured", "summary" ], "title": "Checklist", "type": "object" }, "strict": true } } }Response 200 { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "{\"helmets_worn\":{\"result\":\"problem\",\"evidence\":\"worker is visible without a helmet\",\"at\":\"00:00\"},\"safety_vests_worn\":{\"result\":\"safe\",\"evidence\":\"worker is wearing an orange safety vest\",\"at\":\"none\"},\"exits_clear\":{\"result\":\"problem\",\"evidence\":\"fire exit is blocked by several cardboard boxes\",\"at\":\"00:00\"},\"floor_free_of_spills\":{\"result\":\"safe\",\"evidence\":\"no spills are visible; floor appears dry and clean\",\"at\":\"none\"},\"cables_secured\":{\"result\":\"safe\",\"evidence\":\"no loose cables are visible\",\"at\":\"none\"},\"summary\":\"The site walk-through reveals safety violations regarding missing PPE (helmet) and a blocked fire exit, while the floor remains clear of spills and hazards.\"}", "role": "assistant" } } ], "object": "chat.completion", "usage": { "completion_tokens": 170, "prompt_tokens": 490, "total_tokens": 660 } }
What the program printed
Raise a ticket: helmets_worn at 00:00 | worker is visible without a helmet Raise a ticket: exits_clear at 00:00 | fire exit is blocked by several cardboard boxes The site walk-through reveals safety violations regarding missing PPE (helmet) and a blocked fire exit, while the floor remains clear of spills and hazards.
How it works
Section titled “How it works”- See, then sort.
/chatwatches and listens. A second, cheaper call on Fast turns the description into fixed checklist values. - Every finding has a time. The ticket says when the problem first shows, such as at 01:36, so the person fixing it can jump straight to that moment in the video.
- “Not visible” is an allowed answer. The agent doesn’t have to guess about things the camera missed.
- Your code raises the tickets. The model reports; your program decides who is told.
When a video is refused
Section titled “When a video is refused”POST /chat answers 400 with a fixed error_code, so your code can tell the person what to do:
error_code |
What it means | What to do |
|---|---|---|
video_too_large |
The file is over 100 MB. | Cut it into shorter clips, or record at a lower quality. |
video_too_long |
The video is over 5 minutes. | Cut it into clips of a minute or two. |
video_unreadable |
The file is damaged or not really a video. | Check that it plays on your computer, then send it again. |
video_unavailable |
This server can’t read videos at the moment. | Ask your ZenithAI administrator to check video support. |
Where else this works
Section titled “Where else this works”Checking that a product demo video shows the right steps, reviewing a short training clip, describing CCTV clips for an incident report, or checking that a delivery video shows the seal intact.
Common questions
How long a video can ZenithAI watch?
A video can be up to 5 minutes and 100 MB, and ZenithAI watches all of it. It looks at one frame every second in a video up to a minute long, and one frame every two seconds in a longer one. For a recording over 5 minutes, cut it into clips in your own code and send them one by one.
Can ZenithAI tell me when something happens in a video?
Yes. Every frame ZenithAI looks at carries its time, so you can ask when something first appears or what happens at the end, and get an answer like 01:36. In a video over a minute the frames are two seconds apart, so a time can be off by a second or two.
Can ZenithAI read a video recorded on a phone?
Yes. Send the file just as the phone saved it. Android phones usually save H.264 MP4 and iPhones save H.265 MOV, and ZenithAI reads both. There is no need to convert the video first.
Does ZenithAI hear what people say in a video?
Yes, when the video has a soundtrack. The speech is turned into text and used with the frames, so an answer can cover both what was shown and what was said.
