Features

Vision

Give your character eyes on the real (or virtual) world.

Most conversational characters are blind — they only know what you type or say. IAMX Vision changes that. Your character can look at whatever is in front of it, and that image becomes part of the conversation. Ask it what it sees, hold up an object, walk into frame, change the scene behind you — the character notices, describes, and reacts, all in the same natural back-and-forth it already uses for speech.

Under the hood, Vision grabs a single frame at the moment it matters — typically right when the user finishes speaking — and hands that frame to the language model alongside the rest of the conversation. The model reasons about the picture and the words together, so a question like "What do you think of this one?" can be answered by actually looking at the thing the visitor is holding. No extra prompt engineering, no separate flow: it is the same turn, now with sight.

Vision works with any Vision-capable language model — GPT, Claude, and Gemini all accept images. Ollama models with multimodal support work too, so you can run the whole loop locally if you need to.

How it works at a glance

Capture — At the right moment in the turn, IAMX pulls the current frame from your chosen camera source.
Encode — The frame is compressed to a compact image and attached to the outgoing message, invisibly to the user.
Reason — The language model receives the image plus the conversation so far and produces a reply that accounts for both.
Respond — The character speaks the answer with full lip sync, exactly as it would for any other reply.

 Camera source ──▶ frame ──▶ encode ──▶ ┐
                                         ├─▶  LLM  ──▶ spoken reply
 User speech ─────────────▶ transcript ─┘

Camera sources

Vision is deliberately flexible about where the picture comes from. You pick one of three sources depending on what you want the character to perceive. Every source ends up as the same kind of frame, so the reasoning side never changes — only the eyes do.

Source	What the character sees	Typical use
Its own viewpoint	A scene capture rendered from the character's own perspective — literally what it would see if it had eyes in its head, looking out into the scene in front of it.	In-world NPCs, virtual guides, characters that should react to what is physically near them in the level.
The player's camera	The exact view the player is looking at right now. The character sees what you see.	First-person experiences, co-op "look at this" moments, tutorials where the character comments on wherever the player points the camera.
A render target you assign	Any texture you feed in — a live webcam feed, a media player playing back video, a security-camera stream, a mini-map, or anything else you can render to a target.	Kiosks and installations where the character watches a real-world camera, or apps that pipe in an external video source.

Its own viewpoint

The character carries a capture that renders the scene from where it is standing and which way it is facing. As the character moves or turns, its field of view moves with it. This is the most "alive" option: the character genuinely perceives its surroundings, so objects placed in the level, other actors, and environmental changes are all fair game for it to comment on.

The player's camera

Here the character borrows the player's eyes. Whatever is on screen for the player is what gets captured. This is ideal when the character is a companion or assistant rather than a physical presence in the world — it can talk about the thing the player is currently aiming at without needing its own body positioned to see it.

A render target you assign

The most open-ended option. You hand IAMX a render target and it reads frames from that. The most common setup for a physical kiosk is a webcam: point the camera at the visitor and the character can see the person, the product they are holding, or the document they present. Because it is just a render target, anything you can draw into one becomes the character's eyes.

A webcam feed and a media player are both just render targets. That means you can prototype a kiosk's vision behaviour by playing back a recorded video into a media player, then swap in the live camera for production — the character's logic never changes.

What it does with it

Capturing a frame is only half the story. The point of Vision is what happens next: the current view is given to the AI so it can describe, react to, or answer questions about what it sees, in the same conversational turn as everything else.

Describe

"What's in front of you?" — the character narrates the scene, lists objects, reads visible text, and summarizes what it perceives.

React

The character responds to changes on its own: someone walks up, an item appears, a light turns on. Vision gives it something to react to beyond words.

Answer

"Is this the right size?" or "What color is this?" — questions grounded in the image are answered by actually looking, not guessing.

Because the frame travels with the conversation, follow-up questions keep working. A visitor can hold up a box, ask "what is this?", then ask "and is it in stock?" — the character understood the object from the picture and can carry that understanding forward through the rest of the exchange.

The image is attached to the same message the user's words are attached to, so the model sees both together. That is why the character can resolve vague references like "this one" or "the red one" — the pronoun points at something in the frame.

Reading text and labels

Vision-capable models are good at reading text inside an image. That means the character can pick out a product name off a label, read a serial number, or make sense of a sign — no separate step required. For kiosks this is where Vision starts to feel magical: a visitor shows a package, and the character reads the model name straight off it.

Pairing Vision with the rest of the pipeline

Vision gets dramatically more useful when you combine it with Knowledge and Memory. Picture a retail kiosk: Vision recognizes the product a visitor is holding, the character looks that product up in its Knowledge bank, and answers detailed questions about it — price, features, compatibility — while Memory keeps track of everything the visitor has asked so far. The eyes tell it what, the knowledge base tells it everything about it, and memory keeps the conversation coherent.

The three features stack cleanly:

Vision + Knowledge — the character sees an item, then grounds its answer in your uploaded product catalogue or documentation instead of general world knowledge. See /docs/knowledge-bank.
Vision + Memory — what the character saw earlier in the session stays in context, so a visitor can put the product down and keep talking about it. See /docs/memory.
Vision + Actions — seeing something can lead to doing something: recognize a product and trigger a lookup, a form, or a hand-off. See /docs/actions.

Setup

Enable Vision — Turn on Vision for the character in the panel or on the IAMX component.
Pick a source — Choose its own viewpoint, the player's camera, or a render target.
Wire the source — For the render-target option, assign the texture (your webcam feed, media player, etc.). The other two need no wiring.
Choose a Vision-capable model — Make sure the character's language model supports images (GPT, Claude, and Gemini vision models all do).
Test it — Put something in view and ask the character what it sees.

Configuration reference

Setting	What it controls
Enable Vision	Master toggle. When off, no frame is ever captured or sent.
Camera source	Its own viewpoint, the player's camera, or a custom render target.
Render target	The texture used when the source is set to custom render target (webcam, media player, etc.).
Capture timing	When in the turn the frame is grabbed — typically at the end of the user's input, so the picture matches the question.

Performance and privacy

Vision only captures a frame when there is a reason to — it is not streaming a live video feed to the model every tick. A single compact still per relevant turn keeps bandwidth and cost low while still giving the model everything it needs to reason about the moment.

A camera pointed at real people is sensitive by nature. For public kiosks, be transparent that a camera is in use, keep the feed on your own hardware where you can, and only enable Vision on turns where the character genuinely needs to see. Captured frames are used to answer the current conversation, not retained as a video record.

One frame, on demand

No continuous streaming — a still is captured at the moment it matters, keeping cost and latency down.

Model-agnostic

Any Vision-capable model works, including local Ollama models if you want everything on-device.

Same turn, no glue

The image rides along with the user's message — no separate pipeline for you to build or maintain.

Where to go next

Quickstart — get a character talking, then switch Vision on.
Knowledge Bank — ground what the character sees in your own product data.
Memory — keep visual context alive across the whole conversation.
Blueprint API — drive Vision and react to what the character sees from Blueprint.
Try it live at iamx.live, or ask on the forum.