Aliases: Vision language model, VLM
Multimodal AI
Models that process and generate multiple data types — text, images, audio, video — in a single system rather than text alone.
What is multimodal AI?
A multimodal model accepts and/or produces more than text: images in (document understanding, screenshots, charts), audio in/out (voice agents), video in, images out. Frontier chat models are now natively multimodal — images are encoded into tokens and reasoned over in the same context window as text.
What it’s used for in production
The unglamorous winners: document extraction (invoices, forms, tables from PDFs — replacing brittle OCR pipelines), screenshot understanding (computer-use agents, UI testing), visual QA on product images and defect photos, and voice interfaces with speech-native models that skip the transcribe→LLM→synthesize chain and its compounding latency.
Cost reality
Images are billed as tokens — a detailed image often costs 1,000–2,000 input tokens, so vision at scale is priced like very long prompts. Video is that per frame sampled. Audio-native models bill per second/token at higher rates than text. The standard optimization: preprocess (crop, downscale, sample frames) before sending, and route to text-only models whenever the task doesn’t truly need pixels.
What people get wrong
- Using vision where structure exists. If you have the HTML/JSON, send that — parsing a screenshot of data you already hold in structured form is slower, costlier, and less accurate.
- Expecting pixel-perfect reading. Models still misread dense tables, small fonts, and precise coordinates; validate extracted numbers with guardrails.
- Ignoring prompt injection via images. Instructions embedded in images work as attacks too — screenshots and user uploads are untrusted input; see prompt injection.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.