Building With Vision-Language Models: Multimodal Apps in Production

A vision-language model (VLM) is a language model that can also see. You hand it an image — a receipt, a screenshot, a chart, a photo — alongside your text prompt, and it answers in words. The newest open models ship with this built in, and the big APIs expose it through the same chat endpoint you already use. For a developer, multimodal stopped being a separate stack and became one more field in the request.

That makes a whole class of features suddenly cheap to build: pulling fields off an invoice, describing a UI for an agent to click, reading a graph, checking whether a photo matches a description. But VLMs fail in specific, predictable ways, and the difference between a demo and a product is knowing exactly where.

How a VLM actually sees

Under the hood, a VLM does not bolt a separate vision system onto a chatbot. It turns the image into tokens the language model can attend to, right next to the words. The pipeline is short:

• An image encoder (usually a vision transformer) turns the picture into a grid of feature vectors.
• A small projection layer maps those vectors into the language model's token space, so an image patch looks like a word to the rest of the network.
• Those image tokens are concatenated with the text tokens and read by the same attention layers — the model reasons over pixels and words together.
• Big or high-resolution images get split into tiles, each producing its own tokens, which is why images can quietly cost hundreds or thousands of tokens each.

What they are good at, and what they are not

VLMs are excellent at the gist: what is in this image, what does this document say, what is this screen showing, summarize this chart's trend. For reading text in images they are often better than classic OCR because they use context — they know a total belongs near the word total.

They are unreliable at precision. Exact pixel coordinates, counting many similar objects, tiny dense text, and fine spatial relationships are where they slip — and, worse, they slip confidently, describing detail that is not there. Treat a VLM as a smart reader of images, not a measuring instrument.

Calling one in practice

The request is barely different from a text call — you add an image part and, crucially, ask for structured output so you can trust the result downstream:

# Extract structured fields from a document image, not free-form prose.
from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},   # force machine-readable output
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Return JSON: vendor, date, total. Use null if unsure."},
            {"type": "image_url",
             "image_url": {"url": "data:image/jpeg;base64," + img_b64}},
        ],
    }],
)
data = json.loads(resp.choices[0].message.content)   # {"vendor": ..., "total": ...}

The null if unsure instruction matters more than it looks: it gives the model an exit instead of inventing a value, which is the single biggest source of bad multimodal output.

A vision-language model is a brilliant reader and an unreliable ruler. Ask it what an image means, not exactly where things are, and most of the failure modes disappear.

Patterns that work in production

• Always request structured output. A JSON schema with explicit null options turns confident hallucinations into honest gaps you can handle.
• Downscale and tile deliberately. Send the smallest image that is still legible; image tokens dominate your bill and your latency.
• Ground the answer. Ask the model to quote the text it read or point to a region, so a human or a check can verify it instead of trusting a summary.
• Keep a classic OCR fallback for dense, layout-heavy documents where exact characters matter more than understanding.
• Evaluate on real images, not clean samples. Glare, rotation, handwriting, and screenshots at odd resolutions are where accuracy actually lives or dies.

The bottom line

Vision-language models collapse a lot of old computer-vision plumbing into a single API call, and the open ones are now good enough to run yourself. The work has shifted from building pipelines to designing prompts and schemas that play to the model's strengths — understanding — while fencing off its weaknesses — precision and overconfidence.

Start with a narrow, high-value task: one document type, one screen, one kind of chart. Force structured output, measure on messy real inputs, and add a verification step for anything that has to be exact. Done that way, multimodal is one of the fastest features you can ship this year.