Running a Vision Model 100% in the Browser: What I Learned Shipping PassportPhotoStudio

The usual pitch for vision models in 2026 is that there is nothing to build anymore. A vision-language model is a language model that can also see: you hand it an image alongside your text prompt, it answers in words, and it rides the same chat endpoint you already use. Document parsing, chart reading, UI agents — all of it collapses into one more field in the request. That is genuinely true, and for most teams it is the right default.

I went the other direction. When I built PassportPhotoStudio — a browser-only passport photo editor that removes the background from a headshot and checks it against a country's document spec — I did not want the image to leave the user's device at all. A passport photo is about as personal as an image gets. So instead of posting pixels to a hosted VLM, I run a real vision model, RMBG-1.4, one hundred percent client-side: no server round-trip for inference, no upload, no API bill per photo. This post is the honest version of what that costs and what it buys.

Two ways to 'do vision', and why I picked the harder one

It is worth being precise, because the phrase 'vision model' hides two very different things. A vision-language model reasons over pixels and words together — you ask it what an image means and it replies in language. A vision-only model like RMBG-1.4 does one narrow perceptual job and emits pixels or a mask, not prose. My app is the second kind: given a photo, it produces a per-pixel foreground mask that says 'this is the person, that is the background. ' No language, no prompt, just a cutout.

That distinction matters for where the model runs. A frontier VLM is far too large to ship to a browser, so it lives behind an API. A focused segmentation model is small enough to download and execute on the user's own GPU — which is exactly what makes a privacy-first, offline-capable app possible. The tradeoff is that everything the cloud normally handles for you (the runtime, the fallbacks, the memory budget) becomes your problem, in the browser, on hardware you do not control.

A hosted VLM asks 'what does this image mean? ' My in-browser model asks 'which pixels are the person? ' — and answering that on a stranger's laptop is a systems problem, not a prompt problem.

The download budget decides everything

When the model runs in the cloud, its size is invisible to the user. When it runs in the browser, the model IS the app's first-load cost — every visitor pays the download before anything works. The full-precision RMBG-1.4 weights are around 168MB according to the worker's own notes. That is a non-starter for a web page. So the pipeline loads the model at dtype q8 (int8 quantization), and the code comment is blunt about why: it calls this 'the single biggest first-load win. '

The numbers back that up. The quantized ONNX file on disk is 44,403,226 bytes — I have seen it rounded to both ~42MB and ~44MB in the repo depending on whether you go by `du` or the comment, and honestly both describe the same file. Either way it is roughly a quarter of the fp32 size, which is the difference between a web app that loads and one nobody waits for.

I also stopped trusting a CDN for it. Early on the model was fetched from the HuggingFace Hub at runtime, which meant production availability depended on someone else's download staying fast and up. In commit 81fe7bd I committed the quantized model same-origin under /models and serve it myself; the runtime-Hub path survives only as a dev fallback. A model your users can't download is a model your app doesn't have.

WebGPU when you can, WASM when you must, MediaPipe when all else fails

You cannot assume the browser has a GPU. So inference runs on three tiers, in order: RMBG-1.4 on WebGPU, RMBG-1.4 on WASM/CPU if there is no `navigator. gpu` or the GPU path throws, and — if the primary model won't load at all — a completely separate, always-available MediaPipe selfie-segmenter. The user never sees this negotiation; they just get a cutout. The worker builds the pipeline for a device, tries WebGPU first, and quietly rebuilds on WASM if it fails:

const build = (device: "webgpu" | "wasm") =>
  pipeline("image-segmentation", model, {
    device,
    dtype: "q8",
    progress_callback: (info: any) => postProgress(id, info)
  });

const hasWebGPU = typeof navigator !== "undefined" && "gpu" in (navigator as any);
if (hasWebGPU) {
  try {
    segmenter = await build("webgpu");
    self.postMessage({ type: "ready", id, device: "webgpu" });
    return;
  } catch (err) {
    self.postMessage({ type: "progress", id, status: "webgpu-unavailable" });
    console.warn("WebGPU segmentation unavailable, falling back to WASM/CPU", err);
  }
}
segmenter = await build("wasm");

On my machine — and I want to be clear these are numbers from my own commit messages, verified against the real worker but not a controlled benchmark across devices — the WebGPU model load measured 233ms and a single segmentation pass took about 6.5 seconds, producing a clean cutout that split roughly 44% background to 55% foreground. Your mileage will vary wildly by GPU. The point is not the exact figure; it is that 6.5 seconds is long enough that where you spend it becomes an architecture decision, not a footnote.

The two bugs that cost me the most seconds

Running a model in the browser has failure modes the cloud simply hides. Two of mine (both fixed in commit 8589c86) are the kind you only hit once you leave the tutorial.

The first was a mask-format bug. My worker emitted an RGBA buffer, but the main thread read it as a single-channel grayscale mask. So a white pixel `[255,255,255,alpha]` got misread, and roughly 75% of the background stayed opaque — the cutout barely cut anything out. The fix was to emit a single-channel grayscale mask sized to the original image, and read it as exactly that. Channel-layout mismatches between a Web Worker and the main thread are invisible until the output is visibly wrong.

The second was pure, silent slowness. Handing transformers. js a large hand-built four-channel RawImage made it run its own pure-JS resize — which took tens of seconds and looked like a hang. RMBG-1.4 wants a fixed 1024x1024 input, so the fix is to resize the frame to exactly that size with OffscreenCanvas up front, which turns the library's internal resize into a no-op:

// Handing transformers.js a large hand-built 4-channel RawImage makes it run a
// pure-JS resize/preprocess that takes tens of seconds - that was the hang.
// With the input already at 1024x1024 the processor's own resize is a no-op.
const inCanvas = new OffscreenCanvas(MODEL_INPUT_SIZE, MODEL_INPUT_SIZE);
const inCtx = inCanvas.getContext("2d", { willReadFrequently: true })!;
inCtx.drawImage(srcCanvas, 0, 0, MODEL_INPUT_SIZE, MODEL_INPUT_SIZE);
const inputRgba = new Uint8ClampedArray(
  inCtx.getImageData(0, 0, MODEL_INPUT_SIZE, MODEL_INPUT_SIZE).data
);
const image = new RawImage(inputRgba, MODEL_INPUT_SIZE, MODEL_INPUT_SIZE, 4).rgb();

That one generalizes: if transformers. js feels mysteriously slow in the browser, check whether you are making the library resize your image in JavaScript instead of doing it on a canvas first.

Don't re-run a 6-second model for a slider

The subtlest lesson was about when NOT to invoke the model. In the editor a user drags sliders — background color, edge refine, crop — after the cutout is made. Originally each of those tweaks triggered a full segmentation pass, so adjusting the background color meant waiting ~6 seconds for the AI to re-segment an image it had already segmented. But the raw mask only depends on the source image and a few resolution/quality settings, not on what color you picked afterward.

So (commit a503405) I cache the mask, keyed by image + resolution + quality + threshold + engine, and clone it on read and write so refinement can never corrupt the cached copy. After that, moving a color slider reprocesses in about 2 seconds with zero segmentation calls, versus the ~6 seconds a full re-segment cost before. Only the interactive preview caches — the live camera and batch export still segment per frame and per item, because there the input genuinely changes every time.

None of the perception logic is the hard part here. The hard part is the plumbing around a model that is slow, large, and running on hardware you don't own.

What actually transfers to a hosted VLM

My app is vision-only, but a few of these lessons apply directly to the multimodal-API world everyone else is living in:

• Image size is your real cost. In the browser it is the download and the resize; in a hosted VLM it is the token count, since big or high-resolution images quietly become hundreds or thousands of tokens. Downscale to the smallest input that is still legible either way.
• Match the model's native input size on purpose. RMBG-1.4 wants 1024x1024; I resize to exactly that. VLMs have their own tiling thresholds — feeding them off-size images just wastes tokens or triggers extra tiles.
• Cache the expensive step. Whether it is a 6-second segmentation or a paid API call, if the output only depends on the input, don't recompute it every time the UI changes.
• Always have a fallback path. Mine is WebGPU -> WASM -> MediaPipe. Yours might be VLM -> classic OCR for dense documents. Either way, assume the smart path will sometimes be unavailable.
• Never claim precision the model can't back. My requirement checker is framed as guidance, not a legal guarantee, and facial retouching is disabled on purpose. A VLM that confidently reports a value it half-invented is the same failure in a different costume.

For most features, treating vision as one field in an API call is the correct, boring, fast answer. But if privacy or offline capability actually matters — as it does for a passport photo — running the model yourself is not exotic anymore. It is a 42MB download, a Web Worker, three fallback tiers, and a real respect for the fact that the model is slow and the hardware is a stranger's. That last part is the whole job.