In-Browser Image Segmentation: How Client-Side Background Removal Actually Works
Every background-removal tool used to mean the same trade: upload your photo to someone's server, wait for a round trip, and hope their retention policy matches their marketing. Modern browsers have quietly removed that trade. This is a technical look at how client-side image segmentation works, why it became viable, and where it still struggles.
The model that changed things
The workhorse behind in-browser background removal is a class of lightweight segmentation models — U^2-Net derivatives and MobileNet-based variants — exported to ONNX and executed with ONNX Runtime Web (or TensorFlow.js) on WebAssembly. A quantized model lands around 40-80 MB, downloads once, and is cached by the browser afterwards.
The pipeline is short:
Decode the image to an offscreen canvas, downscale to the model's input size (typically 320x320).
Normalize pixel values to the model's expected range and run inference — a segmentation mask comes out as a grayscale alpha map.
Upscale the mask back to full resolution, feather the edges (a small blur on the alpha channel), and composite: original RGB + mask as alpha = cutout PNG.
Step 3 is pure canvas work. The only "AI" is step 2, and it runs on the user's CPU — or GPU, if WebGPU is available.
Why WebAssembly threads needed a header (the fragile part)
WASM inference is 2-6x faster with threads, and threads require the page to be cross-origin isolated — the COOP/COEP header pair. COEP: credentialless was the pragmatic choice for us because the strict variant (require-corp) breaks every third-party image and CDN script that does not send CORP headers.
The failure modes are invisible: a missing COEP header on one worker route kills SharedArrayBuffer with an empty console error, and the fix is a route pattern that exactly matches your worker paths. This class of deployment bug is the real cost of client-side ML — the model is easy, the headers are not.
Where client-side segmentation still loses to servers
Honest assessment, because the trade-offs matter:
Hair and fur edges — server models with matting refinement still produce cleaner alpha than a 320px-mask upscaled to 4000px. Feathering hides it at social-media sizes, not in print.
Very large images — decoding a 10000x10000px photo into a canvas can allocate hundreds of MB; tiling strategies help but add complexity.
Speed on old hardware — a 2015 laptop without WASM threads will wait several seconds; the same image on a server GPU is sub-second.
The compensating advantage is architectural: the image never leaves the device. For personal photos, ID scans, and client assets, that is not a feature — it is the requirement.
Seeing it in practice
I ship this pipeline in ToolsMonk's Background Remover — drag an image in, the ~80 MB model is fetched and cached, inference runs locally, and a transparent PNG downloads. The tool reports "100% private: processed in your browser, never uploaded" because that is literally the execution model. The same suite is available as a Windows app and an Android app for offline use.
If you're implementing this yourself
Start with ONNX Runtime Web and a U2-Netp ONNX export; measure before optimizing.
Put inference in a Worker from day one, and get your COOP/COEP headers right before debugging anything else.
Cache the model with the Cache API — first-load is a one-time cost, but make it explicit to the user.
Test the mask on hair. It will tell you immediately whether your feathering radius is right.
The browser is now a perfectly good ML runtime for single-image workloads. The next time a feature "needs a backend", check whether an 80 MB model and a canvas can do it locally instead — the privacy story writes itself.