AI masks (Object and Describe)

Imago selects objects with SAM 3 (Segment Anything with Concepts, Meta 2025), run in pure Rust by crates/segment on candle: on the GPU through Metal on macOS, on the CPU elsewhere (for now). If the Mac's GPU can't run it, it runs on the CPU (see the worker below).

The model is optional. It is not part of Imago, and nothing requires it: without it, Object and Describe offer to download it, every other feature (and every existing mask, see below) works, and no render, export or command ever waits for it.

Using them

In the Masking panel (M):

• Object: click the thing you want; every click refines the selection. ⌥-click (Alt) a part to leave it out. Clicks show as green (include) and red (leave out) dots.
• Describe: type what to select — sky, trees, the red car — and press Return. Every instance the model finds (score above 0.5) joins the mask; if nothing matches, nothing is created and you are told so. Around what it found, the mask follows the model's semantic mask for the phrase, which is sure inside and runs on where two instances touch (the instances' own masks alone left a sky or a net about 70–90 % selected and a thin line between two people side by side).
• Several things at once: car, road (comma-separated) selects each and merges them.
• + / − under the selected mask (and the Add / Subtract / Intersect menus) combine selections: Describe car, click the mask, + ▸ Describe… road; − ▸ Describe… people takes the people out of a sky mask; − ▸ Object then clicks remove one object.
• Edge (per selection, −100…100): below 0 a crisper border, above 0 a feathered one (up to 2 % of the long edge, so previews and exports match).
• Detail: about a second after the last click (or right after a description), the photo around the selection is analyzed again, zoomed in, from a sharper source (~5 s on an M4 Pro, in the background); the result is kept as a high-resolution patch (at most 4 per selection), so small objects get 5–10× finer edges than one pass over the whole photo gives.

The first Object or Describe on a photo analyzes it (the image encoder runs once per photo and look: ~3 s on an Apple M4 Pro, plus a one-time ~6 s for compiling GPU kernels and loading the model after launch). That starts in the background as soon as you pick Object; after it, each click takes ~35 ms and each description ~0.6 s. Everything the model does runs on its own worker thread: the window stays responsive, a click shows its dot at once and the selection follows when it's computed ("Selecting…" / "Analyzing the photo…" under the mask tiles). The model is unloaded after 10 minutes without use (it takes several GB of memory) and loaded again when needed.

The result is stored with the mask: the model's 288 × 288 mask logits over the uncropped photo (quantized, compressed, ~10–30 KB). Renders and exports sample them at any resolution and never need the model, so masks survive moving the library to a machine without it, and the web build renders them. Editing the photo's look later does not recompute a mask (click again to update).

Remote Metal inference

Native builds with sam can keep the editor, RAW files, library, and exports local while a Mac runs SAM 3. Build both ends from the same revision. On the Mac, install the model as described below, then run:

cargo build --release --locked -p imago-segment --bin imago-sam3-worker
target/release/imago-sam3-worker "$HOME/Library/Application Support/Imago/models/sam3" 127.0.0.1:8793

On the editing computer, start an SSH tunnel in one terminal and Imago in another:

ssh -N -T -o ExitOnForwardFailure=yes -L 127.0.0.1:8793:127.0.0.1:8793 user@your-mac
IMAGO_SAM3_REMOTE=127.0.0.1:8793 imago

Object clicks, Describe prompts, and detail refinement use the remote worker. Only the rendered model input (long edge at most 1008 pixels) and prompts cross the tunnel. The worker caches the current photo; detail crops do not replace that cache. Returned masks stay in the local library, so later adjustments and exports work without the connection.

The worker requires Metal and accepts one editing connection at a time. It loads weights on demand and releases them when the client disconnects or remains idle for ten minutes. Both endpoints require loopback addresses: SSH provides authentication and encryption. The listener has no independent authentication; do not expose it through a public proxy. Any process of any user on the worker Mac can connect to it, and on the editing computer a local process listening on the port while the tunnel is down receives the rendered model input, so use this only on machines whose local users you trust.

No local weights are required in remote mode. An unavailable worker produces an explicit error, not a silent CPU fallback. Restore the tunnel and retry. Unset IMAGO_SAM3_REMOTE to use the local model again. This offloads segmentation only, not RAW development, denoising, or upscaling. The model licence below still applies.

Getting the model

In the app (recommended)

The first time you pick Object or Describe without the model, Imago asks:

Download the SAM 3 model? Object and Describe masks use SAM 3, Meta's segmentation model. It isn't part of Imago, and everything else works without it. Download it once (about 3.4 GB) · folder. Licence: SAM License (Meta) — Meta's terms, not Imago's. Downloading it means accepting them. [Read the SAM License] — Not Now / Download

Nothing is downloaded until you choose Download. The download runs in the background (close the dialog and keep working; the Masking panel shows its progress), can be cancelled, resumes where it stopped if interrupted (also after quitting), and tries the next download location when one fails or stalls. Each file is written as <name>.part and moved into the model folder only when complete and verified — model.safetensors must have the exact size and SHA-256 of Meta's official checkpoint; a file that doesn't match is deleted, never used. If no location works, the dialog says why and Imago carries on without the model.

Agents and scripts use the same commands (control channel, imago-cli mcp --connect): segment.model.status, segment.model.download {"acknowledged": true} (refused without the explicit acknowledged: pass it only after the user agreed to the download and the licence), segment.model.cancel.

Download locations

The model is fetched from an ordered list of mirrors: base URLs where <base>/model.safetensors, <base>/vocab.json and <base>/merges.txt live. The list is, in order:

1. IMAGO_SAM3_MIRRORS (environment variable; URLs separated by commas or spaces);
2. the file models/sam3-mirrors.txt in Imago's settings folder (one URL per line, # comments) — ~/Library/Application Support/Imago/ (macOS), %APPDATA%\Imago\ (Windows), ~/.config/imago/ (Linux);
3. the built-in list, DEFAULT_MIRRORS in crates/segment/src/fetch/mod.rs.

Maintainers: the built-in list is empty until Imago's own CDN locations exist (see the TODO(maintainer) there): add them in order of preference, host the three files unchanged, and pin vocab.json / merges.txt (size + SHA-256) in SAM3_FILES at the same time. Until then the in-app download needs a user-configured mirror, and the dialog says so: it names the three files to put in the model folder by hand, with a button that shows that folder (created if needed) and a link to this guide.

Hugging Face's facebook/sam3 can't be a default: it is gated (each person must accept the licence there, wait for approval and download with their own token). Someone with access can use it as their mirror:

IMAGO_SAM3_MIRRORS=https://huggingface.co/facebook/sam3/resolve/main \
IMAGO_SAM3_TOKEN=hf_... imago

(IMAGO_SAM3_TOKEN is sent as a bearer token only over https, and only to the mirror's own host — not to the storage host it redirects to.)

The download is plain HTTP/1.1 over TLS in pure Rust (rustls with the RustCrypto provider and the Mozilla root certificates; no OpenSSL, ring or aws-lc, nothing compiled from C), with a 15 s connect timeout and a 30 s stall timeout; it follows redirects but doesn't use HTTP proxies.

By hand

Put model.safetensors, vocab.json and merges.txt from facebook/sam3 in the model folder: models/sam3/ in the settings folder above, or the folder IMAGO_SAM3_DIR names. Developers can use the installer scripts, which download from Hugging Face with your token and verify the weights:

HF_TOKEN=hf_... tools/install-sam3.sh            # macOS, Linux
tools/install-sam3.sh --check                     # verify an installation

$env:HF_TOKEN = "hf_..."; .\tools\install-sam3.ps1   # Windows

Licence of the model

Imago's code is MIT OR Apache-2.0; crates/segment (a port of Apache-2.0 code) is Apache-2.0. The SAM 3 weights are neither: they are Meta's, released under the SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE), a custom licence that is not an OSI open-source licence. It grants a non-exclusive, worldwide, non-transferable, royalty-free licence to use, reproduce, distribute, copy, modify and create derivative works of the "SAM Materials", subject to its terms, including: anyone redistributing them must provide a copy of the licence with them, and derivative works are distributed under its terms; uses subject to ITAR, military or warfare purposes, nuclear industries, espionage, guns or illegal weapons, and reverse engineering are prohibited, and trade controls (sanctions, export rules) apply; the materials come "as is", without warranty; Meta may terminate the licence on a breach (then the materials must be deleted); California law governs. Read the licence itself — this summary is not legal advice.

Therefore:

• The weights are never part of this repository, its releases or installers, and must not be committed.
• They reach a user's computer only when that user asks for them, after the dialog names the licence (or when they install them by hand).
• Whoever hosts a mirror redistributes the SAM Materials and must follow the licence (ship it next to the files).

Commands (control channel, MCP)

remote reports the configured endpoint, not its health. installed describes local weights only. download is {running, done, total, file, error, cancelled, finished}: cancelled is true when the last download was cancelled (its error says so in words; decide on the flag). An AI mask command refused because the model is not installed (or still downloading) fails with an error that starts with "The SAM 3 model is not installed"; inside Imago it is ErrorKind::ModelMissing(Sam3), which is what the app offers the download on.

Coordinates are normalized to the uncropped, oriented photo, like every mask shape. seg passes a stored segmentation (as mask.list / the develop settings hold it): the mask is made without the model. Without the model, the AI kinds fail with an error starting "The SAM 3 model is not installed" that says how to get it.

In the desktop app the commands don't wait for the model: mask.objectPoint returns {pending: true} and the selection is applied when it's computed (one undo step per settled selection); mask.add {kind: "prompt"} returns {pending: true} and the mask appears when something is found (or a message says nothing was). imago-cli and the headless MCP server wait and return the result (when built with the sam feature; the default CLI has no AI masks).

Implementation notes

• crates/segment ports the Hugging Face transformers implementation (Apache-2.0; see NOTICE — the ported files say so and that they were modified): the 32-layer ViT backbone with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box relative-position bias, presence token), pixel decoder and semantic head for text. The CLIP tokenizer is a small BPE in tokenizer.rs. The downloader (original code) is the imago-fetch crate, shared by any model that is downloaded; fetch/ here says what SAM 3 needs.
• candle is pinned at 0.9.2: later releases make candle-core depend on tokenizers with the Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no memory map: unsafe stays in imago-sysmem), only the tensors a path needs, and every tensor's byte range is checked against the file's length when it is opened (a truncated or damaged file is an error, never a crash).
• The engine (crates/engine/src/segment/) owns one worker thread with the model and the last encoded photo; the session sends it jobs and receives results over channels, never holding a lock the model holds. Each job runs under catch_unwind: a panic inside candle becomes an error and the model is reloaded on the next request; busy counters are reset by drop guards.
• GPU fallback (issue #1974): a job that fails on the GPU (a panic or a candle error, e.g. an AMD GPU in an Intel Mac whose Metal compiler crashes on a matmul kernel) frees the model and runs once more on the CPU. If that works, the model stays on the CPU until Imago restarts and the user is told once ("The GPU couldn't run the AI mask model…"); if the CPU fails too, the error is reported and the next job tries the GPU again. Missing or damaged model files and remote-worker errors are not retried. IMAGO_GPU=0 or IMAGO_GPU_BACKEND=off keep the model on the CPU from the start.
• Hostile documents: a selection keeps at most 4 detail patches (when read and when rendered), each at most 1024² logits; more than 64 clicks are never sent to the model.
• Accuracy: crates/segment/tests/it/reference.rs compares every stage against the Python reference (tools/sam3_reference.py); on Metal the backbone matches to 7e-5 relative error and masks to ~5e-5. f16 was tried and rejected (overflow, ~7 % faster).
• The engine feature sam (on in the desktop app, off for web and CLI) gates the model; without it the commands report that the build has no AI masks.

Developers: the accuracy test against the Python reference needs torch and transformers (tools/requirements-sam3-reference.txt):

python3 -m venv .venv-sam3 && .venv-sam3/bin/pip install -r tools/requirements-sam3-reference.txt
.venv-sam3/bin/python tools/sam3_reference.py photo.jpg ref.safetensors
IMAGO_SAM3_DIR=<model dir> IMAGO_SAM3_REF=ref.safetensors cargo test -p imago-segment --release -- --nocapture