GitHub - Brandon-Collins7/sketchvlm

SketchVLM: Vision Language Models Can Annotate Images to Explain Thoughts and Guide Users

@misc{collins2026sketchvlmvisionlanguagemodels,
      title={SketchVLM: Vision language models can annotate images to explain thoughts and guide users},
      author={Brandon Collins and Logan Bolton and Hung Huy Nguyen and Mohammad Reza Taesiri and Trung Bui and Anh Totti Nguyen},
      year={2026},
      eprint={2604.22875},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.22875},
}

Abstract

When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration.

1. Interactive Demo

A live demo is available at sketch-vlm-demo.vercel.app. You can upload any image, type a question, and watch the model sketch its reasoning directly onto the image.

To run the demo locally, see the SketchVLM_Demo README for local setup and environment configuration.

2. Setup

Install dependencies:

pip install -r requirements.txt

Set your API keys in a .env file at the project root:

ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
GOOGLE_API_KEY=...

3. Running Inference

The main entry point is collab_sketch_with_label.py. It takes a dataset directory, overlays a coordinate grid on each image, sends it to the model, and renders the model's SVG strokes back onto the image. Results are saved under results/.

To run inference on a dataset, specify the model provider (--llm), the model name (--model), and the path to your dataset (--mixed-dir). The --adaptive-grid flag automatically scales the grid to match each image's resolution, which is recommended for most use cases:

python collab_sketch_with_label.py \
  --llm <provider> \
  --model <model> \
  --mixed-dir <dataset_path> \
  --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

--llm choices: claude, gpt, gemini, openrouter

Provider examples

Claude (Anthropic):

python collab_sketch_with_label.py --llm claude --model claude-opus-4-5 --mixed-dir datasets/ball_drop --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

OpenRouter (access models like Gemini via a unified API):

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/maze --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

GPT-5:

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/ball_drop --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

Multi-turn (stepwise) inference

If you want the model to refine its sketch over multiple turns — seeing the canvas update after each stroke — use --mixed-stepwise. This mirrors how a human might iteratively draw and correct. The model sees the updated image after each turn and can add or adjust strokes accordingly:

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --mixed-dir datasets/maze --mixed-stepwise --mixed-max-turns 40

4. Key Parameters

Grid parameters

The grid is a coordinate overlay that helps the model refer to specific locations in the image (e.g., "draw a line from x5y6 to x2y3"). If you want to control how fine or coarse the grid is, use --target-cols and --target-rows (default: 50×50). Use --adaptive-grid to have the grid scale automatically to the image resolution, which prevents cells from becoming too small or too large.

If you want to run the model on a raw image with no grid at all (e.g., to test a baseline), use --no-grid and specify the coordinate resolution explicitly with --res-x and --res-y.

Inference parameters

If you want deterministic outputs for reproducibility, use --deterministic (sets temperature=0). If you want to control how much reasoning effort extended-thinking models use before producing strokes, use --reasoning-effort. To run only a subset of a dataset (e.g., for quick testing), use --only with a comma-separated list or range.

Task mode parameters

If you want the model to produce one stroke per turn and see the updated canvas after each (multi-turn collaboration), use --mixed-stepwise. If you want the model to first produce all its strokes and then give a final answer in a second turn, use --two-turn.

Output parameters

5. Post-processing

If you want to re-render strokes onto images after inference (e.g., to change the base image or stroke style without re-running the model), use render_strokes_postprocess.py. Point it at a results folder and specify whether to render on the grid image (--base grid) or the original image (--base orig):

python render_strokes_postprocess.py \
  --results-dir results/mix_eval/<run_folder> \
  --base grid \
  --origin bottom-left

6. Code Structure

7. Datasets

We built three new benchmarks specifically for SketchVLM, all hosted on HuggingFace. The full collection is at loganbolton/sketchvlm.

Examples