GitHub - aditmag/dragonfly

GitHub

3 min read Original article ↗

Dragonfly

Dragonfly is a LoRA fine-tune of Qwen3-VL with typed output heads that answers many questions about an image in one pass, with a probability for every answer.

demo: adit.run/dragonfly

How it works

Unlike a VLM that generates an answer to each question, Dragonfly reads the image once and scores every question in the same pass. Questions can't see each other, so each answer is the same as asking it alone, in any order.

Typed answers

A VLM can reply with anything. Dragonfly can only answer in the form you asked for, with a probability for each possible answer.

Type Question Qwen3-VL-4B Dragonfly
Yes / no "Is the plane off the ground?" "Yes, the plane is off the ground. In this historical photograph, the Wright Flyer…" yes 80%
no 20%
can't answer 0%
Choice "What time of day is it: morning, noon, or night?" "Based on the visual evidence in the image, it is most likely morning. Here's the reasoning: …" morning 58%
noon 30%
night 12%
can't answer 35%
Scale "How cluttered is the scene, from 1 (tidy) to 3 (cluttered)?" "The scene is 1 (tidy). While there are a few scattered objects — like a small crate or box in the foreground…" 1: 74%
2: 19%
3: 6%
can't answer 5%

Examples picked from the demo.

"Can't answer" is the probability that the question doesn't fit the image. It's trained mostly on questions about things that aren't in the photo, and rarely catches anything else.

Results

Dragonfly against the model it was trained from.

Qwen3-VL-4B Dragonfly 4B Qwen3-VL-8B Dragonfly 8B
Yes / no 85.8% (0.011) 88.1% (0.007) 86.4% (0.020) 88.4% (0.006)
Choice 82.1% (0.010) 83.9% (0.003) 82.3% (0.011) 84.1% (0.008)
Scale 72.7% (0.023) 77.8% (0.009) 73.8% (0.031) 78.2% (0.012)
Open / closed (held out) 70.6% 72.3% 72.0% 73.2%
Material (held out) 87.1% 88.2% 88.0% 88.9%
POPE adversarial 88.0% (0.022) 89.0% (0.031) 87.5% (0.019) 88.3% (0.032)

Accuracy (ECE), all models temperature-scaled. Qwen3-VL is read through its own answer probabilities. 18% of test labels come from the larger Qwen VLMs. Raw numbers.

Data

No dataset asks typed questions with probability answers, so we built one: human-labelled questions from public VQA datasets, plus questions written and labelled by larger Qwen VLMs, calibrated against the human labels.

Source Train Val Test Total
Human vote spreads (VQAv2, KonIQ-10k) 79,105 16,005 18,031 113,141
Exact labels (GQA, A-OKVQA, TallyQA, VizWiz, COCO / Visual Genome) 174,093 30,518 43,855 248,466
Written and labelled by Qwen3-VL-32B and Qwen2.5-VL-32B 325,757 8,176 13,821 347,754
Option-count variants (copies with fewer options) 30,275 – – 30,275
Questions 609,230 54,699 75,707 739,636
Images 123,048 9,082 9,706 141,836

Run it

uv sync
uv run python model/serve.py --run runs/full4b      # then open http://127.0.0.1:8800

runs/full4b is the trained adapter (lora/, heads.pt, log.json); the 4B and 8B adapters will be published on Hugging Face. The 4B needs one GPU with ~12 GB (add --dtype float16 on GPUs without bf16).

License

Code: Apache-2.0 (LICENSE, NOTICE). The data sources keep their own licenses, and images are not redistributed (data/IMAGES.md).

Thanks

We acknowledge CSC – IT Center for Science, Finland, for computational resources. Built on Qwen3-VL and Qwen2.5-VL (Alibaba Qwen team). Data from COCO, Visual Genome, VQAv2, GQA, VizWiz (CC BY 4.0), A-OKVQA, TallyQA (Apache-2.0), KonIQ-10k, and POPE (MIT, evaluation only).