Dragonfly is a LoRA fine-tune of Qwen3-VL with typed output heads that answers many questions about an image in one pass, with a probability for every answer.
How it works
Unlike a VLM that generates an answer to each question, Dragonfly reads the image once and scores every question in the same pass. Questions can't see each other, so each answer is the same as asking it alone, in any order.
Typed answers
A VLM can reply with anything. Dragonfly can only answer in the form you asked for, with a probability for each possible answer.
| Type | Question | Qwen3-VL-4B | Dragonfly |
|---|---|---|---|
| Yes / no | "Is the plane off the ground?" | "Yes, the plane is off the ground. In this historical photograph, the Wright Flyer…" | yes 80% no 20% can't answer 0% |
| Choice | "What time of day is it: morning, noon, or night?" | "Based on the visual evidence in the image, it is most likely morning. Here's the reasoning: …" | morning 58% noon 30% night 12% can't answer 35% |
| Scale | "How cluttered is the scene, from 1 (tidy) to 3 (cluttered)?" | "The scene is 1 (tidy). While there are a few scattered objects — like a small crate or box in the foreground…" | 1: 74% 2: 19% 3: 6% can't answer 5% |
Examples picked from the demo.
"Can't answer" is the probability that the question doesn't fit the image. It's trained mostly on questions about things that aren't in the photo, and rarely catches anything else.
Results
Dragonfly against the model it was trained from.
| Qwen3-VL-4B | Dragonfly 4B | Qwen3-VL-8B | Dragonfly 8B | |
|---|---|---|---|---|
| Yes / no | 85.8% (0.011) | 88.1% (0.007) | 86.4% (0.020) | 88.4% (0.006) |
| Choice | 82.1% (0.010) | 83.9% (0.003) | 82.3% (0.011) | 84.1% (0.008) |
| Scale | 72.7% (0.023) | 77.8% (0.009) | 73.8% (0.031) | 78.2% (0.012) |
| Open / closed (held out) | 70.6% | 72.3% | 72.0% | 73.2% |
| Material (held out) | 87.1% | 88.2% | 88.0% | 88.9% |
| POPE adversarial | 88.0% (0.022) | 89.0% (0.031) | 87.5% (0.019) | 88.3% (0.032) |
Accuracy (ECE), all models temperature-scaled. Qwen3-VL is read through its own answer probabilities. 18% of test labels come from the larger Qwen VLMs. Raw numbers.
Data
No dataset asks typed questions with probability answers, so we built one: human-labelled questions from public VQA datasets, plus questions written and labelled by larger Qwen VLMs, calibrated against the human labels.
| Source | Train | Val | Test | Total |
|---|---|---|---|---|
| Human vote spreads (VQAv2, KonIQ-10k) | 79,105 | 16,005 | 18,031 | 113,141 |
| Exact labels (GQA, A-OKVQA, TallyQA, VizWiz, COCO / Visual Genome) | 174,093 | 30,518 | 43,855 | 248,466 |
| Written and labelled by Qwen3-VL-32B and Qwen2.5-VL-32B | 325,757 | 8,176 | 13,821 | 347,754 |
| Option-count variants (copies with fewer options) | 30,275 | – | – | 30,275 |
| Questions | 609,230 | 54,699 | 75,707 | 739,636 |
| Images | 123,048 | 9,082 | 9,706 | 141,836 |
Run it
uv sync
uv run python model/serve.py --run runs/full4b # then open http://127.0.0.1:8800
runs/full4b is the trained adapter (lora/, heads.pt, log.json); the 4B and 8B adapters will be published
on Hugging Face. The 4B needs one GPU with ~12 GB (add --dtype float16 on GPUs without bf16).
License
Code: Apache-2.0 (LICENSE, NOTICE). The data sources keep their own licenses, and images are not redistributed (data/IMAGES.md).
Thanks
We acknowledge CSC – IT Center for Science, Finland, for computational resources. Built on Qwen3-VL and Qwen2.5-VL (Alibaba Qwen team). Data from COCO, Visual Genome, VQAv2, GQA, VizWiz (CC BY 4.0), A-OKVQA, TallyQA (Apache-2.0), KonIQ-10k, and POPE (MIT, evaluation only).
