GitHub - QwenLM/Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Qwen Team  ·  Huazhong University of Science and Technology

📑 Technical Report  |  📖 Blog  |  🤗 Hugging Face  |  🤖 ModelScope

Welcome to the GitHub repository of Qwen-Drive-1.0. Here you can find official information about Qwen-Drive, and post your questions (Issues).

Introduction

Qwen-Drive-1.0 retains the architecture of the pretrained Qwen3.5 vision-language model and integrates 3D perception, visual question answering, and motion planning within a unified framework. The natively multimodal Qwen3.5-4B serves as the shared VLM, with two external modules attached:

• A BEV Perception Head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure.
• A Planning Expert conditions on shared VLM representations to generate future ego trajectories.
• The original VLM's LLM Decoder remains unchanged, and can handle both General VQA and Driving VQA tasks.

We propose a staged training strategy that integrates perception, language, and planning objectives. By combining driving-specific supervision with general-purpose vision-language data, the model achieves specialized driving competence while retaining broad visual understanding and instruction-following capabilities. This approach is supported by a unified data pipeline that: (1) maps heterogeneous perception annotations into a shared label space; (2) re-annotates driving VQA responses to ensure format and factual consistency; and (3) standardizes trajectories from multiple public driving datasets into a unified waypoint representation.

Performance

Planning

SFT is the imitation-trained Planning Expert. RL is the same expert after reward optimization on the benchmark objectives. Both share one VLM. 3D detection, occupancy and map segmentation results are in the technical report, full planning tables in docs/evaluation.md.

Driving VQA

LingoQA is scored with Qwen-Plus as the judge instead of the official LingoJudge, which we found to score leniently and inconsistently across scenarios. Under the official LingoJudge protocol Qwen-Drive-1.0-SFT obtains a LingoScore of 79.4. – marks an invalid or unparsable response.

On driving-scene understanding, Qwen-Drive-1.0 improves markedly over its Qwen3.5-4B base while keeping general vision-language ability intact. Driving QA and spatial understanding, with overall causal-reasoning accuracy on PAI-AV Chain-of-Causation (CoC) and an in-house Chinese urban driving-decision set (IH):

General VQA and Reasoning:

Spatial Understanding and Grounding:

Comparisons are reproduced under one protocol with near-deterministic decoding; see the technical report for the full setup.

Models

The model can be downloaded from Hugging Face or ModelScope. Everything ships in one directory. The VLM sits at its root, shared by every task, and each task head in a subfolder beside it.

Qwen-Drive-1.0-4B/          9.1 GB  the VLM, which on its own serves the VQA mode
├── planner-sft/            2.1 GB  Planning Expert, imitation-trained
├── planner-rl/             2.1 GB  Planning Expert after reward optimization
└── perception/             0.5 GB  BEV perception head

A head is attached when the VLM is loaded:

model = QwenDriveForPlanning.from_pretrained(
    "Qwen-Drive-1.0-4B", planner="Qwen-Drive-1.0-4B/planner-rl", dtype=torch.bfloat16
)

planner-rl was reward-optimized only on reasoning-conditioned rollouts, so run it in the reasoning planning mode. planner-sft covers both direct and reasoning planning.

Install

A GPU with 24 GB+ of memory is recommended.

git clone <repository-url> qwen-drive && cd qwen-drive

# Any Python virtual environment works; conda is shown here
conda create -n qwen-drive python=3.10
conda activate qwen-drive

pip install -e. --no-build-isolation        # or: pip install -r requirements.txt

Quick start

data/demo/ bundles four WOD-E2E planning scenes with their frames in one Parquet file and six perception frames, so the commands below need nothing but the weights. The scenes cover a night intersection whose light turns green, a left turn, a right turn, and a slow-down past a parked truck.

scripts/demo.py --plot writes a summary figure: the camera ring of the scene on the left, one row per view and one column per timestep, the predicted trajectories against the ground truth on the right, and the generated reasoning underneath.

export PYTHONPATH=src
python scripts/demo.py --model Qwen-Drive-1.0-4B --planner Qwen-Drive-1.0-4B/planner-rl \
    --scenes data/demo/planning_scenes.jsonl --image-archive data/demo/frames.parquet \
    --plot demo.png

import torch
from qwen_drive import InferenceMode, QwenDriveForPlanning
from qwen_drive.benchmarks import read_scene_file
from qwen_drive.images import ImageArchive

model = QwenDriveForPlanning.from_pretrained(
    "Qwen-Drive-1.0-4B",
    planner="Qwen-Drive-1.0-4B/planner-rl",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to("cuda").eval()

scene = next(
    read_scene_file(
        "data/demo/planning_scenes.jsonl",
        image_archive=ImageArchive.open("data/demo/frames.parquet"),
    )
).scene

result = model.run(InferenceMode.REASONING_PLANNING, scene=scene, num_samples=6)
print(result.reasoning)
print(result.trajectories.shape)   # (6, 50, 3) -> (x, y, heading), 5 s at 10 Hz

docs/cookbook.md has recipes for VQA, best-of-N planning, benchmark runs, multi-GPU evaluation, perception inference and visualization.

Documentation

Repository layout

qwen-drive/
├── src/qwen_drive/             # VQA + planning: model, scenes, benchmarks, metrics
├── src/qwen_drive_perception/  # perception mode (+ CUDA kernels under ops/)
├── scripts/                    # demo, prediction and scoring, visualization
├── data/demo/                  # bundled demo scenes and perception frames
├── data/benchmarks/            # the four benchmark scene files (relative frame paths)
├── assets/                     # figures
└── docs/

Citation

If you find our work helpful, feel free to cite us.

@misc{zhou2026qwendrive10initialstepvisionlanguage,
      title={Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving}, 
      author={Xin Zhou and Zongchuang Zhao and Zhibo Yang and Mingsheng Li and Humen Zhong and Shuai Bai and Du Chu and Ruizhe Chen and Zhaohai Li and Jun Tang and Qiuyue Wang and Mingkun Yang and Jiazhao Zhang and Dayiheng Liu and Dingkang Liang and Xiang Bai},
      year={2026},
      eprint={2609.00111},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.00111}, 
}

License

Qwen-Drive-1.0 is released under the Apache 2.0 license. Please find more details in the LICENSE file.