Qwen3.8 - How to Run Locally | Unsloth Documentation

Qwen3.8 is Qwen’s new model family, featuring Qwen3.8-27B, Qwen3.8-2.4T-A95B and Qwen3.8-Max. Qwen3.8-27B has vision and reasoning capabilities, a 256K context window, and runs locally on 17GB RAM/VRAM setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and Unsloth Desktop. Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol.

Run Qwen3.8 GuideDownload Unsloth

Qwen3.8 GGUFs use Unsloth Dynamic V3.0 (preview) for SOTA quant performance. Thank you Qwen for day zero access. Unsloth quants also include:

• Developer Role Support for agentic tools like Codex

• Tool calling: Improved parsing nested objects to make tools succeed more

Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit Unsloth Dynamic GGUFs takes 397GB (91% smaller), and larger IQ1_S takes 508GB.

Unsloth quants:

⚙️ Usage Guide

Qwen3.8-27B Requirements:

Qwen3.8-27B 4-bit quants will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM. Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory)

Qwen3.8-2.4T Requirements:

Recommended Settings

Qwen3.8-27B Settings:

Qwen3.8-27B is a hybrid thinking model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can adjust the thinking effort:

• Maximum context window: 262,144 (can be extended to 1M via YaRN)

• Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

• Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Qwen3.8-2.4T Settings:

Qwen3.8-2.4T is thinking-only, while Qwen3.8-Max is hybrid.

• Context length = up to 1,010,000

• temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

If the model fits, you will get ~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.

💡 Thinking + Preserve Thinking

Qwen3.8 has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right):

Qwen3.8-27B comes with support for reasoning_effort, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:

• xhigh (default): for complex tasks demanding thorough analysis

• medium: balancing accuracy and speed

• low: efficient reasoning optimizing for speed and cost

Run Qwen3.8 Guide

You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB IQ1_XXXS quant (named Q1_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: Qwen3.8-GGUF

Run in Unsloth DesktopRun in llama.cppNVFP4 Guide

🦥 Run Qwen3.8 in Unsloth Desktop

Qwen3.8 can run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux and:

• Search, download, run GGUFs and safetensor models

• Fast CPU + GPU inference via MLX and llama.cpp

Search and download Qwen3.8

Go to Unsloth Chat or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.

Run Qwen3.8

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth inference guide.

For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.

Serve Qwen3.8 with Unsloth API

You can use unsloth run command and serve Qwen3.8 via an API using llama-server runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our API docs.

Unsloth is now ready

You can also do many other things with Qwen3.8 via Unsloth Desktop like:

• Train models: Fine-tune text, diffusion, embedding, and more

• Generate media: Create and train images, video, TTS locally

Qwen3.8-2.4T-A95B New 1-bit data-types

We extended IQ1_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation)

Due to naming issues, we used TQ2_0, TQ1_0 and Q1_0 otherwise it won't pop up in the HF repo.

We are still running benchmarks for the new data-types, but for other large models, we get good results without any QAT / QAD:

🦙 Run Qwen3.8 in llama.cpp

⚡️NVFP4

Like Qwen3.6, we’re also releasing new dynamic NVFP4 Qwen3.8-27B quants that run ~1.5× faster than BF16 checkpoints, with better performance and comparable file sizes. Run Qwen3.8-27B NVFP4 1.5x faster on 24GB VRAM. We also added FP8 KV cache calibration for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see Qwen3.8), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in vLLM only for now (SGLang is not supported).

See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:

All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.

For accuracy benchmarks we ran KLD and Top-1% agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16

For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

For more information, you can read our Dynamic NVFP4 quants blog.

To run NVFP4 quants, see below for commands to run Qwen3.8-27B in vLLM or SGLang:

vLLM:

To install vLLM in a separate venv:

Then to serve the 27B variant:

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

SGLang:

SGLang is not yet supported since we quantize the lm_head to FP8.

vLLM has a CompressedTensorsW8A8Fp8 kernel which supports this, whilst SGLang cannot load the FP8 lm_head.

🤯Quantization Analysis

We ran top-1% and KLD for Qwen3.8 GGUFs, and we retain 82.5% accuracy (IQ2_XXS 9GB) whilst being 83.5% smaller (BF16 54.7GB).

NVFP4 quants are 1.5x faster than BF16 and retains 92 to 97% top-1% accuracy.

📊 Benchmarks

Qwen3.8-27B

See further below for table benchmarks:

Text Performance

Qwen3.8-2.4T-A95B