DeepSeek-V4-Flash-0731 GGUF: Run a 284B AI Model Locally with Just 110 GB RAM
Learn how to run DeepSeek-V4-Flash-0731 GGUF locally using Unsloth and llama.cpp. Compare GGUF quantizations, hardware requirements, benchmarks, and performance of DeepSeek's 284B Mixture of Experts model.
DeepSeek-V4-Flash-0731 GGUF: run a 284B model locally on 110 GB RAM
The July 31 update to DeepSeek-V4-Flash isn’t a minor patch. On DeepSWE, it went from 7.3 to 54.4. That’s a 7.4x jump in agentic coding ability in a single release. Unsloth’s GGUF quantizations, published the same day, bring the full model down to 103 GB at 3-bit precision, meaning a 128 GB Mac or a Ryzen AI Max workstation can run it.
Here’s the full breakdown: what changed, what quantization you should pick, and exactly how to get it running.
What changed in the July 31 update?
The 0731 in the name stands for the release date: July 31, 2026. This is the official, stable release of DeepSeek-V4-Flash, superseding the April 2026 preview version.
DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.
The benchmark numbers tell the story better than any description. Compared to the preview version:
| Benchmark | V4-Flash Preview | V4-Flash-0731 | Improvement |
|---|---|---|---|
| DeepSWE | 7.3 | 54.4 | 7.4x |
| Cybergym | 38.7 | 76.7 | 2.0x |
| Terminal Bench 2.1 | 61.8 | 82.7 | 1.3x |
| AutomationBench | 10.8 | 25.1 | 2.3x |
| NL2Repo | 39.4 | 54.2 | 1.4x |
The DeepSWE number deserves attention on its own. It’s a coding agent benchmark measuring real software engineering tasks. Going from 7.3 to 54.4 in one update means the agentic coding capability was fundamentally rebuilt, not tweaked.
DeepSeek-V4-Flash-0731 has the same model structure as DeepSeek-V4-Flash-DSpark, meaning it ships with a speculative decoding module attached. DSpark runs 7 speculative tokens ahead using a greedy draft method, which meaningfully speeds up generation on vLLM without touching output quality.
What is DeepSeek-V4-Flash-0731?
DeepSeek-V4-Flash-0731 has 284B parameters with 13B active per token, a 1M context window (1,048,576 tokens), and is designed for coding, agentic, and chat workflows.
It’s MIT licensed. Full weights are on HuggingFace. The model was trained for a 1-million-token context window, though for Think Max reasoning mode you’ll want to set context to at least 384K to avoid cutting off the model’s reasoning chain.
The model sits between consumer accessibility and frontier capability. V4-Pro, the bigger sibling, has 1.6T parameters and 49B active, and needs over 800 GB of RAM at Q4. Flash-0731 at 284B total and 13B active is what you can actually run on hardware most serious developers already own or can rent cheaply.
Why does 13B active parameters matter?
DeepSeek-V4-Flash has 284B parameters but only 13B active per token due to its Mixture of Experts (MoE) architecture.
In a standard dense model, every parameter fires on every token. In a MoE model, a routing layer picks a small subset of “experts” for each token and only runs those. Flash-0731 routes through 13B parameters per forward pass, which is why the memory bandwidth requirements stay manageable and why Apple Silicon handles it well. Only 13B weights need to move per token, not 284B.
DeepSeek-V4-Flash-0731 outperforms V4-Pro (Preview) despite its far smaller activated parameter count. A 49B-active model lost to a 13B-active model. That gap is what DeepSeek’s training improvements bought between the preview and this release.
Why are Unsloth’s GGUFs different from community conversions?
This matters more than it seems, and the numbers prove it.
DeepSeek-V4-Flash is quantization-aware trained: the official checkpoint stores its routed experts (96% of the model) natively in MXFP4 and everything else in FP8 or BF16. Unsloth’s UD-Q8_K_XL repacks the experts bit-for-bit, and FP8 dequantizes into BF16 with no rounding, checked across all 1,328 tensors against the official DeepSeek weights.
Community converters re-quantize those MXFP4 experts to Q4_K or IQ2_XXS, which means rounding every single weight. The quality loss compounds across 96% of the model.
Here’s the full benchmark table from Unsloth’s measurements:
| Quant | Size (GB) | Perplexity | Same top token | Bit-exact weights |
|---|---|---|---|---|
| Official reference | 156.4 | 4.5319 | 100% | 100% |
| Unsloth UD-Q8_K_XL | 161.9 | 4.5319 | 100.000% | 100% |
| Unsloth UD-Q4_K_XL | 155.1 | 4.5335 | 96.28% | 97.46% |
| bartowski MXFP4 | 156.0 | 4.5351 | 96.18% | 97.57% |
| antirez Q4KExperts-F16 | 164.6 | 4.5726 | 93.94% | 0.93% |
| antirez IQ2XXS | 86.7 | 6.1518 | 77.92% | 0.47% |
The antirez IQ2XXS at 86.7 GB has a perplexity of 6.15 compared to Unsloth’s lossless 4.53. The antirez Q4K at 164.6 GB is actually larger than Unsloth’s lossless Q8 at 161.9 GB, and still delivers worse quality. Download size alone tells you nothing without the benchmark table.
Use Unsloth’s GGUFs.
What GGUF quantizations does Unsloth offer?
Unsloth offers quantizations from 1-bit to lossless 8-bit. The lossless 8-bit GGUF is 162 GB and the 3-bit is 103 GB, which can run on a 110 GB RAM device.
Full hardware requirements:
| Quantization | File size | Minimum RAM+VRAM |
|---|---|---|
| 1-bit | 92 GB | 92 GB |
| 2-bit | 102 GB | 102 GB |
| 3-bit UD-IQ3_XXS | 103 GB | 110 GB |
| 4-bit UD-Q4_K_XL | 155.1 GB | 162 GB |
| Q8 UD-Q8_K_XL (lossless) | 161.9 GB | 169 GB |
UD-IQ3_XXS at 103 GB is recommended for best results, fitting comfortably on a 128 GB device after system overhead.
One counterintuitive note: UD-Q4_K_XL (155.1 GB) and UD-Q8_K_XL (161.9 GB) are only 7 GB apart. If your system can fit Q4, you can almost certainly fit Q8 too, and Q8 is fully lossless. That gap almost never justifies stopping at Q4.
What hardware can actually run this?
The 3-bit quant at 110 GB minimum opens up real consumer options:
Realistic options include a Mac with enough unified memory, or a 128 GB Ryzen AI Max “Strix Halo” mini-PC, the 2026 favorite for exactly this class of model. Apple Silicon is particularly well-suited: only 13B parameters activate per token, so memory bandwidth efficiency is high.
For the lossless Q8 at 169 GB:
The full weights need roughly 170 GB for full weights plus KV cache, achievable on 2×H200 GPUs for production serving.
On vLLM in production, the model can be served on a single 4×GB300 node with DSpark speculative decoding enabled.
If you don’t have local hardware in range, the DeepSeek API, Together AI, Fireworks AI, and OpenRouter all serve V4-Flash-0731. The V4-Pro API costs $1.74 per million input tokens and $3.48 per million output tokens, compared to GPT-5.5 at $5 and $30, which is an 80 to 90% cost reduction. Flash is cheaper still.
How do you run it in Unsloth Studio?
Unsloth Studio is the easier path. It’s an open-source web UI with built-in RAM offloading, multi-GPU detection, and a thinking-mode toggle.
Install:
1
2
3
4
5
# macOS, Linux, WSL
curl -fsSL https://unsloth.ai/install.sh | sh
# Windows PowerShell
irm https://unsloth.ai/install.ps1 | iex
Launch:
1
unsloth studio -H 0.0.0.0 -p 8888
Open http://127.0.0.1:8888 in your browser, search for DeepSeek-V4-Flash in the Model hub, and download your quantization tier.
Think High is on by default. You can toggle it to Non-think or Think Max from the right dropdown in Unsloth Studio’s interface.
Inference parameters auto-set on load. For agentic use, override top-p to 0.95. For everything else, leave it at 1.0.
How do you run it with llama.cpp?
Mainline llama.cpp added V4 support in pull request 24162, and a July 7 follow-up fixed quantized KV caches, the bug quietly causing problems in multi-turn conversations across every provider’s GGUF files. You don’t need a custom fork anymore.
Build llama.cpp:
1
2
3
4
5
6
7
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
--target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
For Apple Silicon, swap -DGGML_CUDA=ON to -DGGML_CUDA=OFF. Metal runs by default on Mac.
Download the 3-bit quant:
1
2
3
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
--local-dir unsloth/DeepSeek-V4-Flash-0731-GGUF \
--include "*UD-IQ3_S*"
Run:
1
2
3
4
5
./llama.cpp/llama-cli \
--model unsloth/DeepSeek-V4-Flash-0731-GGUF/blob/main/UD-IQ3_S/DeepSeek-V4-Flash-0731-UD-IQ3_S-00001-of-00004.gguf \
--temp 1.0 \
--top-p 1.0 \
--min-p 0.0
To disable thinking mode: --reasoning off. To use Think Max: --chat-template-kwargs '{"reasoning_effort":"max"}'
For GPU offloading, tune --n-gpu-layers. Start at 2 and increase until you hit VRAM limits. Add --threads 32 to control CPU thread count for CPU-heavy inference.
How does it benchmark against Claude and other models?
Full benchmark table from DeepSeek’s official report, tested with the minimal mode of DeepSeek Harness at max reasoning effort, temperature = 1.0, top_p = 0.95:
| Benchmark | V4-Flash-0731 | V4-Pro Preview | GLM-5.2 | Opus-4.8 |
|---|---|---|---|---|
| Terminal Bench 2.1 | 82.7 | 72.1 | 81.0 | 85.0 |
| DeepSWE | 54.4 | 12.8 | 46.2 | 58.0 |
| Cybergym | 76.7 | 52.7 | - | 83.1 |
| NL2Repo | 54.2 | 38.5 | 48.9 | 69.7 |
| Toolathlon-Verified | 70.3 | 55.9 | 59.9 | 76.2 |
| Agents’ Last Exam | 25.2 | 16.5 | 23.8 | 25.7 |
| AutomationBench Public | 25.1 | 12.8 | 12.9 | 27.2 |
| DSBench-FullStack | 68.7 | 41.8 | 61.8 | 71.6 |
| DSBench-Hard | 59.6 | 31.1 | 54.5 | 71.7 |
V4-Flash-0731 beats V4-Pro Preview across every benchmark despite activating 13B parameters per token versus 49B. On Agents’ Last Exam (25.2 vs 25.7), it essentially ties Opus-4.8 at a fraction of the API cost.
The gap against Opus-4.8 is real on hard coding tasks like DSBench-Hard (59.6 vs 71.7), but Flash-0731 closes most of the distance the preview left open.
Is the 3-bit version good enough for real work?
Yes. The 3-bit UD-IQ3_XXS at 103 GB doesn’t appear in the quality benchmark table above because it isn’t a lossless quant, but it’s the sweet spot Unsloth explicitly recommends. The perplexity and KLD figures for 3-bit land between the lossless Q8 and the near-lossless Q4, well within the range where you won’t notice degradation on coding or agentic tasks.
Unsloth’s tool-calling test results showed 4 of 15 successful calls before the July llama.cpp KV cache fix, and 15 of 15 after, matching DeepSeek’s official results. That fix is now in mainline. If you download today, you’re not inheriting the multi-turn conversation bugs from spring.
The 1-bit (92 GB) and 2-bit (102 GB) options save only 1-11 GB over the 3-bit while giving up meaningful quality. Skip them unless your RAM ceiling is exactly 92 or 102 GB.
Key facts at a glance
- Model: DeepSeek-V4-Flash-0731 by DeepSeek AI, released July 31, 2026
- Parameters: 284B total, 13B active per token (Mixture of Experts)
- Context window: 1,048,576 tokens
- Speculative decoding: DSpark module, 7 speculative tokens, greedy draft
- License: MIT (full commercial use)
- Reasoning modes: Non-think, Think High (default), Think Max
- Recommended quant: UD-IQ3_XXS (103 GB, 110 GB RAM floor)
- Lossless quant: UD-Q8_K_XL (161.9 GB, bit-identical to official weights)
- Biggest benchmark jump: DeepSWE: 7.3 (preview) → 54.4 (0731), a 7.4x improvement
- Recommended settings: temp 1.0, top-p 1.0 (general), top-p 0.95 (agentic)
- Think Max max context: 384K tokens minimum
- QAT format: routed experts in MXFP4 (96% of model), rest in FP8/BF16
- Technical paper: arxiv.org/abs/2606.19348
- Community: discord.gg/unsloth
GGUF repo: huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Run guide: unsloth.ai/docs/models/deepseek-v4