Recommendation for Self-hosted & open
Best Open-Weight LLM for Self-Hosting
Our top recommendation for Best Open-Weight LLM for Self-Hosting, based on the public evidence we track, is Qwen: Qwen3.8 27B.[1][2][3] Deploy on a single H200 GPU at full 262144 context length with TP1 and no NCCL, simplifying multi-tenant setups where you want fast queuing without node-level orchestration. Watch out: Expect ~17GB of weights to stream through UMA on every token due to dense architecture, which saturates bandwidth and limits decode speed on consumer hardware compared to MoE alternatives. DeepSeek: DeepSeek V4 Flash 0423 is the next-ranked alternative. Run the 85GB IQ2_M GGUF on RTX 2080 Ti (11GB) or RTX 3080 (10GB) with fixed-expert-count LRU eviction, capping host RAM at 11-12GB RSS for practical local inference.
About this recommendation
- Updated
- Sep 6, 2026
- Evidence through
- Sep 6, 2026
- Sources
- 21
- Revision
- v60
Decision audit
Why this result
Inspect the inputs and the computed order behind the recommendation.
Models screened
20
live candidates
Evaluation feeds
5
task-weighted
Winner coverage
73%
intended feed weight
Largest provider share
2 of 8
Z.ai
Sources evaluated
The task sets these weights before any model is scored.
| Evaluation feed | Weight | Winner result | Field measured |
|---|---|---|---|
| price weight | 33% | 87/100 | 20/20 |
| LiveBench Instruction Following | 22% | #1 | 9/20 |
| Route reliabilityunavailable | 22% | feed unavailable | 0/20 |
| LMArena Text | 17% | #16 | 19/20 |
| OpenRouter usage | 6% | 86/100 | 20/20 |
Provider concentration
Each exact model is scored separately; provider identity is not a ranking input.
- Z.ai2 models
- deepseek1 model
- Google1 model
- Moonshot AI1 model
- Qwen1 model
- tencent1 model
- thinkingmachines1 model
Decision table
Every published model is shown in computed order. Practitioner sources are distinct community threads, not the citations repeated in the prose below.
| Rank | Model | Relative score | Coverage | Practitioner evidence | Strongest measured reason |
|---|---|---|---|---|---|
| 01 | Qwen3.8 27BQwen | 75 | 73% | 26 threads · 22 families · 5 cautions | #1 LiveBench Instruction Following · #16 LMArena Text |
| 02 | DeepSeek V4 Flash 0423deepseek | 74 | 73% | 15 threads · 12 families · 4 cautions | #7 LiveBench Instruction Following · #16 LMArena Text |
| 03 | GLM 5.3 FlashZ.ai | 72 | 73% | 6 threads · 6 families · 2 cautions | #3 LMArena Text · #13 LiveBench Instruction Following |
| 04 | Hy3tencent | 70 | 67% | 2 threads · 2 families · 1 cautions | #9 LMArena Text |
| 05 | Gemma 4 26B A4B Google | 69 | 67% | 1 threads · 1 families · 0 cautions | #15 LMArena Text |
| 06 | Inklingthinkingmachines | 58 | 73% | 1 threads · 1 families · 0 cautions | #3 LiveBench Instruction Following · #14 LMArena Text |
| 07 | GLM 5.3Z.ai | 56 | 73% | 4 threads · 4 families · 1 cautions | #2 LMArena Text · #4 LiveBench Instruction Following |
| 08 | Kimi K3Moonshot AI | 36 | 73% | 10 threads · 7 families · 3 cautions | #1 LMArena Text · #2 LiveBench Instruction Following |
Relative score combines normalized benchmark quality and signal coverage; independent practitioner evidence and freshness are bounded tie-breakers. It is an ordering score, not an absolute quality percentage. The writing model receives this order and cannot change it.
Qwen3.8 27B is a dense 27.78B parameter hybrid-attention model with vision capabilities that achieves 122 tok/s single-caller throughput on one H200 GPU at full 262K context, making it viable for high-throughput self-hosted deployments where queue latency matters.
Best when: Deploy on a single H200 GPU at full 262144 context length with TP1 and no NCCL, simplifying multi-tenant setups where you want fast queuing without node-level orchestration.
Tips
- Deploy on a single H200 GPU at full 262144 context length with TP1 and no NCCL, simplifying multi-tenant setups where you want fast queuing without node-level orchestration.
- Use SGLang's `--sleep-on-idle` flag to eliminate the busy-spin CPU consumption that otherwise burns one core per idle model instance.
Watch out for
- Expect ~17GB of weights to stream through UMA on every token due to dense architecture, which saturates bandwidth and limits decode speed on consumer hardware compared to MoE alternatives.
- Avoid the experimental `--gdn-replay` flag with MTP speculative decoding, as it causes degenerate looping after ~900 characters despite appearing to offer speedup.
- Watch for `thinking_token_budget` being ignored when using MTP speculative decoding with NVFP4 ModelOpt variants, causing the model to consume the entire completion allowance as reasoning tokens.
DeepSeek V4 Flash is a 157B MoE (85GB IQ2_M) that now runs on 11GB VRAM cards via community quantization and expert eviction, making frontier-class models accessible on consumer hardware without per-token costs.
Best when: Run the 85GB IQ2_M GGUF on RTX 2080 Ti (11GB) or RTX 3080 (10GB) with fixed-expert-count LRU eviction, capping host RAM at 11-12GB RSS for practical local inference.
Tips
- Run the 85GB IQ2_M GGUF on RTX 2080 Ti (11GB) or RTX 3080 (10GB) with fixed-expert-count LRU eviction, capping host RAM at 11-12GB RSS for practical local inference.
- Use as a fast open-model baseline for comparison testing against other local deployments, with established provider routes through OpenRouter and LiteLLM.
Watch out for
- Monitor for VRAM memory leaks when using GGUF with DSpark speculative decoding, which accumulates ~10MB per PP+TG cycle until OOM.
- Handle intermittent SSE stream stalls mid-generation without terminal `finish_reason` or `[DONE]` chunks, requiring client-side timeout and retry logic.
GLM 5.3 Flash is a 320B total / 18B active MoE with 1M context that ranks #20 on LMArena and runs on consumer GPUs via llama.cpp PR builds, offering strong preference scores at modest active parameter cost.
Best when: Test locally on CPU and RAM using llama.cpp's active PR branch for GLM5-Next architecture, making it accessible before official upstream merge.
Tips
- Test locally on CPU and RAM using llama.cpp's active PR branch for GLM5-Next architecture, making it accessible before official upstream merge.
- Benchmark against Qwen3.8-Flash-Next in real LangGraph agent workloads to determine which better fits your specific tool-calling and latency requirements.
Watch out for
- Avoid `developer` message role when using OpenAI-compatible providers, as upstream gateways return deterministic `[1214] Incorrect role information` errors.
- Watch for `ROLLUP_TURN_TIMEOUT` errors in long-running memory-daily tasks, with EventEmitter memory leak warnings suggesting resource exhaustion under sustained load.
Tencent Hy3 is available through Zen/Go routes with non-standard
reasoning_effortvalues (no_think/low/high), though pricing is unpublished and the/zen/v1endpoint 404s for this model.Best when: Use `no_think` default for fastest generation when reasoning is unnecessary, with explicit control over thinking depth via non-standard parameters.
Tips
- Use `no_think` default for fastest generation when reasoning is unnecessary, with explicit control over thinking depth via non-standard parameters.
Watch out for
- Route only through `zen/go` endpoints, as `/zen/v1/chat/completions` returns 404 and `hy3-preview` is defunct in model listings.
- Treat pricing as uncertain estimates carried from OpenRouter, with no official published rates from TokenHub international host.
Gemma 4 26B-A4B remains the incumbent local model with established tooling, though Qwen3.8-27B outperforms it on benchmarks; adoption blocked by dense architecture bandwidth costs rather than capability.
Best when: Consider only after reviewing the cited caution.
Watch out for
- Accept lower benchmark scores (GPQA Diamond, LiveCodeBench, Terminal-Bench, SWE-bench Pro) compared to Qwen3.8-27B if you need cutting-edge reasoning performance.
Inkling uses numeric
reasoning_effortvalues (0.1-0.99) incompatible with preset string vocabularies, making configuration impossible on platforms expectinglow/medium/high/max.Best when: Consider only after reviewing the cited caution.
Watch out for
- Avoid platforms with fixed-string effort presets, as numeric-effort models cannot be configured and stored values may be silently rewritten to unrecognized defaults.
GLM 5.3 ranks #15 on LMArena with Elo 1482 and offers
low/high/maxreasoning effort presets, though it was temporarily removed from rotation due to provider availability issues later found to be transient.Best when: Consider only after reviewing the cited caution.
Watch out for
- Watch for temporary provider unavailability that may cause session failures with `provider_unavailable` errors, requiring fallback model configuration.
Kimi K3 is a 2.78T parameter MoE with 1.42TB weights that ranks #10 on LMArena, made runnable on Apple Silicon via WASTE expert pruning and REAP calibration that keeps only salient experts.
Best when: Run on Apple Silicon with ~32-58GB peak RAM via REAP expert pruning, calibrating expert salience and letting the router renormalize to avoid loading full 1.56TB.
Tips
- Run on Apple Silicon with ~32-58GB peak RAM via REAP expert pruning, calibrating expert salience and letting the router renormalize to avoid loading full 1.56TB.
Watch out for
- Expect API rejection when sending `temperature` parameter to certain endpoints, requiring client-side parameter filtering for compatibility.
- Accept bandwidth-constrained decode speeds (~15 tok/s) on DDR5-based systems like DGX Spark when the model fills most of 128GB unified memory, compared to HBM-based inference hardware.
Frequently asked
- What is the top-ranked model for Best Open-Weight LLM for Self-Hosting?
- Qwen: Qwen3.8 27B ranks first in the current evidence-weighted comparison. Deploy on a single H200 GPU at full 262144 context length with TP1 and no NCCL, simplifying multi-tenant setups where you want fast queuing without node-level orchestration.[1]
- What should I watch out for with Qwen: Qwen3.8 27B?
- Expect ~17GB of weights to stream through UMA on every token due to dense architecture, which saturates bandwidth and limits decode speed on consumer hardware compared to MoE alternatives.[2]
- What is an alternative to Qwen: Qwen3.8 27B?
- DeepSeek: DeepSeek V4 Flash 0423 is the next-ranked option. Run the 85GB IQ2_M GGUF on RTX 2080 Ti (11GB) or RTX 3080 (10GB) with fixed-expert-count LRU eviction, capping host RAM at 11-12GB RSS for practical local inference.[3]
Sources
- 1
“Qwen3.8-27B is a 27B dense hybrid-attention model with a vision tower and an MTP draft head. It fits one H200 GPU at the full 262144 context, so it queues faster than any recipe that needs a whole node. | | Measured | | --- | --- | | One caller | 122.1 tok/s | | Concurrency 1024 | 3957.3 tok/s, still rising | | Context | 262144, the checkpoint maximum | | KV cache | 1,804,253 tokens, 6.88 full-length requests at once | | Hardware | 1 H200 GPU, TP1, no NCCL | Scope - New recipe at recipes/Qwen3.…”
mmshad · GitHub · Aug 26, 2026 - 2
“## Context Qwen3.8-27B (released 2026-08-14, Apache 2.0, dense 27.78B, hybrid Gated DeltaNet + vision encoder) was evaluated as a replacement for the current local model (Gemma 4 26B-A4B Q8_0). The evaluation surfaced two pre-existing weaknesses in the local LLM layer that are worth fixing on their own merits, independent of whether Qwen ever becomes the default. **Why Qwen is not being made the default:** it is dense, so every token streams all ~17 GB (Q4_K_M) of weights through the UMA. The s…”
nbramia · GitHub · Aug 15, 2026 - 3
“## Update 2026-08-05: now verified on a 157B MoE (85GB) — 16GB-RAM machines included To close the loop on the "TBD" row and show this isn't just small-model magic: **DeepSeek-V4-Flash (157B MoE, 43 layers × 256 experts, 6 active + 1 shared, 85GB IQ2_M GGUF)** now runs end-to-end through the moe-l2 pipeline on **RTX 2080 Ti (11GB)** and **RTX 3080 (10GB)**: - **VRAM: 8.3–9.1 GB** (fits both cards) - **Host RAM: RSS capped at 11–12 GB** via fixed-expert-count LRU eviction (`MOE_L2_LRU_MAX_EXPERTS…”
yalun753 · GitHub · Aug 6, 2026 - 4
“## Summary The pinned SGLang scheduler busy-spins on an idle model and consumes roughly one CPU core. Enabling SGLang's existing `--sleep-on-idle` path removes the zero-timeout polling loop while preserving the active request path. ## Environment - NVIDIA GB10 / DGX Spark - Qwen3.8 27B NVFP4 with DFlash2 - SGLang image: `lmsysorg/sglang@sha256:f6c809a2ebdeea97a3732e8bc139a32c18c9fb00a6e2fd770d257d9f72466ed3` - SGLang build: `0.0.0.dev0+qwen38.27b.g561c8f3` - 262144 context length - FP8 E4M3 KV…”
RyderFreeman4Logos · GitHub · Aug 20, 2026 - 5
“# `--gdn-replay` makes qwen35 generate degenerate looping text On `fb2cc35ab`, enabling `--gdn-replay` with MTP speculative decoding makes Qwen3.8-27B collapse into a repeated phrase until `max_tokens`. The flag is EXPERIMENTAL and defaults to off, so nothing is broken for anyone who leaves it alone — but it currently looks like a large speedup while actually destroying the output, which makes it easy to adopt by mistake. ## Symptom The first ~900 characters are coherent, then it locks into the…”
tmark00 · GitHub · Sep 3, 2026 - 6
“## Your current environment The output of python collect_env.py The repetitive `NVIDIA_REQUIRE_CUDA` constraint and CPU vulnerability table were omitted for readability. I can provide the unabridged collector output if useful. ## Describe the bug `thinking_token_budget` is ignored when Model Runner V2 uses MTP speculative decoding with a Qwen3.8-27B ModelOpt NVFP4 model. The request consumes the entire completion allowance as reasoning and never emits ` ` or final content. With the same image,…”
ma-leezc · GitHub · Sep 2, 2026 - 7
“Part of #18 and #1 Source: `docs/provider-map.md` ## Question Which available Command Code models should be used for adapter smoke testing, the next pilot Match, and later comparison Matches, and which pairings are sufficiently equivalent to support a model-comparison claim? ## Candidate set - `poolside/laguna-s-2.1-free` — free adapter smoke test; locally verified. - `deepseek/deepseek-v4-flash` — fast open-model baseline. - `deepseek/deepseek-v4-pro` — reasoning open-model baseline. - `xiaomi…”
andreamicheli · GitHub · Aug 10, 2026 - 8
“### Name and Version ### Description I'm observing a reproducible VRAM memory leak when running DeepSeek V4 Flash GGUF with DSpark speculative decoding enabled. After each PP+TG (prefill + generate) cycle, VRAM usage increases by approximately **10 MB**, and this accumulation continues until the process eventually runs out of memory (OOM). This issue does **not** occur with: - Step 3.7 + DSpark - Qwen3.8-27B + DSpark Both of the above maintain stable VRAM usage across multiple cycles. ### Envir…”
wxt-coder · GitHub · Aug 16, 2026 - 9
“## Summary The DeepSeek OpenAI-compatible `/chat/completions` SSE endpoint (both `api.deepseek.com` and the `opencode-go` proxy) **intermittently stops sending chunks mid-generation without closing the connection** — typically mid-`reasoning_content` (thinking) — with no terminal `finish_reason` chunk and no `[DONE]`. The stream simply goes silent and stays open indefinitely. Observed across multiple models: - `deepseek-v4-flash` - `deepseek-v4-flash-vision-exp` This does **not** happen with ot…”
iacore · GitHub · Aug 26, 2026 - 10
“Is work on vision in progress? By the way for those who do that for the first time -- anyone can try GLM 5.3 Flash on text only mode taking the current pull request on llama.cpp. This is very easy. The model is very good and fast even on CPU and RAM #1. I Assume you already have Installed llama.cpp #2. cd ~/llama.cpp #3. Fetch the active PR branch for GLM5-Next architecture: git fetch origin pull/27754/head:glm5next-patch git checkout glm5next-patch #4.Clean up the old build files and compile t…”
Dan646464 · Hugging Face · Sep 6, 2026 - 11
“## Problem Statement Two candidate models for StructureClaw's agent — **GLM-5.3-Flash** and **Qwen3.8-Flash-Next (FP8)** — are deployed locally as vLLM OpenAI-compatible endpoints (from the team's model-deploy stack folder, on disjoint GPUs). I need to know which one the agent should use, i.e. benchmark both through the real LangGraph agent benchmark and compare them. Today that is manual and error-prone: - The benchmark runner (`llm-benchmark`) supports exactly one model under test per invocat…”
kornkritpawit · GitHub · Sep 4, 2026 - 12
“## 摘要 / Summary 当使用 OpenAI 兼容提供商、且其上游网关不接受 `developer` 消息角色时，AI 的每一轮都会失败，报错如下： 本报告环境：提供商 `aimom`（base URL `https://platform.aimom.net/v1`，openai_compatible / chat_completions），模型 `GLM-5.3-Flash`（`glm-5.3-flash`）。失败是**确定性 100%**，而同一提供商上的 `deepseek-v4-flash` 每次都成功。 --- ## 根因（已通过直连 API 复现）/ Root cause (reproduced with direct API calls) PI-Desktop 向该提供商发送请求时，把系统提示/指令放在 **`developer`** 角色里。而服务于 `glm-5.3-flash` 的上游路由（“Console Go”）**不接受 `developer` 角色**，返回 `[1214] Incorrect role information`。同一网关下不同模型…”
Tioit-Wang · GitHub · Sep 2, 2026 - 13
“Iva 0.4.0 (commit 817b9cc29d96), Ubuntu 24.04.4, Node v24.19.0, provider openrouter (z-ai/glm-5.3-flash), single-user install on a VPS. Since 2026-08-31 every nightly memory-daily run fails: • schedule-runner: memory-daily finished (code=1) • tail: { code: 'ROLLUP_TURN_TIMEOUT', label: 'main-turn' } Last success in rollup-status.json is 2026-08-31T01:01:02Z. Catch-up runs after restart fail the same way. At the timeout moment the log also shows MaxListenersExceededWarning: Possible EventEmitter…”
enjoy2k · GitHub · Sep 2, 2026 - 14
“## Tencent Hunyuan — 0.4.0 Provider Audit **Connection:** `tencent` | **Transport:** `openai-chat` **Source:** `models/builtin/tencent.toml` ### Current state - **Models shipped:** 2 (hy3, hy3-preview) via TokenHub international host - **Pricing:** UNPUBLISHED — prices are estimates carried from OpenRouter. Marked uncertain at add time. - **Reasoning:** NONE — hy3 uses a non-standard `reasoning_effort` with values `no_think` (default) / `low` / `high`. No matching codec in bonsai. - **Prompt ca…”
strozynskiw · GitHub · Jul 25, 2026 - 15
“Note that HY3 changed it's name for opencode. Now need to call Hy3 instead of the preview one Strangely, - both models.dev and the official /zen/v1/models API still shows hy3 had a free and preview version. - both models.dev and /zen/go/v1/models shows a defunct `hy3-preview`. - /zen/v1/chat/completions 404's for hy3, meaning zen had no support for hy3 - Only HY3 that works is via zen/go with model id `hy3`”
SodaWithoutSparkles · GitHub · Jul 29, 2026 - 16
“**Standing issue — intentionally left open. Revisit periodically; close only if the model is ruled out for good.** ## What this tracks Whether [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) should replace Gemma 4 26B-A4B as the local model, and what would have to change upstream for that to make sense. ## Why it was not adopted on release (2026-08-15) The benchmarks are excellent — GPQA Diamond 89.2, LiveCodeBench v6 90.3, Terminal-Bench 2.1 73.0, SWE-bench Pro 61.7, Apache 2.0, 262K na…”
nbramia · GitHub · Aug 15, 2026 - 17
“## Summary The per-model `reasoning_effort` preset assumes one vocabulary — the fixed strings `low`/`medium`/`high`/`max` — but models ship different ones. On 0.5.7 this makes some efforts impossible to set, and worse, the panel silently rewrites stored values it doesn't understand, so what the platform shows can't be trusted. ## Symptoms (0.5.7, admin panel) 1. **Numeric-effort models (Inkling, 0.1–0.99) can't be configured at all.** There is no mapping from the preset strings to the int/float…”
Pietroski · GitHub · Aug 15, 2026 - 18
“Raised after GLM 5.3 was restored on 2026-08-20 and found to have been working the whole time. Owner asked directly: *"how do we prevent this from ever happening again?"* This is the primary half of the answer. The secondary half is `u2giants/shared-db#1287`. ## What happened `glm-5.3` was removed from the shared-db reviewer rotation on 2026-08-18 after **three consecutive `provider_unavailable` failures** — rotation sequences 161, 164, 167 — each creating a session that never produced a turn,…”
u2giants · GitHub · Aug 20, 2026 - 19
“## Motivation [PipeNetwork/kimi-k3-mlx](https://github.com/PipeNetwork/kimi-k3-mlx) demonstrated that REAP expert pruning ([Cerebras](https://github.com/CerebrasResearch/reap)) is the practical route to running huge MoE models on Apple Silicon: score each expert by `S_j = mean(gate_j · ‖expert_out_j‖₂)` over a calibration corpus, keep the most salient, let the router renormalize. Their measured results on Kimi-K3 (2.78T, 896 experts): - Calibration of a 1.56 TB model in ~32–58 GB peak RAM by st…”
dpalmqvist · GitHub · Jul 28, 2026 - 20
“在使用 Opus 5 、Kimi K3 模型时会发生如题所示的错误。 程序在 server/ai-adapter.js:92 和 server/ai-adapter.js:138 无条件传了 `temperature` 参数，而这两个模型的 API 会拒绝带这个参数的请求。 我正在自己 fork 的仓库中修复这个问题，解决方案是增加用户配置文件 ai-request-parameters.json。发送大模型请求时热重载该文件，替换内存中的 API 请求参数配置。 并且我还增加了新功能：用户可用 CLI 命令来修改软件在 Windows 上存储文件的位置。原先默认在 C 盘，对 C 盘爆满的用户不友好。”
wushuangabao · GitHub · Jul 31, 2026 - 21
“I have one. The limitation (beyond total size of the memory) with the Spark is DDR5. "Real" inference hardware is HBM (high bandwidth memory) which is like 10x the performance. So for prefill -- which is more about compute than bandwidth -- the Spark performs quite admirably. But on decode it's highly bandwidth constrained. Some smaller MoE models (e.g. Gemma4) can do 60-70 tok second but anything dense, or anything that is actually filling up most of that 128GB is going to choke out around 15…”
cmrdporcupine · Hacker News · Jul 27, 2026
Rankings synthesized from community evidence and open benchmarks. See methodology. Not driven by vendor marketing.