Recommendation for Self-hosted & open
Best Open-Weight LLM for Self-Hosting
The best LLM for best open-weight llm for self-hosting in this evidence-weighted comparison is Qwen: Qwen3.5-27B.[1][2] Deploy on a single 24GB GPU (like RTX 3090) using 4-bit quantization for low-barrier entry, or scale to dual L4s with vLLM to serve 10+ developers at 20-40 tok/sec depending on load. Z.ai: GLM 5.2 is the next-ranked alternative. Its currently supported evidence is cautionary: Practical self-hosting demands FP4 or FP2 quantization, which incurs 3% to over 17% accuracy degradation: at FP2, performance drops below Claude Sonnet 4.6 levels, making the model's benchmark standing misleading for typical hardware constraints.
About this recommendation
- Updated
- Jul 22, 2026
- Evidence through
- Jul 22, 2026
- Sources
- 2
- Revision
- v4
Qwen3.5-27B runs efficiently at 4-bit quantization on single 24GB consumer GPUs and scales to multi-GPU deployments serving production developer workloads with acceptable throughput.
Best when: Deploy on a single 24GB GPU (like RTX 3090) using 4-bit quantization for low-barrier entry, or scale to dual L4s with vLLM to serve 10+ developers at 20-40 tok/sec depending on load.
Tips
- Deploy on a single 24GB GPU (like RTX 3090) using 4-bit quantization for low-barrier entry, or scale to dual L4s with vLLM to serve 10+ developers at 20-40 tok/sec depending on load.
GLM 5.2 approaches frontier closed-model performance at full precision but requires massive memory footprints that home and small-team deployments rarely achieve.
Best when: Consider only after reviewing the cited caution.
Watch out for
- Practical self-hosting demands FP4 or FP2 quantization, which incurs 3% to over 17% accuracy degradation: at FP2, performance drops below Claude Sonnet 4.6 levels, making the model's benchmark standing misleading for typical hardware constraints.
Frequently asked
- What is the top-ranked model for best open-weight llm for self-hosting?
- Qwen: Qwen3.5-27B ranks first in the current evidence-weighted comparison. Deploy on a single 24GB GPU (like RTX 3090) using 4-bit quantization for low-barrier entry, or scale to dual L4s with vLLM to serve 10+ developers at 20-40 tok/sec depending on load.[1]
Sources
- 1
“Qwen3.5-27B with a 4bit quant can be run on a 24G card with no problem. With 2 Nvidia L4 cards and some additional vllm flags, i am serving 10 developers at 20-25tok sek, off-peak is around 40tok sek. Developers are ok with that performance, but ofc they requested more GPU's for added throughput.”
proxysna · Hacker News · Apr 22, 2026 - 2
“"GLM 5.2 is just shy of GPT 5.4"... If your running the full model. As in have 750 (FP8) to 1.5TB(FP16) of memory available. Do not mix the benchmark results of GLM 5.2 FP16 FP8 with FP4 or FP2. * FP4 will mean a accuracy loss of about 3%. Not noticeable but more chance for mistakes. * FP2 ... what is what most people are able to run at home, for a "reasonable" price. Your looking at over 17% loss in accuracy. At that point, your running at less then claude-sonnet-4.6, as the issues compound wi…”
benjiro29 · Hacker News · Jun 22, 2026
Rankings synthesized from community evidence and open benchmarks. See methodology. Not driven by vendor marketing.