Recommendation for RAG / Long context
RAG & Long Context
Our top recommendation for RAG & Long Context, based on the public evidence we track, is Qwen: Qwen3.8 27B.[1][2][3][4] Use for single-GPU long-context serving where the hybrid attention (48 SSM layers, 17 full attention layers) reduces KV cache pressure compared to dense alternatives. Watch out: Watch for arithmetic bugs that kill long runs at iteration 6 when using Ollama with pinned windows above 32K tokens. Anthropic: Claude Opus 4.6 is the next-ranked alternative. Select when LMArena document-task preference rankings align with your quality requirements, as it scores 1510 on that benchmark.
About this recommendation
- Updated
- Sep 4, 2026
- Evidence through
- Sep 4, 2026
- Sources
- 18
- Revision
- v59
Decision audit
Why this result
Inspect the inputs and the computed order behind the recommendation.
Models screened
20
live candidates
Evaluation feeds
7
task-weighted
Winner coverage
38%
intended feed weight
Largest provider share
3 of 8
Anthropic
Sources evaluated
The task sets these weights before any model is scored.
| Evaluation feed | Weight | Winner result | Field measured |
|---|---|---|---|
| LongBench v2unavailable | 25% | feed unavailable | 0/20 |
| FACTS Groundingunavailable | 20% | feed unavailable | 0/20 |
| context length | 15% | 50/100 | 20/20 |
| LMArena Document | 15% | not measured | 8/20 |
| LMArena Long Query | 10% | #30 | 18/20 |
| LMArena Search | 10% | not measured | 4/20 |
| OpenRouter usage | 5% | 88/100 | 20/20 |
Provider concentration
Each exact model is scored separately; provider identity is not a ranking input.
- Anthropic3 models
- deepseek1 model
- Google1 model
- OpenAI1 model
- Qwen1 model
- xiaomi1 model
Decision table
Every published model is shown in computed order. Practitioner sources are distinct community threads, not the citations repeated in the prose below.
| Rank | Model | Relative score | Coverage | Practitioner evidence | Strongest measured reason |
|---|---|---|---|---|---|
| 01 | Qwen3.8 27BQwen | 49 | 38% | 10 threads · 8 families · 2 cautions | #30 LMArena Long Query |
| 02 | Claude Opus 4.6Anthropic | 45 | 42% | no linked practitioner threads | #1 LMArena Long Query · #2 LMArena Document |
| 03 | Claude Fable 5Anthropic | 45 | 42% | 1 threads · 1 families · 0 cautions | #3 LMArena Document · #3 LMArena Long Query |
| 04 | GPT-5.6 SolOpenAI | 44 | 42% | 1 threads · 1 families · 0 cautions | #1 LMArena Search · #6 LMArena Document |
| 05 | Claude Sonnet 4.6Anthropic | 43 | 42% | 1 threads · 1 families · 0 cautions | #5 LMArena Document · #15 LMArena Long Query |
| 06 | DeepSeek V4 Flash 0423deepseek | 43 | 38% | 3 threads · 2 families · 1 cautions | #47 LMArena Long Query |
| 07 | MiMo-V2.5-Proxiaomi | 43 | 38% | no linked practitioner threads | #13 LMArena Long Query |
| 08 | Gemini 3.6 FlashGoogle | 42 | 38% | 2 threads · 2 families · 1 cautions | #19 LMArena Long Query |
Relative score combines normalized benchmark quality and signal coverage; independent practitioner evidence and freshness are bounded tie-breakers. It is an ordering score, not an absolute quality percentage. The writing model receives this order and cannot change it.
Qwen3.8 27B offers a native 262K context window extensible to 1M tokens with hybrid attention that makes long-context inference cheaper than typical 27B models, though arithmetic bugs can crash long runs on locally pinned windows.
Best when: Use for single-GPU long-context serving where the hybrid attention (48 SSM layers, 17 full attention layers) reduces KV cache pressure compared to dense alternatives.
Tips
- Use for single-GPU long-context serving where the hybrid attention (48 SSM layers, 17 full attention layers) reduces KV cache pressure compared to dense alternatives.
- Deploy on H200 for 122 tok/s single-caller throughput at full 262K context with queue-friendly single-GPU occupancy.
- Run 60K context retrieval workloads with verified functional correctness on vLLM with NVFP4 compression for 1.6× faster decode.
Watch out for
- Watch for arithmetic bugs that kill long runs at iteration 6 when using Ollama with pinned windows above 32K tokens.
- Expect context condensation failures at ~181K tokens in llama.cpp when the reported token count exceeds available context size.
- Accept small quality degradation with NVFP4 compression (q_avg 92.5 vs 93.2, HumanEval 89.6 vs 94.5) versus GGUF Q4_K_M.
Claude Opus 4.6 ranks second on LMArena's document arena for blind preference on document tasks.
Best when: Select when LMArena document-task preference rankings align with your quality requirements, as it scores 1510 on that benchmark.
Tips
- Select when LMArena document-task preference rankings align with your quality requirements, as it scores 1510 on that benchmark.
Claude Fable 5 ranks third on LMArena's document arena and has been evaluated on 250K+ token poetry collections for long-context coherence.
Best when: Use for long-form creative analysis where models must process 250K+ token unstructured collections and identify chronological patterns.
Tips
- Use for long-form creative analysis where models must process 250K+ token unstructured collections and identify chronological patterns.
- Select when LMArena document-task preference rankings matter, as it scores 1504 on that benchmark.
GPT-5.6 Sol ranks eighth on LMArena's document arena but has a context window reporting discrepancy where some endpoints advertise 272K instead of 372K tokens.
Best when: Select when LMArena document-task preference rankings matter, as it scores 1479 on that benchmark.
Tips
- Select when LMArena document-task preference rankings matter, as it scores 1479 on that benchmark.
Watch out for
- Verify actual context window in your deployment, as discovery may report 272K instead of the expected 372K tokens.
Claude Sonnet 4.6 ranks sixth on LMArena's document arena and achieved 0.851-0.867 scores on documentation evaluation across multiple server backends.
Best when: Deploy for documentation-heavy RAG where Mintlify MCP server integration yielded 0.867 evaluation scores.
Tips
- Deploy for documentation-heavy RAG where Mintlify MCP server integration yielded 0.867 evaluation scores.
- Select when LMArena document-task preference rankings inform your decision, as it scores 1483 on that benchmark.
DeepSeek V4 Flash 0423 powers production agentic workflows with map/reduce compilation over 30-50 document gates, though latency at 64K context reaches 2+ minutes.
Best when: Use for cost-sensitive agentic RAG where it delivers acceptable hallucination rates and correct information retrieval as the cheapest viable option in the stack.
Tips
- Use for cost-sensitive agentic RAG where it delivers acceptable hallucination rates and correct information retrieval as the cheapest viable option in the stack.
- Deploy for document compilation workflows with progressive UAT from 10-doc smoke tests to full corpus processing.
Watch out for
- Expect 2+ minute time-to-first-token at 64K agentic context (450 tok/s prefill), which compounds with larger contexts.
MiMo-V2.5-Pro ranks 15th on LMArena's long-query category for blind human preference on longer prompts.
Best when: Select when LMArena long-query preference rankings inform your decision, as it scores 1482 Elo on that benchmark.
Tips
- Select when LMArena long-query preference rankings inform your decision, as it scores 1482 Elo on that benchmark.
Gemini 3.6 Flash ranks 21st on LMArena's long-query category and offers competitive tok/s latency at high context, though free tier limits to 20 requests per day per project.
Best when: Use when tok/s latency with high context uptime matters more than peak benchmark scores, as reported by production users.
Tips
- Use when tok/s latency with high context uptime matters more than peak benchmark scores, as reported by production users.
Watch out for
- Avoid free tier for multi-turn RAG bots, as 20 requests per day per project limits practical use to ~10 messages with 2 LLM calls each.
Frequently asked
- What is the top-ranked model for RAG & Long Context?
- Qwen: Qwen3.8 27B ranks first in the current evidence-weighted comparison. Use for single-GPU long-context serving where the hybrid attention (48 SSM layers, 17 full attention layers) reduces KV cache pressure compared to dense alternatives.[1][2]
- What should I watch out for with Qwen: Qwen3.8 27B?
- Watch for arithmetic bugs that kill long runs at iteration 6 when using Ollama with pinned windows above 32K tokens.[3]
- What is an alternative to Qwen: Qwen3.8 27B?
- Anthropic: Claude Opus 4.6 is the next-ranked option. Select when LMArena document-task preference rankings align with your quality requirements, as it scores 1510 on that benchmark.[4]
Sources
- 1
“## 64K context: fits with room to spare — 16K was over-cautious Parsed the full GGUF metadata header (51 KV pairs) instead of estimating. The KV geometry makes long context far cheaper on this model than on a normal 27B. ### Architecture (from GGUF metadata, `general.architecture = qwen35`) | key | value | |---|---| | `qwen35.block_count` | 65 | | `qwen35.full_attention_interval` | **4** — so only every 4th layer is full attention → **17 attention layers**, 48 SSM | | `qwen35.attention.head_cou…”
changtimwu · GitHub · Aug 19, 2026 - 2
“Qwen3.8-27B is a 27B dense hybrid-attention model with a vision tower and an MTP draft head. It fits one H200 GPU at the full 262144 context, so it queues faster than any recipe that needs a whole node. | | Measured | | --- | --- | | One caller | 122.1 tok/s | | Concurrency 1024 | 3957.3 tok/s, still rising | | Context | 262144, the checkpoint maximum | | KV cache | 1,804,253 tokens, 6.88 full-length requests at once | | Hardware | 1 H200 GPU, TP1, no NCCL | Scope - New recipe at recipes/Qwen3.…”
mmshad · GitHub · Aug 26, 2026 - 3
“Split out from the #464/#469 family: those were message-shape bugs and are fixed (wire-verified). This one is arithmetic, and it kills every long run on a locally pinned window. ## Repro - Ollama, Qwen3.8 27B derived model with `PARAMETER num_ctx 32768` baked, served window correctly discovered since d1df0ada. - `flat-coder` blueprint (91% conversation budget), a ~368k-token code corpus, ordinary read-heavy investigation, main @6eed3db7. - Dies at iteration 6, every time, with the Ollama 500 `{…”
GEMISIS · GitHub · Aug 17, 2026 - 4
“Ranks #2 of 29 on LMArena's document arena (score 1510), based on blind preference for document tasks.”
LMArena document arena · Benchmark · Jul 30, 2026 - 5
“Hardware: 1xRTX 5090 32 GB (SM 12.0), driver 595.84, CUDA 13.2 toolkit Engine: vLLM 0.27.1 (pip) + FlashInfer 0.6.17, torch 2.13.0+cu130 Model: Qwen3.8-27B, compressed-tensors NVFP4 (W4A16, group 16), MTP speculative head in BF16 ### Quality functional correctness verified (math, tool calling, 60K context retrieval, multi-turn); NVFP4 carries a small quality tax vs GGUF Q4_K_M on evals (independent measurement: q_avg 92.5 vs 93.2, HumanEval 89.6 vs 94.5) — the trade for ~1.6x faster decode than…”
marco9899 · Hugging Face · Aug 18, 2026 - 6
“**Describe the bug** When using Qwen3.8:27B running in llama.cpp as an openai compatible endpoint, Although I have the max context window set to 262144 in the server, when I attempt to condense context at 181.1k tokens used, I get an error for `Failed to condense context - (304897 tokens) exceeds the available context size 262144 tokens` **To Reproduce** Steps to reproduce the behavior: 1. Run Qwen3.8:27B server via llama.cpp using the following command: 2. setup OpenAI Compatible provider to c…”
RemainIndoors1 · GitHub · Aug 22, 2026 - 7
“So, in the past I've shared that I evaluate AI models by feeding them my ever-growing large collection of personal poems that span well over 800 poems (1000 depending on how you count) and over 250k tokens. What I do is feed it some initial prompt asking it to simply discuss what can be said when faced with this unedited, unseen collection of poetry. I ask the model to evaluate who the author is (or claims to be), what they went through in life, if there are different chronological poetic "phas…”
jorl17 · Hacker News · Jun 9, 2026 - 8
“Ranks #3 of 29 on LMArena's document arena (score 1504), based on blind preference for document tasks.”
LMArena document arena · Benchmark · Jul 30, 2026 - 9
“Ranks #8 of 29 on LMArena's document arena (score 1479), based on blind preference for document tasks.”
LMArena document arena · Benchmark · Jul 30, 2026 - 10
“Reproduced. With `fetchCodexModels` fed a payload where `gpt-5.6-sol` reports `context_window: 272000`, discovery yields `contextWindow = 272000` (expected 372000). The `?? fallbackContextWindow` at `src/discovery/codex.ts:233` only applies when the field is absent, so an actively-reported 272000 passes through untouched. Fix: override to 372K for the 5.6 SKUs rather than only falling back on absence.”
roboomp · GitHub · Jul 22, 2026 - 11
“# 📊 Documentation Evaluation Report **Run ID**: 20260726-074322 **Timestamp**: 2026-07-26T07:43:22.470933+00:00 **Total Evaluations**: 200 **Servers**: foundry-docs-vnext, foundry-docs, microsoft-learn, mintlify-hosted **Models**: claude-sonnet-4.6, gpt-5.4 --- ## Scoreboard: Server × Model Matrix | Server | claude-sonnet-4.6 | gpt-5.4 | Average | |---|---|---|---| | Mintlify MCP (Control B) 🥇 | 0.867 | 0.865 | **0.866** | | FastMCP docs-vnext/ (Treatment) 🥈 | 0.851 | 0.865 | **0.858** | | M…”
github-actions[bot] · GitHub · Jul 26, 2026 - 12
“Ranks #6 of 29 on LMArena's document arena (score 1483), based on blind preference for document tasks.”
LMArena document arena · Benchmark · Jul 30, 2026 - 13
“I've been extremely impressed with DeepSeek V4 flash. We've been working on a project which can be thought of as an agent, just not for coding. So we've been building everything: agents, sub-agents, RAG, dynamic intent detection, changing models based on what's being done, etc. In our tests, DeepSeek V4-flash is the cheapest model with acceptable replies (few hallucinations, while finding the right information). It's not the cheapest one we run overall (we're actually surviving with 3B models f…”
jorl17 · Hacker News · May 22, 2026 - 14
“## Destination Fork personal-mode compile loop re-plumbed to run through **pi** as its LLM backend (A段: pi as stateless executor, serial; B段: orchestration moves inside pi as a TS workflow via the workflow extension, 2-lane parallel map/reduce), with the map/reduce prompts ported to **deepseek/deepseek-v4-flash**. Proven by progressive UAT: UAT-2 sample (~10-doc smoke → 30–50-doc gate) then UAT-1 full corpus, ending in a **visitable wiki** (web view + Q&A over compiled output). Execution is car…”
proletariat64 · GitHub · Aug 16, 2026 - 15
“> From GLM 4.7 flash GLM 4.7 Flash is a 30b model that was far behind SOTA at launch, and I know that because I pay for z.ai inference and have run the model locally. Qwen and Deepseek V4 Flash have the same issue, and beg the question; are you really going to process a 64k agentic context at 450tok s? That's 2+ minutes that you spend waiting for the first token to generate! Of course nobody can sell that as competitive inference, and it only gets worse with larger models. We're talking about n…”
bigyabai · Hacker News · Jun 25, 2026 - 16
“Ranks #15 of 144 on LMArena's long-query category (Elo 1482), based on blind human preference for longer prompts.”
LMArena long-query category · Benchmark · Sep 1, 2026 - 17
“Agree, with you but I'm still using 3.6 Flash because of tok s latency uptime with high context. Tried Grok 4.6 and it was scoring lower on some internal benchmarks or slower.”
cahaya · Hacker News · Aug 13, 2026 - 18
“## 문제 요약 `gemini-3.6-flash` 무료 티어가 프로젝트+모델 기준 하루 20건 요청으로 제한되어 있는데(`GenerateRequestsPerDayPerProjectPerModel-FreeTier`), 메시지 하나당 LLM을 최소 2번(1차 긴급도 판단 + RAG 재검색 후 2차 최종 판단) 호출하므로 실질적으로 하루 메시지 10개 정도면 한도가 참. 실사용 메신저 봇으로는 감당 불가. ## 에러 로그 ## 해결 방안 (적용함) 채팅 생성 LLM을 Gemini에서 **Groq**(`llama-3.3-70b-versatile`)로 전환. Groq 무료 티어는 분당 요청수 기준이라 하루 단위로 막히는 지금 상황보다 훨씬 넉넉함. - `app/pipelines/shared/llm.py` 신규 — `get_chat_llm()` 공용 팩토리 - `realtime_graph.py`(3곳), `feedback_graph.py`(2곳), `memory_gc_graph.py`(1곳)…”
oh0227 · GitHub · Jul 24, 2026
Rankings synthesized from community evidence and open benchmarks. See methodology. Not driven by vendor marketing.