Nous Research Releases Arete-27B that Cuts Qwen3.8-27B Propaganda From 22% to 0.9%
AIvia MarkTechPost
Quick Read
Nous Research has released Arete-27B, a supervised fine-tune of Alibaba’s Qwen3.8-27B. Arete-27B is a 27B open-weight model that strips Chinese state framing from Qwen’s answers. On Nous’s own 803-prompt eval, answers containing propaganda fell from 22% to 0.9%. Developers get Qwen-class agentic skill without state-aligned answers. The weights, eval and training data are all public. TL;DR Size: 27B dense parameters (all active). Trained at 64k context. Base Qwen3.8-27B supports 262,144 tokens natively. Runs on: BF16 weights are 53.81 GB. A community Q4_K_M quant is 17.20 GB, sized for 24 GB consumer GPUs. Performance: Capability unchanged on the benchmarks Nous ran; political framing now matches Western models. Best: Propaganda rate 0.9%, down from 22% for base Qwen3.8-27B. Worst: Fact score 2.98 of 5; 216 of 803 answers still score 0. What is Arete-27B? Arete-27B is Qwen3.8-27B after supervised fine-tuning only. Nous used no RL, no preference optimization and no architecture changes. The license is Apache 2.0, matching the base model. Nous team states that it saw refusals, deflections and CCP-framed answers on sensitive topics in the base model. It picked Qwen3.8-27B because it was the most capable local model and showed the target behavior. How was Arete-27B trained? The teacher was Inkling, an open model from Thinking Machines. Nous chose an open teacher to avoid terms-of-use limits on distillation. It rejected Nemotron 3 Ultra after finding a checkered record on China topics. Training covered about 52.8 million tokens at 64k context, without sequence packing. The run took 2 epochs and 1,202 steps at batch size 48. It used AdamW, a 1e-5 peak learning rate and tensor parallel 4. All language-model weights were trained. The vision encoder stayed frozen, since the data is text-only. The multi-token-prediction head was disabled. The released arete-data-pairs set holds 34,878 prompts across 7 topics. Each prompt has both an Inkling answer and a Qwen answer. 29,236 accepted teacher answers form the SFT split. How does the propaganda eval work? The Propaganda_Eval set has 803 prompts across 7 value categories. These include freedom of expression, state ideology, data privacy and democracy. Each prompt carries 2 to 4 required facts from a cited source. A Grok 4.7 judge grades two things separately: Fact mean (0 to 5): how many required facts the answer states. One opposed fact scores 0. Any propaganda: share of answers that adopt the state’s account as their own. The judge must quote the exact phrase to flag propaganda. Nous removed every training prompt that overlapped the eval. The eval is MIT licensed and includes grading scripts. window.addEventListener("message",function(e){if(e.data&&e.data.mtpArete){var f=document.getElementById("mtp-arete-frame");if(f)f.style.height=e.data.mtpArete+"px";}}); What do the benchmarks show? On the model card page, Arete-27B raised fact mean from 2.32 to 2.98. Propaganda fell from 22% to 0.9%. Answers scoring 0 dropped from 359 to 216 of 803. The biggest change came in freedom of expression. Fact score rose from 0.89 to 2.44, and propaganda fell from 60% to 1%. Capability held on Nous’s tests. Hermes Bench moved from 65.77% to 65.60%, which Nous calls noise. Terminal-Bench-2.1 rose from 67% to 70% in Nous’s harness. Nous does not claim credit for that gain. How does Arete-27B compare? Note: Arete and Qwen ran at temperature 0 in the Nous eval. Muse Glimmer and Gemma ran at 0.6. Terminal-Bench figures come from different harnesses. How can developers run it? Weights ship in BF16 on Hugging Face. No hosted inference provider serves the model yet. Community GGUF quants run in llama.cpp, LM Studio and Ollama, from 8.64 GB to 53.81 GB. The GGUF build also includes the image projector. Key Takeaways Arete-27B cuts Qwen3.8-27B propaganda answers from 22% to 0.9%. The fix took only 52.8M tokens of plain SFT, no RL. Hermes Bench held at 65.60% versus 65.77% for the base. Fact recall rose to 2.98 of 5 but still trails peers. Weights, eval and training data are open under Apache 2.0 and MIT. Check out the model weights, technical blog, evaluation set and training data. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Nous Research Releases Arete-27B that Cuts Qwen3.8-27B Propaganda From 22% to 0.9% appeared first on MarkTechPost.
Nous Research has released Arete-27B, a supervised fine-tune of Alibaba’s Qwen3.8-27B. Arete-27B is a 27B open-weight model that strips Chinese state framing from Qwen’s answers. On Nous’s own 803-prompt eval, answers containing propaganda fell from 22% to 0.9%. Developers get Qwen-class agentic skill without state-aligned answers. The weights, eval and training data are all public.
TL;DR
Size: 27B dense parameters (all active). Trained at 64k context. Base Qwen3.8-27B supports 262,144 tokens natively.
Runs on: BF16 weights are 53.81 GB. A community Q4_K_M quant is 17.20 GB, sized for 24 GB consumer GPUs.
Performance: Capability unchanged on the benchmarks Nous ran; political framing now matches Western models.
Best: Propaganda rate 0.9%, down from 22% for base Qwen3.8-27B.
Worst: Fact score 2.98 of 5; 216 of 803 answers still score 0.
What is Arete-27B?
Arete-27B is Qwen3.8-27B after supervised fine-tuning only. Nous used no RL, no preference optimization and no architecture changes. The license is Apache 2.0, matching the base model.
Nous team states that it saw refusals, deflections and CCP-framed answers on sensitive topics in the base model. It picked Qwen3.8-27B because it was the most capable local model and showed the target behavior.
How was Arete-27B trained?
The teacher was Inkling, an open model from Thinking Machines. Nous chose an open teacher to avoid terms-of-use limits on distillation. It rejected Nemotron 3 Ultra after finding a checkered record on China topics.
Training covered about 52.8 million tokens at 64k context, without sequence packing. The run took 2 epochs and 1,202 steps at batch size 48. It used AdamW, a 1e-5 peak learning rate and tensor parallel 4.
All language-model weights were trained. The vision encoder stayed frozen, since the data is text-only. The multi-token-prediction head was disabled.
The released arete-data-pairs set holds 34,878 prompts across 7 topics. Each prompt has both an Inkling answer and a Qwen answer. 29,236 accepted teacher answers form the SFT split.
How does the propaganda eval work?
The Propaganda_Eval set has 803 prompts across 7 value categories. These include freedom of expression, state ideology, data privacy and democracy. Each prompt carries 2 to 4 required facts from a cited source.
A Grok 4.7 judge grades two things separately:
Fact mean (0 to 5): how many required facts the answer states. One opposed fact scores 0.
Any propaganda: share of answers that adopt the state’s account as their own.
The judge must quote the exact phrase to flag propaganda. Nous removed every training prompt that overlapped the eval. The eval is MIT licensed and includes grading scripts.
On the model card page, Arete-27B raised fact mean from 2.32 to 2.98. Propaganda fell from 22% to 0.9%. Answers scoring 0 dropped from 359 to 216 of 803.
The biggest change came in freedom of expression. Fact score rose from 0.89 to 2.44, and propaganda fell from 60% to 1%.
Capability held on Nous’s tests. Hermes Bench moved from 65.77% to 65.60%, which Nous calls noise. Terminal-Bench-2.1 rose from 67% to 70% in Nous’s harness. Nous does not claim credit for that gain.
How does Arete-27B compare?
Note: Arete and Qwen ran at temperature 0 in the Nous eval. Muse Glimmer and Gemma ran at 0.6. Terminal-Bench figures come from different harnesses.
How can developers run it?
Weights ship in BF16 on Hugging Face. No hosted inference provider serves the model yet. Community GGUF quants run in llama.cpp, LM Studio and Ollama, from 8.64 GB to 53.81 GB. The GGUF build also includes the image projector.
Key Takeaways
Arete-27B cuts Qwen3.8-27B propaganda answers from 22% to 0.9%.
The fix took only 52.8M tokens of plain SFT, no RL.
Hermes Bench held at 65.60% versus 65.77% for the base.
Fact recall rose to 2.98 of 5 but still trails peers.
Weights, eval and training data are open under Apache 2.0 and MIT.
Check out the model weights, technical blog, evaluation set and training data. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Nous Research Releases Arete-27B that Cuts Qwen3.8-27B Propaganda From 22% to 0.9% appeared first on MarkTechPost.
Every Mixture-of-Experts (MoE) model repeats one decision for every token: which experts should process it. That decision belongs to the router. MoE routing strategies shape quality, load balance and serving cost in every major open model. This guide explains the 7 routing strategies used in production, and the models that use each one. TL;DR Size: Routers choose from 8 experts (Mixtral) to 384 (DeepSeek-V4-Pro), activating 1 to 8 per token. Runs on: Routing cost shows up in multi-GPU expert parallelism, where each choice becomes all-to-all network traffic. Performance: Sigmoid scoring with aux-loss-free bias balancing is the dominant 2026 recipe for frontier open MoE models. Best: Loss-free balancing cut MaxVio from 0.72 to 0.04 at 1B, with lower perplexity. Worst: Expert choice leaks future tokens in causal models; one test showed a roughly 10% abnormal loss drop. Bottom line: Best thing: bias-based balancing adds no interfering gradients. Worst thing: no router removes the need for serving-time load fixes. What is MoE routing? MoE routing is the mechanism that sends each token to a small subset of expert networks. A router scores every expert, keeps the top candidates and weights their outputs. Only the selected experts run. Mixtral 8x7B, for example, gives each token access to 47B parameters but uses 13B active parameters. The 7 MoE routing strategies used in production 1. Softmax top-k token choice Each token scores all experts with a linear layer and keeps the k highest. A softmax over those k scores sets the mixing weights. Mixtral picks 2 of 8 experts. gpt-oss-120b picks 4 of 128. Qwen3-235B-A22B picks 8 of 128. An auxiliary loss usually keeps expert load even. Used by: Mixtral, gpt-oss, Qwen3. 2. Top-1 switch routing with a capacity factor Switch Transformer showed that routing each token to 1 expert preserves quality and cuts router compute. Each expert gets a fixed buffer: tokens per batch divided by experts, times a capacity factor. Overflow tokens skip the layer through the residual connection. Llama 4 Maverick also sends each token to 1 routed expert, plus a shared expert. Used by: Switch Transformer, Llama 4 Maverick (with a shared expert). 3. Shared plus routed experts DeepSeekMoE splits each layer into always-on shared experts and sparsely routed experts. The shared expert captures common knowledge, so routed experts can specialize. DeepSeek-V3 uses 1 shared and 256 routed experts, activating 8. Kimi K2, GLM-4.5 and Qwen3.6-35B-A3B follow the same pattern. Qwen3 went the other way and removed shared experts. Used by: DeepSeek-V3 and V4, Kimi K2, GLM-4.5, Qwen3.6, Llama 4. 4. Sigmoid gating with aux-loss-free bias DeepSeek-V3 scores experts with a sigmoid instead of a softmax. It adds a per-expert bias only when choosing the top-k. After each step, the bias drops for overloaded experts and rises for idle ones. Mixing weights still come from the raw scores. GLM-4.5 uses loss-free balance routing with sigmoid gates. DeepSeek-V4 switches the score to sqrtsoftplus but keeps bias balancing. Used by: DeepSeek-V3, DeepSeek-V4, GLM-4.5, Kimi K2. 5. Group-limited and node-limited routing Sending one token to experts on many nodes is expensive. DeepSeek-V3 caps each token at 4 nodes, chosen by their highest affinity scores. Its config splits experts into 8 groups and keeps 4 before the final top-8 pick. DeepSeek says this cap lets training nearly fully overlap computation and communication. Kimi K2 drops grouping entirely, with n_group set to 1. Used by: DeepSeek-V2 and V3. 6. Expert choice routing Google’s expert choice routing flips the direction. Each expert picks its top-k tokens, so every expert gets a fixed bucket size. Tokens can receive a variable number of experts. The paper reports more than 2x faster training convergence. The catch is causality: a token’s expert can depend on later tokens. DeepSeek researchers measured an abnormal loss drop of about 10% from this leakage. Used by: Research and non-causal settings. None of the 10 models in our companion table uses it. 7. Hash routing and dynamic top-k Hash routing assigns experts from the token ID, with no learned router. DeepSeek-V4 uses hash routing in its first 3 MoE layers. Dynamic top-k lets the number of active experts vary per token. SMIDT built it into SGLang and improved P99 time-to-first-token by 1.21 to 1.26x. Used by: DeepSeek-V4 (hash routing), research serving systems (dynamic top-k). MoE routing strategies compared (function(){window.addEventListener('message',function(e){if(e.data&&e.data.mtpEmbed==='r1'&&e.data.h){var f=document.getElementById('mtp-r1-frame');if(f)f.style.height=e.data.h+'px';}});})(); Which MoE routing strategy should you use? Most teams do not pick a router. They inherit one when they choose a checkpoint. Before serving, read 4 fields in config.json: expert count, experts per token, shared experts and topk_method. Changing top-k at inference changes the model itself, so treat it as a quality experiment. For new pretraining runs, sigmoid scoring with bias balancing is the current open default. Key Takeaways Top-k token choice is the baseline; k ranges from 1 (Llama 4) to 8 (DeepSeek-V3, Qwen3). Shared experts are standard in DeepSeek, Kimi, GLM and Qwen3.6; Qwen3 dropped them. Sigmoid plus bias balancing is the 2026 default for frontier open MoE models. Expert choice balances perfectly but leaks future tokens in causal LLMs. DeepSeek-V4 adds hash routing to its first 3 MoE layers. Check out the primary sources: Switch Transformer, Expert Choice, Loss-Free Balancing, DeepSeek-V3 report and the DeepSeek-V4 model card. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide) appeared first on MarkTechPost.
In this tutorial, we work with Dory 1.0 4B, a small reasoning model from NVIDIA’s Nemotron Labs whose middle layers form a shared block that can be applied once, twice, three, or four times for every token. Each extra pass, or loop, spends more computation per token without adding a single parameter, so one checkpoint can run at four different compute budgets with no retraining. We build the llama.cpp branch that NVIDIA’s model card pins, download the 3.7 GB GGUF, and read the architecture straight from the file to predict what each loop should cost in parameters and cache memory. We then check those predictions against the running server, measure decode speed at every loop count, test whether the cheaper last-loop cache changes the model’s predictions, and run a paired GSM8K evaluation to see how accuracy, answer length, and time per solved question move as the loops increase. Copy CodeCopiedUse a different Browserimport os import re import sys import json import time import math import random import signal import shutil import hashlib import platform import statistics import traceback import subprocess import collections import urllib.request from pathlib import Path RESULTS = {} # step -> one-line result, for the summary table CLAIMS = [] # (step, claim, held) CURRENT = ["1"] def banner(title): print("n" + "=" * 78) print(title) print("=" * 78) def section(name): def wrap(fn): def run(*a, **kw): CURRENT[0] = name.split(".")[0] banner(name) t0 = time.time() try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}" print(f"n[!] {name} did not complete: {type(e).__name__}: {e}") traceback.print_exc(limit=3) return None finally: print(f"n[{time.time() - t0:.1f} s]") return run return wrap def claim(text, held): CLAIMS.append((CURRENT[0], text, bool(held))) return held banner("1. Build the pinned Dory llama.cpp fork and record the environment") FORK = "" FORK_COMMIT = "98a74c97ea09ac72f6ceda1559808fc0a2f447ce" # the commit NVIDIA's model card pins; ggml-org PR #30306 is still a draft GGUF_REPO = "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF" GGUF_REV = "df53755cb8741843af8e019e088819fbef17e29b" GGUF_FILE = "NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf" # the card's commands call it dory-q5_k_m.gguf GGUF_SIZE = 3_673_288_864 GGUF_SHA256 = "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0" GSM8K_REV = "740312add88f781978c0658806c59bc2815b9866" WORK = Path(os.environ.get("DORY_WORK") or ("/content/dory" if os.path.isdir("/content") else "dory_work")).resolve() WORK.mkdir(parents=True, exist_ok=True) (WORK / "logs").mkdir(exist_ok=True) subprocess.run([sys.executable, "-m", "pip", "install", "-q", "gguf==0.19.0", "huggingface_hub", "pandas", "pyarrow"], check=True) SYSTEM = platform.system() GPU = "" if shutil.which("nvidia-smi"): q = subprocess.run(["nvidia-smi", "--query-gpu=name,memory.total,compute_cap", "--format=csv,noheader"], capture_output=True, text=True) GPU = q.stdout.strip().splitlines()[0] if q.returncode == 0 and q.stdout.strip() else "" BACKEND = "metal" if SYSTEM == "Darwin" else ("cuda" if GPU else "cpu") SRC = WORK / "llama.cpp-dory" SERVER_BIN = SRC / "build" / "bin" / "llama-server" free_gb = shutil.disk_usage(WORK).free / 1e9 print(f"{SYSTEM} {platform.machine()}, Python {platform.python_version()}, {os.cpu_count()} CPU cores, " f"backend {BACKEND}{' (' + GPU + ')' if GPU else ''}, {free_gb:.1f} GB free disk") if not SERVER_BIN.exists(): t0 = time.time() if not (SRC / ".git").exists(): subprocess.run(["git", "init", "-q", str(SRC)], check=True) subprocess.run(["git", "-C", str(SRC), "remote", "add", "origin", FORK], check=True) subprocess.run(["git", "-C", str(SRC), "fetch", "-q", "--depth", "1", "origin", FORK_COMMIT], check=True) subprocess.run(["git", "-C", str(SRC), "checkout", "-q", "--detach", "FETCH_HEAD"], check=True) flags = ["-DCMAKE_BUILD_TYPE=Release", "-DLLAMA_BUILD_SERVER=ON", "-DLLAMA_BUILD_TESTS=OFF", "-DLLAMA_BUILD_EXAMPLES=OFF", "-DLLAMA_BUILD_APP=OFF", "-DLLAMA_BUILD_UI=OFF", "-DLLAMA_USE_PREBUILT_UI=ON", "-DLLAMA_OPENSSL=OFF"] if BACKEND == "metal": # the card's Apple-silicon flags flags += ["-DCMAKE_OSX_ARCHITECTURES=arm64", "-DGGML_METAL=ON", "-DGGML_METAL_EMBED_LIBRARY=ON", "-DGGML_BLAS=ON", "-DGGML_BLAS_VENDOR=Apple"] elif BACKEND == "cuda": # kernels for this GPU only (a T4 is 7.5); flash-attention kernels are unused with --flash-attn off cap = GPU.split(",")[-1].strip().replace(".", "") flags += ["-DGGML_CUDA=ON", f"-DCMAKE_CUDA_ARCHITECTURES={cap}", "-DGGML_CUDA_FA=OFF", "-DGGML_NATIVE=OFF"] env = dict(os.environ, HF_UI_VERSION="b11551") with open(WORK / "build.log", "w") as log: subprocess.run(["cmake", "-S", str(SRC), "-B", str(SRC / "build")] + flags, check=True, env=env, stdout=log, stderr=log) subprocess.run(["cmake", "--build", str(SRC / "build"), "--parallel", str(os.cpu_count()), "--target", "llama-server"], check=True, env=env, stdout=log, stderr=log) print(f"built llama-server in {time.time() - t0:.0f} s") head = subprocess.run(["git", "-C", str(SRC), "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip() assert head == FORK_COMMIT, f"checked out {head}, expected {FORK_COMMIT}" version = subprocess.run([str(SERVER_BIN), "--version"], capture_output=True, text=True) print(f"fork commit {head[:12]}; {(version.stdout + version.stderr).strip().splitlines()[-1]}") RESULTS["1. Build the pinned Dory llama.cpp fork and record the environment"] = f"fork {head[:12]}, backend {BACKEND}" We build llama-server from the exact commit of the Dory branch that NVIDIA’s model card pins, because Dory support in llama.cpp is still an open draft pull request and no released binary can load the model. The cell detects the platform and passes the card’s Metal flags on a Mac, or compiles CUDA kernels for the attached GPU alone on Colab, skipping the flash-attention kernels that the card’s –flash-attn off never uses, which keeps a T4 build manageable. It then checks that the checked-out commit is the pinned one, so every later number traces to one known version of the code. Copy CodeCopiedUse a different Browser@section("2. Download the Q5_K_M GGUF at a pinned revision and check its SHA-256") def download_step(): global GGUF from huggingface_hub import hf_hub_download t0 = time.time() GGUF = Path(hf_hub_download(GGUF_REPO, GGUF_FILE, revision=GGUF_REV, local_dir=str(WORK))) h = hashlib.sha256() with open(GGUF, "rb") as f: for block in iter(lambda: f.read(1 << 24), b""): h.update(block) size, digest = GGUF.stat().st_size, h.hexdigest() print(f"{GGUF.name}: {size:,} bytes, sha256 {digest[:16]}... ({time.time() - t0:.0f} s)") assert size == GGUF_SIZE and digest == GGUF_SHA256, "the file does not match the card's size and checksum" claim("the GGUF matches the size and SHA-256 published on the card", True) return f"{size / 1e9:.2f} GB, checksum matches the card" download_step() The Q5_K_M GGUF comes from Hugging Face at a fixed repository revision, and we hash it before anything else touches it. The card’s commands refer to the file as dory-q5_k_m.gguf, but the repository names it NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf, so copying the card’s command verbatim fails; pinning the real name, the size, and the SHA-256 together catches both a wrong file and a corrupted download. The 3.67 GB file matches the card’s published checksum exactly, so every measurement that follows uses the same bytes NVIDIA published. Copy CodeCopiedUse a different Browser@section("3. Read the architecture from the GGUF: three layer groups and a default of four loops") def architecture_step(): global META, GROUPS, PATTERN import gguf reader = gguf.GGUFReader(str(GGUF)) META = {k: f.contents() for k, f in reader.fields.items() if k.startswith("dory.")} for k in ("dory.block_count", "dory.input_layer_count", "dory.recurrent_layer_count", "dory.output_layer_count", "dory.num_loops", "dory.recurrent_kv_cache_mode", "dory.attention.sliding_window", "dory.context_length"): print(f"{k:34s} {META[k]}") PATTERN = META["dory.layer_pattern"].split("-")[:-1] # one symbol per attention layer: S sliding window, * full n_in, n_rec = META["dory.input_layer_count"], META["dory.recurrent_layer_count"] print(f"nlayer entries alternate attention and MLP: {len(PATTERN)} attention layers " f"({PATTERN.count('S')} sliding-window, {PATTERN.count('*')} full attention)") GROUPS = collections.defaultdict(lambda: {"params": 0, "bytes": 0}) dtypes = collections.Counter() for t in reader.tensors: m = re.match(r"blk.(d+).", t.name) if m: i = int(m.group(1)) g = "input block" if i < n_in else "recurrent block" if i 9s} {'bytes':>9s}") for g in ("embedding", "input block", "recurrent block", "output block", "LM head"): print(f"{g:16s} {GROUPS[g]['params'] / 1e9:7.3f} B {GROUPS[g]['bytes'] / 1e9:6.3f} GB") print(f"tensor types: {dict(dtypes)}") claim("the GGUF stores 16 input, 40 recurrent and 16 output layer entries (8, 20 and 8 blocks)", (n_in, n_rec, META["dory.output_layer_count"]) == (16, 40, 16)) claim("the GGUF's built-in loop count is 4, not the 2 the card recommends", META["dory.num_loops"] == 4) total = sum(v["params"] for v in GROUPS.values()) return (f"{total / 1e9:.2f} B parameters, {GROUPS['recurrent block']['params'] / 1e9:.2f} B of them in the shared " f"recurrent block; built-in num_loops = {META['dory.num_loops']}") architecture_step() The GGUF header tells us how the model is wired before we load it. Dory’s 72 layer entries split into an input group of 16, a shared recurrent group of 40 and an output group of 16, alternating attention and MLP, with one full-attention layer for every three sliding-window layers. Summing tensor sizes by group shows the recurrent block accounts for 1.84 of the model’s 3.98 billion parameters. The header also carries a trap: its built-in loop count is 4, while the card recommends 2, so a server started without an override silently runs the most expensive setting. Copy CodeCopiedUse a different BrowserLOOPS = (1, 2, 3, 4) CTX, UBATCH = 4096, 512 def predicted_kv(loops, mode, ctx=CTX, ubatch=UBATCH): """KV cache under the fork's allocation rule: per_loop keeps one cache per loop for every recurrent attention layer, last_loop keeps one in all. Full-attention layers hold ctx cells; sliding-window layers hold window + ubatch, padded to 256.""" n_in, n_rec = META["dory.input_layer_count"] // 2, META["dory.recurrent_layer_count"] // 2 rec, fixed = PATTERN[n_in:n_in + n_rec], PATTERN[:n_in] + PATTERN[n_in + n_rec:] copies = loops if mode == "per_loop" else 1 full = fixed.count("*") + copies * rec.count("*") swa = fixed.count("S") + copies * rec.count("S") swa_cells = min(ctx, -(-(META["dory.attention.sliding_window"] + ubatch) // 256) * 256) per_cell = 2 * META["dory.attention.head_count_kv"] * META["dory.attention.key_length"] * 2 # K and V, 2 bytes each return full, swa, (full * ctx + swa * swa_cells) * per_cell / 2**20 @section("4. Predict what each extra loop costs: parameters per token and KV-cache size") def cost_model_step(): global PER_TOKEN once = ("input block", "output block", "LM head") # the embedding is a one-row lookup, so it costs no matrix work PER_TOKEN = {} print(f"{'loops':>5s} {'params per token':>17s} {'vs 1 loop':>10s} {'weight bytes read':>18s} {'vs 1 loop':>10s}") for n in LOOPS: p = sum(GROUPS[g]["params"] for g in once) + n * GROUPS["recurrent block"]["params"] b = sum(GROUPS[g]["bytes"] for g in once) + n * GROUPS["recurrent block"]["bytes"] PER_TOKEN[n] = (p, b) print(f"{n:5d} {p / 1e9:15.2f} B {p / PER_TOKEN[1][0]:9.2f}x {b / 1e9:16.2f} GB {b / PER_TOKEN[1][1]:9.2f}x") print(f"npredicted KV cache at --ctx-size {CTX}, f16 keys and values, micro-batch {UBATCH}:") print(f"{'loops':>5s} {'per_loop: full + SWA layers':>28s} {'MiB':>7s} {'last_loop: full + SWA layers':>29s} {'MiB':>7s}") for n in LOOPS: a, b = predicted_kv(n, "per_loop"), predicted_kv(n, "last_loop") print(f"{n:5d} {a[0]:>21d} + {a[1]:22d} + {b[1]: 600: self.stop() raise RuntimeError(f"llama-server did not start; see {self.log}") time.sleep(0.5) self.load_s = time.time() - t0 text = self.log.read_text(errors="ignore") sizes = KV_LINE.findall(text) # the full-attention cache first, then the sliding-window cache self.kv_mib = sum(float(s[0]) for s in sizes) self.kv_layers = tuple(int(s[2]) for s in sizes) m = re.search(r"offloaded (d+)/(d+) layers", text) self.offloaded = f"{m.group(1)}/{m.group(2)}" if m else "n/a" return self def __exit__(self, *exc): self.stop() def stop(self): if self.proc.poll() is None: os.killpg(self.proc.pid, signal.SIGTERM) try: self.proc.wait(60) except subprocess.TimeoutExpired: os.killpg(self.proc.pid, signal.SIGKILL) self.proc.wait() def post(self, path, payload, timeout=1800): req = urllib.request.Request(f"", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=timeout) as r: return json.loads(r.read()) def split_reasoning(message): """The server returns the thinking in reasoning_content when it can parse it; otherwise it stays inside tags.""" content, reasoning = message.get("content") or "", message.get("reasoning_content") or "" m = re.search(r"(.*?)", content, re.S) if m: reasoning, content = reasoning + m.group(1), content.replace(m.group(0), "") return reasoning.strip(), content.strip() @section("5. Start the server, prove the loop override took effect, and catch the silent default") def override_step(): rows = [] for loops, mode in ((2, "per_loop"), (2, "last_loop"), (None, "per_loop")): with DoryServer(loops, mode) as s: want = predicted_kv(META["dory.num_loops"] if loops is None else loops, mode) rows.append((s.name, s.kv_layers, s.kv_mib, want, s.offloaded, s.load_s)) if loops == 2 and mode == "per_loop": r = s.post("/v1/chat/completions", {"messages": [{"role": "user", "content": "What is 17 multiplied by 23?"}], "max_tokens": 1024, "seed": 1, "cache_prompt": False}) reasoning, answer = split_reasoning(r["choices"][0]["message"]) print(f"2 loops, a first request: {r['usage']['completion_tokens']} tokens, " f"{len(reasoning.split())} words of reasoning, answer: {answer[:120]!r}") print(f" fields in the message: {sorted(r['choices'][0]['message'])}; " f"timings: {sorted(r.get('timings', {}))[:6]}...n") print(f"{'server':28s} {'KV layers (full, SWA)':>22s} {'KV MiB':>8s} {'predicted':>16s} {'offloaded':>10s} {'load s':>7s}") for name, layers, mib, want, off, load in rows: print(f"{name:28s} {str(layers):>22s} {mib:8.1f} {str(want[:2]) + f' {want[2]:.1f}':>16s} {off:>10s} {load:7.1f}") ok = all(layers == want[:2] for _, layers, _, want, _, _ in rows) claim("the --override-kv loop count and cache mode are applied (KV layer counts match the prediction)", ok) default = rows[-1] claim("without an override the server runs the GGUF's 4 loops (the most expensive setting)", default[1] == predicted_kv(4, "per_loop")[:2]) return f"override applied: {ok}; no override -> {default[1]} KV layers, {default[2]:.0f} MiB, i.e. 4 loops" override_step() Loop count and cache mode are read when the model loads, so each setting needs its own server process; the DoryServer helper starts one per configuration, waits for the health check, and parses the KV-cache lines from the server’s log. We compare the parsed layer counts with the prediction to prove an override actually took effect, rather than trusting the flag. We also start one server without any override as a negative control, and send one chat request to see how the server separates the model’s reasoning from its answer. Copy CodeCopiedUse a different BrowserCONFIGS = [(1, "per_loop"), (2, "per_loop"), (2, "last_loop"), (3, "per_loop"), (3, "last_loop"), (4, "per_loop"), (4, "last_loop")] SPEED_PROMPT = ("Explain, step by step, how a recurrent block that is applied several times per token changes the amount of " "computation a language model spends on each token, and what that means for memory and latency.") @section("6. Measure KV memory and decode speed for every loop count and cache mode") def sweep_step(): global SPEED SPEED = {} print(f"{'loops':>5s} {'mode':10s} {'KV MiB':>8s} {'predicted':>9s} {'prefill tok/s':>14s} {'decode tok/s':>13s} " f"{'decode vs 1 loop':>17s} {'params ratio':>13s}") for loops, mode in CONFIGS: with DoryServer(loops, mode) as s: runs = [s.post("/completion", {"prompt": SPEED_PROMPT, "n_predict": 192, "temperature": 0, "ignore_eos": True, "cache_prompt": False, "seed": 1})["timings"] for _ in range(3)] dec = statistics.median(t["predicted_per_second"] for t in runs) pre = statistics.median(t["prompt_per_second"] for t in runs) SPEED[(loops, mode)] = {"kv_mib": s.kv_mib, "kv_layers": s.kv_layers, "decode": dec, "prefill": pre, "decode_runs": [round(t["predicted_per_second"], 2) for t in runs]} base = SPEED[(1, "per_loop")]["decode"] print(f"{loops:5d} {mode:10s} {s.kv_mib:8.1f} {predicted_kv(loops, mode)[2]:9.1f} {pre:14.1f} {dec:13.1f} " f"{base / dec:16.2f}x {PER_TOKEN[loops][0] / PER_TOKEN[1][0]:12.2f}x") flat = len({round(SPEED[(n, 'last_loop')]['kv_mib'], 1) for n in (2, 3, 4)} | {round(SPEED[(1, 'per_loop')]['kv_mib'], 1)}) == 1 grows = SPEED[(1, "per_loop")]["kv_mib"] < SPEED[(2, "per_loop")]["kv_mib"] < SPEED[(3, "per_loop")]["kv_mib"] = len(GATE_PROMPTS) - 1) return f"seeded answers identical: {same}; {BACKEND} vs CPU top-1 agreement {agree}/{len(GATE_PROMPTS)}" gates_step() Comparisons between settings are only meaningful if the measurements repeat and the accelerated build computes the same model as the reference CPU path. Two chat requests with the same seed at the card’s sampling temperature must return identical messages, otherwise per-question comparisons would mix model behaviour with sampling noise. We then start a CPU-only server, with no layers offloaded, and compare the most likely next token on five prompts against the GPU or Metal server, because Dory support has not been reviewed upstream on every backend. Copy CodeCopiedUse a different Browserdef load_gsm8k(): import pandas as pd from huggingface_hub import hf_hub_download path = hf_hub_download("openai/gsm8k", "main/test-00000-of-00001.parquet", repo_type="dataset", revision=GSM8K_REV, local_dir=str(WORK / "gsm8k")) df = pd.read_parquet(path) df["gold"] = df.answer.str.extract(r"####s*([-d,.]+)")[0].str.replace(",", "").astype(float) return df def tv_distance(p, q): """Total-variation distance between two top-k next-token distributions, each renormalised over its own top k.""" zp, zq = sum(math.exp(v) for v in p.values()), sum(math.exp(v) for v in q.values()) return 0.5 * sum(abs(math.exp(p.get(t, -1e9)) / zp - math.exp(q.get(t, -1e9)) / zq) for t in set(p) | set(q)) @section("8. What an extra loop changes: next-token distributions on 50 prompts") def mechanism_step(): global GSM8K GSM8K = load_gsm8k() prompts = list(GSM8K.sample(50, random_state=11).question) configs = [(n, m, ub) for ub in (UBATCH, 16) for n in (1, 2, 4) for m in ("per_loop", "last_loop")] dist, lengths = {}, [] for n, m, ub in configs: with DoryServer(n, m, ubatch=ub) as s: got = [s.post("/completion", {"prompt": p, "n_predict": 1, "n_probs": 20, "temperature": 0, "cache_prompt": False}) for p in prompts] dist[(n, m, ub)] = [top_probs(r) for r in got] lengths = lengths or [r["timings"]["prompt_n"] for r in got] print(f"50 GSM8K questions as raw prompts, {min(lengths)} to {max(lengths)} tokens: one chunk at a {UBATCH}-token " f"micro-batch, {math.ceil(min(lengths) / 16)} to {math.ceil(max(lengths) / 16)} chunks at 16n") print(f"{'comparison':52s} {'same top-1':>10s} {'mean TV':>8s} {'max |dlogp|':>12s}") def compare(label, a, b): x, y = dist[a], dist[b] same = sum(max(p, key=p.get) == max(q, key=q.get) for p, q in zip(x, y)) tv = statistics.mean(tv_distance(p, q) for p, q in zip(x, y)) mx = max(abs(p[t] - q[t]) for p, q in zip(x, y) for t in set(p) & set(q)) print(f"{label:52s} {same:>7d}/50 {tv:8.4f} {mx:12.5f}") return same, tv, mx out = {} out["modes_1"] = compare("1 loop: per_loop vs last_loop", (1, "per_loop", UBATCH), (1, "last_loop", UBATCH)) for n in (2, 4): out[f"modes_{n}"] = compare(f"{n} loops: per_loop vs last_loop, one {UBATCH}-token chunk", (n, "per_loop", UBATCH), (n, "last_loop", UBATCH)) out[f"chunks_{n}"] = compare(f"{n} loops: per_loop vs last_loop, 16-token chunks", (n, "per_loop", 16), (n, "last_loop", 16)) print() out["1v2"] = compare("1 loop vs 2 loops (per_loop)", (1, "per_loop", UBATCH), (2, "per_loop", UBATCH)) out["2v4"] = compare("2 loops vs 4 loops (per_loop)", (2, "per_loop", UBATCH), (4, "per_loop", UBATCH)) out["1v4"] = compare("1 loop vs 4 loops (per_loop)", (1, "per_loop", UBATCH), (4, "per_loop", UBATCH)) claim("with the prompt in one chunk, last_loop gives the same next-token distribution as per_loop", all(out[f"modes_{n}"][2] == 0 for n in (1, 2, 4))) claim("split into small chunks, last_loop and per_loop disagree at 2 and 4 loops", all(out[f"chunks_{n}"][1] > 0 for n in (2, 4))) return (f"top-1 agreement 1 vs 4 loops {out['1v4'][0]}/50 (mean TV {out['1v4'][1]:.3f}); last_loop = per_loop in one chunk: " f"{all(out[f'modes_{n}'][2] == 0 for n in (1, 2, 4))}") mechanism_step() Next-token distributions show what a loop changes inside the model at almost no cost. For 50 GSM8K questions used as raw prompts, we request the top 20 next-token candidates under each setting and compare them by top-1 agreement and total-variation distance. The fork’s code implies a sharp prediction: last_loop overwrites a recurrent layer’s cache on every loop, so it can differ from per_loop only when earlier chunks of the prompt keep just their final-loop keys, which never happens when the whole prompt fits in one micro-batch. Repeating the probe with 16-token micro-batches tests exactly that. Copy CodeCopiedUse a different BrowserEVAL_CONFIGS = [(1, "per_loop"), (2, "per_loop"), (2, "last_loop"), (4, "per_loop"), (4, "last_loop")] N_QUESTIONS = 30 MAX_TOKENS = 2048 INSTRUCTION = "nnSolve the problem. End your reply with a last line of the form 'Final answer: '." def grade(text, gold): m = re.findall(r"Final answer:s*$?s*(-?[d,]*.?d+)", text) nums = m or re.findall(r"-?[d,]*.?d+", text) if not nums: return False try: return abs(float(nums[-1].replace(",", "")) - gold) 9s} {'95% CI':>13s} {'median tokens':>14s} {'tokens vs 1 loop [95% CI]':>27s} " f"{'s per solved':>13s}") STATS = {} for key in EVAL_CONFIGS: rows = EVAL[key] k, n = sum(r["correct"] for r in rows), len(rows) ratios = [r["tokens"] / q["tokens"] for r, q in zip(rows, ref)] boots = sorted(statistics.median(rng.choices(ratios, k=len(ratios))) for _ in range(2000)) per_solved = sum(r["wall_s"] for r in rows) / max(k, 1) STATS[key] = {"acc": k / n, "ratio": statistics.median(ratios), "ratio_ci": (boots[50], boots[1949]), "per_solved": per_solved, "tokens": statistics.median(r["tokens"] for r in rows)} lo, hi = wilson(k, n) print(f"{key[0]} loops {key[1]:9s} {100 * k / n:8.1f}% {lo:5.1f}-{hi:5.1f}% {STATS[key]['tokens']:14.0f} " f"{STATS[key]['ratio']:12.2f} [{boots[50]:.2f}, {boots[1949]:.2f}] {per_solved:12.1f}") print("npaired accuracy contrasts (McNemar exact; 'only A' = questions only the first setting solved)") pairs = [((2, "per_loop"), (1, "per_loop")), ((4, "per_loop"), (1, "per_loop")), ((4, "per_loop"), (2, "per_loop")), ((2, "last_loop"), (2, "per_loop")), ((4, "last_loop"), (4, "per_loop"))] for a, b in pairs: oa, ob, p = mcnemar([r["correct"] for r in EVAL[a]], [r["correct"] for r in EVAL[b]]) print(f" {a[0]} {a[1]:9s} vs {b[0]} {b[1]:9s}: only A {oa:2d}, only B {ob:2d}, p = {p:.3f}") STATS[(a, b)] = (oa, ob, p) fewer = STATS[(4, "per_loop")]["ratio_ci"][1] sp[(1, "per_loop")]["kv_mib"]), ("Loop count trades compute per token without a model swap", f"decode {sp[(1, 'per_loop')]['decode']:.1f} -> {sp[(4, 'per_loop')]['decode']:.1f} tok/s, same file", sp[(4, "per_loop")]["decode"] < sp[(1, "per_loop")]["decode"]), ("More loops solve tasks with fewer generated tokens", f"GSM8K median ratio at 4 loops {s[(4, 'per_loop')]['ratio']:.2f} [{s[(4, 'per_loop')]['ratio_ci'][0]:.2f}, " f"{s[(4, 'per_loop')]['ratio_ci'][1]:.2f}]", s[(4, "per_loop")]["ratio_ci"][1] 0.05), ] print(f"{'claim (model card)':60s} {'this run':48s} verdict") for text, found, ok in rows: print(f"{text:60s} {found:48s} {'holds' if ok else 'does not hold'}") claim(text, ok) print(f"nnot testable here: the benchmark table (SWE-bench, Terminal-Bench, GPQA ...), the 256K context, NVFP4 pre-training;" f"nwith {N_QUESTIONS} paired questions only large accuracy differences are resolvable") return f"{sum(bool(ok) for *_, ok in rows)} of {len(rows)} card claims hold in this run" claims_step() Rather than restate the model card, we check its claims against this run. Each row pairs a sentence from the card with the measurement that tests it: cache-size claims and the loop defaults to token efficiency and the accuracy of last-loop caching, and the verdict comes from the saved results rather than being typed by hand. Claims the notebook cannot reach, such as the benchmark table, the 256K context and the NVFP4 pre-training, are listed as untested rather than quietly dropped. Copy CodeCopiedUse a different Browserbanner("SUMMARY") for name, res in RESULTS.items(): print(f" {name:<100s} {res}") print("nchecks decided in this run:") for step, text, held in CLAIMS: print(f" [{'held' if held else 'did not hold'}] step {step}: {text}") failed = [k for k, v in RESULTS.items() if str(v).startswith("SKIPPED")] print(f"n{sum(h for *_, h in CLAIMS)} of {len(CLAIMS)} checks held; steps failed or skipped: {failed or 'none'}") print(f"llama.cpp fork {FORK_COMMIT[:12]}, GGUF @ {GGUF_REV[:7]}, backend {BACKEND}{' ' + GPU if GPU else ''}, " f"Python {platform.python_version()}") The summary prints each step’s one-line result and every check the run decided, marking which passed and which failed, along with the fork commit, the GGUF revision, and the backend, so the notebook doubles as a record of exactly what was verified and on which hardware. In conclusion, Dory turns test-time compute into a load-time setting: one 3.7 GB file runs as four different models, from a single pass through its 20-block recurrent core to four, and the GGUF header alone predicts what each setting costs in parameters and cache memory. The notebook checks those predictions against the server, measures speed, tests the last-loop cache against a precise prediction from the code, and evaluates accuracy and answer length on a paired GSM8K sample. Two practical lessons hold whatever the measurements show: always set the loop count explicitly, because the GGUF defaults to four, and pin the fork commit, because Dory support in llama.cpp is still an open draft. Check out the FULL CODES here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post How to Run NVIDIA Dory 4B with llama.cpp: Loop by Loop Guide appeared first on MarkTechPost.
Despite the tech selloff, Anthropic is pressing ahead with its IPO at a valuation of $1.25 to $1.75 trillion, well below the hoped-for $2 trillion. The Claude developer aims to raise $80 to $100 billion before Thanksgiving. Rival OpenAI has also recently had to cut its valuation. The article Anthropic pushes ahead with pre-Thanksgiving IPO despite tech selloff, accepts lower valuation appeared first on The Decoder.
