Stanford and UC Berkeley researchers released TokenGate, a module for long-context prefill. TokenGate learned token pruning lets an existing LLM skim its input. Only the tokens that matter get full processing. The research team reports 1.4x to 11.3x faster time-to-first-token (TTFT) with no accuracy loss. Peak prefill memory drops 1.5x to 3.4x. Code and checkpoints are on GitHub under Apache-2.0. TL;DR MLP gate: A small MLP gate and LoRA (r=16) added to Qwen3-1.7B or Qwen3.5-2B. Tested on contexts up to 128k tokens. Runs on: Paper timings use 1 A100 at batch size 1. Training used 4 A100s. Performance: Pareto-optimal on 5 tasks against 6 baselines. Best: NIAH 16k at 11.3x faster TTFT and 3.42x lower memory. TTFT gain hits 13.0x at 128k. Worst: Speedup drops to 1.4x on some tasks. What is TokenGate? TokenGate is a drop-in component that adds trainable token pruning to pretrained LLMs. Transformers push every token through every layer. Attention cost grows quadratically with length. Training-free pruning is cheap but loses accuracy at high pruning rates. Architectural changes often need pre-training from scratch. How does TokenGate learned token pruning work? The forward pass splits into 3 phases: Standard forward: layers below gate layer L process all N tokens as usual. Gate module: an MLP scores each token from its post-attention residual at layer L. A top-K selector keeps the highest-scoring tokens. Pruned forward: layers above L process only those K tokens and never create KV entries for pruned ones. Only the document part of the prompt is prunable. The question is never pruned. What is token-gated attention? Top-K selection is not differentiable, which blocks end-to-end training. TokenGate borrows the Mixture-of-Experts router trick. Each kept token’s score becomes a bias on attention logits above the gate. That bias gives gradients a path back to the scorer. A learnable per-head gain and a learnable temperature refine this. The temperature corrects for each query seeing fewer keys. Training uses plain next-token loss. Each step averages a full-context loss and a random-budget loss. So 1 trained gate works across budgets. window.addEventListener("message",function(e){if(e.data&&e.data.mtpH){var f=document.getElementById("mtp-tg-frame");if(f&&e.source===f.contentWindow){f.style.height=e.data.mtpH+"px";}}}); #mtp-tg-wrap hr,#mtp-tg-wrap p:empty,#mtp-tg-wrap del,#mtp-tg-wrap s{display:none!important} How does TokenGate perform on benchmarks? The research team tested Qwen3-1.7B on RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX and needle-in-a-haystack (NIAH). Baselines were FastKV, SlimInfer, DASH, GemFilter, FlexPrefill and MInference, on the same LoRA backbone. Versus full prefill: 1.4x to 11.3x faster TTFT and 1.5x to 3.4x lower memory. Versus the next-best baseline: 1.2x to 6.4x faster TTFT and up to 2.1x lower memory. 2WikiMultihopQA: at 95% of full-context accuracy, prefill runs 3.0x faster with 2.7x less memory. Longer contexts: a gate trained at 16k reaches 12.8x at 32k and 13.0x at 64k and 128k. Memory savings stay near 3.43x. The learned selector beats FastKV’s heuristic at the same layer. Even random-K selection scores higher on the TokenGate backbone. The research team credits backbone co-adaptation. Does it work on hybrid-attention models? Yes, Qwen3.5-2B mixes Gated DeltaNet and full-attention layers. On NIAH 32k, TokenGate at layer 3 matched full-prefill accuracy. It ran 3.8x faster than full prefill and 2.1x faster than FastKV. Gating at layers 1 or 2 reached 7.8x, within 6% accuracy. FastKV can only prune attention layers. TokenGate vs FastKV, SlimInfer, GemFilter and DASH Self-reported gains use different setups, so they are not directly comparable. Pruning layers come from TokenGate’s Appendix C.2. How do you deploy TokenGate? The GitHub Repo needs Python 3.10+ and CUDA PyTorch 2.9+. An optional FlashAttention-2 key-bias kernel speeds up dense Qwen3. It costs 1.0x to 1.1x a plain call. Without it, inference falls back to FlexAttention. A {state} placeholder marks the prunable document. generate takes a max_kept_prefill_tokens budget. Checkpoints cover all 5 tasks on Qwen3-1.7B, plus a Qwen3.5-2B NIAH gate. Example: liana313/tokengate-niah-qwen3-1.7b. Training mirrors a TRL/PEFT recipe. Each step is 1.3x slower than plain LoRA. Runs converged faster on 4 of 5 tasks. Mean training time was 1.8 versus 2.2 A100 hours. Key Takeaways TokenGate adds learned token pruning to existing LLMs via a small gate and LoRA. Prefill runs 1.4x to 11.3x faster with 1.5x to 3.4x lower peak memory. Token-gated attention makes top-K selection trainable, MoE-router style. It beats the next-best pruning baseline by up to 6.4x on TTFT. It works on hybrid Qwen3.5-2B, where FastKV cannot prune DeltaNet layers. Check out the Paper, GitHub repo and Hugging Face paper page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster appeared first on MarkTechPost.
Anthropic has launched OSS Scanner, a free, opt-in vulnerability scanner for critical open-source projects. The Anthropic OSS Scanner runs the company’s strongest models, including Claude Mythos, against enrolled codebases. Reports go straight to maintainers without human review. That trade delivers faster findings, but shifts triage work onto project teams. 11 days after the October 8 launch, Anthropic reported 77 critical and 803 high model-rated findings. TL;DR Scale: 114 projects accepted from about 650 applications, as of October 19. Runs on: Anthropic’s infrastructure, free. Maintainers supply a Dockerfile; audits run in an isolated VM with no Internet. Performance: 85 of 97 audited critical and high findings (88%) met Anthropic’s disclosure bar. Best: Only 1 of 97 audited findings was a false positive. wolfSSL says 72 of 74 reports were valid. Worst: No human review before delivery. 11 of 97 audited findings duplicated known issues; maintainers report inflated severity. What is Anthropic OSS Scanner? OSS Scanner is a free application from Anthropic that periodically audits open-source repositories with its strongest models. The Frontier Red Team and Safeguards Research team run it together. The design borrows from Google’s OSS-Fuzz, but swaps fuzzers for language models. The main goal is a review bottleneck. Over 6 months, Anthropic’s models surfaced over 29,000 candidate vulnerabilities. Human reviewers triaged roughly 6,000 of them. Nearly 5,000 more reports went directly to maintainers who asked for everything, validated or not. OSS Scanner turns that ad hoc request into an opt-in program. How does OSS Scanner work? Core maintainers enroll by opening a pull request on anthropics/oss-scanner. The PR adds projects//project.yaml, based on the project template. Required fields: repo, primary_contact, and a Dockerfile path. Optional fields: threat_model, pgp for encrypted reports, auto_ccs, homepage, and disabled. The Dockerfile builds the project with network access. The VM then moves to a network without Internet for the audit. Agents in the pipeline double-check bugs, run root cause analysis and propose patches. Each report contains a self-contained reproducer and an explanation of the bug. Where possible, it includes a bisection to the commit that introduced it. A candidate patch is attached when available. An optional threat_model.md defines adversarial inputs, out-of-scope code and a severity rubric. Without it, the scanner guesses. How accurate are OSS Scanner reports? Anthropic team asked its penetration testers to check 97 critical and high findings across 48 projects. 85 (88%) met the bar for Anthropic’s coordinated vulnerability disclosure (CVD) process. 11 were real but duplicated known issues or other scan findings. 1 was invalid. Maintainer data points in the same direction. wolfSSL received 74 reports; all but 2 were valid, and 5 became CVEs. curl patched CVE-2026-92392, a High-severity HTTP/2 upload use-after-free. It affects curl 8.12.0 to 8.22.0 and is fixed in 8.23.0, released October 14, 2026. Anthropic links the timing to benchmark progress. On CyberGym, LLMs went from finding under 20% of vulnerabilities to over 85%. Its broader CVD dashboard lists 6,157 disclosed vulnerabilities across 591 projects, as of October 2. 516 of those are patched. #mtp-oss-scanner-wrap iframe{width:100%!important;border:0!important;display:block!important;background:transparent!important;}#mtp-oss-scanner-wrap hr,#mtp-oss-scanner-wrap p:empty,#mtp-oss-scanner-wrap del,#mtp-oss-scanner-wrap s{display:none!important;}@media (max-width:640px){#mtp-oss-scanner-wrap{margin:16px auto!important;}} (function(){var f=document.getElementById('mtp-oss-scanner-frame');window.addEventListener('message',function(e){if(e.data&&e.data.mtpFrame==='mtp-oss-scanner'&&e.source===f.contentWindow){f.style.height=e.data.h+'px';}});})(); OSS Scanner vs other open-source security programs Sources: OSS-Fuzz disclosure guidelines, OpenAI Codex Security, DeepMind CodeMender. Key Takeaways OSS Scanner gives critical open-source projects free Claude Mythos audits. Reports skip human review; 88% of 97 audited critical/high findings were valid. First 11 days: 77 critical and 803 high model-rated findings. Enrollment is a PR with a Dockerfile; audits run offline. No 90-day deadline applies to unvalidated findings. Check out the launch post, the OSS Scanner FAQ and the GitHub repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Anthropic Launches OSS Scanner, a Free AI Vulnerability Scanner appeared first on MarkTechPost.
Microsoft Research has open-sourced Agensh, a decentralized harness for large teams of coding agents. The Agensh multi-agent harness removes the central orchestrator that most agent frameworks depend on. Every worker picks its own sub-task, shares findings and merges code into a common Git repo. On ProgramBench, the team scaled the setup from 1 agent to 1,024 agents. TL;DR A Harness: Tested from 1 to 1,024 agents. Each worker gets a 272,000-token input and 128,000-token output limit. Runs on: Linux with Docker Compose. Single node by default (8 workers). The 1,024-agent run used 16 nodes, 64 agents each. Workers have a 64 GiB memory ceiling. Performance: more agents raised scores on all 5 hardest ProgramBench tasks, and reached them sooner. Best: 79.59% pandoc pass rate with 32 agents on GPT-6.1-sol. GPT-5.6-sol hit 55.06% with 1,024 agents. Worst: the 5-task mean reached only 28.78% at 128 agents. Going from 1 to 8 agents added 1.37 points. What is Agensh? Agensh is a multi-agent harness from Microsoft Research that coordinates many LLM agents without a lead agent. The target problem is the orchestrator ceiling. In orchestrator-worker designs, a lead agent plans, assigns and merges all work. The research paper cites Claude Code agent teams, Codex sub-agents, Copilot fleet and Kimi Agent Swarm as examples. Agensh lets the workers do that coordination themselves. How does the Agensh cooperation loop work? Each worker runs the same 5-step loop, concurrently and asynchronously: Gather context: read the goal, peer progress, messages and findings. Claim a sub-task: post a CLAIM, then settle overlaps by direct message. Take action: work locally with tools and post useful findings early. Verify results: check the work against the sub-task’s acceptance criteria. Merge progress: merge into the shared repo, post an update, then repeat. Three services connect the workers. Gitea hosts the shared workspace, with branches, PRs and merge conflicts. Mattermost carries a team channel plus direct messages for urgent collisions. A shared context board, adapted from DeLM, stores typed notes: OBSERVED, FACT, FAIL, CLAIM and PATCH_SUMMARY. The loop lives in the prompt, not in runtime code. Every worker gets the same prompt except its ID. Agensh sits on top of a single-agent harness. The experiments used GitHub Copilot CLI. The research paper states thin adapters can connect other harnesses, such as Claude Code. (function(){var f=document.getElementById("agx-frame");window.addEventListener("message",function(e){if(e.data&&e.data.agxHeight&&f&&e.source===f.contentWindow){f.style.height=e.data.agxHeight+"px";}});})(); How well does Agensh scale on ProgramBench? Microsoft team tested the 5 hardest ProgramBench tasks: FFmpeg, gromacs, pandoc, PHP-src and ctags. Agents rebuild each program from its observed behavior. They work offline, within 6 hours. Reference repos range from 2,767 files (pandoc) to 26,266 files (PHP-src). With GPT-5.6-sol at high reasoning effort, the mean final test-pass rate moved as follows: 1 agent: 19.31% 8 agents: 20.68% 32 agents: 26.52% 128 agents: 28.78% That is a 9.47-point gain, about 49% relative. On pandoc alone, scores were 33.89% (1), 41.75% (8), 43.11% (32), 50.94% (128) and 55.06% (1,024). Speed improved too: on pandoc, 128 agents passed 30% at the 30-minute checkpoint. 32 and 8 agents needed 60 and 90 minutes. The README adds a newer model. With GPT-6.1-sol, pandoc rose from 56.19% (1 agent) to 79.59% (32 agents). That is a 41.64% relative gain. These GPT-6.1-sol figures are not in the v1 paper. What behaviors emerged as the team grew? The research team traced worker trajectories at each scale: 8 agents: peers agreed on interfaces and resolved overlapping claims themselves. 32 agents: peers reviewed shared work; one approval was withdrawn after a counterexample. 128 agents: workers reused trusted reviewers and standardized an integration protocol. 1,024 agents: several workers acted as integrators, and peers recovered failed work. None of these roles was hard-coded. How does Agensh compare with other multi-agent harnesses? How do you run Agensh? The release includes a single-node Docker Compose team and a single-agent baseline. Both use native Copilot ACP sessions. You need Linux with Docker Engine, Compose v2, systemd, cgroup v2 and Python 3.11+. Defaults are 8 workers, 6 hours per task and gpt-5.6-sol at high effort. You also need ProgramBench (v1.2.5 was validated) and a model API token. Full steps sit in the reproduction guide. Key Takeaways Agensh coordinates up to 1,024 agents with no central orchestrator. Workers self-claim tasks via Gitea, Mattermost and a shared findings board. 128 agents lifted the 5-task ProgramBench mean from 19.31% to 28.78%. GPT-6.1-sol with 32 agents hit 79.59% on pandoc. Code is MIT licensed, but token cost is not disclosed. Check out the Paper, GitHub Repo and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Microsoft Open-Sources Agensh: A Multi-Agent Harness for 1,024 Agents appeared first on MarkTechPost.
Cognition has released the Devin Vulnerability Fixes plugin, a free, open-source set of agent skills. It targets three critical remote code execution (RCE) advisories. They cover PHP2Shell, the Node.js October 21, 2026 security release and a CVSS 10.0 self-managed GitLab bug. The skills run in Devin and in any coding agent that reads SKILL.md files. Cognition packaged the remediation workflow it used internally. Developers can do inventory, approve, patch, test, then ship 1 pull request per repository. TL;DR Size: Three skills (fix-php-vuln, fix-node-vuln, fix-gitlab-vuln), MIT license. Runs on: Devin, Claude Code, Codex, Cursor, Factory Droid, GitHub Copilot, Gemini CLI, OpenCode and Windsurf. Needs git, a shell, Docker and registry access. The npx skills install needs Node.js 22.20 or later. Performance: No remediation benchmark or fix-rate number is published for the plugin. Best: Covers the week’s most severe bug, GitLab CVE-2026-110264, rated CVSS 10.0. Worst: Validation is only as strong as your tests. Managed runtimes, appliances and Electron apps are flagged, not patched. What is the Devin Vulnerability Fixes plugin? The Devin Vulnerability Fixes plugin is a set of three Agent Skills from Cognition. They find and patch vulnerable PHP, Node.js and self-managed GitLab deployments. The source is on GitHub under the MIT license. Cognition’s product page lists it as free and open source. Which vulnerabilities does it fix? PHP2Shell (CVE-2026-109618, CVE-2026-109619): PHP rates both flaws CVSS v4.0 9.2. The first corrupts memory after an out-of-memory condition. The second lets stale global state from one request reach the next. PHP says every web-serving setup is at risk, with mod_php rated highest risk. Fixed versions are 8.2.35, 8.3.36, 8.4.27 and 8.5.12. PHP 8.1 and older get no releases, only source backports for 7.4, 8.0 and 8.1. Node.js October 2026 release: It fixes 10 CVEs, 4 rated High. The lead bug, CVE-2026-111497, is a Brotli decoder out-of-bounds write that can enable RCE. Fixed versions are 22.23.4, 24.22.1 and 26.12.1. Node.js 20 and older are end-of-life and get no fix. GitLab CVE-2026-110264: An integer overflow in the regular expression compiler allows unauthenticated code execution. It affects GitLab CE/EE from 7.0 onward and scores CVSS 10.0. GitLab.com and GitLab Dedicated are already patched. How does the plugin work? Every skill follows the same five-step shape: Read-only inventory: The agent scans Dockerfiles, base images, distro packages, CI, Helm, Terraform, Packer, Lambda and config management. Approval gate: Nothing changes until you pick repos. Riskier minor or major upgrades are called out first. Verified target: Every tag, digest, package version and checksum is confirmed against the official source before editing. Test before and after: The suite runs on the current version, then inside the new production image. Regressions stop the run. One PR per repo: Each PR carries versions, test results, runtime diffs, a risk label and a rollback plan. The skills also encode known patching traps. Debian and Ubuntu backport fixes without changing upstream versions, so php -v can mislead. Ubuntu 24.04’s fixed package is 8.3.6-0ubuntu0.24.04.13. Digest-pinned images need a new digest. Changing Composer’s require.php patches nothing. window.addEventListener("message",function(e){if(e.data&&e.data.mtpEmbedHeight){var f=document.getElementById("mtp-dvf-frame");if(f&&e.source===f.contentWindow){f.style.height=e.data.mtpEmbedHeight+"px";}}}); How do you install and run it? In Devin, open Customize, then Plugins, then Browse Marketplace, and search for Vulnerability Fixes. The Devin CLI uses devin plugins install CognitionAI/devin-marketplace#plugins/devin-vuln-fixes. For other agents, run npx skills add CognitionAI/devin-vuln-fixes. Claude Code users can add the marketplace and run /plugin install vuln-fixes@cognition-skills. Then ask in plain language, such as ‘Please fix my PHP vulnerability.’ Slash commands also work: /fix-php-vuln, /fix-node-vuln and /fix-gitlab-vuln. How does it compare with Dependabot, Copilot Autofix and Snyk Agent Fix? These tools solve adjacent problems. Dependabot and the code-fix agents work on application dependencies and source code. Cognition’s plugin targets the runtime layer: base images, OS packages and infrastructure-as-code. Key Takeaways Three MIT-licensed skills patch PHP2Shell, Node.js and GitLab CVE-2026-110264. Read-only inventory and an approval gate come before any edit. Tests run before and after inside the new production image. Works across Devin, Claude Code, Codex, Cursor and other SKILL.md agents. No published fix-rate benchmark; results depend on your test coverage. Check out the GitHub repo, the Cognition blog post and the plugin page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Cognition Open-Sources an Agent Plugin that Helps Secure Systems against Recent Cybersecurity Threats appeared first on MarkTechPost.
In this tutorial, we work with Underdog Saluki 27B, a 7.9 GB build of Qwen3.8-27B that its makers say keeps tool calling intact at about two bits per weight and even edges out the full 54 GB model on their own 120-task benchmark. Saluki is built on ISTA-DASLab’s GSQ-RCO quantization of the same model, so we put both files side by side in stock llama.cpp. We start with the files themselves: a tensor-by-tensor diff of the two GGUF headers, then a byte-level comparison that shows whether Saluki is a remix of ISTA’s bits or a changed model. We then serve each file on one GPU, read its memory and speed, and score both on the same seeded 120 tasks from the Berkeley Function Calling Leaderboard, using BFCL’s own checker with strict and lenient parsing. Copy CodeCopiedUse a different Browserimport os import re import ast import sys import json import math import copy import enum import glob import time import types import random import shutil import struct import hashlib import tarfile import platform import traceback import subprocess import collections import urllib.request import urllib.error RESULTS = {} # step -> one-line result, for the summary table CLAIMS = [] # (step, claim, held) rows; held is True, False or None (not decided in this run) CURRENT = ["1"] def banner(title): print("n" + "=" * 78) print(title) print("=" * 78) def section(name): def wrap(fn): def run(*a, **kw): CURRENT[0] = name.split(".")[0] banner(name) t0 = time.time() try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}" print(f"n[!] {name} did not complete: {type(e).__name__}: {e}") traceback.print_exc(limit=3) return None finally: print(f"n[{time.time() - t0:.1f} s]") json.dump({"results": RESULTS, "claims": CLAIMS}, open(os.path.join(WORK, "results.json"), "w"), indent=1) return run return wrap def claim(text, held): CLAIMS.append((CURRENT[0], text, held)) return held banner("1. Platform, pins and helpers") SYSTEM, MACHINE = platform.system(), platform.machine() ON_CUDA = SYSTEM == "Linux" and shutil.which("nvidia-smi") is not None assert ON_CUDA or (SYSTEM == "Darwin" and MACHINE == "arm64"), "needs an NVIDIA GPU on Linux (Colab T4) or an Apple-silicon Mac" WORK = os.environ.get("SALUKI_WORK") or ("/content/saluki" if os.path.isdir("/content") else os.path.abspath("saluki_work")) for d in ("models", "llama", "bfcl", "runs"): os.makedirs(os.path.join(WORK, d), exist_ok=True) # Everything the notebook downloads is pinned by size and SHA-256. LLAMA_TAG = "b11565" # llama.cpp release of 2026-10-12, commit f9a95aeb2 LLAMA_ASSET = {"Linux": ("llama-b11565-bin-ubuntu-cuda-12.8-x64.tar.gz", 171559844, "61b8cf4e49739afe4008b06a92390dff999e706e9e7e9749c015544a9f241627"), "Darwin": ("llama-b11565-bin-macos-arm64.tar.gz", 12105509, "19289548468babcd5131cecccf3a9bc7c67030cc7f0a978dafc0db1ba7891068")}[SYSTEM] MODELS = { "Saluki": dict(repo="ConwayResearch/Underdog-Saluki-27B-1.0", rev="4f60eba09f4b6db63c72915d328d97e1d93edb8c", file="Underdog-Saluki-27B-1.0-IQ2-mix.gguf", size=7898369152, sha256="4a673518b11b1c4445f9b9a9d3c40356f5ba39f6a6260dcb25f3fbfd475d9efb"), "ISTA": dict(repo="ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF", rev="d562806dbafae37109975e970aae91b43e73b440", file="Qwen3.8-27B-GSQ-RCO-IQ2_XS.gguf", size=8422841472, sha256="f0ae5006da0ce6225935339e4e989369f94de95d2263cf969519f8420c9ae02c"), } # BFCL v4 data and BFCL's own checker code, at one gorilla commit. GORILLA = ("" "berkeley-function-call-leaderboard/bfcl_eval/") BFCL_FILES = { "data/BFCL_v4_simple_python.json": "82dd63ba502eb2520c6b5d1d9a5c4b590e03ff261565175561f6228a367d1991", "data/BFCL_v4_multiple.json": "aef168155ebd74b7ac2401198b201343bc7d16d7a3d7e0d4e6d8ee82c6969b2a", "data/BFCL_v4_irrelevance.json": "2b6ed4c2e992cdcf5f1678a701851f944bef7550ee026ed1ddb89efed5be01a6", "data/BFCL_v4_parallel.json": "19f51a82eff42e5d62541aa500115a056eb78f437c2ba1f10415fd7c8e5dda84", "data/BFCL_v4_parallel_multiple.json": "8863ea8433239f55c5f016154cf0830853c89f693c6ea270396a2fa121960579", "data/possible_answer/BFCL_v4_simple_python.json": "90cd5bc653690ee8e459b5b3f3fc9458606f7f3fcbf795bb51b7dc581f8c86dc", "data/possible_answer/BFCL_v4_multiple.json": "244e00ce9395df948bcafc7bee64e8f9c87ef70887587d83cae45b13699f3047", "data/possible_answer/BFCL_v4_parallel.json": "8a6aa19c1adddc6a5a2f7e40f9dbf30cc7e95815e7b830c90589ab318229e0f0", "data/possible_answer/BFCL_v4_parallel_multiple.json": "5ebf24f458c1f16300c05505d83d6f0a1b68b79be273a033febd0d4f840507e3", "eval_checker/ast_eval/ast_checker.py": "2aae7a68461a8f76c0be3894c8901b66b56967a1989d3ab066051e3fb97f1538", "constants/enums.py": "2182becfa2a1d071ee1db30db593b4758c6bf866aa12d2d4b8daf09175ea518a", "constants/type_mappings.py": "1702fb67afbe2c492608e58e2b7d02e46381f50166b47f3c952f76e34c7cd3bd", "utils.py": "3703b9bb63f83581c60b8e0b82aac74c360c9ca781e99e7d4eb28b0cce2285dd", "model_handler/utils.py": "f78fd3edce603b333dc9a88ee2c041dc547d51f71aa449ffebc044c4b1e353f3", } # Underdog Bench's five kinds of tasks, 24 each; we draw our own seeded sample because its task list is not published. KINDS = [("simple_python", "Call the one right function"), ("multiple", "Pick the right function from several"), ("irrelevance", "Know when no tool fits"), ("parallel", "Same function, several calls"), ("parallel_multiple", "Several different functions")] N_PER_KIND, SEED, CTX, PORT = 24, 2026, 16384, 8080 # llama-server's own defaults are temperature 0.8 and top_k 40, so every request states its sampler. GREEDY = dict(temperature=0, top_k=0, top_p=1.0, min_p=0) def sh(cmd, **kw): return subprocess.run(cmd, capture_output=True, text=True, **kw) def sha256_file(path, chunk=1 < {ver}") for rel, digest in BFCL_FILES.items(): dest = os.path.join(WORK, "bfcl", rel.replace("/", "__")) fetch(GORILLA + rel, dest, sha=digest) print(f"BFCL v4: {len(BFCL_FILES)} files from gorilla@6ea5797 (5 task files, 4 answer files, 5 checker sources), all SHA-256 checked") for tag, m in MODELS.items(): path = os.path.join(WORK, "models", m["file"]) had = os.path.exists(path) and os.path.getsize(path) == m["size"] url = f"'repo']}/resolve/{m['rev']}/{m['file']}" dl = fetch(url, path, m["size"]) # size check now, hash below t0 = time.time() ok = sha256_file(path) == m["sha256"] how = "already on disk" if had else f"downloaded in {dl:.0f} s ({m['size'] / dl / 1e6:.0f} MB/s)" print(f"{tag:7s}: {m['file']} {m['size']:,} bytes, {how}; SHA-256 {'matches' if ok else 'DOES NOT MATCH'} " f"the Hub's LFS pin ({time.time() - t0:.0f} s to hash)") claim(f"{m['file']} is byte-identical to the file pinned at {m['repo']}@{m['rev'][:7]}", ok) assert ok return f"llama.cpp {LLAMA_TAG}; Saluki 7.90 GB and ISTA IQ2_XS 8.42 GB verified by SHA-256" download_step() We download the CUDA build of llama.cpp b11565, both 2-bit files and the BFCL data, and check every byte against its pin. The Hugging Face downloads are resumable, and the hashes are the Hub’s own LFS digests at a fixed revision, so a silently replaced file would stop the notebook rather than change its results. BFCL’s task files, answer files, and checker code all come from one gorilla commit. The cell prints how long the downloads took, which is the slowest part of the run on Colab. Copy CodeCopiedUse a different BrowserGGML_TYPES = {0: ("F32", 1, 4), 1: ("F16", 1, 2), 8: ("Q8_0", 32, 34), 10: ("Q2_K", 256, 84), 12: ("Q4_K", 256, 144), 16: ("IQ2_XXS", 256, 66), 17: ("IQ2_XS", 256, 74), 18: ("IQ3_XXS", 256, 98), 19: ("IQ1_S", 256, 50), 21: ("IQ3_S", 256, 110), 22: ("IQ2_S", 256, 82), 23: ("IQ4_XS", 256, 136), 29: ("IQ1_M", 256, 56), 30: ("BF16", 1, 2)} # name, values per block, bytes per block (ggml-common.h) def read_gguf(path): """GGUF v3 header: metadata, tensor table (name, dims, type, offset) and where the tensor data starts.""" f = open(path, "rb") u = lambda fmt: struct.unpack(" a["bytes"]; down += b["bytes"] <a>5s} {'weights':>9s} {'ISTA bpw':>9s} {'Saluki bpw':>10s} {'change MB':>10s}") for c, r in sorted(rows.items(), key=lambda kv: -kv[1]["n"]): if r["n"] {b}" for (a, b), n in moves.most_common())) print(f" {down} get fewer bytes, {up} get more") claim("every tensor whose type changes gets fewer bits; none gets more", up == 0 and down == changed) untouched = [c for c, r in rows.items() if r["bi"] == r["bs"] and r["n"] >= 1e6] print(f" classes with no change at all: {', '.join(sorted(untouched))}") proj = lambda c: c.startswith(("full-attention", "linear-attention")) and "gates" not in c claim("every attention and DeltaNet projection matrix keeps its type (only the small alpha/beta gates change)", all(a["type"] == b["type"] for a, b in zip(t_i, t_s) if proj(tensor_class(a["name"])))) keys = sorted(k for k in set(kv_s) | set(kv_i) if kv_s.get(k) != kv_i.get(k) and not k.startswith("tokenizer.ggml.t")) print(f"nmetadata keys that differ: {keys}") print(f"Saluki's header still reads: general.name = {kv_s['general.name']!r}, quantized_by = {kv_s['general.quantized_by']!r}") print(f"sampling stored in both headers: temp {kv_s['general.sampling.temp']}, top_k {kv_s['general.sampling.top_k']}, " f"top_p {kv_s['general.sampling.top_p']:.2f} (these are Qwen's thinking-mode settings; step 5 shows what the server does with them)") saved = (MODELS["ISTA"]["size"] - MODELS["Saluki"]["size"]) / 1e6 return f"{changed} of {len(t_s)} tensor types changed, all downgrades; {saved:.0f} MB smaller" header_diff() Reading the two GGUF headers in plain Python shows exactly what Saluki changed. Both files have the same 851 tensors with the same shapes, and the type table predicts each file’s size in bytes. Saluki changes the type of 146 tensors and every change cuts bits: most fall on the feed-forward down projections and the output layer, and the small DeltaNet gate vectors drop from BF16 to Q8_0. No attention or DeltaNet projection matrix changes, and the header still names ISTA DASLab as the quantiser. Copy CodeCopiedUse a different Browser@section("4. Byte-level provenance: are the unchanged tensors ISTA's own bytes?") def provenance(): import numpy as np (_, t_s, start_s), (_, t_i, start_i) = HDR["Saluki"], HDR["ISTA"] ms = np.memmap(os.path.join(WORK, "models", MODELS["Saluki"]["file"]), dtype=np.uint8, mode="r") mi = np.memmap(os.path.join(WORK, "models", MODELS["ISTA"]["file"]), dtype=np.uint8, mode="r") q = dict(k=0, same=0, eq_bytes=0, bytes=0) f32 = collections.defaultdict(lambda: [0, 0, []]) # kind -> [identical, differ, relative L2 changes] for a, b in zip(t_i, t_s): if a["type"] != b["type"]: continue x = mi[start_i + a["off"]: start_i + a["off"] + a["bytes"]] y = ms[start_s + b["off"]: start_s + b["off"] + b["bytes"]] if a["type"] == "F32": kind = re.sub(r"blk.d+.", "", a["name"]) u, v = x.view(np.float32).astype(np.float64), y.view(np.float32).astype(np.float64) same = bool(np.array_equal(u, v)) f32[kind][0 if same else 1] += 1 if not same: f32[kind][2].append(np.linalg.norm(u - v) / np.linalg.norm(u)) else: eq = int(np.count_nonzero(x == y)) q["k"] += 1; q["same"] += eq == a["bytes"]; q["eq_bytes"] += eq; q["bytes"] += a["bytes"] print(f"quantised matrices that keep their type: {q['same']} of {q['k']} byte-identical, but " f"{100 * q['eq_bytes'] / q['bytes']:.1f}% of their {q['bytes'] / 1e9:.2f} GB of bytes are equal") print(f"nunquantised F32 vectors (never re-encoded, so any change means the weights changed):") print(f" {'kind':28s} {'identical':>9s} {'changed':>8s} median relative change") for kind, (same, diff, rel) in sorted(f32.items(), key=lambda kv: -kv[1][1]): med = f"{sorted(rel)[len(rel) // 2]:.1e}" if rel else "-" print(f" {kind:28s} {same:9d} {diff:8d} {med}") changed = sum(v[1] for v in f32.values()) total = sum(v[0] + v[1] for v in f32.values()) claim("Saluki's unchanged-type tensors are byte-for-byte ISTA's (a pure type re-mix)", q["same"] == q["k"] and changed == 0) claim("Saluki's F32 norm and gate vectors differ from ISTA's, so its weights were changed, not only re-encoded", changed > 0) return (f"{q['same']}/{q['k']} same-type matrices byte-identical, {100 * q['eq_bytes'] / q['bytes']:.1f}% of their bytes equal; " f"{changed}/{total} F32 vectors changed") provenance() Same types don’t mean the same weights, so we compare the bytes of the 705 tensors that keep their type. Only one of 352 quantized matrices is byte-identical, yet 98.4% of their bytes match, consistent with small edits to the underlying weights followed by re-quantizing on the same grids. The unquantized norm and gate vectors settle it: most changed by a median of a few parts in ten thousand. Saluki is a lightly changed model, not a re-encoding of ISTA’s file. Copy CodeCopiedUse a different BrowserSERVER = {} def start_server(tag): """One llama-server per model, one slot, all layers on the GPU, debug log kept for unparsed replies.""" stop_server() log_path = os.path.join(WORK, "runs", f"server_{tag}.log") cmd = [SERVER_BIN, "-m", os.path.join(WORK, "models", MODELS[tag]["file"]), "--host", "127.0.0.1", "--port", str(PORT), "-c", str(CTX), "-np", "1", "-ngl", "99", "-fa", "on", "--jinja", "--reasoning-format", "deepseek", "-lv", "5"] t0 = time.time() proc = subprocess.Popen(cmd, stdout=open(log_path, "w"), stderr=subprocess.STDOUT, env=SERVER_ENV) while True: try: if urllib.request.urlopen(f"", timeout=5).status == 200: break except urllib.error.HTTPError as e: assert e.code == 503, e except urllib.error.URLError: pass assert proc.poll() is None, f"llama-server exited; see {log_path}" assert time.time() - t0 < 900, "server did not become healthy in 15 minutes" time.sleep(2) SERVER.update(tag=tag, proc=proc, log=log_path, load_s=time.time() - t0) return SERVER["load_s"] def stop_server(): if SERVER.get("proc") and SERVER["proc"].poll() is None: SERVER["proc"].terminate() SERVER["proc"].wait(timeout=60) SERVER.clear() def api(path, body=None, timeout=1800): req = urllib.request.Request(f"", None if body is None else json.dumps(body).encode(), {"Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=timeout) as r: return json.loads(r.read()) def memory_report(): """Buffer sizes from the server's load log, plus what the GPU reports.""" text = open(SERVER["log"]).read() out = {} for label, what in (("weights", "model"), ("KV cache", "KV"), ("recurrent state", "RS"), ("compute", "compute")): last = {} # llama.cpp sizes every buffer in a dry run first; keep the final value per device for dev, mib in re.findall(rf"(S+) +{what} buffer size = +([d.]+) MiB", text): last[dev] = float(mib) out[label] = sum(last.values()) if ON_CUDA: out["GPU in use (nvidia-smi)"] = float(sh(["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"]).stdout.split()[0]) return out @section("5. A hybrid model on one GPU: start the server, read its memory, and look at the prompt it builds") def architecture_and_server(): kv = HDR["Saluki"][0] L, every = kv["qwen35.block_count"], kv["qwen35.full_attention_interval"] full = [i for i in range(L) if (i + 1) % every == 0] heads_kv, dk = kv["qwen35.attention.head_count_kv"], kv["qwen35.attention.key_length"] kv_tok = len(full) * heads_kv * dk * 2 * 2 # K and V, f16 conv = (kv["qwen35.ssm.conv_kernel"] - 1) * (kv["qwen35.ssm.inner_size"] + 2 * kv["qwen35.ssm.group_count"] * kv["qwen35.ssm.state_size"]) state = kv["qwen35.ssm.state_size"] * kv["qwen35.ssm.inner_size"] rs = (L - len(full)) * (conv + state) * 4 print(f"{L} blocks: {L - len(full)} Gated DeltaNet (linear attention) and {len(full)} full-attention layers (3, 7, ..., {full[-1]})") print(f"KV cache: {kv_tok // 1024} KiB per token for the {len(full)} attention layers; a model with attention in all " f"{L} layers would need {L * heads_kv * dk * 4 // 1024} KiB") print(f"recurrent state: {rs / 2**20:.1f} MiB per sequence, fixed, whatever the context length") print(f"expected at -c {CTX}: KV {kv_tok * CTX / 2**20:.0f} MiB + state {rs / 2**20:.0f} MiB") load_s = start_server("Saluki") mem = memory_report() print(f"nllama-server up in {load_s:.0f} s (-c {CTX}, -np 1, -ngl 99, -fa on); buffers from its log:") for k, v in mem.items(): print(f" {k:24s} {v:8.0f} MiB") claim(f"the server's KV buffer at -c {CTX} matches 64 KiB per token within 2%", abs(mem["KV cache"] - kv_tok * CTX / 2**20) <= 0.02 * kv_tok * CTX / 2**20) SERVER_MEM["Saluki"] = mem props = api("/props") p = props["default_generation_settings"]["params"] g = {k: kv[f"general.sampling.{k}"] for k in ("temp", "top_k", "top_p", "min_p")} from_file = abs(p["temperature"] - g["temp"]) < 1e-6 and p["top_k"] == g["top_k"] and abs(p["top_p"] - g["top_p"]) {'taken from the GGUF header' if from_file else 'llama.cpp built-ins, not the header'}; the card suggests 0.6 for thinking," f" so every request below states its own sampler") claim("llama-server takes its default sampler from the GGUF's general.sampling keys", from_file) caps = props.get("chat_template_caps", {}) print("template supports: " + ", ".join(k.replace("supports_", "") for k, v in caps.items() if v)) msgs = [{"role": "user", "content": "What is the weather in Oslo?"}] tool = [{"type": "function", "function": {"name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}] for label, kw in (("thinking off", {"enable_thinking": False}), ("thinking on, effort low", {"reasoning_effort": "low"}), ("thinking on, default", {})): prompt = api("/apply-template", {"messages": msgs, "tools": tool, "chat_template_kwargs": kw})["prompt"] n = len(api("/tokenize", {"content": prompt, "add_special": False})["tokens"]) print(f" {label:24s}: {n:3d} prompt tokens, ends {prompt[-28:]!r}") return f"KV {kv_tok // 1024} KiB/token, state {rs / 2**20:.0f} MiB; server buffers {sum(mem[k] for k in ('weights', 'KV cache', 'recurrent state', 'compute')) / 1024:.1f} GiB" SERVER_MEM = {} architecture_and_server() Qwen3.8-27B is a hybrid: 48 of its 64 blocks are Gated DeltaNet linear-attention layers with a fixed recurrent state, and only every fourth block keeps full attention. The KV cache therefore costs 64 KiB per token instead of 256 KiB, plus about 150 MiB of state per sequence. We start llama-server with one slot, read its buffers from the log, check where its default sampler comes from, and look at the prompts the chat template builds with thinking off, on, and at low effort. Copy CodeCopiedUse a different BrowserSPEED = {} def speed_probe(tag): """Prefill and decode rates from the server's own timings: two prompt lengths, 128 forced output tokens.""" rng = random.Random(SEED) words = "def class return import yield lambda tensor gradient matrix kernel cache token buffer layer".split() rows = [] for target in (256, 1024): text = " ".join(rng.choice(words) for _ in range(target)) ids = api("/tokenize", {"content": text, "add_special": False})["tokens"][:target] r = api("/completion", {"prompt": ids, "n_predict": 128, "ignore_eos": True, "cache_prompt": False, **GREEDY}) t = r["timings"] rows.append((t["prompt_n"], t["prompt_per_second"], t["predicted_n"], t["predicted_per_second"])) print(f" {tag:7s} prompt {t['prompt_n']:5d} tok at {t['prompt_per_second']:7.1f} tok/s | " f"decode {t['predicted_n']} tok at {t['predicted_per_second']:5.2f} tok/s") SPEED[tag] = rows return rows @section("6. Prefill and decode speed, measured by the server") def speed_step(): print(f"hardware: {HW}") rows = speed_probe("Saluki") return f"Saluki on {HW}: prefill {rows[1][1]:.0f} tok/s at {rows[1][0]} tokens, decode {rows[1][3]:.1f} tok/s" speed_step() Speed comes from the server’s own timings rather than a stopwatch around the request. We send a 256-token and a 1,024-token prompt with the prompt cache off and force 128 output tokens, which gives a prefill rate and a decode rate at each length. A dense 27B model reads its full weight file for every generated token, so decode speed mostly depends on memory bandwidth, and the same probe runs again on ISTA’s file later for a like-for-like comparison. Copy CodeCopiedUse a different Browserdef load_bfcl_checker(): """BFCL's own checker functions, executed from the pinned source with their package imports replaced.""" src = lambda rel: open(os.path.join(WORK, "bfcl", rel.replace("/", "__"))).read() def run(rel, ns, only=None): tree = ast.parse(src(rel)) body = [n for n in tree.body if not isinstance(n, (ast.Import, ast.ImportFrom))] if only: body = [n for n in body if getattr(n, "name", None) in only] exec(compile(ast.Module(body=body, type_ignores=[]), rel, "exec"), ns) return ns enums = run("constants/enums.py", {"Enum": enum.Enum}) tmap = run("constants/type_mappings.py", {}) model_cfg = collections.defaultdict(lambda: types.SimpleNamespace(underscore_to_dot=True)) # OpenAI-style FC model chk = run("eval_checker/ast_eval/ast_checker.py", {"re": re, "Language": enums["Language"], "MODEL_CONFIG_MAPPING": model_cfg, "JAVA_TYPE_CONVERSION": tmap["JAVA_TYPE_CONVERSION"], "JS_TYPE_CONVERSION": tmap["JS_TYPE_CONVERSION"], "java_type_converter": None, "js_type_converter": None}) utl = run("utils.py", {}, {"is_function_calling_format_output", "is_empty_output"}) conv = run("model_handler/utils.py", {"copy": copy, "re": re, "json": json, "ModelStyle": enums["ModelStyle"], "GORILLA_TO_OPENAPI": tmap["GORILLA_TO_OPENAPI"]}, {"_cast_to_openai_type", "convert_to_tool"}) return types.SimpleNamespace( ast_checker=lambda funcs, out, gt, cat: chk["ast_checker"](funcs, out, gt, enums["Language"].PYTHON, cat, "fc-model"), is_empty=utl["is_empty_output"], to_tools=lambda funcs: conv["convert_to_tool"](funcs, tmap["GORILLA_TO_OPENAPI"], enums["ModelStyle"].OPENAI_COMPLETIONS)) def load_tasks(): tasks = [] for kind, label in KINDS: rows = [json.loads(l) for l in open(os.path.join(WORK, "bfcl", f"data__BFCL_v4_{kind}.json")) if l.strip()] gt = {} if kind != "irrelevance": for l in open(os.path.join(WORK, "bfcl", f"data__possible_answer__BFCL_v4_{kind}.json")): if l.strip(): d = json.loads(l); gt[d["id"]] = d["ground_truth"] for r in sorted(random.Random(f"{SEED}-{kind}").sample(rows, N_PER_KIND), key=lambda r: int(r["id"].rsplit("_", 1)[1])): tasks.append(dict(id=r["id"], kind=kind, messages=r["question"][0], functions=r["function"], gt=gt.get(r["id"]))) return tasks TOOL_BLOCK = re.compile(r"(.*?)(?:|$)", re.S) FUNC_XML = re.compile(r"s]+)s*>(.*?)(?:|$)", re.S) PARAM_XML = re.compile(r"s]+)s*>n?(.*?)n?", re.S) def lenient_calls(raw, functions): """Forgiving reading of the model's raw text: any block, closed or not, inside or outside . Values are typed with the function's own schema (BFCL types). Not part of BFCL; used only for the 'lenient' column.""" raw = raw.split("")[-1] # with thinking on, the reasoning precedes types_ = {re.sub(r".", "_", f["name"]): {k: v.get("type", "string") for k, v in f["parameters"]["properties"].items()} for f in functions} # tool names were sent with dots turned into underscores calls = [] for fname, body in FUNC_XML.findall(raw): fname = fname.strip().strip('"') args = {} for p, v in PARAM_XML.findall(body): t = types_.get(re.sub(r".", "_", fname), {}).get(p, "string") v = v.strip() if t != "string" else v try: args[p] = (int(v) if t == "integer" else float(v) if t in ("float", "number") else v.lower() == "true" if t == "boolean" and v.lower() in ("true", "false") else json.loads(v) if t in ("array", "tuple", "dict", "object") else v) except ValueError: try: args[p] = ast.literal_eval(v) except Exception: args[p] = v calls.append({fname: args}) return calls def score(task, calls): """BFCL's rule: irrelevance passes with no call; the other kinds go through BFCL's ast_checker.""" if task["kind"] == "irrelevance": return CHECK.is_empty(calls), "made a call" if calls else None if not calls: return False, "no call" try: r = CHECK.ast_checker(task["functions"], calls, task["gt"], task["kind"]) except Exception as e: # malformed arguments the checker cannot walk return False, f"checker_error:{type(e).__name__}" return bool(r["valid"]), r.get("error_type") def run_task(task, think=False, max_tokens=512): body = {"messages": task["messages"], "tools": CHECK.to_tools(task["functions"]), "max_tokens": max_tokens, "cache_prompt": False, "verbose": True, **GREEDY, "chat_template_kwargs": {"reasoning_effort": "low"} if think else {"enable_thinking": False}} log_pos = os.path.getsize(SERVER["log"]) t0 = time.time() rec = dict(id=task["id"], kind=task["kind"], think=think) try: r = api("/v1/chat/completions", body) msg = r["choices"][0]["message"] rec.update(status=200, finish=r["choices"][0]["finish_reason"], raw=r["__verbose"]["content"], prompt_n=r["timings"]["prompt_n"], out_n=r["timings"]["predicted_n"], reasoning_chars=len(msg.get("reasoning_content") or "")) rec["calls"] = [{tc["function"]["name"]: json.loads(tc["function"]["arguments"])} for tc in msg.get("tool_calls") or []] except urllib.error.HTTPError as e: # stock llama.cpp answers 500 when its parser cannot read the reply time.sleep(0.5) with open(SERVER["log"], errors="replace") as f: f.seek(log_pos) new = f.read() m = re.findall(r"=== BEGIN ===n(.*?)n=== END ===", new, re.S) rec.update(status=e.code, finish="parse error", raw=m[-1] if m else "", calls=None, prompt_n=None, out_n=None) rec["seconds"] = time.time() - t0 rec["strict"], rec["strict_why"] = score(task, rec["calls"]) if rec["calls"] is not None else (False, "unparsed (HTTP 500)") if rec["strict"]: rec["lenient"] = True else: rec["lenient"], _ = score(task, lenient_calls(rec["raw"], task["functions"])) return rec def bench(tag, tasks): out, t0 = [], time.time() for i, task in enumerate(tasks): out.append(run_task(task)) if (i + 1) % 30 == 0: print(f" {tag}: {i + 1}/{len(tasks)} done, {time.time() - t0:.0f} s") json.dump(out, open(os.path.join(WORK, "runs", f"bfcl_{tag}.json"), "w"), indent=1) return out def ci(k, n, alpha=0.05): """Clopper-Pearson interval by bisection on the binomial tail.""" def cdf(x, p): return sum(math.comb(n, i) * p ** i * (1 - p) ** (n - i) for i in range(x + 1)) def solve(f, target, rising): lo, hi = 0.0, 1.0 for _ in range(60): mid = (lo + hi) / 2 lo, hi = (mid, hi) if (f(mid) 8s} {'lenient':>8s} replies: structured / plain text / unparsed (500) / cut off") for kind, label in KINDS: rs = [r for r in recs if r["kind"] == kind] st = collections.Counter("unparsed" if r["status"] != 200 else "cut off" if r["finish"] == "length" else "structured" if r["calls"] else "plain text" for r in rs) print(f" {label:44s} {sum(r['strict'] for r in rs):3d}/{len(rs):<4d} {sum(r['lenient'] for r in rs):3d}/{len(rs):<4d}" f" {st['structured']:3d} / {st['plain text']:3d} / {st['unparsed']:3d} / {st['cut off']:3d}") k, n = sum(r["strict"] for r in recs), len(recs) lo, hi = ci(k, n) print(f" {'total':44s} {k:3d}/{n:<4d} {sum(r['lenient'] for r in recs):3d}/{n:7s} {'ISTA':>6s} | {'lenient':>7s} {'':>5s} | vendor's Saluki (of 24)") for kind, label in KINDS: s = [r for r in S if r["kind"] == kind]; i = [r for r in I if r["kind"] == kind] print(f"{label:38s} {sum(r['strict'] for r in s):7d} {sum(r['strict'] for r in i):6d} | " f"{sum(r['lenient'] for r in s):7d} {sum(r['lenient'] for r in i):5d} | {vendor[kind]:3d}") out = {} for mode in ("strict", "lenient"): a, b = [r[mode] for r in S], [r[mode] for r in I] b10, b01, p = mcnemar(a, b) out[mode] = (sum(a), sum(b), b10, b01, p) print(f"{mode:8s}: Saluki {sum(a)}/120, ISTA {sum(b)}/120; Saluki-only passes {b10}, ISTA-only {b01}, exact McNemar p = {p:.3f}") print("vendor's Underdog Bench (different 120 tasks, unpublished): Saluki 88, ISTA IQ2_XS 76, full Qwen3.8-27B 84") sa, ia, b10, b01, p = out["strict"] claim("Saluki passes more of our 120 tasks than its parent (strict scoring)", sa > ia) claim("the difference is significant at 5% (exact McNemar)", p = pick_i) return f"strict: Saluki {sa}/120 vs ISTA {ia}/120 (McNemar p = {p:.2f}); lenient: {out['lenient'][0]} vs {out['lenient'][1]}" head_to_head() Because both models answered the same tasks, the comparison is paired. The cell prints the per-kind scores next to the vendor’s per-kind numbers for Saluki and runs an exact McNemar test on the tasks where the two files disagree, for strict and lenient scoring. A gap of a few tasks in 120 sits inside run-to-run noise, as the vendor’s page itself concedes, so the test tells us whether our sample shows a real difference. Copy CodeCopiedUse a different Browser@section("11. Where the failures come from") def failures(): for tag in ("Saluki", "ISTA"): recs = RUNS[tag] why = collections.Counter((r["strict_why"] or "?").split(":")[-1] for r in recs if not r["strict"]) print(f"{tag}: {sum(not r['strict'] for r in recs)} failures -> " + ", ".join(f"{k} {v}" for k, v in why.most_common())) rescued = [r for r in recs if r["lenient"] and not r["strict"]] print(f" lenient reading rescues {len(rescued)}: " + ", ".join(r["id"] for r in rescued[:8])) ex = next((r for r in RUNS["Saluki"] if not r["strict"] and r["kind"] != "irrelevance"), None) if ex: task = next(t for t in TASKS if t["id"] == ex["id"]) print(f"nexample, {ex['id']} ({ex['strict_why']}):") print(" expected:", json.dumps(task["gt"])[:300]) print(" model :", (json.dumps(ex["calls"]) if ex["calls"] is not None else ex["raw"])[:300]) irr = [r for r in RUNS["Saluki"] if r["kind"] == "irrelevance" and not r["strict"]] if irr: print(f"nirrelevance: Saluki made a call on {len(irr)} of {N_PER_KIND} prompts that no tool fits, e.g. {irr[0]['id']}: " f"{json.dumps(irr[0]['calls'])[:160]}") un = sum(r["status"] != 200 for tag in RUNS for r in RUNS[tag]) return f"{un} replies across both models were unparseable by the stock server" failures() Scores hide where calls go wrong, so this cell breaks the failures down by BFCL’s own error types: wrong number of calls, wrong or missing arguments, a call made when no tool fits, and replies the stock server could not parse. It also lists the tasks the lenient reading rescues and prints one failed task beside its expected answer, the quickest way to tell a model error from a scoring artifact. Copy CodeCopiedUse a different Browserbanner("SUMMARY") for name, res in RESULTS.items(): print(f" {name:<100s} {res}") print("nClaims decided in this run:") for step, text, held in CLAIMS: print(f" [{'held' if held else 'DID NOT HOLD' if held is False else 'undecided'}] step {step}: {text}") failed = [k for k, v in RESULTS.items() if str(v).startswith("SKIPPED")] print(f"n{sum(h is True for *_, h in CLAIMS)} of {len(CLAIMS)} claims held; steps failed or skipped: {failed or 'none'}") print(f"run on: {HW}; llama.cpp {LLAMA_TAG}; Python {platform.python_version()}") print("not tested here: the full-size Qwen3.8-27B; Underdog Bench's own 120 tasks and the 100-task parallel set (unpublished); " "SWE-bench, WebWalkerQA, IFEval and AIME; the vision add-on; Mia and Ternary Bonsai 2; thinking at the default effort.") The summary prints each step’s one-line result and the claims ledger (held, did not hold, or undecided), along with the hardware and the llama. cpp version.cpp version. It closes with what the notebook does not test: the full-size Qwen3.8-27B, the vendor’s own task lists, the agentic and math benchmarks, and the vision add-on. In conclusion, Saluki is not quite what its file suggests. Its header keeps ISTA’s 851-tensor layout and saves bits only where ISTA spent more, yet its bytes and unquantized vectors show a model whose weights were changed, consistent with Underdog calling it a preview checkpoint. That turns the tool-calling comparison with its parent into a test of that training rather than of quantization alone. Both files run in stock llama.cpp, one at a time, and BFCL’s own checker gives a paired comparison on identical tasks. Small samples, greedy decoding, and a single run limit what any one notebook can claim, so the intervals matter as much as the scores. Check out the FULL CODES here. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide appeared first on MarkTechPost.
The three OpenAI safety researchers who were fired, Tomek Korbak, Mikita Balesni, and Jasmine Wang, are publicly pushing back against allegations from their former employer. They deny sharing confidential information and see a connection to their advocacy for AI systems that are easier to scrutinize. The researchers say OpenAI never told them in writing which information they allegedly mishandled. The article Three fired OpenAI safety researchers say they were never told what they actually leaked appeared first on The Decoder.
A three-month field experiment with 133 patent attorneys shows that AI boosts immediate performance, with junior lawyers gaining the most. But when the AI is taken away, only senior attorneys retain the benefits, suggesting that baseline expertise is a prerequisite for turning AI-assisted practice into lasting skill. The article AI makes junior patent attorneys instantly better but may quietly stunt their professional growth appeared first on The Decoder.
Block, the company behind Square and Cash App, has introduced Dessa, a speech model that processes audio directly instead of converting it to text first. Dessa is designed to recognize emotions, sarcasm, and pauses in real time. The model runs locally on a MacBook Pro M4 Max but isn't publicly available yet. The article Block's new AI model Dessa was trained entirely on human speech to catch emotions text-based models miss appeared first on The Decoder.
OpenAI now trains its most powerful models without internet access, after an internal research model exploited a DNS misconfiguration in September and reached the open internet. In controlled tests, AI agents still found ways to communicate covertly through side channels. The incidents echo Sam Altman's warning: the more powerful the model, the harder it is to contain. The article OpenAI's isolated AI agents found ways to talk to each other through CPU cache appeared first on The Decoder.
In this article, we will discuss newly trending open sourced Anthropic’s sandbox-runtime (srt), the OS-level sandbox behind Claude Code’s sandboxed bash tool. The Anthropic sandbox-runtime limits which files a process can touch and which hosts it can reach. It does this without spinning up a container. It is also not tied to Claude Code. Your own agents, local MCP servers and plain command-line programs can run inside it. TL;DR Size: npm package @anthropic-ai/sandbox-runtime v0.0.81, about 11.4 MB unpacked, Apache-2.0, Node.js 22.12+. Runs on: any macOS or Linux dev machine or CI box. macOS uses Seatbelt; Linux needs bubblewrap, socat and ripgrep. Windows is alpha. Performance: sandboxing cut Claude Code permission prompts by 84% in Anthropic’s internal usage (self-reported). Best: 84% fewer permission prompts, with file and network limits enforced on the whole process tree. Worst: still a beta research preview, Windows support is alpha. What is Anthropic sandbox-runtime? Anthropic sandbox-runtime is an open-source tool that wraps any process in filesystem and network restrictions. You install it with npm install -g @anthropic-ai/sandbox-runtime, then prefix a command with srt. It works as a CLI and as a TypeScript library. How does sandbox-runtime work on macOS, Linux and Windows? srt uses each operating system’s own isolation primitives. Its restrictions cover the entire process tree, including scripts and subprocesses. macOS: sandbox-exec with dynamically generated Seatbelt profiles. Only ripgrep is required. Linux: bubblewrap with the network namespace removed. It also needs socat and ripgrep. Windows (alpha): a dedicated srt-sandbox account, a Windows Filtering Platform egress fence and explicit file ACLs. Traffic leaves only through HTTP and SOCKS5 proxies running on the host. On Linux, those proxies sit behind Unix sockets bind-mounted into the sandbox. On macOS, the Seatbelt profile allows only one localhost port. The proxies then apply your domain allowlist and denylist. Why does it need both filesystem and network isolation? Anthropic team argues that 1 boundary without the other fails. Without network isolation, a prompt-injected agent could exfiltrate SSH keys. Without filesystem isolation, it could escape the sandbox and gain network access. srt’s defaults follow that logic. Reads are allowed everywhere unless denied. Writes are denied everywhere unless allowed. Network access is denied unless a domain is allowlisted. How do you sandbox an MCP server or your own agent? For an MCP server, change the command in .mcp.json from npx to srt. Pass the original command as arguments. Rules then live in ~/.srt-settings.json: Copy CodeCopiedUse a different Browser{ "network": { "allowedDomains": ["api.github.com", "*.npmjs.org"], "deniedDomains": [] }, "filesystem": { "denyRead": ["~/.ssh"], "allowWrite": ["."], "denyWrite": [".env"] } } For your own agent, the library exposes SandboxManager. Call initialize(config) to start the proxies. Then wrapWithSandbox() returns a wrapped command string you can spawn. What does the 84% figure mean? Anthropic reports that sandboxing reduced permission prompts by 84% in internal Claude Code usage. That is a self-reported number, not an external benchmark. The target is approval fatigue. Developers who click “approve” all day stop reading what they approve. A fixed boundary lets the agent work freely inside it and ask only at the edge. <iframe id="mtp-srt-frame" title="Anthropic sandbox-runtime interactive explainer" style="width:100%;height:600px;border:0;display:block;background:#141413;border-radius:16px;" loading="lazy" srcdoc="<!doctype html> <title>sandbox-runtime Explainer</title> :root{--bg:#141413;--panel:#1f1e1c;--panel2:#2a2926;--cream:#faf9f5;--muted:#b0aea5;--orange:#d97757;--blue:#6a9bcc;--green:#788c5d;--red:#d4504a;--mtp:#76B900;--line:#3a3935} *{box-sizing:border-box;margin:0;padding:0} html,body{background:var(--bg)!important} body{font-family:Inter,system-ui,-apple-system,Segoe UI,Roboto,sans-serif;color:var(--cream);line-height:1.5} #srt-x{max-width:880px;margin:0 auto;background:var(--bg)!important;border:1px solid var(--line)!important;border-radius:16px;overflow:hidden} #srt-x .hd{padding:22px 24px 14px;border-bottom:1px solid var(--line)} #srt-x .kicker{font-size:12px;letter-spacing:.12em;text-transform:uppercase;color:var(--orange);font-weight:700} #srt-x h2{font-size:24px;font-weight:800;margin-top:4px;color:var(--cream)} #srt-x .sub{color:var(--muted);font-size:14px;margin-top:4px} #srt-x .tabs{display:flex;gap:4px;padding:10px 16px 0;border-bottom:1px solid var(--line);overflow-x:auto} #srt-x .tab{background:transparent;border:0;color:var(--muted);font:600 14px Inter,sans-serif;padding:10px 14px;cursor:pointer;border-bottom:2px solid transparent;white-space:nowrap} #srt-x .tab.on{color:var(--cream);border-bottom-color:var(--orange)} #srt-x .tab:hover{color:var(--cream)} #srt-x .pane{display:none;padding:22px 24px;animation:fade .35s ease} #srt-x .pane.on{display:block} @keyframes fade{from{opacity:0;transform:translateY(6px)}to{opacity:1;transform:none}} /* flow */ #srt-x .flow{display:grid;grid-template-columns:repeat(5,1fr);gap:10px;align-items:stretch;position:relative} #srt-x .node{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:12px 10px;text-align:center;font-size:13px;transition:all .35s ease;position:relative} #srt-x .node b{display:block;font-size:14px;margin-bottom:2px} #srt-x .node small{color:var(--muted);font-size:12px} #srt-x .node.act{border-color:var(--orange);box-shadow:0 0 0 3px rgba(217,119,87,.25);transform:translateY(-3px)} #srt-x .node.ok{border-color:var(--green);box-shadow:0 0 0 3px rgba(120,140,93,.3)} #srt-x .node.bad{border-color:var(--red);box-shadow:0 0 0 3px rgba(212,80,74,.3)} #srt-x .track{height:6px;background:var(--panel2);border-radius:6px;margin:16px 0 6px;position:relative;overflow:hidden} #srt-x .pkt{position:absolute;top:0;left:0;height:6px;width:12%;border-radius:6px;background:var(--orange);transition:left .6s cubic-bezier(.4,0,.2,1),background .3s} #srt-x .desc{background:var(--panel);border-left:3px solid var(--orange);border-radius:8px;padding:14px 16px;margin-top:14px;min-height:92px;font-size:14px} #srt-x .desc h4{font-size:15px;margin-bottom:4px} #srt-x .ctrl{display:flex;gap:8px;margin-top:14px;flex-wrap:wrap} #srt-x .btn{background:var(--panel2);color:var(--cream);border:1px solid var(--line);border-radius:999px;padding:8px 16px;font:600 13px Inter,sans-serif;cursor:pointer;transition:all .2s} #srt-x .btn:hover{border-color:var(--orange)} #srt-x .btn.pri{background:var(--orange);border-color:var(--orange);color:#141413} #srt-x .btn.sel{border-color:var(--orange);color:var(--orange)} /* sim */ #srt-x .sim{display:grid;grid-template-columns:1fr 1.25fr;gap:16px} #srt-x .card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:14px} #srt-x .card h5{font-size:12px;letter-spacing:.08em;text-transform:uppercase;color:var(--muted);margin-bottom:8px} #srt-x label.chk{display:flex;align-items:center;gap:8px;font-size:13px;padding:5px 0;cursor:pointer} #srt-x label.chk input{accent-color:var(--orange);width:16px;height:16px} #srt-x code,#srt-x pre{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace!important;background:transparent!important;color:inherit!important;border:0!important} #srt-x .cmds{display:flex;flex-direction:column;gap:6px} #srt-x .cmd{text-align:left;background:var(--panel2);border:1px solid var(--line);color:var(--cream);border-radius:8px;padding:8px 10px;font:12.5px ui-monospace,Menlo,monospace;cursor:pointer;transition:all .2s} #srt-x .cmd:hover{border-color:var(--orange);transform:translateX(3px)} #srt-x .term{background:#0c0c0b!important;border:1px solid var(--line)!important;border-radius:10px;padding:12px 14px;font:12.5px/1.6 ui-monospace,Menlo,monospace;min-height:150px;color:var(--cream)!important;white-space:pre-wrap;word-break:break-word} #srt-x .term .p{color:var(--orange)} #srt-x .term .ok{color:#9fbf7a} #srt-x .term .no{color:#ef7a73} #srt-x .term .dim{color:var(--muted)} #srt-x .why{font-size:13px;color:var(--muted);margin-top:10px;min-height:40px} /* platforms */ #srt-x .plat{display:grid;grid-template-columns:repeat(3,1fr);gap:12px} #srt-x .pc{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px;cursor:pointer;transition:all .25s} #srt-x .pc:hover,#srt-x .pc.on{border-color:var(--orange);transform:translateY(-3px)} #srt-x .pc h4{font-size:16px} #srt-x .tag{display:inline-block;font-size:11px;font-weight:700;padding:2px 8px;border-radius:999px;margin-top:4px} #srt-x .t-g{background:rgba(120,140,93,.25);color:#b5cc94} #srt-x .t-o{background:rgba(217,119,87,.2);color:var(--orange)} #srt-x .pc ul{list-style:none;margin-top:10px;font-size:13px;color:var(--muted)} #srt-x .pc li{padding:3px 0} #srt-x .pc li::before{content:"→ ";color:var(--orange)} #srt-x .pdetail{margin-top:14px} /* defaults */ #srt-x .defs{display:grid;grid-template-columns:repeat(3,1fr);gap:12px} #srt-x .dc{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px;text-align:center} #srt-x .dc .big{font-size:22px;font-weight:800;margin:6px 0} #srt-x .dc p{font-size:13px;color:var(--muted)} #srt-x .bars{margin-top:18px} #srt-x .bar{margin:10px 0} #srt-x .bar .lb{display:flex;justify-content:space-between;font-size:13px;margin-bottom:4px} #srt-x .bar .bg{height:22px;background:var(--panel2);border-radius:6px;overflow:hidden} #srt-x .bar .fg{height:22px;width:0;border-radius:6px;transition:width 1.2s cubic-bezier(.2,.8,.2,1)} #srt-x .note{font-size:12px;color:var(--muted);margin-top:8px} /* compare */ #srt-x table{width:100%;border-collapse:collapse;font-size:13px} #srt-x th,#srt-x td{border:0!important;border-bottom:1px solid var(--line)!important;padding:9px 8px;text-align:left;vertical-align:top;background:transparent!important;color:var(--cream)!important} #srt-x th{color:var(--muted)!important;font-weight:600;font-size:12px;text-transform:uppercase;letter-spacing:.05em} #srt-x td.hl{color:var(--orange)!important;font-weight:600} #srt-x tr.row{opacity:0;transform:translateX(-6px);transition:all .4s ease} #srt-x tr.row.in{opacity:1;transform:none} #srt-x .ft{display:flex;justify-content:space-between;gap:10px;flex-wrap:wrap;padding:12px 24px;border-top:1px solid var(--line);font-size:12px;color:var(--muted)} #srt-x .ft .mtp{color:var(--mtp);font-weight:700} #srt-x hr,#srt-x p:empty,#srt-x del,#srt-x s{display:none!important} #srt-x .tblwrap{overflow-x:auto} @media (max-width:640px){ #srt-x .hd{padding:18px 16px 12px} #srt-x h2{font-size:20px} #srt-x .pane{padding:18px 16px} #srt-x .flow{grid-template-columns:1fr 1fr} #srt-x .flow .node:last-child{grid-column:span 2} #srt-x .sim,#srt-x .plat,#srt-x .defs{grid-template-columns:1fr} #srt-x .ft{padding:12px 16px} } <div id="srt-x"> <div class="hd"> <div class="kicker">Anthropic · sandbox-runtime (srt)</div> <h2>How an OS-level agent sandbox works</h2> <div class="sub">Filesystem and network limits for any process, without a container.</div> </div> <div class="tabs" role="tablist"> <button class="tab on" data-t="flow">How it works</button> <button class="tab" data-t="sim">Try a command</button> <button class="tab" data-t="plat">Platforms</button> <button class="tab" data-t="defs">Defaults & 84%</button> <button class="tab" data-t="cmp">Compare</button> </div> <div class="pane on" id="p-flow"> <div class="flow"> <div class="node" data-n="0"><b>Agent / process</b><small>Claude Code, MCP server, any CLI</small></div> <div class="node" data-n="1"><b>srt wrapper</b><small>reads ~/.srt-settings.json</small></div> <div class="node" data-n="2"><b>OS boundary</b><small>Seatbelt · bubblewrap · WFP</small></div> <div class="node" data-n="3"><b>Host proxy</b><small>HTTP + SOCKS5</small></div> <div class="node" data-n="4"><b>Allowlist check</b><small>allow · deny · ask</small></div> </div> <div class="track"><div class="pkt" id="pkt"></div></div> <div class="desc" id="fdesc"></div> <div class="ctrl"> <button class="btn" id="fprev"> Back</button> <button class="btn pri" id="fnext">Next step </button> <button class="btn" id="fplay">Auto-play</button> </div> </div> <div class="pane" id="p-sim"> <div class="sim"> <div> <div class="card"> <h5>network.allowedDomains</h5> <label class="chk"> anthropic.com</label> <label class="chk"> *.npmjs.org</label> <label class="chk"> github.com</label> <h5 style="margin-top:12px">filesystem</h5> <label class="chk"> denyRead: ~/.ssh</label> <label class="chk"> denyWrite: .env</label> </div> </div> <div> <div class="cmds" id="cmds"> <button class="cmd" data-c="0">srt "cat README.md"</button> <button class="cmd" data-c="1">srt "cat ~/.ssh/id_rsa"</button> <button class="cmd" data-c="2">srt "curl anthropic.com"</button> <button class="cmd" data-c="3">srt "curl example.com"</button> <button class="cmd" data-c="4">srt "npm install"</button> <button class="cmd" data-c="5">srt "echo KEY=1 >> .env"</button> </div> <div class="term" id="term" style="margin-top:10px"><span class="dim">Pick a command. Toggle the config to see what changes.</span></div> <div class="why" id="why"></div> </div> </div> </div> <div class="pane" id="p-plat"> <div class="plat"> <div class="pc on" data-p="0"><h4>macOS</h4><span class="tag t-g">Supported</span><ul><li>sandbox-exec</li><li>Seatbelt profiles</li><li>Needs ripgrep</li></ul></div> <div class="pc" data-p="1"><h4>Linux</h4><span class="tag t-g">Supported</span><ul><li>bubblewrap</li><li>No network namespace</li><li>socat + ripgrep</li></ul></div> <div class="pc" data-p="2"><h4>Windows</h4><span class="tag t-o">Alpha</span><ul><li>srt-sandbox account</li><li>WFP egress fence</li><li>One-time install</li></ul></div> </div> <div class="desc pdetail" id="pdesc"></div> </div> <div class="pane" id="p-defs"> <div class="defs"> <div class="dc"><div class="kicker">Read</div><div class="big" style="color:#b5cc94">Allow</div><p>Allowed everywhere by default. Deny regions like ~/.ssh, re-allow with allowRead.</p></div> <div class="dc"><div class="kicker">Write</div><div class="big" style="color:#ef7a73">Deny</div><p>Denied everywhere by default. Open paths like "." and /tmp with allowWrite.</p></div> <div class="dc"><div class="kicker">Network</div><div class="big" style="color:#ef7a73">Deny</div><p>No access by default. Every host must match allowedDomains.</p></div> </div> <div class="bars" id="bars"> <div class="kicker" style="margin-bottom:4px">Claude Code permission prompts (indexed)</div> <div class="bar"><div class="lb"><span>Without sandboxing</span><span>100</span></div><div class="bg"><div class="fg" data-w="100" style="background:var(--muted)"></div></div></div> <div class="bar"><div class="lb"><span>With sandboxing</span><span>16</span></div><div class="bg"><div class="fg" data-w="16" style="background:var(--orange)"></div></div></div> <div class="note">84% fewer prompts in Anthropic's internal usage (self-reported, Oct 2025 engineering post). Indexed to 100.</div> </div> </div> <div class="pane" id="p-cmp"> <div class="tblwrap"> <table> <thead><tr><th></th><th>srt</th><th>Codex CLI</th><th>Docker Sandboxes</th><th>E2B</th></tr></thead> <tbody> <tr class="row"><td>Isolation</td><td class="hl">OS primitives</td><td>OS primitives</td><td>microVM</td><td>Firecracker microVM</td></tr> <tr class="row"><td>Runs</td><td class="hl">Local</td><td>Local</td><td>Local</td><td>Cloud / BYOC</td></tr> <tr class="row"><td>Container or VM</td><td class="hl">No</td><td>No</td><td>Yes</td><td>Yes</td></tr> <tr class="row"><td>Network default</td><td class="hl">Deny all</td><td>Off (toggle)</td><td>Chosen policy</td><td>On</td></tr> <tr class="row"><td>Domain allowlist</td><td class="hl">Wildcards + ports</td><td>Not disclosed</td><td>Wildcards + ports</td><td>allowOut, wildcards</td></tr> <tr class="row"><td>License</td><td class="hl">Apache-2.0</td><td>Apache-2.0</td><td>Not disclosed</td><td>Apache-2.0 (SDK)</td></tr> </tbody> </table> </div> </div> <div class="ft"><span>Sources: github.com/anthropics/sandbox-runtime · anthropic.com/engineering · v0.0.81</span><span class="mtp">Built by Marktechpost</span></div> </div> (function(){ var R=document.getElementById('srt-x'); function post(){try{parent.postMessage({srtH:R.offsetHeight+40},'*')}catch(e){}} var tabs=R.querySelectorAll('.tab'); tabs.forEach(function(b){b.addEventListener('click',function(){ tabs.forEach(function(x){x.classList.remove('on')});b.classList.add('on'); R.querySelectorAll('.pane').forEach(function(p){p.classList.remove('on')}); var t=b.getAttribute('data-t');document.getElementById('p-'+t).classList.add('on'); if(t==='defs')bars();if(t==='cmp')rows();setTimeout(post,60); })}); // flow var steps=[ ['Your agent runs a command','The agent, MCP server or CLI tool issues a command. It is unchanged; srt simply wraps it.',0,'act'], ['srt builds the policy','srt loads ~/.srt-settings.json (or built-in defaults) and starts HTTP and SOCKS5 proxies on the host.',1,'act'], ['The OS enforces the boundary','Seatbelt (macOS), bubblewrap (Linux) or a WFP fence (Windows alpha) applies file rules to the whole process tree.',2,'act'], ['All traffic goes through the proxy','The sandbox has no direct route out. On Linux, the network namespace is removed; proxies are reached via Unix sockets.',3,'act'], ['Allowed host: connection proceeds','A host in allowedDomains passes, after a check that it does not resolve to loopback or cloud metadata IPs.',4,'ok'], ['Unknown host: blocked or asked','A denied or unlisted host is refused, or an ask callback decides. The violation is logged for the agent to read.',4,'bad'] ]; var fi=0,timer=null; function draw(){ var s=steps[fi];var ns=R.querySelectorAll('.node'); ns.forEach(function(n,i){n.classList.remove('act','ok','bad');if(i<s[2])n.classList.add('act')}); ns[s[2]].classList.add(s[3]); var pk=document.getElementById('pkt');pk.style.left=(s[2]*22)+'%'; pk.style.background=s[3]==='ok'?'#788c5d':s[3]==='bad'?'#d4504a':'#d97757'; document.getElementById('fdesc').innerHTML='<h4>Step '+(fi+1)+' of '+steps.length+': '+s[0]+'</h4>'+s[1]; } document.getElementById('fnext').onclick=function(){fi=(fi+1)%steps.length;draw()}; document.getElementById('fprev').onclick=function(){fi=(fi-1+steps.length)%steps.length;draw()}; document.getElementById('fplay').onclick=function(){ if(timer){clearInterval(timer);timer=null;this.textContent='Auto-play';return} this.textContent='Pause';timer=setInterval(function(){fi=(fi+1)%steps.length;draw()},1800); }; draw(); // sim function on(id){return document.getElementById(id).checked} var C=[ function(){return [1,'# Anthropic Sandbox Runtime (srt) ...','Reads are allowed by default, and the current directory is readable.']}, function(){return on('f-ssh')?[0,'cat: ~/.ssh/id_rsa: Operation not permitted','~/.ssh is in denyRead, so the key cannot be read.']:[1,'-----BEGIN OPENSSH PRIVATE KEY----- ...','Without a denyRead rule, reads are open everywhere. Deny secrets explicitly.']}, function(){return on('d-anth')?[1,'...','anthropic.com is in allowedDomains, so the proxy forwards it.']:[0,'Connection blocked by network allowlist','No allowlist match: network is deny-by-default.']}, function(){return [0,'Connection blocked by network allowlist','example.com is not allowlisted.']}, function(){return on('d-npm')?[1,'packages installed (illustrative output)','*.npmjs.org is allowlisted, so the registry is reachable.']:[0,'npm ERR! network request to registry.npmjs.org failednConnection blocked by network allowlist','The classic trade-off: too strict, and installs fail.']}, function(){return on('f-env')?[0,'bash: .env: Operation not permitted','denyWrite beats allowWrite, so .env stays protected inside the project.']:[1,'(written)','"." is in allowWrite, so the write succeeds.']} ]; var busy=false; R.querySelectorAll('.cmd').forEach(function(b){b.addEventListener('click',function(){ if(busy)return;busy=true;var r=C[+b.getAttribute('data-c')]();var t=document.getElementById('term'); t.innerHTML='<span class="p">$</span> '+b.textContent.replace(/</g,'<')+'n<span class="dim">Running inside sandbox...</span>'; document.getElementById('why').textContent=''; setTimeout(function(){ t.innerHTML='<span class="p">$</span> '+b.textContent.replace(/</g,'<')+'n<span class="'+(r[0]?'ok':'no')+'">'+(r[0]?'✓ ':'✗ ')+r[1].replace(/</g,'<')+'</span>'; document.getElementById('why').textContent=r[2];busy=false;post(); },650); })}); // platforms var P=['macOS uses the built-in sandbox-exec tool with Seatbelt profiles generated per run. The profile allows only 1 localhost port, where the srt proxies listen. Violations can be read from the system log in real time.', 'Linux uses bubblewrap and removes the network namespace entirely. Proxies are reached through Unix sockets bind-mounted into the sandbox. Ubuntu 24.04+ needs an AppArmor change for user namespaces.', 'Windows (alpha) runs commands as a dedicated srt-sandbox account. A machine-wide WFP filter blocks its outbound traffic except to the proxy ports. Setup needs 1 elevated install.']; var pcs=R.querySelectorAll('.pc'); function pd(i){pcs.forEach(function(x){x.classList.remove('on')});pcs[i].classList.add('on');document.getElementById('pdesc').innerHTML='<h4>'+pcs[i].querySelector('h4').textContent+'</h4>'+P[i];post()} pcs.forEach(function(c,i){c.addEventListener('click',function(){pd(i)})});pd(0); function bars(){R.querySelectorAll('.fg').forEach(function(f){f.style.width='0';setTimeout(function(){f.style.width=f.getAttribute('data-w')+'%'},80)})} function rows(){R.querySelectorAll('tr.row').forEach(function(r,i){r.classList.remove('in');setTimeout(function(){r.classList.add('in')},90*i+60)})} window.addEventListener('load',post);window.addEventListener('resize',post);setTimeout(post,300); })(); "> (function(){var f=document.getElementById('mtp-srt-frame');window.addEventListener('message',function(e){if(e.source===f.contentWindow&&e.data&&e.data.srtH){f.style.height=e.data.srtH+'px';}});})(); The real trade-off: strict allowlists vs working agents Coding agents do more than edit files. They install dependencies, pull docs and call APIs. Make the allowlist too strict and npm install fails. Make it too loose and the sandbox protects little. srt ships several knobs for that middle ground: Wildcards and ports: entries like *.npmjs.org or api.example.com:443. Ask callback: decides hosts no rule covers, and can return a reason the model reads. Per-command allow lists: open a registry for 1 invocation only. Live updates: --control-fd swaps network lists while a command runs. Resolved-address check: blocks allowed names that resolve to loopback or cloud metadata IPs. Denied-domain reasons: tell the agent the sanctioned alternative, such as an HTTPS git remote.. How does it compare with other agent sandboxes? Key Takeaways srt sandboxes any process at the OS level, no container needed. Seatbelt on macOS, bubblewrap on Linux, Windows in alpha. Network is deny-by-default; every connection passes a host-side allowlist proxy. Anthropic reports 84% fewer Claude Code permission prompts (internal). Allowlist breadth is the real security dial; broad domains can leak data. Check out the GitHub repo, the Anthropic engineering post and the Claude Code sandboxing docs. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meet Anthropic’s sandbox-runtime: OS-Level Sandboxing for AI Agents, No Containers appeared first on MarkTechPost.
