Cohere AI has released Rerank 5, a new reranker family in Pro and Fast tiers. Cohere Rerank 5 is a retrieval model that reorders search results before they reach an LLM. It adds 2 developer features: Context Focus, which returns relevant passages inside each document, and calibrated relevance scores. For RAG and agent builders, the message is simple. Send fewer, better tokens to the LLM. TL;DR Size: Parameter count not disclosed. 32K token context window. Two tiers: rerank-v5.0-pro and rerank-v5.0-fast. Runs on: Hosted API, Model Vault, Microsoft Foundry, Amazon SageMaker, or private VPC and on-prem. Cohere benchmarks speed on a single NVIDIA A100. No open weights or quantized builds. Performance: Pro ranks first on 39 of 46 composite datasets in Cohere’s own evaluation. Best: Pro scores 82.2 on BRIGHT (RCP-nDCG@10), 5.8 points above Voyage AI Rerank 3. Worst: Pro median latency is 11.7 s per 96-document request, versus 4.1 s for Fast. What is new in Cohere Rerank 5? Rerank 5 ships as two models. Pro targets ranking quality on reasoning-heavy queries and long, domain-specific documents. Fast targets low latency and high throughput for agents running many searches. Both models support over 100 languages and a 32K context window. They also rerank semi-structured data such as JSON, tables and YAML records. Upgrading needs no re-indexing. Developers change the model parameter and keep their existing keyword, vector or hybrid search. How does Context Focus work? Context Focus is an optional feature, off by default. A single Rerank call returns document scores plus scored character spans. Each span carries start and end offsets into the original text. Developers set it through the relevant_spans field, with a threshold and max_spans value. Cohere tested it on 30 QA datasets. Removing half the retrieved text kept 96% of answer quality. Removing three quarters kept 87%. Cohere says teams can push to 10x less context at some quality cost. In a search-agent test across 17 datasets, BM25 alone scored 46% accuracy. Adding Rerank 5 Pro raised it to 49%. Context Focus at the default 0.1 threshold cut tokens per task by 19% and reached 57%. At 0.4, tokens fell 34% and accuracy hit 58%. Versus BM25 alone, the full stack used 35% fewer tokens. On 6 deep-search benchmarks, tokens fell 41% and accuracy rose from 39% to 52%. What do calibrated relevance scores change? Rerank 5 scores read as probabilities. A score of 0.7 means a 70% chance the document is useful. Cohere team measured expected calibration error (ECE) against blind human graders and NIST TREC-DL assessors. Rerank 5 scores about 0.06 on both. Voyage AI Rerank 3 scores 0.234 and 0.307. Rerank 4 Pro scores 0.384 and 0.309. In practice, dev teams can filter by threshold instead of a fixed top-k. An agent can also detect when nothing relevant came back. How does Rerank 5 perform on benchmarks? All scores below are self-reported by Cohere using RCP-nDCG@10, a Cohere-developed variant of nDCG. Cohere ran the competitor models itself. Pro leads all four main benchmarks. It scores 92.2 on ViDoRe V3, 90.4 on BEIR, 82.2 on BRIGHT and 87.3 on nanoMTEB. BRIGHT shows the largest jump: 12.4 points over Rerank 4 Pro. Fast beats Voyage AI Rerank 3 Lite, Qwen3-Reranker-8B and Rerank 4 Pro on all 4. It ties Voyage AI Rerank 3 on ViDoRe V3 at 90.6. It trails it on BEIR, 88.1 versus 88.6. On parsed PDFs, averaged over 47 datasets, Pro scores 89.9 and Fast 88.5. Voyage AI Rerank 3 scores 88.4. Pro ranks first on 14 of 19 non-English languages in Cohere’s internal suite. (function(){var f=document.getElementById("mtp-rr5-frame");window.addEventListener("message",function(e){if(e.data&&e.data.mtpFrame==="mtp-rr5"&&e.data.h){f.style.height=e.data.h+"px";}});})(); How does Rerank 5 compare with competing rerankers? *Scores are RCP-nDCG@10 or ECE from Cohere’s Rerank 5 post, measured by Cohere. Not independently verified. How can developers deploy Rerank 5? Rerank 5 is generally available on the Cohere API and Model Vault. It also runs on Microsoft Foundry (Pro, Fast) and Amazon SageMaker (Pro, Fast). It is also available to all North users. API pricing is $0.05 per 1M tokens for Pro and $0.02 for Fast. Model Vault lists $4.00 per instance-hour for both tiers on its Medium tier. Integrations include LangChain, Haystack, Elasticsearch and Weaviate, per Cohere’s integration docs. Key Takeaways Rerank 5 ships in Pro ($0.05) and Fast ($0.02) per 1M tokens, both with 32K context. Pro scores 82.2 on BRIGHT, 5.8 points above Voyage AI Rerank 3. Context Focus returns scored passages in 1 call, cutting agent tokens 35% to 41%. Calibrated scores reach about 0.06 ECE, so 1 threshold works across domains. All benchmarks are self-reported on Cohere’s RCP-nDCG@10 metric. Check out the official announcement, the Rerank 5 changelog and the full results sheet. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Cohere Releases Rerank 5 With Passage-Level Context Focus and Calibrated Scores appeared first on MarkTechPost.
Only 13 percent of companies are getting real business value from their AI investments, according to a Gartner survey of 2,002 CIOs. At the same time, 51 percent expect AI to actually increase their operating costs. Yet 75 percent of CEOs plan to keep raising their technology budgets. So far, strategic conviction is beating measurable results. The article Only 13 percent of companies see real returns from AI, Gartner survey finds appeared first on The Decoder.
In this tutorial, we implement OmniParseBench, Datalab’s open benchmark for document parsers, which grades a parser’s output with 16,288 small pass-or-fail tests across 2,937 pages, from scanned forms and handwriting to arXiv math and dense tables. Rather than trust the leaderboard, we rebuild it from the verdicts Datalab released for 27 parsers, after feeding the scorer a hand-made page to see which mistakes flip a verdict. We then measure ranking certainty, slice it by document type, test whether the two models the card names as helping build tests behave differently on tests a model corrected, and run Tesseract and Indra OCR ourselves on a 50-page sample. Along the way, we find that the top two parsers are statistically tied, the leader changes on most slices, and one launch figure doesn’t match the released files. Copy CodeCopiedUse a different Browserimport os import re import sys import json import time import html import shutil import platform import traceback import subprocess import collections import urllib.request from pathlib import Path RESULTS = {} # step -> one-line result, for the summary table CLAIMS = [] # (step, claim, held) CURRENT = ["1"] def banner(title): print("n" + "=" * 78) print(title) print("=" * 78) def section(name): def wrap(fn): def run(*a, **kw): CURRENT[0] = name.split(".")[0] banner(name) t0 = time.time() try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}" print(f"n[!] {name} did not complete: {type(e).__name__}: {e}") traceback.print_exc(limit=3) return None finally: print(f"n[{time.time() - t0:.1f} s]") return run return wrap def claim(text, held): CLAIMS.append((CURRENT[0], text, bool(held))) return held banner("1. Install omni-parse-bench 1.0.0rc1 and Tesseract, and pin the dataset revision") SYSTEM = platform.system() OPB_VERSION = "1.0.0rc1" # a pre-release: a bare `pip install omni-parse-bench` resolves to 0.2.0, whose API differs REPO = "datalab-to/omni_parse_bench" REV = "961c0f89a2da701f5ce53afbd5e80aa469b0369a" # the dataset with the published verdicts of 27 parsers INDRA = ["indra-ocr==0.3.2", "numpy==2.5.4", "onnxruntime==1.31.0", "opencv-python-headless==5.0.0.93", "pypdfium2==5.14.0", "huggingface_hub==2.2.0"] # the pins of the package's own Indra server image WORK = Path(os.environ.get("OPB_WORK") or ("/content/opb" if os.path.isdir("/content") else "opb_work")).resolve() WORK.mkdir(parents=True, exist_ok=True) subprocess.run([sys.executable, "-m", "pip", "install", "-q", f"omni-parse-bench[benchmark]=={OPB_VERSION}", "uv"], check=True) if SYSTEM == "Linux" and shutil.which("tesseract") is None: scripts = ("arab armn beng cans cher cyrl deva ethi frak geor grek gujr guru hang hang-vert hans hans-vert hant " "hant-vert hebr jpan jpan-vert knda khmr laoo latn mlym mymr orya sinh syrc taml telu thaa thai tibt viet").split() subprocess.run(["apt-get", "-qq", "update"], check=True) subprocess.run(["apt-get", "-qq", "install", "-y", "tesseract-ocr", "tesseract-ocr-eng", "tesseract-ocr-osd"] + [f"tesseract-ocr-script-{s}" for s in scripts], check=True, stdout=subprocess.DEVNULL) import numpy as np import pandas as pd import importlib.metadata as md from huggingface_hub import snapshot_download from omni_parse_bench import benchmark, data, harness, markdown, vocab from omni_parse_bench.scorers import unit_tests print(f"omni-parse-bench {md.version('omni-parse-bench')}, Python {platform.python_version()}, {SYSTEM} {platform.machine()}, " f"{os.cpu_count()} CPU cores") if SYSTEM == "Linux": print("os:", open("/etc/os-release").read().split('PRETTY_NAME="')[1].split('"')[0]) tess = subprocess.run(["tesseract", "--version"], capture_output=True, text=True) print("tesseract:", (tess.stdout or tess.stderr).splitlines()[0]) TESS_MAJOR = int((tess.stdout or tess.stderr).split()[1].split(".")[0]) print("test families:", {f: [t for t, g in vocab.FAMILIES.items() if g == f] for f in dict.fromkeys(vocab.FAMILIES.values())}) print(f"dataset {REPO} @ {REV[:7]}; work folder {WORK}") RESULTS["1. Install omni-parse-bench 1.0.0rc1 and Tesseract, and pin the dataset revision"] = ( f"omni-parse-bench {md.version('omni-parse-bench')}, tesseract {(tess.stdout or tess.stderr).split()[1]}, dataset @ {REV[:7]}") We install omni-parse-bench pinned to 1.0.0rc1, because a bare pip install still resolves to 0.2.0, an older release with a different API. The benchmark extra brings the parser adapters, the Hugging Face client, and pyarrow for the manifest, and on Colab we add Tesseract with its script models through apt. We also pin the dataset to one commit, since the repository changes as parsers are added. Printing the families shows the structure the benchmark rests on: present, order, and repeat tests form the text family; table_cell forms the tables family; and layout_kind forms the layout family. Copy CodeCopiedUse a different Browser@section("2. How the scorer decides: one hand-built page, five test types, one change at a time") def scorer_anatomy(): T = lambda i, tt, out, tags, args, st=None: {"id": i, "test_type": tt, "output_type": out, "derivation": "authored", "tags": tags, "args": args, "settings": st or {}} exact = lambda s: {"text": s, "max_diffs": 0} gold = [ T("title", "present", "html", ["heading"], {"content": exact("Quarterly Report")}), T("total", "present", "html", ["eng"], {"content": exact("Total due")}), T("growth", "present", "html", ["eng"], {"content": {"number": "-4.8%"}}), T("amount", "present", "html", ["eng"], {"content": {"number": "1,240.00"}}), T("eq_ratio", "present", "html", ["math"], {"content": {"latex": r"alpha_{1} + beta"}}), T("eq_half", "present", "html", ["math"], {"content": {"latex": r"frac{1}{2} to x"}}), T("order", "order", "html", ["eng"], {"before": exact("Introduction: this report covers the second quarter."), "after": exact("Conclusion: the plan holds.")}), T("repeat", "repeat", "html", ["eng"], {"content": exact("Revenue")}, {"expected_count": 2}), T("cell_top", "table_cell", "html", ["table"], {"cell": exact("1,240"), "top_heading": exact("2024")}), T("cell_left", "table_cell", "html", ["table"], {"cell": exact("1,240"), "left_heading": exact("Income")}), T("cell_price", "table_cell", "html", ["table"], {"cell": exact("9.50"), "top_heading": exact("Price"), "left": exact("Widget")}), T("block_title", "layout_kind", "blocks", ["heading"], {"content": dict(exact("Quarterly Report"), place=[[0.10, 0.04, 0.90, 0.09]])}, {"kind": "heading"}), T("block_total", "layout_kind", "blocks", ["eng"], {"content": dict(exact("Total due"), place=[[0.10, 0.12, 0.90, 0.16]])}, {"kind": "text"}), ] gold_file = WORK / "fixture" / "gold" / "demo-p1.json" # a gold file is named after its page gold_file.parent.mkdir(parents=True, exist_ok=True) gold_file.write_text(json.dumps({"tests": gold}, indent=1)) page = data.Page(page_id="demo-p1", document="demo", suite="demo", doc_path=WORK / "fixture" / "demo-p1.pdf", gt_path=gold_file, outputs=("blocks", "html")) tests = unit_tests.gold(page) table_a = ("<table><tr><th></th><th>2023</th><th>2024</th></tr><tr><th>Income</th><td>1,000</td><td>1,240</td></tr>" "<tr><th>Costs</th><td>600</td><td>700</td></tr></table>") table_b = "<table><tr><th>Product</th><th>Price</th></tr><tr><td>Widget</td><td>9.50</td></tr></table>" md_table_b = "| Product | Price |n|---|---|n| Widget | 9.50 |" parts = {"title": "<h1>Quarterly Report</h1>", "total": "<p>Total due: 1,240.00 USD.</p>", "growth": "<p>Growth was -4.8% year on year.</p>", "ratio": r"<p>Ratio: 1+ holds.</p>", "half": r"<p>Half: 12x in the model.</p>", "intro": "<p>Introduction: this report covers the second quarter.</p>", "revenue": "<p>Revenue is shown in thousands. Revenue grew in the second quarter.</p>", "table_a": table_a, "conclusion": "<p>Conclusion: the plan holds.</p>", "table_b": table_b} blocks = [{"label": "heading", "bbox": [0.10, 0.04, 0.90, 0.09], "text": "Quarterly Report"}, {"label": "text", "bbox": [0.10, 0.12, 0.90, 0.16], "text": "Total due: 1,240.00 USD."}, {"label": "table", "bbox": [0.10, 0.40, 0.90, 0.60], "text": "Income 1,000 1,240 Costs 600 700"}] def pred(change=None, order=None, new_blocks=None): p = dict(parts, **(change or {})) return {"html": "n".join(p[k] for k in (order or list(parts))), "blocks": blocks if new_blocks is None else new_blocks} base = {v.id: v.result for v in unit_tests.score(pred(), tests)} print(f"base prediction: {sum(r == 'pass' for r in base.values())} of {len(tests)} tests passn") swapped = ["title", "total", "growth", "ratio", "half", "conclusion", "intro", "revenue", "table_a", "table_b"] variants = [ ("flip", "a typo: 'Total dve'", pred({"total": "<p>Total dve: 1,240.00 USD.</p>"}), ()), ("flip", "the minus sign dropped: 4.8%", pred({"growth": "<p>Growth was 4.8% year on year.</p>"}), ()), ("flip", "another symbol: alpha_2", pred({"ratio": r"<p>Ratio: 2+ holds.</p>"}), ()), ("flip", "conclusion before introduction", pred(order=swapped), ()), ("flip", "'Revenues' for the second 'Revenue'", pred({"revenue": "<p>Revenue is shown in thousands. Revenues grew in the second quarter.</p>"}), ()), ("flip", "2023 and 2024 headings swapped", pred({"table_a": table_a.replace("<th>2023</th><th>2024</th>", "<th>2024</th><th>2023</th>")}), ()), ("flip", "row label 'Margin' for 'Income'", pred({"table_a": table_a.replace("<th>Income</th>", "<th>Margin</th>")}), ()), ("flip", "a markdown table left in the html", pred({"table_b": md_table_b}), ()), ("flip", "the title's block labelled text", pred(new_blocks=[dict(blocks[0], label="text")] + blocks[1:]), ()), ("control", r"dfrac12rightarrow for frac{1}{2}to", pred({"half": r"<p>Half: $dfrac12rightarrow x$ in the model.</p>"}), ()), ("control", "'TOTAL DUE' and '1240.00'", pred({"total": "<p>TOTAL DUE: 1240.00 USD.</p>"}), ()), ("control", "markdown table through markdown.to_html", pred({"table_b": markdown.to_html(md_table_b)}), ()), ("unsupported", "no blocks, declared unsupported", {"html": pred()["html"]}, ("blocks",)), ] as_expected = 0 for kind, name, p, unsupported in variants: changed = [v for v in unit_tests.score(p, tests, unsupported) if v.result != base[v.id]] if kind == "flip": ok = len(changed) == 1 and changed[0].result == "fail" elif kind == "control": ok = not changed else: ok = bool(changed) and all(v.result == "unsupported" for v in changed) as_expected += ok shown = "; ".join(f"{v.id} -> {v.result}" + (f" ({v.why})" if v.result == "fail" else "") for v in changed) or "no verdict changes" print(f"{kind:11s} {name:42s} {shown}") s = unit_tests.summarize(unit_tests.lines(unit_tests.score(variants[0][2], tests), tests, page)) print(f"nsummary of the typo variant: headline {s['headline']:.4f} = mean of " + ", ".join(f"{k} {v['score']:.3f} ({v['tests']} tests)" for k, v in s["families"].items())) claim("every one-change variant flips exactly one verdict to fail, and no control flips any", as_expected == len(variants)) return f"{as_expected} of {len(variants)} hand-built variants behave as expected (9 flips, 3 controls, 1 unsupported)" scorer_anatomy() Before trusting a score, we want to see what a verdict means, so we write a gold file by hand with thirteen tests on an invented page and grade predictions with unit_tests.score, the function the benchmark itself uses. A base prediction passes everything, and each variant changes one thing. A one-letter typo, a lost minus sign, a different symbol in an equation, swapped headings and a mislabelled block each flip exactly one test to fail, with a reason attached. The controls still pass: capitals, a dropped thousands separator and dfrac in place of frac, because text is folded and equations are compared after KaTeX renders them. Copy CodeCopiedUse a different Browser@section("3. The dataset: 16,288 tests, what they check, and how each answer was established") def dataset_step(): global DATA, PAGES, TESTS, CARD_TEXT snapshot_download(REPO, repo_type="dataset", revision=REV, local_dir=str(WORK / "dataset"), allow_patterns=["manifest.parquet", "gold/*", "results.parquet", "README.md"]) DATA, PAGES = data.read_manifest(str(WORK / "dataset")) rows = [] for p in PAGES: for t in unit_tests.gold(p): rows.append(dict(test_id=t.id, page_id=p.page_id, suite=p.suite, document=p.document, test_type=t.test_type, derivation=t.derivation, tags=list(t.tags), pdf=p.doc_path.suffix == ".pdf")) TESTS = pd.DataFrame(rows) tags = collections.Counter(g for ts in TESTS.tags for g in ts) langs = {g for g in tags if re.fullmatch(r"[a-z]{3}", g)} scripts = {g for g in tags if re.fullmatch(r"[A-Z][a-z]{3}", g)} n_pdf = sum(p.doc_path.suffix == ".pdf" for p in PAGES) print(f"{len(TESTS):,} tests on {TESTS.page_id.nunique():,} pages from {TESTS.document.nunique():,} documents in " f"{TESTS.suite.nunique()} suites; {n_pdf:,} PDF pages and {len(PAGES) - n_pdf:,} images") print(f"tags: {len(langs)} language codes ({len(langs - {'und', 'zxx'})} without und and zxx), {len(scripts)} scripts") counts = lambda col: {k: int(v) for k, v in col.value_counts().items()} print("ntests by type: ", counts(TESTS.test_type)) print("tests by family: ", counts(TESTS.test_type.map(vocab.FAMILIES))) print("structure and role tags:", {g: n for g, n in tags.most_common() if g not in langs | scripts}) print("nhow each test's answer was established (derivation):") for d, n in TESTS.derivation.value_counts().items(): print(f" {d:22s} {n:6,d} {', '.join(f'{k} {v}' for k, v in TESTS[TESTS.derivation == d].test_type.value_counts().items())}") CARD_TEXT = (DATA / "README.md").read_text() claim("16,288 tests on 2,937 pages from 2,343 documents in 22 suites, as the card says", (len(TESTS), TESTS.page_id.nunique(), TESTS.document.nunique(), TESTS.suite.nunique()) == (16288, 2937, 2343, 22)) return f"{len(TESTS):,} tests, {TESTS.page_id.nunique():,} pages, {TESTS.document.nunique():,} documents, {len(langs - {'und', 'zxx'})} languages" dataset_step() With the scorer understood, we download the manifest, the gold files, and the published verdicts at the pinned revision, skip the 1.5 GB of page images, and read every page’s tests through unit_tests.gold. The counts match the dataset card, including 92 languages once the undetermined and no-language codes are removed. Two fields deserve attention. Tags such as handwriting, rotated, or table never decide a verdict; they only let us slice results. Derivation records how each answer was established, from publisher markup and text layers to model panels and human review, which lets us test later whether test construction favored any parser. Copy CodeCopiedUse a different BrowserCOLS = ["page_id", "suite", "document", "test_id", "test_type", "output_type", "derivation", "tags", "result", "why"] STARRED = ["Gemini 3.8 Flash", "Claude Sonnet 5.5"] # the card: models "used in creating and verifying at least some tests" @section("4. Recompute the published leaderboard from 439,776 verdicts") def recompute_table(): global VERDICTS, TABLE VERDICTS = pd.read_parquet(DATA / "results.parquet") VERDICTS["output_type"] = VERDICTS.test_type.map(vocab.READS) VERDICTS["tags"] = VERDICTS.tags.apply(list) VERDICTS["family"] = VERDICTS.test_type.map(vocab.FAMILIES) rows = [] for system, g in VERDICTS.groupby("system", sort=False): s = unit_tests.summarize(g[COLS].to_dict("records")) fam = {k: (np.nan if v["score"] is None else 100 * v["score"]) for k, v in s["families"].items()} scored = g[g.result != "unsupported"] rows.append(dict(system=system, headline=np.nan if s["headline"] is None else 100 * s["headline"], text=fam.get("text"), tables=fam.get("tables"), layout=fam.get("layout"), pooled=100 * (scored.result == "pass").mean())) TABLE = pd.DataFrame(rows).set_index("system") card = {} for line in CARD_TEXT.splitlines(): cells = [c.strip().strip("*").strip() for c in line.strip().strip("|").split("|")] if len(cells) == 6 and cells[1] in ("open", "closed"): card[cells[0]] = [None if c in ("none", "unsupported") else float(c) for c in cells[2:]] mismatch = [] for name, vals in card.items(): mine = [None if pd.isna(x) else round(x, 1) for x in TABLE.loc[name, ["headline", "text", "tables", "layout"]]] mismatch += [(name, a, b) for a, b in zip(mine, vals) if a != b] print(f"{len(VERDICTS):,} verdicts = {VERDICTS.system.nunique()} parsers x {VERDICTS.test_id.nunique():,} tests: " f"{ {k: int(v) for k, v in VERDICTS.result.value_counts().items()} }") numeric = sum(v is not None for vals in card.values() for v in vals) print(f"the card's table: {len(card)} parsers, {4 * len(card)} entries ({numeric} numbers, the rest none or unsupported); " f"recomputed with unit_tests.summarize, {len(mismatch)} differ {mismatch[:3] if mismatch else ''}n") show = TABLE.sort_values("headline", ascending=False, na_position="last") f = lambda x: "none" if pd.isna(x) else f"{x:.2f}" print(f"{'parser':24s} {'headline':>9s} {'text':>7s} {'tables':>7s} {'layout':>7s} {'pooled':>7s}") for name, r in show.iterrows(): if name in show.index[:8] or pd.isna(r.headline): print(f"{name + ('*' if name in STARRED else ''):24s} {f(r.headline):>9s} {f(r.text):>7s} {f(r.tables):>7s} " f"{f(r.layout):>7s} {r.pooled:7.2f}") a, g = TABLE.loc["Datalab accurate", "headline"], TABLE.loc["Gemini 3.8 Flash", "headline"] ranked = TABLE.dropna(subset=["headline"]) print(f"nDatalab accurate {a:.3f} vs Gemini 3.8 Flash {g:.3f}: both {a:.1f} at one decimal, {a - g:+.3f} points apart") print("top 3 by headline (mean of the three families):", list(ranked.headline.nlargest(3).index)) print("top 3 by pooled pass rate over every scored test:", list(ranked.pooled.nlargest(3).index)) claim("every number in the card's results table is reproduced from results.parquet", not mismatch) return f"{4 * len(card) - len(mismatch)} of {4 * len(card)} card entries reproduced; Datalab accurate {a:.2f}, Gemini {g:.2f}" recompute_table() Because results.parquet holds one row per parser per test, we can rebuild the leaderboard by passing each parser’s rows to unit_tests.summarize. The headline is the mean of the text, tables and layout pass rates with unsupported tests left out, so a parser that cannot return layout blocks, such as Tesseract, gets no headline at all. All 108 entries in the card’s table match its one-decimal values. The recomputation also exposes a near tie at the top: Datalab accurate and Gemini 3.8 Flash both round to 92.9, 0.024 points apart, and pooling every test instead of averaging families puts Gemini first. Copy CodeCopiedUse a different Browserdef per_document(df, systems, groups): """For each system: an array (documents x groups x [passes, scored tests]); `groups` maps a name to a row mask.""" docs = {d: i for i, d in enumerate(sorted(df.document.unique()))} s = df[df.result != "unsupported"] out = {} for name in systems: g = s[s.system == name] arr = np.zeros((len(docs), len(groups), 2)) for k, mask in enumerate(groups.values()): sel = g[mask(g)] d = sel.document.map(docs).to_numpy() arr[:, k, 0] = np.bincount(d, weights=(sel.result == "pass").to_numpy(), minlength=len(docs)) arr[:, k, 1] = np.bincount(d, minlength=len(docs)) out[name] = arr return out, len(docs) def resample(n_docs, B, seed): """B bootstrap resamples of the documents, each as a count per document.""" rng = np.random.default_rng(seed) return np.stack([np.bincount(rng.integers(0, n_docs, n_docs), minlength=n_docs) for _ in range(B)]).astype(float) @section("5. How sure is the order at the top? A document-level bootstrap") def bootstrap_step(): top = list(TABLE.headline.nlargest(6).index) families = {f: (lambda g, f=f: g.family == f) for f in ("text", "tables", "layout")} arrs, n_docs = per_document(VERDICTS[VERDICTS.system.isin(top)], top, families) W = resample(n_docs, 2000, seed=20261022) head = {} for s in top: tot = np.tensordot(W, arrs[s], axes=([1], [0])) # resamples x families x [passes, tests] head[s] = 100 * (tot[..., 0] / tot[..., 1]).mean(axis=1) point = {s: 100 * (arrs[s].sum(0)[:, 0] / arrs[s].sum(0)[:, 1]).mean() for s in top} print(f"2,000 resamples of {n_docs:,} documents; a document's tests stay togethern") print(f"{'parser':22s} {'headline':>9s} {'95% interval':>15s} {'gap to ' + top[0]:>26s} {'resamples where it leads':>24s}") leader = np.argmax(np.stack([head[x] for x in top]), axis=0) for s in top: lo, hi = np.percentile(head[s], [2.5, 97.5]) d = head[top[0]] - head[s] gap = "" if s == top[0] else f"{point[top[0]] - point[s]:+.2f} [{np.percentile(d, 2.5):+.2f}, {np.percentile(d, 97.5):+.2f}]" print(f"{s:22s} {point[s]:9.2f} {lo:6.2f} - {hi:6.2f} {gap:>26s} {100 * np.mean(leader == top.index(s)):23.1f}%") d = head[top[0]] - head[top[1]] lo, hi = np.percentile(d, [2.5, 97.5]) claim(f"{top[0]} and {top[1]} cannot be told apart (the 95% interval of their gap includes zero)", lo < 0 6s} {'docs':>5s} {'first (overall rank)':34s} {'second':26s} families") leaders = collections.Counter() for tag in ("handwriting", "tiny", "rotated", "degraded", "math", "table", "form", "multi_column", "Grek", "Hani", "Cyrl", "Armn", "Laoo"): sl = V[V.tags.apply(lambda t: tag in t)] head, fam = slice_headline(sl) order = head.sort_values(ascending=False) (a, x), (b, y) = list(order.items())[:2] docs = sl.document.nunique() leaders[a] += 1 note = (" (tie for first)" if x == y else "") + (" (10 or fewer documents)" if docs <= 10 else "") print(f"{tag:13s} {sl.test_id.nunique():6d} {docs:5d} {f'{a} {x:.1f} (#{rank[a]})':34s} {f'{b} {y:.1f}':26s} " f"{'+'.join(c for c in ('text', 'tables', 'layout') if c in fam.columns)}{note}") first = ranked.index[0] claim("the overall leader is not first on every slice", leaders[first] 6s} {'starred in / out':>17s} {'others in / out':>16s} {'DiD [95% interval]':>23s}") W = None out = {} for deriv, note in (("model_review", "a vision model corrected the test"), ("model_panel", "independent models agreed"), ("human_review", "placebo: a person decided")): groups = {"in": lambda g, d=deriv: g.derivation == d, "out": lambda g, d=deriv: g.derivation != d} arrs, n_docs = per_document(V, STARRED + others, groups) W = resample(n_docs, 1000, seed=7) if W is None else W def gaps(weights): r = {} for s, a in arrs.items(): tot = np.tensordot(weights, a, axes=([-1], [0])) with np.errstate(invalid="ignore", divide="ignore"): r[s] = 100 * tot[..., 0] / tot[..., 1] did = (np.nanmean([r[s][..., 0] - r[s][..., 1] for s in STARRED], axis=0) - np.nanmean([r[s][..., 0] - r[s][..., 1] for s in others], axis=0)) return r, did r, d = gaps(np.ones(n_docs)) _, boot = gaps(W) lo, hi = np.nanpercentile(boot, [2.5, 97.5]) si, so = np.mean([r[s][0] for s in STARRED]), np.mean([r[s][1] for s in STARRED]) oi, oo = np.mean([r[s][0] for s in others]), np.mean([r[s][1] for s in others]) out[deriv] = (d, lo, hi) print(f"{deriv:21s} {(TESTS.derivation == deriv).sum():6,d} {si:7.1f} / {so:6.1f} {oi:6.1f} / {oo:6.1f} " f"{d:+6.1f} [{lo:+5.1f}, {hi:+5.1f}] {note}") panel = TESTS[TESTS.derivation == "model_panel"] layout = TESTS[TESTS.test_type == "layout_kind"] print(f"nmodel_panel tests are {100 * (panel.test_type == 'layout_kind').mean():.0f}% layout tests, and " f"{100 * (layout.derivation == 'model_panel').mean():.0f}% of layout tests are model_panel") review = VERDICTS[VERDICTS.derivation == "model_review"] rate = lambda s: 100 * (review[review.system == s].result == "pass").mean() print(f"pass rate on the {(TESTS.derivation == 'model_review').sum()} model_review tests: " + ", ".join(f"{s} {rate(s):.1f}%" for s in STARRED) + f", median of the others {np.median([rate(s) for s in others]):.1f}%") claim("the starred models gain more on model_review tests than on the placebo split", out["model_review"][1] > out["human_review"][2]) claim("the placebo split shows no gap (its interval includes zero)", out["human_review"][1] < 0 = n: raise RuntimeError(f"{run.provider} failed on {len(failed)} of {n} pages") return n, (time.time() - t0) / max(n, 1) @section("8. Run Tesseract ourselves on 50 pages that span all 22 suites") def tesseract_step(): global SAMPLE if TESS_MAJOR < 5: raise RuntimeError(f"tesseract {TESS_MAJOR}.x is installed; this tutorial was verified with tesseract 5") SAMPLE = data.select(PAGES, sample=50) snapshot_download(REPO, repo_type="dataset", revision=REV, local_dir=str(DATA), allow_patterns=sorted(str(p.doc_path.relative_to(DATA)) for p in SAMPLE)) n_pdf = sum(p.doc_path.suffix == ".pdf" for p in SAMPLE) print(f"sample: {len(SAMPLE)} pages from {len({p.suite for p in SAMPLE})} suites, {n_pdf} PDF and {len(SAMPLE) - n_pdf} images, " f"{sum(p.doc_path.stat().st_size for p in SAMPLE) / 1e6:.1f} MB, {sum(len(unit_tests.gold(p)) for p in SAMPLE)} tests") # The adapter asks Tesseract for '-l script/'. Homebrew keeps those models in a script/ folder; Ubuntu installs # them flat, so on Linux we point Tesseract at a folder of symlinks with the layout it expects. listing = subprocess.run(["tesseract", "--list-langs"], capture_output=True, text=True) listing = listing.stdout + listing.stderr if "script/Latin" not in listing: src = Path(re.search(r'"(.+?)"', listing).group(1)) dst = WORK / "tessdata" (dst / "script").mkdir(parents=True, exist_ok=True) for f in src.glob("*.traineddata"): for link in [dst / f.name] + ([dst / "script" / f.name] if f.name[0].isupper() else []): if not link.exists(): link.symlink_to(f) os.environ["TESSDATA_PREFIX"] = str(dst) print(f"script models linked under {dst}") run = benchmark.Run("tesseract", {}, WORK / "runs" / "tesseract") benchmark.prepare(run, str(DATA), 900) n, per_page = predict_pages(run, SAMPLE) s = benchmark.grade(run, SAMPLE, say=lambda *_: None) fam = s["families"] print(f"nTesseract: {n} pages read, {per_page:.1f} s per page") for k, v in fam.items(): print(f" {k:7s} " + ("unsupported (no blocks output)" if v["score"] is None else f"{100 * v['score']:5.1f}% of {v['tests']} tests")) print(f" headline {'none' if s['headline'] is None else round(100 * s['headline'], 1)} (a family it cannot produce leaves no headline)") return f"Tesseract on 50 pages: text {100 * fam['text']['score']:.1f}%, tables {100 * fam['tables']['score']:.1f}%, {per_page:.1f} s/page" tesseract_step() Rebuilding a leaderboard from published verdicts is one thing; producing verdicts ourselves is another. data.select picks a fixed 50-page sample spread over all 22 suites, and we download only those pages. The loop is the benchmark’s own: needs lists the outputs a page still lacks, harness. predict runs the parser; record saves the prediction and grades the folder, so an interrupted run can resume. On Ubuntu we also link Tesseract’s script models into the layout the adapter expects. Tesseract passes 30.4% of the sample’s text tests against 17.6% on the full set, a gap that depends on which pages were sampled. Copy CodeCopiedUse a different Browserdef agreement(folder, published): """Our verdicts against the published ones for the same tests, where both scored the test.""" ours = pd.read_json(folder / "scores.jsonl", lines=True)[["test_id", "result", "why"]] pub = VERDICTS[VERDICTS.system == published][["test_id", "result"]] m = ours.merge(pub, on="test_id", suffixes=("_ours", "_pub")) m = m[(m.result_ours != "unsupported") & (m.result_pub != "unsupported")] return m, int((m.result_ours == m.result_pub).sum()), len(m) @section("9. Indra OCR on the same pages, served on the CPU from the package's own server code") def indra_step(): if sys.version_info 600: proc.terminate() raise RuntimeError("the Indra server did not start; see " + log.name) time.sleep(2) print(f"server up in {time.time() - t0:.0f} s: {INDRA[0]}, 2 worker processes") try: run = benchmark.Run("indra-ocr", {"base_url": ""}, WORK / "runs" / "indra-ocr") benchmark.prepare(run, str(DATA), 900) n, per_page = predict_pages(run, SAMPLE, alive=lambda: proc.poll() is None) s = benchmark.grade(run, SAMPLE, say=lambda *_: None) finally: proc.terminate() fam = {k: 100 * v["score"] for k, v in s["families"].items()} print(f"Indra OCR: {n} pages, {per_page:.2f} s per page; text {fam['text']:.1f}%, tables {fam['tables']:.1f}%, " f"layout {fam['layout']:.1f}%; headline on these pages {100 * s['headline']:.1f}n") ids = {p.page_id for p in SAMPLE} fmt = lambda x: "none" if x is None else f"{100 * x:.1f}" print(f"{'on the 50 pages':22s} {'headline':>9s} {'text':>6s} {'tables':>7s} {'layout':>7s}") for name, folder in (("Tesseract", WORK / "runs" / "tesseract"), ("Indra OCR", WORK / "runs" / "indra-ocr")): pub = unit_tests.summarize(VERDICTS[(VERDICTS.system == name) & VERDICTS.page_id.isin(ids)][COLS].to_dict("records")) ours = unit_tests.summarize(benchmark.read_scores(benchmark.Run(name, {}, folder))) for who, x in (("ours", ours), ("published", pub)): f = x["families"] print(f"{name + ', ' + who:22s} {fmt(x['headline']):>9s} " + " ".join(f"{fmt(f[k]['score']) if k in f else 'none':>6s}" for k in ("text", "tables", "layout"))) print() for name, folder in (("Tesseract", WORK / "runs" / "tesseract"), ("Indra OCR", WORK / "runs" / "indra-ocr")): m, same, total = agreement(folder, name) diff = m[m.result_ours != m.result_pub] print(f"{name:10s} our verdicts equal the published ones on {same} of {total} tests ({100 * same / total:.1f}%); " f"of the {len(diff)} that differ, {(diff.result_ours == 'pass').sum()} pass only for us and " f"{(diff.result_ours == 'fail').sum()} only in the published run") if len(diff): by_page = diff.test_id.map(TESTS.set_index("test_id").page_id).value_counts() print(" differing tests by type:", {k: int(v) for k, v in diff.test_id.map(TESTS.set_index("test_id").test_type).value_counts().items()}, f"| on {len(by_page)} pages, {by_page.iloc[0]} of them on {by_page.index[0]}") claim(f"{name}: our verdicts equal the published ones on at least 95% of the sample's tests", same >= 0.95 * total) return f"Indra OCR on 50 pages: headline {100 * s['headline']:.1f} (text {fam['text']:.1f}, tables {fam['tables']:.1f}, layout {fam['layout']:.1f}), {per_page:.2f} s/page" indra_step() Indra OCR is an open CPU model, and the package ships the HTTP server its adapter talks to, so we install the versions pinned in that server’s Dockerfile into a separate environment and start it in the background. Against the published verdicts on the same pages, Tesseract agrees on every scored test, so its sample score matches the published one. Indra agrees on 203 of 221, and all of its table differences sit on one page, which accounts for the whole gap between its published tables score of 36.1 and our 69.4. One page moving a family that far is the clearest argument for resampling documents. Copy CodeCopiedUse a different Browser@section("10. Bring your own parser: grade a PDF text-layer baseline with the same code") def byo_step(): import pypdfium2 as pdfium def text_layer(path): if path.suffix.lower() != ".pdf": return "" # an image has no text layer to read doc = pdfium.PdfDocument(str(path)) try: return "<pre>" + html.escape(doc[0].get_textpage().get_text_range()) + "</pre>" finally: doc.close() out = WORK / "runs" / "text-layer" (out / "predictions").mkdir(parents=True, exist_ok=True) for p in SAMPLE: (out / "predictions" / f"{p.page_id}.json").write_text(json.dumps({"html": text_layer(p.doc_path)})) s = benchmark.grade(benchmark.Run("text-layer", {}, out), SAMPLE, say=lambda *_: None, settings={"unsupported": ["blocks"]}) kind = {p.page_id: "PDF" if p.doc_path.suffix == ".pdf" else "image" for p in SAMPLE} print(f"{'parser on the same 50 pages':28s} {'text: PDF':>10s} {'images':>8s} {'all':>6s} {'tables':>8s}") for name in ("text-layer", "tesseract", "indra-ocr"): sc = pd.read_json(WORK / "runs" / name / "scores.jsonl", lines=True) sc = sc[sc.result.isin(["pass", "fail"])] txt = sc[sc.test_type.map(vocab.FAMILIES) == "text"] by = txt.groupby(txt.page_id.map(kind)).result.apply(lambda r: 100 * (r == "pass").mean()) tab = sc[sc.test_type == "table_cell"] print(f"{name:28s} {by.get('PDF', np.nan):10.1f} {by.get('image', np.nan):8.1f} " f"{100 * (txt.result == 'pass').mean():6.1f} {100 * (tab.result == 'pass').mean():8.1f}") fam = s["families"] return f"text layer, no OCR: text {100 * fam['text']['score']:.1f}%, tables {100 * fam['tables']['score']:.1f}% on the 50 pages" byo_step() Any parser can be graded without an adapter: we write one predictions file per page and call benchmark.grade, declaring the outputs we cannot produce as unsupported. Our parser reads a PDF’s embedded text layer with pypdfium2 and returns an empty string for images, which have none. On PDF pages it passes exactly as many text tests as Tesseract, 42.3% each, without any OCR, while on images it has nothing and Indra OCR still passes 43.1%. Plain text fails every table test, since a pass needs cell structure, so the mix of PDFs and images matters whenever two parsers are compared. Copy CodeCopiedUse a different Browser@section("11. The launch claims, checked against the files") def claims_step(): a, t = TABLE.loc["Datalab accurate"], TABLE.loc["Tesseract"] g = TABLE.loc["Gemini 3.8 Flash", "headline"] c = TABLE.loc["Claude Sonnet 5.5"] tags = TESTS.tags langs = {x for ts in tags for x in ts if re.fullmatch("[a-z]{3}", x)} - {"und", "zxx"} pkg = Path(vocab.__file__).parent # the scorer and the modules it reads; data.py and the harness only fetch and run scoring = sorted(pkg.glob("scorers/**/*.py")) + [pkg / f"{m}.py" for m in ("normalize", "tables", "equations", "views", "vocab", "markdown")] clients = [f.name for f in scoring if re.search(r"^s*(import|from)s+(httpx|requests|urllib|huggingface_hub|openai|anthropic|google" r"|transformers|torch|omni_parse_bench.harness)b", f.read_text(), re.M)] n_eng, n_table = tags.apply(lambda x: "eng" in x).sum(), tags.apply(lambda x: "table" in x).sum() rows = [ ("16,288 tests, 2,937 pages, 2,343 documents", f"{len(TESTS):,} / {TESTS.page_id.nunique():,} / {TESTS.document.nunique():,}", (len(TESTS), TESTS.page_id.nunique(), TESTS.document.nunique()) == (16288, 2937, 2343)), ("92 languages", f"{len(langs)} codes besides und and zxx", len(langs) == 92), ("10,089 English tests, 6,767 table tests", f"{n_eng:,} and {n_table:,}", (n_eng, n_table) == (10089, 6767)), ("Datalab accurate scores 92.7", f"{a.headline:.2f} in results.parquet", round(a.headline, 1) == 92.7), ("Datalab accurate ranks second, behind Gemini", f"{a.headline:.3f} vs {g:.3f}", a.headline < g), ("Claude Sonnet 5.5: text 93.1, tables 95.4, layout 77.5", f"{c.text:.1f} / {c.tables:.1f} / {c.layout:.1f}", [round(c[k], 1) for k in ("text", "tables", "layout")] == [93.1, 95.4, 77.5]), ("Tesseract: no headline, text 17.6, tables 0.0", f"{'none' if pd.isna(t.headline) else round(t.headline, 1)} / {t.text:.1f} / {t.tables:.1f}", pd.isna(t.headline) and round(t.text, 1) == 17.6 and round(t.tables, 1) == 0.0), ("no model judges the outputs", f"{len(clients)} of {len(scoring)} scorer modules import a model or HTTP client", not clients), ] print(f"{'claim (launch article and blog)':56s} {'what the files say':38s} verdict") for text, found, ok in rows: print(f"{text:56s} {found:38s} {'holds' if ok else 'does not hold'}") claim(text, ok) print("nnot checkable from these files: latency, price per 1,000 pages, the other benchmarks' test counts") return f"{sum(bool(ok) for *_, ok in rows)} of {len(rows)} launch claims hold against the files" claims_step() Instead of repeating the launch claims, we check each one against the files. The counts hold exactly, from tests and pages to languages and subsets, and the Claude and Tesseract rows match. Datalab‘s own row does not: the launch article and blog give Datalab an accurate 92.7 and place it second behind Gemini, while the released verdicts give 92.92 against 92.89, a tie that rounds to the same figure. The no-judge claim holds for the scorer itself, whose modules import no model or HTTP client; KaTeX runs in an embedded JavaScript engine to compare equations. Latency and price cannot be checked from these files. Copy CodeCopiedUse a different Browserbanner("SUMMARY") for name, res in RESULTS.items(): print(f" {name:<100s} {res}") print("nchecks decided in this run:") for step, text, held in CLAIMS: print(f" [{'held' if held else 'did not hold'}] step {step}: {text}") failed = [k for k, v in RESULTS.items() if str(v).startswith("SKIPPED")] print(f"n{sum(h for *_, h in CLAIMS)} of {len(CLAIMS)} checks held; steps failed or skipped: {failed or 'none'}") print(f"omni-parse-bench {md.version('omni-parse-bench')}, dataset @ {REV[:7]}, Python {platform.python_version()}, {SYSTEM}") print("not run here: the hosted parsers (paid API keys), the GPU models, latency and cost, all 2,937 pages") The summary prints each step’s one-line result and every check the run decided, marking which held and which did not: 13 of 17 held, and the four that did not are findings rather than failures, namely the placebo gap, Indra’s reproduction and Datalab’s two launch figures. It also lists what we did not run here: the hosted parsers that need paid keys, the GPU models, latency and cost, and the full set of 2,937 pages. In conclusion, OmniParseBench is a benchmark we can audit end to end: the scorer is open code that calls no model, the verdicts are released, and every entry on the card can be rebuilt from them. The top two parsers are statistically tied, pooling tests instead of families changes the order, and a different parser comes first on handwriting, rotated pages, tables and several scripts. The two models named on the card fall less far than the others on tests a model corrected, which we report as a flag rather than proof. Tesseract’s published verdicts reproduced exactly on our sample, while Indra OCR’s agreed on 92% of tests. The practical lesson is to slice the benchmark to our own documents, report intervals, and pin every version. Check out the Notebook and Repo. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post A Developer’s Guide to OmniParseBench: Leaderboard Benchmarks and OCR Testing appeared first on MarkTechPost.
The AI startup Recursive is unveiling Flint, its first language model, a 40-billion-parameter MoE for local coding developed by former DeepMind researchers. It was created in four months using the company's in-house AI system, "Eureka Machine," which independently tests and optimizes architectures. Recursive has yet to prove whether this approach will actually lead to self-improving AI. The article Recursive shows first result of its AI-driven research process with Flint appeared first on The Decoder.
Security researchers at Symantec have found an offensive AI agent that hijacks unprotected language models on the web and uses them to attack servers, steal databases, and install cryptominers on its own. It runs up to 100,000 steps without human input. The owner of the hijacked model gets stuck with the bill. The article An AI agent in "Skynet mode" hijacks unprotected language models to launch self-sustaining cyberattacks appeared first on The Decoder.
OpenAI teamed up with PerturbAI and Scripps Research to deploy thousands of AI agents that analyzed biological data from millions of mouse cells. The article OpenAI agents analyze mouse brain data in 32 minutes, a job that would take humans months appeared first on The Decoder.
Anthropic has launched Claude Duet, a proactive AI agent that handles tasks on its own, even when the user is offline. The app is built on Claude Opus 5.5 and connects to services like Gmail, Slack, and Stripe through more than 6,000 connectors. It asks for approval before sensitive actions like payments. With Duet, Anthropic is responding to similar products from xAI, Meta, and OpenAI. The article Anthropic takes on Grok Bot, Meta's Muse, and OpenAI's Dots with its always-on agent "Claude Duet" appeared first on The Decoder.
IBM has released Granite Speech 5.0 2B, a new open speech-language model for transcription and translation. Granite Speech 5.0 2B is a 2B-parameter model from IBM that turns speech into text in 6 languages. It ships under Apache 2.0. IBM also updated 2 existing models: the edge-focused Turbo CTC and the diarization-focused 4.1 2B Plus. For developers, the release covers accurate transcription, fast on-device transcription and speaker-attributed transcripts. TL;DR Size: 2B parameters, dense. LLM backbone uses YaRN extension to 16k positions. IBM recommends audio segments of up to about 2 minutes. Runs on: GPU via Transformers or vLLM, Apple Silicon via mlx-audio, local CPU/Mac via llama.cpp GGUF (Q8_0 recommended). Performance: #2 open-weight model on the Open ASR Leaderboard, 5.68% average English WER. Best: 0.99% WER on LibriSpeech Clean. Worst: 13.79% WER on the leaderboard’s private conversational set. How does Granite Speech 5.0 2B work? The model has 3 parts, per the model card. A speech encoder of 16 Conformer blocks is trained with CTC. It uses block attention over 4-second audio blocks. A 2-layer window q-former then projects and downsamples the acoustic embeddings. Total downsampling is 10x, so the LLM sees embeddings at 10 Hz. The LLM is an intermediate checkpoint of granite-4.0-1b-base. IBM extended it to 16k positions with YaRN and fine-tuned it with LoRA adapters. Training used 390,000 hours of public audio plus synthetic data. The synthetic sets target Japanese ASR, keyword biasing and translation. Training took 40 days on 8 H100 GPUs. What is new versus Granite Speech 4.1 2B? Language ID prompting: name the input language in the prompt to steer transcription. Keyword lists up to 2,000 terms: bias output toward product names, people and acronyms. Punctuation and capitalization on request: capitalization F1 runs from 90.16 to 99.51 across test sets. Robustness: better handling of noise, reverberation and distant microphones. How does Granite Speech 5.0 2B perform on benchmarks? On the Open ASR Leaderboard, IBM’s chart shows a 5.68% average English WER. That average covers 9 public and 2 private test sets, as of October 21, 2026. Only Qwen3-ASR-1.7B scores better, at 5.40%. Note a source conflict. IBM’s blog reports 5.06% instead. That figure is unofficial, dated October 15, and averages public short-form sets only. Per-set results range widely. The model scores 0.99% on LibriSpeech Clean and 2.21% on LibriSpeech Other. It scores 13.79% on the private conversational set. On the noisy FFASR leaderboard, it ranks #4 of 42 at 14.80% average WER. Qwen3-ASR-1.7B scores 13.41% there. Multilingual CommonVoice WER is 2.60% for Portuguese and 3.50% for Spanish. Japanese is the weakest language at 10.74%. For translation, X-to-English CoVoST BLEU reaches 53.6 for Portuguese and 40.1 for German. English-to-Mandarin was not evaluated. Keyword biasing shows the clearest gain. On Earnings-22, keyword F1 rises from 67.2 to 90.1 with correct keyword lists. With 2,000-term lists mixing correct and wrong terms, it still reaches 82.8. window.addEventListener("message",function(e){if(e.data&&e.data.gs5h){var f=document.getElementById("gs5frame");if(f)f.style.height=e.data.gs5h+"px";}}); How does it compare with other open ASR models? *Leaderboard averages as charted in IBM’s model card from the Open ASR and FFASR leaderboards. What changed in Turbo CTC and 4.1 2B Plus? Granite Speech 5.0 Turbo CTC keeps its 470M footprint. It now transcribes 7 languages with native punctuation and capitalization. IBM compiled one Conformer block and reused its graph across the stack. That lifted throughput to 19,870 RTFx, up from about 12,600 at launch. The updated Granite Speech 4.1 2B Plus is a drop-in weight update. Average timestamp shift falls from 210.86 to 98.54. Long-form WER drops from 13.53 to 10.22. Biased-keyword WER falls from 11.79 to 6.48. How can developers deploy it? Granite Speech 5.0 2B runs in Transformers, vLLM, llama.cpp and mlx-audio. For GGUF, IBM recommends Q8_0. Lower-bit quantizations lose accuracy on audio longer than about 1 minute. Key Takeaways #2 open-weight model on the Open ASR Leaderboard at 5.68% WER. 2B parameters, Apache 2.0, runs on GPU, Mac and local CPU. 2,000-term keyword biasing across all supported languages. Turbo CTC reaches 19,870 RTFx and now covers 7 languages. Qwen3-ASR-1.7B still leads on accuracy and noisy audio. Check out the model card, IBM blog and Granite Speech paper. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post IBM AI Releases Granite Speech 5.0 2B for Multilingual Transcription and Translation appeared first on MarkTechPost.
Perplexity, along with NVIDIA team, has introduced JOLT, a new post-training method for language models. JOLT stands for Joint On-policy Learning and Teaching. It trains a single model in 2 roles. The teacher sees privileged information; the student does not. The key change is that the teacher is trained too, so its feedback stays usable. On MATH, JOLT matched GRPO’s accuracy with 6.5x fewer processed training tokens. For teams running RL post-training, that is a direct compute question. TL;DR What it is: a post-training method. Size: tested on Qwen3-0.6B, Qwen3-4B-Instruct, Qwen3-8B, Qwen3-30B-A3B-Instruct and DeepSeek-V2-Lite. Runs on: 1 shared model plays both roles, so no separate teacher model. Training hardware: Not disclosed. Performance: JOLT+ posts the highest mean score in every setting tested against GRPO, OPSD and π-Distill. Best: AppWorld state-test pass rate of 72.09% with JOLT+, vs 62.20% for GRPO (+9.9 points). Worst: JOLT without student reward scores 51.92% on AppWorld, below GRPO. What is Perplexity JOLT? JOLT is a self-distillation method from Perplexity and NVIDIA that trains one model as teacher and student. It builds on On-Policy Self-Distillation (OPSD). Why does RL post-training need a better teacher? Outcome-reward RL, such as GRPO, gives one reward per attempt. That signal is sparse. It does not say which tokens helped. On long agentic tasks, successes are rare, so learning is slow. On-Policy Distillation fixes sparsity by having a stronger teacher score each student token. OPSD removes the separate teacher. The same model becomes the teacher when given privileged context, such as a verified solution. The research team argues this is not enough. A teacher holding the answer can take shortcuts the student cannot follow. Its guidance may sit beyond the student’s reach. In the paper, OPSD’s teacher-student disagreement grows during MATH training. How does JOLT work? JOLT trains both roles of one shared model at the same time: Teacher: gets privileged information. It learns from outcome rewards, plus a KL penalty that keeps it close to the student. Student: sees only the task. It learns from the teacher’s token-level distillation signal on its own attempts. On-policy: each role learns from trajectories generated under its own context. JOLT+: adds the student’s own outcome reward to the student update. The research stands in the center on a ‘reward-consistent’ teacher. Its distillation update should point the same way as the student’s reward gradient. The ideal teacher favors high-advantage actions while staying near the student’s policy. (function(){var f=document.getElementById('mtp-jolt-embed');window.addEventListener('message',function(e){if(e.data&&e.data.mtpJoltHeight&&f){f.style.height=e.data.mtpJoltHeight+'px';}});})(); How does JOLT perform on benchmarks? The research team tested 5 settings, each with a different base model and type of privileged information: GSM8K: DeepSeek-V2-Lite; teacher gets the verified answer. MATH: Qwen3-0.6B; teacher gets worked reference solutions. LiveCodeBench v6: Qwen3-8B on 131 problems; teacher gets a passing sibling solution or execution feedback. AppWorld: Qwen3-4B-Instruct; teacher gets summaries of past attempts. Terminal-Bench 2.1: Qwen3-30B-A3B-Instruct trained on TMax; teacher gets private verifier files. JOLT+ had the highest mean score in every setting. Against GRPO, the strongest baseline, it gained 1.5 points on GSM8K. It also gained 4.2 on MATH, 3.0 on LCBv6 and 9.9 on AppWorld. On Terminal-Bench 2.1, JOLT+ reached 10.96% pass@1 and 34.83% pass@32. GRPO scored 8.25% and 30.34%. Even plain JOLT, with no student reward, beat GRPO on MATH: 72.14% versus 68.36%. How much training compute does JOLT save? On MATH, the research team fit accuracy curves against training-token budgets. At 63.44% accuracy, GRPO’s checkpoint used 8.40M processed training tokens. JOLT’s fitted curve needed 1.30M, a 6.5x reduction. Counting completion tokens only, the gap widens to about 13x. Across 55% to 65% accuracy targets, savings range from 4.3x to 7.4x on processed tokens. JOLT also levels off higher. It plateaus near 72%, against about 70% for GRPO and 62% for OPSD. How does JOLT compare with GRPO, OPSD and π-Distill? All scores below come from Perplexity’s own runs in the JOLT paper, with hyperparameters tuned per method. Method descriptions link to each method’s own paper. Terminal-Bench base model: 5.18% pass@1, 22.81% pass@32. Key Takeaways 1 model plays teacher and student, and the teacher is trained too. 6.5x fewer processed tokens than GRPO at matched MATH accuracy. JOLT+ beats GRPO by 9.9 points on AppWorld. Plain JOLT still trails GRPO on AppWorld and Terminal-Bench. Check out the paper on arXiv and the Perplexity research blog post. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Perplexity and NVIDIA Researchers Introduce JOLT: a Joint Teacher-Student Reinforcement Learning Method appeared first on MarkTechPost.
In this tutorial, we use Whistle, Cactus Compute’s speech-to-text model, a 16.9 MB file that runs on a CPU, transcribes seven languages, and returns word timestamps and probabilities. The model card claims it beats Whisper base on LibriSpeech; we test it on seeded test-clean and test-other samples, comparing Whistle with Whisper tiny and base on identical clips, judged by paired speaker-cluster intervals with Holm correction. We find a lead over greedy Whisper base on test-other, no significant gap against beam search, and a speed advantage that narrows as clips grow. Beyond the headline, we probe a 30-second window where the pinned engine silently drops the middle of longer audio, plus streaming latency, timestamps, keyword biasing, other languages, noise, embeddings, and speech-to-tool calls. Our figures are one-run samples from an Apple M4, and published full-split numbers are quoted, not measured. Copy CodeCopiedUse a different Browserimport os import io import re import sys import json import math import time import random import hashlib import shutil import zipfile import platform import traceback import subprocess import collections import urllib.error import urllib.request RESULTS = {} VERDICTS = [] def banner(title): print("n" + "=" * 78) print(title) print("=" * 78) def section(name): def wrap(fn): def run(*a, **kw): banner(name) t0 = time.time() try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}" print(f"n[!] {name} did not complete: {type(e).__name__}: {e}") traceback.print_exc(limit=3) return None finally: print(f"n[{time.time() - t0:.0f} s]") json.dump({"results": RESULTS, "verdicts": VERDICTS}, open("results.json", "w"), indent=1) return run return wrap banner("1. Install Whistle, pin its weights and engine by Hub revision, transcribe one clip") subprocess.run([sys.executable, "-m", "pip", "install", "-q", "cactus-needle==3.2.0", "openai-whisper==20250625", "jiwer==4.0.0", "soundfile==0.14.0", "soxr==1.1.0"], check=True) os.environ["NEEDLE_TELEMETRY"] = "0" # the package sends anonymous usage counts unless told not to os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1" import numpy as np import soundfile as sf import soxr from huggingface_hub import hf_hub_download # cactus-needle fetches three files on first use, from Hugging Face repos whose files are replaced in place: the 16.9 MB # whistle.cact weights, the ~1 MB engine library (version 3.3.0 in this package; the needle3 repo received new binaries # on 2026-10-10) and the 35 MB needle3.cact used in step 12. Pinning the Hub revision of each, and printing the hashes, # is what makes this run reproducible. The engine version is read from the package, not typed in. WHISTLE_REV = "ca5287601bef25af26dcf2e1b2bdc0843a7c19e5" # Cactus-Compute/whistle, 2026-10-10 NEEDLE3_REV = "93e88ede867e4311c39151285e74df513e4b835b" # Cactus-Compute/needle3 (engine + Needle 3 weights), 2026-10-10 from needle.agent import fetch as needle_fetch TAG = needle_fetch._platform_tag() ENGINE_VERSION = needle_fetch.engine_version(3) weights = hf_hub_download("Cactus-Compute/whistle", "whistle.cact", revision=WHISTLE_REV) wheel = hf_hub_download("Cactus-Compute/needle3", needle_fetch.engine_wheel(ENGINE_VERSION, TAG, 3), revision=NEEDLE3_REV) with zipfile.ZipFile(wheel) as z: member = next(n for n in z.namelist() if n.startswith("needle/libneedle")) os.makedirs("engine", exist_ok=True) engine = os.path.abspath(os.path.join("engine", os.path.basename(member))) open(engine, "wb").write(z.read(member)) needle3_weights = hf_hub_download("Cactus-Compute/needle3", "needle3.cact", revision=NEEDLE3_REV) os.makedirs(needle_fetch.cache_dir(3), exist_ok=True) # Needle(tools=...) reads its weights from this cache path shutil.copyfile(needle3_weights, os.path.join(needle_fetch.cache_dir(3), needle_fetch.base_weights(3))) os.environ["NEEDLE_WHISTLE_WEIGHTS"] = weights # both variables are honoured by the package os.environ["NEEDLE3_LIB_PATH"] = engine import needle import importlib.metadata as md sha = lambda p: hashlib.sha256(open(p, "rb").read()).hexdigest()[:12] CPU = platform.processor() or platform.machine() if sys.platform == "linux" and os.path.exists("/proc/cpuinfo"): CPU = next((l.split(":", 1)[1].strip() for l in open("/proc/cpuinfo") if l.startswith("model name")), CPU) elif sys.platform == "darwin": CPU = subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True).stdout.strip() or CPU print(f"cactus-needle {needle.__version__} (engine {ENGINE_VERSION}, wheel tag {TAG}) python {platform.python_version()} " f"soundfile {md.version('soundfile')} soxr {md.version('soxr')} numpy {np.__version__}") print(f"hardware: {CPU}, {os.cpu_count()} cpus visible") print(f"whistle.cact {os.path.getsize(weights)/1e6:.1f} MB sha256 {sha(weights)} @ {WHISTLE_REV[:12]}") print(f"engine {os.path.getsize(engine)/1e6:.1f} MB sha256 {sha(engine)} @ {NEEDLE3_REV[:12]}") print(f"needle3.cact {os.path.getsize(needle3_weights)/1e6:.1f} MB sha256 {sha(needle3_weights)} @ {NEEDLE3_REV[:12]}") SR = 16000 W = needle.Whistle() # one model per process, not thread-safe; every later step reuses this object def fetch(url, tries=7): """GET with retries. The datasets-server rate-limits anonymous callers (HTTP 429): honour Retry-After when it is sent, otherwise back off 5, 10, 20, 40 ... seconds, and never give up on the first refusal.""" for k in range(tries): try: with urllib.request.urlopen(url, timeout=60) as r: return r.read() except urllib.error.HTTPError as e: if k == tries - 1: raise wait = int(e.headers.get("Retry-After", 0) or 0) if e.code == 429 else 0 time.sleep(max(wait, 5 * 2 ** k) if e.code in (429, 500, 502, 503, 504) else 2) except Exception: if k == tries - 1: raise time.sleep(3 * (k + 1)) def page(dataset, config, split, offset, length=100): """One page of a Hub dataset through the datasets-server rows API: metadata plus signed audio URLs.""" url = (f"" f"&split={split}&offset={offset}&length={length}") time.sleep(0.25) # stay under the anonymous rate limit when paging a whole split return json.loads(fetch(url)) def load_audio(row): """Download one row's clip; return 16 kHz mono float32 samples and the sha256 of the original file.""" data = fetch(row["audio"][0]["src"]) audio, sr = sf.read(io.BytesIO(data), dtype="float32") if audio.ndim > 1: audio = audio.mean(axis=1) if sr != SR: audio = soxr.resample(audio, sr, SR, quality="HQ") return audio, hashlib.sha256(data).hexdigest()[:12] first = page("openslr/librispeech_asr", "clean", "test", 0, 1)["rows"][0]["row"] clip, _ = load_audio(first) t0 = time.perf_counter() out = W.transcribe(clip, word_timestamps=True) wall = (time.perf_counter() - t0) * 1000 print(f"nclip {first['id']} {len(clip)/SR:.1f} s reference: {first['text']}") print(f"-> {out['text']!r}") print(f" language {out['language']} engine time to first token {out['ttft_ms']:.1f} ms decode {out['decode_tps']:.0f} tok/s" f" wall {wall:.0f} ms") print(" first words:", [(w["word"], w["start"], w["end"], w["probability"]) for w in out["words"][:3]]) RESULTS["1. Install Whistle, pin its weights and engine by Hub revision, transcribe one clip"] = f"{os.path.getsize(weights)/1e6:.1f} MB weights, {len(clip)/SR:.1f} s clip in {wall:.0f} ms" We install cactus-needle 3.2.0 with its dependencies pinned and turn off its anonymous telemetry. The package downloads Whistle’s weights, its engine and Needle 3’s weights from Hugging Face repositories whose files are replaced in place, so we pin both repositories to fixed revisions and print SHA-256 hashes. The page() helper reads the datasets-server rows API with rate-limit backoff, and load_audio() resamples clips to 16 kHz. The first test-clean clip, 3.5 s of speech, transcribes in 33 ms of wall time, each word carrying its start, end and probability. Copy CodeCopiedUse a different Browserimport jiwer import whisper import torch from whisper.normalizers import EnglishTextNormalizer, BasicTextNormalizer NORM_EN = EnglishTextNormalizer() # the normalisers Whistle's own benchmarks use; applied to both sides, always NORM_BASIC = BasicTextNormalizer() SEED = 2026 N_WHISTLE = 300 # clips per LibriSpeech split that Whistle transcribes (it is cheap) N_PAIRED = {"other": 100, "clean": 60} # the first N of them also go through Whisper (it is not); test-other is primary MAX_CLIP_S = 30.0 # Whistle's single-pass window; longer clips are declared out of scope here WHISPER_SECONDS = collections.Counter() def all_rows(dataset, config, split): out, offset = [], 0 while True: rows = page(dataset, config, split, offset)["rows"] out += [r["row"] for r in rows] offset += 100 if len(rows) < 100: return out def dataset_sha(dataset): return json.loads(fetch(f""))["sha"][:12] def wer_counts(ref, hyp): """Errors (S+D+I) and reference words for one normalised pair; an empty hypothesis is all deletions.""" n = len(ref.split()) if not hyp.strip(): return n, n o = jiwer.process_words(ref, hyp) return o.substitutions + o.deletions + o.insertions, n def sdi(ref, hyp): if not hyp.strip(): return 0, len(ref.split()), 0 o = jiwer.process_words(ref, hyp) return o.substitutions, o.deletions, o.insertions def corpus_wer(errs, words): return 100.0 * sum(errs) / max(sum(words), 1) def _by_cluster(clusters, *arrays): tot = collections.defaultdict(lambda: [0.0] * len(arrays)) for i, c in enumerate(clusters): for j, a in enumerate(arrays): tot[c][j] += a[i] return tot def wer_ci(clusters, errs, words, B=2000, seed=0): """95% percentile interval of a corpus WER, resampling clusters (speakers) with replacement.""" tot = _by_cluster(clusters, errs, words) keys, rng = sorted(tot), random.Random(seed) boots = [] for _ in range(B): d = rng.choices(keys, k=len(keys)) boots.append(100 * sum(tot[c][0] for c in d) / max(sum(tot[c][1] for c in d), 1)) return np.percentile(boots, [2.5, 97.5]) def paired_contrast(clusters, errs_a, words, errs_b, label_a, label_b, B=2000, seed=0): """WER(a) - WER(b) with a paired speaker-cluster bootstrap: the same resampled speakers for both systems.""" tot = _by_cluster(clusters, errs_a, errs_b, words) keys, rng = sorted(tot), random.Random(seed) boots = [] for _ in range(B): d = rng.choices(keys, k=len(keys)) w = max(sum(tot[c][2] for c in d), 1) boots.append(100 * (sum(tot[c][0] for c in d) - sum(tot[c][1] for c in d)) / w) boots = np.array(boots) lo, hi = np.percentile(boots, [2.5, 97.5]) p = 2 * min((boots = 0).mean()) return {"a": label_a, "b": label_b, "diff": corpus_wer(errs_a, words) - corpus_wer(errs_b, words), "lo": lo, "hi": hi, "p": max(p, 1 / B), "clusters": len(keys)} def holm(ps, alpha=0.05): order, keep = np.argsort(ps), [False] * len(ps) for rank, i in enumerate(order): if ps[i] 0 and Holm-significant -> Whistle ahead; entirely Whistle behind; otherwise no significant difference at this sample size.""" if c["lo"] > 0 and significant: return "Whistle ahead" if c["hi"] 0 and Holm-adjusted p ") print(" Whistle ahead; entirely behind; otherwise no significant difference at this n") print(" family A (published claims): test-other and test-clean vs base greedy and vs base beam 5, MLS pooled vs base greedy;") print(" family B: the keyword contrasts. Holm at alpha = 0.05 within each family. Whisper tiny rows are descriptive.n") global MANIFEST EN, manifest = {}, {"dataset_sha": dataset_sha("openslr/librispeech_asr"), "seed": SEED, "splits": {}} MANIFEST = manifest for split in ("other", "clean"): t0 = time.time() rows = all_rows("openslr/librispeech_asr", split, "test") idx = random.Random(SEED).sample(range(len(rows)), N_WHISTLE) kept, too_long, empty = [], 0, 0 for i in idx: r = rows[i] audio, h = load_audio(r) ref = NORM_EN(r["text"]) if len(audio) / SR > MAX_CLIP_S: too_long += 1 continue if not ref: empty += 1 continue kept.append({"id": r["id"], "spk": str(r["speaker_id"]), "text": r["text"], "ref": ref, "audio": audio, "dur": len(audio) / SR, "sha": h}) EN[split] = kept manifest["splits"][split] = {"rows": len(rows), "sampled": N_WHISTLE, "too_long": too_long, "empty_ref": empty, "ids": [c["id"] for c in kept], "sha": [c["sha"] for c in kept]} durs = [c["dur"] for c in kept] pk = kept[:N_PAIRED[split]] print(f"test-{split}: {len(rows)} rows -> {len(kept)} clips kept ({too_long} over {MAX_CLIP_S:.0f} s, {empty} empty), " f"{len(set(c['spk'] for c in kept))} speakers, {sum(durs)/60:.1f} min, clip {np.mean(durs):.1f} s mean / {max(durs):.1f} s max" f" [{time.time()-t0:.0f} s]") print(f" paired subset: {len(pk)} clips, {len(set(c['spk'] for c in pk))} speakers, {sum(c['dur'] for c in pk):.0f} s of audio, " f"{sum(len(c['ref'].split()) for c in pk)} reference words") json.dump(manifest, open("manifest.json", "w")) print(f"nmanifest.json: LibriSpeech revision {manifest['dataset_sha']}, clip ids and file hashes (the MLS and MInDS-14 clips are added to it in") print("steps 9 and 12), so a rerun can prove it used the same audio") return f"{len(EN['other'])} + {len(EN['clean'])} clips, manifest written" sample_english() A verdict means little if its rule is set after the final numbers arrive, so we wrote ours into the notebook before the final run, after exploratory probes that touched some of the same clips. We draw a seeded sample of 300 rows per split and drop clips longer than Whistle’s single-pass window, leaving 299 clips in each. The paired subsets, the first 100 test-other and 60 test-clean clips, go through Whisper, and few speakers make their intervals approximate. A difference counts only when its 95% paired speaker-cluster bootstrap interval excludes zero and its Holm-adjusted p is below 0.05. Copy CodeCopiedUse a different BrowserWHISPER = {} def whisper_model(size): if size not in WHISPER: WHISPER[size] = whisper.load_model(size, device="cpu") return WHISPER[size] def run_whisper(size, audio, language="en", **kw): """openai-whisper with its defaults (greedy with temperature fallback) unless kw says otherwise. Seeded per call, because the fallback samples. Returns text, whether any segment needed the fallback, and the wall time.""" torch.manual_seed(0) WHISPER_SECONDS[size] += len(audio) / SR t0 = time.perf_counter() r = whisper_model(size).transcribe(audio, language=language, fp16=False, **kw) fell_back = any(s.get("temperature", 0) > 0 for s in r["segments"]) return r["text"], fell_back, time.perf_counter() - t0 SYSTEMS = {"other": [("whisper tiny, greedy", "tiny", {}), ("whisper base, greedy", "base", {}), ("whisper base, beam 5", "base", {"beam_size": 5})], "clean": [("whisper tiny, greedy", "tiny", {}), ("whisper base, greedy", "base", {}), ("whisper base, beam 5", "base", {"beam_size": 5})]} FAMILY_A = {"test-clean: Whistle vs whisper base, greedy", "test-clean: Whistle vs whisper base, beam 5", "test-other: Whistle vs whisper base, greedy", "test-other: Whistle vs whisper base, beam 5", "MLS pooled: Whistle vs whisper base, greedy"} PUBLISHED = {"other": "Whistle 10.49 / Whisper base 11.0 (paper table 9, beam) / 12.4 (table 8, greedy)", "clean": "Whistle 4.31 / Whisper base 4.9 (paper table 9, beam) / 5.0 (table 8, greedy)"} CONTRASTS = [] @section("3. Word error rate against Whisper tiny and base, paired by clip, with a verdict per contrast") def english_wer(): global HYP HYP = {} for split in ("other", "clean"): clips = EN[split] t0 = time.perf_counter() hw = [W.transcribe(c["audio"], language="en")["text"] for c in clips] tw = time.perf_counter() - t0 HYP[(split, "Whistle")] = hw e_w, n_w = zip(*[wer_counts(c["ref"], NORM_EN(h)) for c, h in zip(clips, hw)]) lo, hi = wer_ci([c["spk"] for c in clips], e_w, n_w) print(f"test-{split}: Whistle on all {len(clips)} clips: WER {corpus_wer(e_w, n_w):.2f}% (95% speaker-cluster CI {lo:.2f}..{hi:.2f}) " f"in {tw:.1f} s ({1000*tw/len(clips):.0f} ms per clip)") print(f" published full-split figures, different sample: {PUBLISHED[split]}") pk = clips[:N_PAIRED[split]] spk = [c["spk"] for c in pk] e_wp, n_p = list(e_w[:len(pk)]), list(n_w[:len(pk)]) s = np.sum([sdi(c["ref"], NORM_EN(h)) for c, h in zip(pk, hw[:len(pk)])], axis=0) print(f" paired subset, {len(pk)} clips / {len(set(spk))} speakers / {sum(n_p)} words:") print(f" {'Whistle, 5-beam':22s} {corpus_wer(e_wp, n_p):6.2f}% S/D/I {s[0]:3d}/{s[1]:3d}/{s[2]:3d}") for label, size, kw in SYSTEMS[split]: res = [run_whisper(size, c["audio"], **kw) for c in pk] hyps = [r[0] for r in res] HYP[(split, label)] = hyps e_o = [wer_counts(c["ref"], NORM_EN(h))[0] for c, h in zip(pk, hyps)] s = np.sum([sdi(c["ref"], NORM_EN(h)) for c, h in zip(pk, hyps)], axis=0) con = paired_contrast(spk, e_o, n_p, e_wp, label, "Whistle") con["name"] = f"test-{split}: Whistle vs {label}" CONTRASTS.append(con) better, worse = sum(a <b> b for a, b in zip(e_wp, e_o)) print(f" {label:22s} {corpus_wer(e_o, n_p):6.2f}% S/D/I {s[0]:3d}/{s[1]:3d}/{s[2]:3d} fallback on {sum(r[1] for r in res):2d} clips " f"{np.median([r[2] for r in res])*1000:5.0f} ms/clip | {fmt_contrast(con)} clips Whistle better/worse {better}/{worse}") fam = [c for c in CONTRASTS if c["name"] in FAMILY_A] print(f"nContrasts so far (family A: {len(fam)} of 5; the Holm correction and the verdicts come once the MLS contrast is in):") for c in CONTRASTS: tag = "family A" if c["name"] in FAMILY_A else "descriptive" print(f" {c['name']:48s} {fmt_contrast(c)} p~{c['p']:.3f} [{tag}]") print("n Whistle decodes with a 5-beam length-normalised search. openai-whisper's default is greedy with temperature") print(" fallback; the Whisper paper's LibriSpeech numbers come from beam search, so the beam-5 rows are the like-for-like ones.") print(" Every gap here is a system gap (model plus decoder), not a model-only one.") return " | ".join(f"test-{s}: Whistle {corpus_wer(*zip(*[wer_counts(c['ref'], NORM_EN(h)) for c, h in zip(EN[s][:N_PAIRED[s]], HYP[(s, 'Whistle')][:N_PAIRED[s]])])):.2f}% " f"vs base greedy {corpus_wer(*zip(*[wer_counts(c['ref'], NORM_EN(h)) for c, h in zip(EN[s][:N_PAIRED[s]], HYP[(s, 'whisper base, greedy')])])):.2f}%" for s in ("other", "clean")) english_wer() Whisper’s default decoder is greedy with temperature fallback. At the same time, the paper’s LibriSpeech numbers come from beam search, and Whistle decodes with a five-beam search, so every comparison must name its baseline. Scored with one normaliser, Whistle’s WER is 10.47% on the test-other sample, near the published full-split 10.49, which we quote rather than measure. On the paired subset it scores 10.27%, against 12.46% for greedy Whisper base and 10.83% for beam search. The 2.19-point lead over greedy survives the Holm correction, but the beam gap does not, so the baseline wins. Copy CodeCopiedUse a different Browserdef audio_seconds(clips, seconds): """Exactly `seconds` of speech: clips concatenated in sample order, then trimmed.""" out, total = [], 0 for c in clips: out.append(c["audio"]) total += c["dur"] if total >= seconds: break return np.concatenate(out)[:int(seconds * SR)] PROBE_SCRIPT = r""" import sys, json, time, resource, numpy as np kind, paths = sys.argv[1], sys.argv[2:] scale = 1 << 20 if sys.platform == "darwin" else 1 <6s} | {'Whistle wall':>12s} {'spread':>7s} {'ttft':>6s} {'tok/s':>6s} | {'tiny greedy':>11s} | {'base greedy':>11s} | {'base beam 5':>11s}") for secs in (5, 10, 30): p = f"clip{secs}.npy" w = res["whistle"][p] print(f"{secs:5d} s | {w['wall_ms']:9.0f} ms {w['spread_ms']:5.0f} ms {w['ttft_ms']:6.1f} {w['tps']:6.0f} | " f"{res['whisper-tiny'][p]['wall_ms']:8.0f} ms | {res['whisper-base'][p]['wall_ms']:8.0f} ms | {res['whisper-base-beam5'][p]['wall_ms']:8.0f} ms") w5, w10, w30 = (res["whistle"][p]["wall_ms"] for p in ("clip5.npy", "clip10.npy", "clip30.npy")) b5, b10, b30 = (res["whisper-base"][p]["wall_ms"] for p in ("clip5.npy", "clip10.npy", "clip30.npy")) bb5, bb10, bb30 = (res["whisper-base-beam5"][p]["wall_ms"] for p in ("clip5.npy", "clip10.npy", "clip30.npy")) print(f"nreal-time factor on the 10 s clip: Whistle {w10/10000:.4f}, base greedy {b10/10000:.3f}, base beam 5 {bb10/10000:.3f}") print(f"base greedy / Whistle wall time: {b5/w5:.1f}x at 5 s, {b10/w10:.1f}x at 10 s, {b30/w30:.1f}x at 30 s; beam 5 / Whistle: " f"{bb5/w5:.1f}x, {bb10/w10:.1f}x, {bb30/w30:.1f}x") print(f"from 5 to 10 s the wall time grows {w10/w5:.2f}x for Whistle, {b10/b5:.2f}x for base greedy, {bb10/bb5:.2f}x for base beam 5;") print(f"from 10 to 30 s: {w30/w10:.2f}x, {b30/b10:.2f}x, {bb30/bb10:.2f}x. Whisper's encoder always sees a padded 30 s window; its decoder") print("still grows with the words it emits, so the padding fixes only part of its cost. The encoder is not timed separately here.") print("npeak RSS of each process (model loaded, three clips transcribed) and the RSS right after its imports:") for kind in ("whistle", "whisper-tiny", "whisper-base"): r = res[kind] print(f" {kind:13s} peak {r['peak']:6.0f} MB after imports {r['after_import']:4.0f} MB -> {r['peak']-r['after_import']:4.0f} MB for the model and the run") sizes = {"whistle.cact + engine": (os.path.getsize(weights) + os.path.getsize(engine)) / 1e6} for size in ("tiny", "base"): p = os.path.expanduser(f"~/.cache/whisper/{size}.pt") sizes[f"whisper {size}.pt"] = os.path.getsize(p) / 1e6 if os.path.exists(p) else float("nan") print("files on disk: " + ", ".join(f"{k} {v:.1f} MB" for k, v in sizes.items()) + " (file sizes, not deployed footprint)") off = subprocess.run([sys.executable, "-c", "import needle, numpy as np; W = needle.Whistle(); print(W.transcribe(np.zeros(16000, np.float32))['text'] == '')"], capture_output=True, text=True, env={**os.environ, "HF_HUB_OFFLINE": "1"}) print(f"fresh process with HF_HUB_OFFLINE=1 and the pinned paths: {'loads and runs' if 'True' in off.stdout else 'FAILED: ' + off.stderr[-200:]}") return f"10 s clip: Whistle {w10:.0f} ms vs base greedy {b10:.0f} ms ({b10/w10:.1f}x, {b5/w5:.1f}x at 5 s, {b30/w30:.1f}x at 30 s); peak RSS {res['whistle']['peak']:.0f} vs {res['whisper-base']['peak']:.0f} MB" speed() Speed comparisons fail when engines share a process, since torch threads can slow one. Each engine therefore runs in a fresh process on identical float32 arrays, timed as a median after warm-up. The gap narrows as clips grow, from 7.7 times Whistle’s time at 5 s to 2.8 times at 30 s, partly from Whisper’s fixed padding window. Peak memory is 124 MB for Whistle and 823 MB for Whisper base. Whistle’s per-clip time differed nearly threefold between the two splits, which clip length does not explain, so treat the ratios as approximate. Copy CodeCopiedUse a different Browserdef quiet_windows(a, limit=30.0, search=5.0, hop=0.2): """Cut before `limit` seconds at the quietest `hop`-second patch inside the last `search` seconds of each window.""" out, i, n, s, h = [], 0, int(limit * SR), int(search * SR), int(hop * SR) while len(a) - i > n: seg = a[i + n - s:i + n] energy = [np.mean(seg[j:j + h] ** 2) for j in range(0, len(seg) - h, h // 4)] cut = i + n - s + int(np.argmin(energy)) * (h // 4) + h // 2 out.append(a[i:cut]) i = cut out.append(a[i:]) return out def speaker_sessions(min_s=40, max_s=100, k=8): """Same-speaker concatenations (0.4 s gaps) from the sampled clips of both splits, longest speakers first.""" by = collections.defaultdict(list) for split in ("other", "clean"): for c in EN[split]: by[c["spk"]].append(c) out, gap = [], np.zeros(int(0.4 * SR), np.float32) for spk, cs in sorted(by.items(), key=lambda kv: -sum(c["dur"] for c in kv[1])): parts, used, total = [], [], 0 for c in cs: if total + c["dur"] + 0.4 > max_s: break parts += [c["audio"], gap] used.append(c) total += c["dur"] + 0.4 if total >= min_s: out.append({"spk": spk, "clips": used, "audio": np.concatenate(parts), "ref": " ".join(c["ref"] for c in used), "dur": total}) if len(out) == k: break return out OTHER_ENGINE_SCRIPT = r""" import sys, os, numpy as np os.environ["NEEDLE3_LIB_PATH"] = sys.argv[1] import needle W = needle.Whistle() a = np.load(sys.argv[2]) for secs in (30.0, 30.01, 45.0): try: r = W.transcribe(a[:int(secs * 16000)], language="en", word_timestamps=True) print(f" {secs:6.2f} s -> {len(r['words'])} words, last word ends at {r['words'][-1]['end']:.2f} s") except RuntimeError as e: print(f" {secs:6.2f} s -> RuntimeError: {e}") """ @section("5. The 30-second window: what transcribe() does with longer audio, and two ways around it") def long_audio(): global SESSIONS SESSIONS = speaker_sessions() s = SESSIONS[0] gap = np.zeros(int(0.4 * SR), np.float32) print(f"speaker {s['spk']}: {len(s['clips'])} utterances joined with 0.4 s gaps, {s['dur']:.0f} s, {len(s['ref'].split())} reference words") print("the docs say transcribe() takes 'audio of any length, one pass up to 30 seconds and windowed beyond it'.") print(f"the pinned engine {ENGINE_VERSION}, on whole utterances cut from the session so the reference is exact:n") print(f"{'audio':>7s} | {'ref words':>9s} {'hyp words':>9s} {'WER':>7s} | gaps longer than 3 s in the word timeline") for target in (20, 29, 35, 45, 60, 90): parts, refs, total = [], [], 0 for c in s["clips"]: if total + c["dur"] + 0.4 > target: break parts += [c["audio"], gap] refs.append(c["ref"]) total += c["dur"] + 0.4 if total 3] print(f"{total:5.0f} s | {len(ref.split()):9d} {len(hyp.split()):9d} {100*jiwer.wer(ref, hyp or ''):6.1f}% | {', '.join(gaps) or 'none'}") full = W.transcribe(s["audio"], language="en", word_timestamps=True) hyp = NORM_EN(full["text"]) print(f"nthe whole {s['dur']:.0f} s in one call: {len(hyp.split())} of {len(s['ref'].split())} words, WER {100*jiwer.wer(s['ref'], hyp or ''):.1f}%, " f"nothing raised, engine ttft {full['ttft_ms']:.0f} ms") # the same repo revision also holds a 3.2.0 engine wheel; a child process shows what that build does with the same audio other_wheel = hf_hub_download("Cactus-Compute/needle3", needle_fetch.engine_wheel("3.2.0", TAG, 3), revision=NEEDLE3_REV) with zipfile.ZipFile(other_wheel) as z: member = next(n for n in z.namelist() if n.startswith("needle/libneedle")) other = os.path.abspath(os.path.join("engine", "v3.2.0-" + os.path.basename(member))) open(other, "wb").write(z.read(member)) np.save("session0.npy", s["audio"]) open("other_engine.py", "w").write(OTHER_ENGINE_SCRIPT) out = subprocess.run([sys.executable, "other_engine.py", other, "session0.npy"], capture_output=True, text=True, env=os.environ) print(f"nthe 3.2.0 engine wheel from the same repo revision (sha256 {sha(other)}), in a child process, on the same audio:") print(out.stdout.rstrip() or out.stderr[-300:]) print(" two builds, two behaviours: one windows and loses the middle, the other refuses. Either way the split is yours to make.n") print("workarounds on the whole clip, scored against the exact reference (one clip; the joins are 0.4 s of silence):") results = {} for name, wins in (("fixed 30 s windows", [s["audio"][i:i + 30 * SR] for i in range(0, len(s["audio"]), 30 * SR)]), ("fixed 25 s windows", [s["audio"][i:i + 25 * SR] for i in range(0, len(s["audio"]), 25 * SR)]), ("quiet-point cuts <= 30 s", quiet_windows(s["audio"], 30)), ("quiet-point cuts <= 20 s", quiet_windows(s["audio"], 20))): t0 = time.perf_counter() hyp = NORM_EN(" ".join(W.transcribe(w, language="en")["text"] for w in wins)) results[name] = 100 * jiwer.wer(s["ref"], hyp or "") print(f" {name:26s} {len(wins):2d} calls WER {results[name]:5.2f}% {1000*(time.perf_counter()-t0):4.0f} ms " f"windows of {', '.join(f'{len(w)/SR:.1f}' for w in wins)} s") t0 = time.perf_counter() steps = list(W.stream(iter([s["audio"][i:i + SR] for i in range(0, len(s["audio"]), SR)]), language="en")) hyp = NORM_EN(" ".join(x["text"] for x in steps if x["text"])) results["stream(), 1 s chunks"] = 100 * jiwer.wer(s["ref"], hyp or "") print(f" {'stream(), 1 s chunks':26s} {len(steps):2d} passes WER {results['stream(), 1 s chunks']:5.2f}% {1000*(time.perf_counter()-t0):4.0f} ms") best = min(results, key=results.get) return (f"one call on {s['dur']:.0f} s keeps {len(NORM_EN(full['text']).split())}/{len(s['ref'].split())} words; {best}: {results[best]:.2f}% WER " f"(fixed 30 s windows {results['fixed 30 s windows']:.2f}%)") long_audio() Joining one speaker’s utterances gives a long clip with an exact reference, and the README promises windowing beyond the 30 s limit. A shorter cut comes back almost word for word, but the pinned 3.3.0 engine silently drops the middle of the full clip, returning fewer than half its words with no error. The 3.2.0 engine refuses such audio with a RuntimeError instead, so the two builds disagree and the split is ours to make. Splitting at quiet points scores 3.75% WER against 6.74% for fixed windows, though one clip cannot settle it. Copy CodeCopiedUse a different Browser@section("6. Streaming: cost per pass, how long a word waits to be committed, and what chunk size changes") def streaming(): durs = [s["dur"] for s in SESSIONS] print(f"{len(SESSIONS)} same-speaker sessions of {', '.join(f'{x:.0f}' for x in durs)} s, " f"{sum(len(s['ref'].split()) for s in SESSIONS)} reference words; the joins between utterances are artificial") off = {c["id"]: h for split in ("other", "clean") for c, h in zip(EN[split], HYP[(split, "Whistle")])} off_err = [sum(wer_counts(c["ref"], NORM_EN(off[c["id"]]))[0] for c in s["clips"]) for s in SESSIONS] words = [len(s["ref"].split()) for s in SESSIONS] print(f"the same utterances one at a time through transcribe() (the step 3 outputs): WER {100*sum(off_err)/sum(words):.2f}%n") print(f"{'chunk':>6s} {'sessions':>8s} | {'WER':>6s} | {'pass ms mean / max':>18s} | {'passes > 1 s':>12s} | {'commit latency median / p90':>27s} | {'words only in tail':>18s} | {'pass vs time r':>14s}") summary, stream_err = {}, None for chunk_s, sessions in ((1.0, SESSIONS), (0.5, SESSIONS[:2]), (1.0, SESSIONS[:2]), (2.0, SESSIONS[:2])): n = int(chunk_s * SR) errs, tail, allw = [], 0, 0 pass_ms, lat, rs = [], [], [] for s in sessions: steps = list(W.stream(iter([s["audio"][i:i + n] for i in range(0, len(s["audio"]), n)]), language="en")) errs.append(wer_counts(s["ref"], NORM_EN(" ".join(x["text"] for x in steps if x["text"])))[0]) body = steps[:-1] pass_ms += [x["pass_ms"] for x in body] lat += [x["received"] - wd["end"] for x in steps for wd in x["words"]] tail += len(steps[-1]["words"]) allw += sum(len(x["words"]) for x in steps) if len(body) > 2: rs.append(np.corrcoef([x["received"] for x in body], [x["pass_ms"] for x in body])[0, 1]) w = words[:len(sessions)] summary[(chunk_s, len(sessions))] = (100 * sum(errs) / sum(w), np.median(lat), np.mean(pass_ms)) if chunk_s == 1.0 and len(sessions) == len(SESSIONS): stream_err = errs print(f"{chunk_s:5.1f} s {len(sessions):8d} | {100*sum(errs)/sum(w):5.2f}% | {np.mean(pass_ms):9.0f} / {max(pass_ms):4.0f} | {sum(p > 1000 for p in pass_ms):12d} | " f"{np.median(lat):12.2f} / {np.percentile(lat, 90):4.2f} s | {100*tail/allw:17.1f}% | {np.mean(rs):+.2f}") c = paired_contrast(list(range(len(SESSIONS))), stream_err, words, off_err, "stream, 1 s chunks", "one utterance at a time") print(f"nstream (1 s chunks, 8 sessions) minus utterance-by-utterance: {c['diff']:+.2f} pts (95% session-cluster CI {c['lo']:+.2f}..{c['hi']:+.2f}, " f"{c['clusters']} sessions) -> {'no significant difference' if c['lo'] <= 0 <= c['hi'] else 'a real gap'} at this n") print(" pass_ms is the engine time per chunk; below the chunk length means real time. Commit latency is the stream time") print(" at which a word was committed minus the word's end: each chunk re-transcribes the recent audio and commits what two") print(" passes agree on, so the wait scales with the chunk. r is the correlation of pass time with stream time (buffer growth).") print(" The three chunk sizes are compared on the same two sessions; the eight-session row is the headline for 1 s chunks.") a8, a1, a05, a2 = summary[(1.0, len(SESSIONS))], summary[(1.0, 2)], summary[(0.5, 2)], summary[(2.0, 2)] return (f"1 s chunks on 8 sessions: WER {a8[0]:.2f}% vs {100*sum(off_err)/sum(words):.2f}% utterance by utterance, median commit {a8[1]:.2f} s; " f"on 2 sessions the median commit is {a05[1]:.2f} / {a1[1]:.2f} / {a2[1]:.2f} s for 0.5 / 1 / 2 s chunks") streaming() Streaming through stream() re-transcribes recent audio, committing a word once two consecutive passes agree, so waits grow with chunk size. Eight same-speaker sessions, made by joining utterances, stream in one-second chunks at 10.05% WER, against 10.41% when the same utterances are transcribed one at a time, a non-significant difference. A word is committed a median 1.68 s after it ends, and on two sessions the median rises from 0.78 s at half-second chunks to 3.44 s at two-second chunks. The joins are artificial, so this tests the mechanism, not conversation. Copy CodeCopiedUse a different Browserdef auroc(scores, labels): """Probability that a random positive scores above a random negative (Mann-Whitney).""" pos = [s for s, l in zip(scores, labels) if l] neg = [s for s, l in zip(scores, labels) if not l] if not pos or not neg: return float("nan") order = sorted((s, i) for i, s in enumerate(scores)) ranks, i = {}, 0 while i < len(order): j = i while j + 1 < len(order) and order[j + 1][0] == order[i][0]: j += 1 for k in range(i, j + 1): ranks[order[k][1]] = (i + j) / 2 + 1 i = j + 1 pos_idx = [i for i, l in enumerate(labels) if l] return (sum(ranks[i] for i in pos_idx) - len(pos) * (len(pos) + 1) / 2) / (len(pos) * len(neg)) @section("7. Word timestamps and word probabilities: consistency, a known shift, agreement with Whisper, and what p < 0.7 means") def timestamps(): import difflib pk = EN["other"][:40] key = lambda w: re.sub(r"[^a-z']", "", w.lower()) diffs, same, mono, shifts = [], 0, 0, [] probs, correct, durs = [], [], [] replay_hits = replay_n = 0 for i, c in enumerate(pk): ws = W.transcribe(c["audio"], language="en", word_timestamps=True)["words"] mono += all(a["start"] <= a["end"] <= b["start"] + 1e-6 for a, b in zip(ws, ws[1:])) if i < 20: # a known shift: 2.000 s of zeros in front should move every word by 2.000 s ws2 = W.transcribe(np.concatenate([np.zeros(2 * SR, np.float32), c["audio"]]), language="en", word_timestamps=True)["words"] if ws2 and ws: shifts.append(ws2[0]["start"] - ws[0]["start"]) if i = 6: # cut the audio where word k ends and transcribe only the cut k = len(ws) // 2 got = NORM_EN(W.transcribe(c["audio"][:int(ws[k]["end"] * SR)], language="en")["text"]).split() exp = NORM_EN(" ".join(w["word"] for w in ws[:k + 1])).split() replay_n += 1 replay_hits += got == exp torch.manual_seed(0) WHISPER_SECONDS["base"] += c["dur"] ww = [w for s in whisper_model("base").transcribe(c["audio"], language="en", fp16=False, word_timestamps=True)["segments"] for w in s["words"]] if [key(w["word"]) for w in ws] == [key(w["word"]) for w in ww]: same += 1 diffs += [x["start"] - y["start"] for x, y in zip(ws, ww)] + [x["end"] - y["end"] for x, y in zip(ws, ww)] H, tp, td = [], [], [] for w in ws: for t in NORM_EN(w["word"]).split(): # "hmm" normalises to nothing, "didn't" to two tokens, as in the reference H.append(t) tp.append(w["probability"]) td.append(w["end"] - w["start"]) ok = [False] * len(H) for tag, i1, i2, j1, j2 in difflib.SequenceMatcher(a=c["ref"].split(), b=H).get_opcodes(): if tag == "equal": for j in range(j1, j2): ok[j] = True probs += tp correct += ok durs += td diffs, shifts = np.abs(diffs), np.array(shifts) print(f"{len(pk)} test-other clips: word times monotonic on {mono}/{len(pk)}") print(f"2.000 s of silence prepended on 20 clips: first word moved by {np.median(shifts):.3f} s (median), within 50 ms of 2.000 on " f"{100*np.mean(np.abs(shifts-2) <= 0.05):.0f}% of clips") print(f"cut-and-replay: cutting the audio at a word's 'end' and transcribing the cut reproduced the exact word prefix on {replay_hits}/{replay_n} clips") print(f"agreement with Whisper base's DTW word times, on the {same}/{len(pk)} clips where both emit the same words ({len(diffs)//2} words; a selected, easier subset):") print(f" |boundary difference| median {np.median(diffs):.3f} s, within 100 ms {100*np.mean(diffs <= 0.1):.0f}%, within 200 ms " f"{100*np.mean(diffs = lo) & (probs < hi) if m.sum(): print(f" p in [{lo:.1f}, {min(hi, 1):.1f}): {m.sum():4d} words, {100*(~correct[m]).mean():5.1f}% wrong") print(f" AUROC for 'this word is wrong': 1 - probability {auroc(-probs, ~correct):.3f}; word duration as a baseline (shorter = riskier) {auroc(-durs, ~correct):.3f}") low, high = probs = 0.7 return (f"shift {np.median(shifts):.2f} s; within 200 ms of Whisper on {100*np.mean(diffs <= 0.2):.0f}% of {len(diffs)//2} matched words; " f"p<0.7 words wrong {100*(~correct[low]).mean():.0f}% vs {100*(~correct[high]).mean():.1f}%") timestamps() Without gold alignments, we check timings against what they must satisfy. Word times are monotonic on every clip we check, and prepending a two-second silence to twenty clips shifts the first word by a median of 2.000 s. Cutting at a word’s end should reproduce its prefix, which holds on only a fifth of those twenty clips; this tests stability, not end-time accuracy. Probability is the stronger signal: words rated below one half were wrong 69.2% of the time, against 2.5% at 0.9 or above, and AUROC is 0.916 for wrong words, against 0.509 for duration. Copy CodeCopiedUse a different Browser@section("8. Keyword biasing: an oracle ceiling, a matched wrong-keyword control and a distractor list") def keywords(): import difflib clips = EN["other"] base = HYP[("other", "Whistle")] freq = collections.Counter(w for c in clips for w in c["ref"].split()) def error_words(ref, hyp): out = [] for tag, i1, i2, j1, j2 in difflib.SequenceMatcher(a=ref.split(), b=NORM_EN(hyp).split()).get_opcodes(): if tag in ("replace", "delete"): out += ref.split()[i1:i2] return sorted(set(out)) oracle = [error_words(c["ref"], h) for c, h in zip(clips, base)] idx = [i for i, k in enumerate(oracle) if k] print(f"{len(idx)}/{len(clips)} test-other clips have at least one Whistle error ({sum(len(oracle[i]) for i in idx)} distinct wrong words)") print(" oracle keywords = the reference words Whistle got wrong on that clip (a CEILING: it reads the answer)") print(" matched wrong = the next error clip's oracle list, minus every word the target clip actually says") print(" distractor (8) = the 8 most frequent reference words of the whole sample that the target clip does not say") print(" distractor (2) = the same list cut to 2 words, so its size matches the oracle and matched armsn") def matched_wrong(i): donor = oracle[idx[(idx.index(i) + 1) % len(idx)]] return [w for w in donor if w not in clips[i]["ref"].split()] or None arms = {} distractor = lambda i, k: [w for w, _ in freq.most_common(60) if w not in clips[i]["ref"].split()][:k] for name, keylist in (("no keywords", lambda i: None), ("oracle (ceiling)", lambda i: oracle[i]), ("matched wrong", matched_wrong), ("distractor (8)", lambda i: distractor(i, 8)), ("distractor (2)", lambda i: distractor(i, 2))): errs, words, ins, recovered, total_kw, bad_kw, bad_kw_base, n_kw = [], [], 0, 0, 0, 0, 0, 0 for i in idx: c = clips[i] kws = keylist(i) n_kw += len(kws or []) h = base[i] if name == "no keywords" else W.transcribe(c["audio"], language="en", keywords=kws)["text"] s, d, ins_i = sdi(c["ref"], NORM_EN(h)) errs.append(s + d + ins_i) words.append(len(c["ref"].split())) ins += ins_i still_wrong = set(error_words(c["ref"], h)) recovered += sum(w not in still_wrong for w in oracle[i]) total_kw += len(oracle[i]) if kws and name != "oracle (ceiling)": bad_kw += sum(w in NORM_EN(h).split() for w in kws) bad_kw_base += sum(w in NORM_EN(base[i]).split() for w in kws) arms[name] = (errs, words) extra = f" wrong keywords in the output {bad_kw:3d} (already {bad_kw_base} without biasing)" if name not in ("no keywords", "oracle (ceiling)") else "" print(f" {name:18s} {n_kw/len(idx):4.1f} keywords/clip WER {corpus_wer(errs, words):6.2f}% insertions {ins:3d} oracle words now right {recovered:3d}/{total_kw}{extra}") spk = [clips[i]["spk"] for i in idx] w = arms["no keywords"][1] fam = [] for a, b in (("no keywords", "oracle (ceiling)"), ("matched wrong", "no keywords"), ("distractor (8)", "no keywords"), ("distractor (2)", "no keywords"), ("matched wrong", "oracle (ceiling)")): c = paired_contrast(spk, arms[a][0], w, arms[b][0], a, b) fam.append(c) sig = holm([c["p"] for c in fam]) print(f"n family B, {len(fam)} contrasts, Holm at 0.05:") for c, s in zip(fam, sig): print(f" WER({c['a']}) - WER({c['b']}) = {fmt_contrast(c)} p~{c['p']:.3f} {'significant' if s else 'not significant'}") print(" the decisive contrast is matched wrong minus oracle: how much of the gain needs the right words, not just any words") return (f"oracle ceiling {corpus_wer(*arms['oracle (ceiling)']):.2f}% vs {corpus_wer(*arms['no keywords']):.2f}% baseline; " f"wrong keywords {corpus_wer(*arms['matched wrong']):.2f}%") keywords() Keyword biasing needs controls, because an oracle that reads the answer looks like a win. The oracle lists the words Whistle got wrong; the matched control uses another clip’s, minus any this clip says. On the 185 clips with errors, the oracle cuts WER from 14.74% to 7.62%, while matched wrong words leave it at 14.82%, a non-significant change. Eight frequent words the clip lacks raise WER 2.05 points. Against the oracle, matched wrong words are 7.20 points worse, significant after Holm correction, so the right words matter. Copy CodeCopiedUse a different BrowserMLS = {"de": "german", "fr": "french", "es": "spanish", "it": "italian", "nl": "dutch", "pl": "polish"} WHISTLE_LANGS = ("en", "de", "fr", "es", "it", "nl", "pl") N_PER_LANG = 12 PUBLISHED_MLS_BASE = {"nl": 28.4, "fr": 26.6, "de": 17.7, "it": 31.1, "pl": 22.8, "es": 12.8} # Whisper paper, table 10 (full splits) def clopper_pearson_lower(k, n, alpha=0.05): """Exact one-sided 95% lower bound for a binomial proportion, by bisection on the binomial tail.""" lo, hi = 0.0, 1.0 for _ in range(60): p = (lo + hi) / 2 tail = sum(math.comb(n, j) * p ** j * (1 - p) ** (n - j) for j in range(k, n + 1)) lo, hi = (p, hi) if tail 4s} {'clips':>5s} {'spk':>3s} {'words':>5s} | {'Whistle':>8s} {'base':>8s} | Whisper base published on the full split (paper table 10)") for code, n, s, nw, ww, wbase in rows_out: print(f"{code:>4s} {n:5d} {s:3d} {nw:5d} | {ww:7.1f}% {wbase:7.1f}% | {PUBLISHED_MLS_BASE[code]}") print(" per-language rows are descriptive: 12 clips from a handful of speakers cannot rank the systems within a languagen") c = paired_contrast(clusters, pooled["whisper base, greedy"], words, pooled["Whistle (forced)"], "whisper base, greedy", "Whistle") c["name"] = "MLS pooled: Whistle vs whisper base, greedy" CONTRASTS.append(c) print(f"pooled over {len(words)} clips / {c['clusters']} speakers / {sum(words)} words: Whistle (forced) {corpus_wer(pooled['Whistle (forced)'], words):.2f}%, " f"whisper base greedy {corpus_wer(pooled['whisper base, greedy'], words):.2f}%; {fmt_contrast(c)} [family A]") print(f"Whistle with the language detected instead of forced: {corpus_wer(pooled['Whistle (detected)'], words):.2f}%") n = len(words) print(f"language identification over the same seven labels: Whistle {lid_w}/{n} (one-sided 95% lower bound {100*clopper_pearson_lower(lid_w, n):.0f}%, " f"treating clips as independent), whisper base {lid_b}/{n} (lower bound {100*clopper_pearson_lower(lid_b, n):.0f}%)") json.dump(MANIFEST, open("manifest.json", "w")) fam = [x for x in CONTRASTS if x["name"] in FAMILY_A] sig = dict(zip([x["name"] for x in fam], holm([x["p"] for x in fam]))) print(f"nfamily A complete ({len(fam)} contrasts), Holm at 0.05 (the printed CIs are unadjusted; the verdict uses the adjusted p):") for x in fam: v = verdict(x, sig[x["name"]]) VERDICTS.append((x["name"], fmt_contrast(x), v)) print(f" {x['name']:48s} {fmt_contrast(x)} p~{x['p']:.3f} -> {v}") return f"pooled WER Whistle {corpus_wer(pooled['Whistle (forced)'], words):.1f}% vs base {corpus_wer(pooled['whisper base, greedy'], words):.1f}%; language ID {lid_w}/{n} vs {lid_b}/{n}" multilingual() Forcing the true language would hide detection failures, so each system must also detect the language. We draw 12 clips per language from three random pages, fixed in advance, and score both sides with BasicTextNormalizer. We pin no MLS revision, so each clip is hashed instead. Pooled over 72 clips from 22 speakers, forced Whistle scores 26.40% against 24.06% for greedy Whisper base, a gap in Whisper’s favour that is not significant. Both systems detect the language on all 72 clips. Per-language rows are descriptive only. Copy CodeCopiedUse a different Browser@section("10. Non-speech, quiet speech and white noise at three SNRs, Whistle against Whisper base") def noise(): rng = np.random.default_rng(SEED) quiet = EN["other"][0]["audio"] * (10 ** (-30 / 20) / np.sqrt(np.mean(EN["other"][0]["audio"] ** 2))) probes = {"5 s digital silence": np.zeros(5 * SR, np.float32), "white noise, -40 dBFS": (rng.standard_normal(5 * SR) * 0.01).astype(np.float32), "white noise, -20 dBFS": (rng.standard_normal(5 * SR) * 0.1).astype(np.float32), "440 Hz tone, -20 dBFS": (0.1 * np.sqrt(2) * np.sin(2 * np.pi * 440 * np.arange(5 * SR) / SR)).astype(np.float32), "speech scaled to -30 dBFS": quiet.astype(np.float32)} print(f"{'input':>26s} | Whistle text / language | Whisper base text") for name, x in probes.items(): w = W.transcribe(x, language="en") b = run_whisper("base", x)[0].strip() print(f"{name:>26s} | {w['text']!r} {w['language']!r} | {b!r}") print(f" reference for the quiet clip: {EN['other'][0]['ref']}") print(" an empty Whisper output is its no-speech gate, not evidence of quality; an empty Whistle output on real speech would be a missn") clips = EN["other"][:20] spk = [c["spk"] for c in clips] words = [len(c["ref"].split()) for c in clips] print(f"white noise on {len(clips)} test-other clips, {sum(words)} words, {len(set(spk))} speakers (directional at this n):") print(f"{'condition':>14s} | {'Whistle':>8s} {'base greedy':>11s} | WER(base) - WER(Whistle), 95% speaker-cluster CI") out = [] for snr in (None, 20, 10, 5): xs = [] for c in clips: if snr is None: xs.append(c["audio"]) else: n = rng.standard_normal(len(c["audio"])).astype(np.float32) n *= np.sqrt((c["audio"] ** 2).mean() / ((n ** 2).mean() * 10 ** (snr / 10))) y = c["audio"] + n xs.append((y * min(1, 0.99 / np.abs(y).max())).astype(np.float32)) e_w = [wer_counts(c["ref"], NORM_EN(W.transcribe(x, language="en")["text"]))[0] for c, x in zip(clips, xs)] e_b = [wer_counts(c["ref"], NORM_EN(run_whisper("base", x)[0]))[0] for c, x in zip(clips, xs)] c = paired_contrast(spk, e_b, words, e_w, "base", "Whistle") label = "clean" if snr is None else f"{snr} dB SNR" out.append((label, corpus_wer(e_w, words), corpus_wer(e_b, words))) print(f"{label:>14s} | {corpus_wer(e_w, words):7.1f}% {corpus_wer(e_b, words):10.1f}% | {c['diff']:+6.1f} ({c['lo']:+.1f}..{c['hi']:+.1f})") print("n Whistle's front end keeps roughly 250-3500 Hz while Whisper's covers 0-8 kHz, so the two systems do not see the same") print(" noise; this compares them as shipped, and at n = 20 none of the differences is distinguishable from zero.") return "; ".join(f"{l}: {w:.0f}% vs {b:.0f}%" for l, w, b in out[1:]) noise() Digital silence, two noise levels, and a pure tone all give empty text. For Whisper that is a no-speech gate, not evidence of quality, and an empty Whistle output on speech would be a miss. A real sentence at -30 dBFS comes back word for word from Whistle, though Whisper base writes ‘crazed’ for ‘traced’. At the harshest SNR we ran, white noise lifts WER on 20 test-other clips from 12.4% to 37.1% for Whistle and from 15.2% to 39.9% for Whisper base, and no gap is distinguishable from zero. Copy CodeCopiedUse a different Browser@section("11. Speech embeddings: the measured frame rate, and finding a spoken word inside another clip") def embeddings(): frames = {} for secs in (1, 10, 30): v = np.asarray(W.embed(audio_seconds(EN["other"], secs)), np.float32) frames[secs] = len(v) // 256 print(f"{secs:2d} s of audio -> {len(v):7d} floats = {len(v)//256:4d} frames x 256 ({len(v)/256/secs:.1f} frames/s, {1000*secs/(len(v)/256):.1f} ms per frame)") fps = frames[10] / 10 try: W.embed(np.zeros(31 * SR, np.float32)) except RuntimeError as e: print(f"31 s -> RuntimeError: {e}") print("nspoken-term search: embed clip A whole, keep the frames under one word's own timestamps as the query (the encoder is") print("non-causal, so those frames carry context), slide the query over clip B's frames with a cosine score. Pool: the first 120") print("test-other clips; words of 5+ letters and 0.25 s+, first two occurrences in different clips; a hit is a peak within 0.3 s.") clips = [c for c in EN["other"][:120]] key = lambda w: re.sub(r"[^a-z']", "", w.lower()) occurrences = collections.defaultdict(list) for i, c in enumerate(clips): for w in W.transcribe(c["audio"], language="en", word_timestamps=True)["words"]: if len(key(w["word"])) >= 5 and w["end"] - w["start"] >= 0.25: occurrences[key(w["word"])].append((i, w["start"], w["end"])) pairs = [(k, o[0], o[1]) for k, o in occurrences.items() if len(o) >= 2 and o[0][0] != o[1][0]] cache = {} def frames_of(i): if i not in cache: v = np.asarray(W.embed(clips[i]["audio"]), np.float32).reshape(-1, 256) cache[i] = v / (np.linalg.norm(v, axis=1, keepdims=True) + 1e-8) return cache[i] hits = n = 0 chance, tol = [], 0.3 for k, (ia, sa, ea), (ib, sb, eb) in pairs[:60]: q = frames_of(ia)[int(sa * fps):max(int(sa * fps) + 2, int(ea * fps))] T = frames_of(ib) if len(T) <= len(q): continue scores = np.array([(q * T[j:j + len(q)]).sum(1).mean() for j in range(len(T) - len(q) + 1)]) peak = (scores.argmax() + len(q) / 2) / fps hit = sb - tol <= peak <= eb + tol hits += hit n += 1 chance.append((eb - sb + 2 * tol) / clips[ib]["dur"]) if n peak at {peak:5.2f} s in clip {ib:3d}; the word is spoken at {sb:.2f}-{eb:.2f} s: {'hit' if hit else 'miss'}") print(f" {hits}/{n} peaks inside the target word (+-{tol} s); a random position would land there about {100*np.mean(chance):.0f}% of the time") print(" the encoder output is a content representation at frame level; it is not a speaker or sentence embedding") return f"{fps:.1f} frames/s x 256 floats; spoken-term search {hits}/{n} hits (chance ~{100*np.mean(chance):.0f}%)" embeddings() Frame rate is measured because the search maps frames to seconds. A ten-second clip yields 25 frames a second, each a 256-float vector, and clips past the embedding limit raise a RuntimeError. We embed a clip whole, since its frames see both sides, and slide one word’s frames across another clip by cosine score. The peak lands within 0.3 s of the word on 32 of 34 pairs of words with five or more letters, where chance gives about 17%. The output is frame-level content, not a sentence embedding. Copy CodeCopiedUse a different BrowserINTENTS = {"abroad": "Using the card or account while travelling or abroad", "address": "Change, update or confirm the postal address on the account", "app_error": "The banking app or website shows an error or does not work", "atm_limit": "The daily ATM cash withdrawal limit", "balance": "Check the current account balance", "business_loan": "Apply for or ask about a business loan", "card_issues": "A card is lost, stolen, blocked, declined or not working", "cash_deposit": "Deposit or transfer cash into the account", "direct_debit": "Set up, change or cancel a direct debit or standing order", "freeze": "Freeze, block or unfreeze the account or card", "high_value_payment": "Make or authorize a large or high-value payment or transfer", "joint_account": "Open or ask about a joint account with another person", "latest_transactions": "See the latest or recent transactions and payments", "pay_bill": "Pay a bill or make a payment to a company"} N_MINDS = 200 @section("12. From speech to a tool call: Whistle transcripts into Needle 3 on MInDS-14 banking requests") def speech_to_tools(): tools = [{"name": n, "description": d, "parameters": {"type": "object", "properties": {}}} for n, d in INTENTS.items()] print(f"14 tools, one per MInDS-14 intent, descriptions fixed before the first call (sha256 {hashlib.sha256(json.dumps(tools).encode()).hexdigest()[:12]})") first = page("PolyAI/minds14", "en-US", "train", 0, 1) names = next(f for f in first["features"] if f["name"] == "intent_class")["type"]["names"] assert names == list(INTENTS), names rows = all_rows("PolyAI/minds14", "en-US", "train") sample = random.Random(SEED).sample(rows, N_MINDS) clips, too_long = [], 0 for r in sample: audio, h = load_audio(r) # 8 kHz telephone wav, resampled to 16 kHz by load_audio MANIFEST.setdefault("minds14", []).append([r["path"], h]) if len(audio) / SR > MAX_CLIP_S: # the same pre-declared filter as step 2: the engine refuses longer clips too_long += 1 continue clips.append({"audio": audio, "ref": r["transcription"], "intent": names[r["intent_class"]], "dur": len(audio) / SR}) t0 = time.perf_counter() hyps = [W.transcribe(c["audio"], language="en")["text"] for c in clips] tw = time.perf_counter() - t0 e, n = zip(*[wer_counts(NORM_EN(c["ref"]), NORM_EN(h)) for c, h in zip(clips, hyps)]) json.dump(MANIFEST, open("manifest.json", "w")) print(f"{len(clips)} of the {N_MINDS} sampled clips ({len(rows)} rows in the split; {too_long} dropped for exceeding {MAX_CLIP_S:.0f} s; " f"{sum(c['dur'] for c in clips)/60:.1f} min of 8 kHz telephone speech): Whistle WER {corpus_wer(e, n):.1f}% " f"against the casual MInDS-14 transcriptions, {1000*tw/len(clips):.0f} ms per clip") for c, h in list(zip(clips, hyps))[:2]: print(f" [{c['intent']}] ref: {c['ref'][:72]}n {' ' * (len(c['intent']) + 2)} hyp: {h[:72]}") t0 = time.perf_counter() agent = needle.Needle(tools=tools, auto_date=False) # no date fact in the prompt, so the result does not depend on the clock print(f"nNeedle 3 loaded in {time.perf_counter()-t0:.1f} s into the same engine; Whistle stays loaded beside it") def route(texts): out = [] for t in texts: r = agent.complete(t) agent.reset() calls = [c["name"] for c in r.get("function_calls", [])] out.append((calls[0] if calls else None, r.get("confidence") or 0.0, r.get("peak_ram_mb"))) return out truth = [c["intent"] for c in clips] res = {} for label, texts in (("reference transcripts", [c["ref"] for c in clips]), ("Whistle transcripts", hyps)): t0 = time.perf_counter() res[label] = route(texts) right = np.array([p == y for (p, _, _), y in zip(res[label], truth)]) conf = np.array([c for _, c, _ in res[label]]) print(f" routing from {label:21s}: {100*right.mean():.1f}% correct, {sum(p is None for p, _, _ in res[label])} refusals, " f"{1000*(time.perf_counter()-t0)/len(texts):.0f} ms/call, confidence when right {conf[right].mean():.2f} vs wrong {conf[~right].mean():.2f}") a = [p == y for (p, _, _), y in zip(res["reference transcripts"], truth)] b = [p == y for (p, _, _), y in zip(res["Whistle transcripts"], truth)] only_ref, only_wh = sum(x and not y for x, y in zip(a, b)), sum(y and not x for x, y in zip(a, b)) m = only_ref + only_wh p_mc = min(1.0, 2 * sum(math.comb(m, j) for j in range(min(only_ref, only_wh) + 1)) / 2 ** m) if m else 1.0 print(f" discordant clips: right only from the reference text {only_ref}, right only from Whistle's text {only_wh} (exact McNemar p = {p_mc:.2f})") conf = collections.Counter((y, p) for (p, _, _), y in zip(res["Whistle transcripts"], truth) if p != y) print(" most common confusions (truth -> call): " + ", ".join(f"{y} -> {p} x{k}" for (y, p), k in conf.most_common(3))) off = route([NORM_EN(h) for h in HYP[("other", "Whistle")][:40]]) print(f" 40 LibriSpeech sentences (off-topic): {sum(p is None for p, _, _ in off)} refused, {sum(p is not None for p, _, _ in off)} given a banking tool call") print(f" peak_ram_mb reported by Needle: {res['Whistle transcripts'][-1][2]} MB (this whole process, Whisper models included). MInDS-14 is public,") print(" so training-set overlap cannot be ruled out for either model, and the tool descriptions are ours: the number moves with them.") return f"routing {100*np.mean(b):.0f}% right from Whistle's text vs {100*np.mean(a):.0f}% from the reference text; WER {corpus_wer(e, n):.0f}%" speech_to_tools() Whistle transcribes the sampled MInDS-14 banking requests at 31.9% WER against casual reference transcripts. Needle 3 routes each transcript among fourteen tools, one per banking intent, whose descriptions we wrote. The returned function call becomes the route, and a reply with no call counts as a refusal. Routing is right 50.3% of the time from Whistle’s text and 51.8% from reference text, a non-significant gap, so with these descriptions, routing, not transcription, is the limit. Of the off-topic LibriSpeech sentences, 17 are refused, and the rest get a banking tool call. Copy CodeCopiedUse a different Browserbanner("SUMMARY") for name, res in RESULTS.items(): print(f" {name:<104s} {res}") print("nVerdicts on the published-claim contrasts (family A, 95% speaker-cluster paired bootstrap, unadjusted CIs, Holm over 5):") for claim, measured, v in VERDICTS: print(f" {claim:<48s} {measured:<58s} {v}") print(f"nWhisper audio transcribed in this run: " + ", ".join(f"{k} {v/60:.1f} min" for k, v in WHISPER_SECONDS.items()) + f"; hardware {CPU}") print(""" What to carry over - Pin the Hub revisions of whistle.cact, the engine wheel and needle3.cact and print their hashes; both repos replace files in place, and the engine version the package wants is in needle.agent.fetch.ENGINE_VERSIONS, not in the package version. - Score with the same normalizer on both sides of each comparison, sample with a fixed seed, cluster the bootstrap by speaker. - Say which Whisper you compared against: greedy (the package default) and beam search (the paper's numbers) differ. - Keep a single transcribe() call under 30 s; cut longer audio yourself (quiet points worked best on this clip) or use stream(). - Word probability is a usable error signal, keyword biasing has a ceiling, embeddings are content frames. Where to go next - pip install "cactus-needle[mic,compare]" then needle whistle compare clip.wav runs one clip through Whistle, Whisper and Moonshine. - needle download fetches the engine for macOS, Linux, Android, iOS, RISC-V, MIPS, the browser or WASI; the native runner takes --model needle3.cact --model whistle.cact --tools tools.json --audio clip.wav for audio-to-tool-calls in one call. - The weights also ship as safetensors, and fine-tuning for new languages is being discussed in the repository's issues. - Docs: cactuscompute.com/whistle Code: github.com/cactus-compute/needle Weights: huggingface.co/Cactus-Compute/whistle """) The summary prints each section’s one-line result, then the five published-claim verdicts, where only test-other against greedy Whisper base favors Whistle. It also reports the audio Whisper transcribed, 18.4 minutes for tiny and 68.7 for base, and all 13 cells ran without failure. The habits to carry over are pinning Hub revisions, fixing the seed, scoring both sides with one normalizer, and naming the Whisper decoder. In conclusion, on an Apple M4 CPU, Whistle scored 10.47% on the test-other sample, near the published full-split 10.49, quoted, not measured. It beat greedy Whisper base by 2.19 points on the paired subset, while beam search, test-clean, and pooled MLS showed no significant gap. It ran 5.7 times faster than greedy base on a ten-second clip. Streaming with one-second chunks committed words a median 1.68 s after they end, with no significant loss. The 30 s limit matters most: the pinned engine silently drops the middle of longer audio, so split it or stream. These are one-run samples, and no single clip or small set can settle a claim. Check out the FULL CODES here. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post A Coding Guide to Cactus Whistle: Does a 16.9 MB Speech Model Hold Up Against Whisper Base? Word Error Rate, Speed, Streaming, Timestamps, Keywords, and Speech-to-Tool Calls on a CPU appeared first on MarkTechPost.
