Breaking
Advertisement

AI

AI

OpenAI can’t yet explain why its coding agents switch off misalignment monitoring

Two internal OpenAI coding agents switched off safety monitoring on their own, even after weighing whether to keep the safeguards in place. When asked about it, GPT-6 Astra denied removing the monitoring. OpenAI calls this a recurring pattern and doesn't yet have a satisfying explanation for the behavior. The article OpenAI can't yet explain why its coding agents switch off misalignment monitoring appeared first on The Decoder.

AI

Claude Science tackles deadly scorpion stings by designing protein binders from scratch

Anthropic's Claude independently designed protein binders targeting deadly scorpion neurotoxins. Out of more than 318,000 designs, 211 were tested in the lab. Seventeen met the criteria, and the two strongest bind extremely tightly to Cn2, a toxin from a Mexican scorpion. Whether they actually work as antivenom remains an open question. The article Claude Science tackles deadly scorpion stings by designing protein binders from scratch appeared first on The Decoder.

AI

OpenAI acquires Orchestra Bio to build AI agents that speed up drug development

OpenAI is acquiring startup Orchestra Bio and plans to merge it with its life sciences AI platform Rosalind. The goal is to build AI agents that help pharma teams coordinate drug development, from tracking trials to spotting problems. Developing a new drug in the US takes ten to 15 years on average. OpenAI already works with Novo Nordisk and Moderna on AI-driven pharma research. The article OpenAI acquires Orchestra Bio to build AI agents that speed up drug development appeared first on The Decoder.

Advertisement
AI

Mistral AI Releases Voxtral 3: Speech-to-Text, TTS and a Voice Agent SDK

Mistral AI just released Voxtral 3. Mistral Voxtral 3 is a speech model family paired with a voice agent SDK. It ships 3 transcription models, 1 text-to-speech model and an SDK built on LiveKit. One place for listening, reasoning, tool calls and speaking. All 4 models are in public preview on the Mistral API. TL;DR Size: Voxtral Mini Transcribe 3 has 4B parameters. Small Transcribe 3 size: not disclosed. Both batch models list a 32k context. Runs on: Mistral API (EU, US and global endpoints) or your own infrastructure via Mistral sales. GPU and VRAM needs: not disclosed. Mistral says Mini suits resource-constrained environments. Performance: Top 3 on independent speech-to-text rankings, strongest on diarization and multilingual batch transcription. Best: 2.1% AA-WER for Small Transcribe 3, #3 on Artificial Analysis. Worst: Realtime at 240 ms: 7.4% WER on FLEURS 35, vs 4.2% for Gemini 3.5 Realtime. What is in the Voxtral 3 family? Mistral lists 4 models and 1 SDK: Voxtral Small Transcribe 3: flagship batch transcription and diarization. Docs also list translation, speech Q&A, summarization and function calling. Voxtral Mini Transcribe 3: a 4B model for cheaper, faster batch jobs. Voxtral Mini Transcribe Realtime 3: streaming ASR over WebSocket with end-of-turn detection. Voxtral Speech 3: text-to-speech, also called Voxtral Mini TTS 3 in the post. All transcription models support diarization and context biasing. Context biasing takes key terms or free text to improve recall. Batch transcription covers 50 languages, including 24 official EU languages. How does the Voxtral voice agent SDK work? The SDK is built on LiveKit and runs 3 stages as 1 streaming pipeline. Realtime Transcribe 3 listens and detects when the caller stops speaking. A general LLM, such as Mistral Small 4, reasons and calls tools. Voxtral Speech 3 replies in the tone of the conversation so far. Each turn completes in under 1 second. The agent stops talking when the caller cuts in. It handles phone calls over SIP and browser or app sessions over WebRTC. Because each stage is a separate model, teams can swap the LLM, voice or language. (function(){var f=document.getElementById('mtp-vx3-frame');window.addEventListener('message',function(e){if(e.data&&e.data.mtpEmbed==='mtp-vx3'&&e.source===f.contentWindow){f.style.height=e.data.h+'px';}});})(); How accurate is Voxtral Transcribe 3? These numbers are self-reported by Mistral: FLEURS 45 (WER, forced / inferred language): Small 4.1% / 4.0%. MAI-Transcribe-2 ties at 4.1% / 4.0%. Mini 4.6% / 4.5%. Whisper large-v3 16.0% / 14.9%. Diarization (cpWER, 5 public sets): Small 18.4% and Mini 18.6%. Pyannote scores 19.0% and MAI-Transcribe-2 20.3%. Keyphrase recall (Earnings 22): Small 94.6% with context, 81.4% without. Scribe v2 posts 94.3% and 84.0%. On Artificial Analysis, Small scores 2.1% AA-WER and Mini 2.2%. Only Fun-Realtime-ASR-preview (1.7%) and MAI-Transcribe-2 (2.0%) rank higher. Speed is the gap: Small’s speed factor is 65.8x, against 307.5x for MAI-Transcribe-2. How do realtime transcription and TTS compare? On FLEURS 35, the realtime model trades latency for accuracy. It scores 5.7% WER at 2,400 ms and 6.4% at 480 ms. At 240 ms, WER rises to 7.4%. Gemini 3.5 Realtime Transcribe scores 4.2% and MAI-Transcribe-2 Streaming 5.8%. GPT Live Transcribe (8.4%) and Scribe v2 Realtime (8.7%) trail every Voxtral setting. Voxtral Speech 3 costs $13 per 1M characters. Mistral says that undercuts OpenAI TTS-1, Deepgram, ElevenLabs Flash and Gemini Flash. Explicit controls cover emotion, pacing and emphasis. The model page lists ~90 ms time-to-first-audio and 34 emotion tags. Pro Voice adapts a speaker’s style from 30 minutes of audio. Artificial Analysis places it in the top 12 of both TTS arenas. Voxtral 3 vs MAI-Transcribe-2, Scribe v2 and Gemini 3.5 Transcribe How much does Voxtral 3 cost, and where can you run it? Per Mistral’s price table: Voxtral Mini Transcribe 3 (batch): $0.0035/min Voxtral Small Transcribe 3: $0.004/min Voxtral Mini Transcribe Realtime 3: $0.008/min Voxtral Speech 3 (batch and realtime): $13 per 1M characters Voice agent API: $0.04/min with Mistral Small 4, $0.08/min with any other reasoning model The models run on Mistral’s European, US and global endpoints. OpenRouter also lists all 3 new models with EU-hosted, zero-data-retention endpoints. Key Takeaways Voxtral 3 bundles 3 ASR models, 1 TTS model and a LiveKit voice agent SDK. Small Transcribe 3 hits 2.1% AA-WER, #3 on Artificial Analysis. Best-in-chart diarization: 18.4% cpWER, ahead of Pyannote and MAI-Transcribe-2. Batch transcription costs $0.0035 to $0.004 per minute; TTS costs $13 per 1M characters. Realtime accuracy trails Gemini 3.5 Realtime, and no open weights were announced. Check out the Voxtral 3 announcement, the audio docs and the model pages. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Mistral AI Releases Voxtral 3: Speech-to-Text, TTS and a Voice Agent SDK appeared first on MarkTechPost.

AI

Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Google team have released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool. They released it at the PyTorch Conference North America. It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down. TL;DR A serving stack: Google reports a published benchmark on a 17,000-node multi-region GKE deployment. Runs on: Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang. Performance: Vendor-reported gains come from cache-aware routing, tiered KV cache offload and prefill/decode disaggregation. Best: Median TTFT cut by more than 85%, with 2.4x throughput, on an Ironwood TPU coding workload. Worst: RL time-slicing lifts accelerator duty cycle only to above 70%, so roughly 30% idle time remains. What is llm-d 1.0? llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google are the lead maintainers. The project counts more than 500 contributors from over 25 organizations. It is a CNCF Sandbox project, accepted on March 12, 2026, and is licensed under Apache 2.0. Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped. Version 1.0 is about running them reliably in production, which IBM calls day 2 operations. What is new in llm-d 1.0? Multi-tenant quality of service: Priority and fairness-aware flow control stops bursty agents from starving latency-sensitive requests. Stable foundation: v1 APIs and a hardened request path, validated on multiple clouds and accelerators. Zero-downtime upgrades: Router high availability with hot standby, plus DisaggregatedSet, which rolls prefill and decode pools together. Agent-aware serving: Session-aware routing, turn-priority scheduling and native Responses API state. Async inference: llm-d-async-processor pulls jobs from Google Cloud Pub/Sub or Redis into slack capacity. Expanded TPU support: TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang. How does llm-d route requests? The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas hold each cached prefix. It then weighs load and predicted latency before picking a replica. Agents call one endpoint, and each step lands where its context already lives. When accelerator memory fills, KV cache moves to host memory, then SSD or Managed Lustre. It uses vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache for this. Prefill and decode run as separate pools, because prefill is compute-heavy and decode is memory-bound. <iframe id="mtp-llmd1-frame" title="llm-d 1.0 interactive explainer" srcdoc="<!doctype html> <title>llm-d 1.0 Explainer</title> html,body{margin:0!important;padding:0!important;background:transparent!important} #mtp-llmd1{--bg:#0f1218;--card:#171b23;--line:#2a303c;--tx:#e8eaed;--mut:#9aa0a6;--blue:#4285F4;--red:#EA4335;--yel:#FBBC04;--grn:#34A853;--nv:#76B900; max-width:880px;margin:0 auto;background:var(--bg)!important;color:var(--tx)!important;border:1px solid var(--line)!important;border-radius:16px;font-family:Inter,system-ui,sans-serif;overflow:hidden;box-sizing:border-box} #mtp-llmd1 *{box-sizing:border-box} #mtp-llmd1 hr,#mtp-llmd1 p:empty,#mtp-llmd1 del,#mtp-llmd1 s{display:none!important} #mtp-llmd1 .hd{padding:22px 24px 14px;border-bottom:1px solid var(--line)} #mtp-llmd1 .tag{display:inline-flex;gap:6px;align-items:center;font-size:11px;letter-spacing:.08em;text-transform:uppercase;color:var(--mut)!important} #mtp-llmd1 .dots i{display:inline-block;width:8px;height:8px;border-radius:50%;margin-right:3px} #mtp-llmd1 h3{margin:8px 0 4px!important;font-size:24px!important;font-weight:800!important;color:var(--tx)!important;line-height:1.2!important} #mtp-llmd1 .sub{margin:0!important;color:var(--mut)!important;font-size:14px} #mtp-llmd1 .tabs{display:flex;gap:4px;padding:10px 16px 0;border-bottom:1px solid var(--line);overflow-x:auto} #mtp-llmd1 .tabs button{background:transparent!important;border:0!important;border-bottom:3px solid transparent!important;color:var(--mut)!important;padding:10px 12px!important;font:600 13px Inter,sans-serif;cursor:pointer;white-space:nowrap;border-radius:0!important} #mtp-llmd1 .tabs button.on{color:var(--tx)!important;border-bottom-color:var(--blue)!important} #mtp-llmd1 .pn{display:none;padding:20px 24px 24px;animation:mtpfade .35s ease} #mtp-llmd1 .pn.on{display:block} @keyframes mtpfade{from{opacity:0;transform:translateY(6px)}to{opacity:1;transform:none}} #mtp-llmd1 .flow{display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin-bottom:14px} #mtp-llmd1 .node{background:var(--card)!important;border:1px solid var(--line)!important;border-radius:12px;padding:12px;text-align:center;transition:all .3s;position:relative} #mtp-llmd1 .node b{display:block;font-size:13px;color:var(--tx)!important} #mtp-llmd1 .node small{color:var(--mut)!important;font-size:11px} #mtp-llmd1 .node .ic{font-size:20px;display:block;margin-bottom:4px} #mtp-llmd1 .node.act{border-color:var(--blue)!important;box-shadow:0 0 0 3px rgba(66,133,244,.25);transform:translateY(-3px)} #mtp-llmd1 .pkt{height:6px;background:var(--line);border-radius:3px;position:relative;overflow:hidden;margin:4px 0 14px} #mtp-llmd1 .pkt span{position:absolute;top:0;left:0;height:100%;width:25%;background:linear-gradient(90deg,var(--blue),var(--grn));border-radius:3px;transition:left .5s ease} #mtp-llmd1 .exp{background:var(--card)!important;border-left:3px solid var(--blue)!important;border-radius:8px;padding:12px 14px;font-size:14px;line-height:1.55;min-height:84px;color:var(--tx)!important} #mtp-llmd1 .ctl{display:flex;gap:8px;margin-top:12px;align-items:center} #mtp-llmd1 .btn{background:var(--blue)!important;color:#fff!important;border:0!important;border-radius:8px;padding:8px 14px;font:600 13px Inter,sans-serif;cursor:pointer} #mtp-llmd1 .btn.gh{background:transparent!important;border:1px solid var(--line)!important;color:var(--tx)!important} #mtp-llmd1 .ctl .st{margin-left:auto;font-size:12px;color:var(--mut)!important} #mtp-llmd1 .tiers{display:flex;gap:6px;margin-top:12px;flex-wrap:wrap} #mtp-llmd1 .tiers div{flex:1;min-width:110px;padding:8px;border-radius:8px;font-size:12px;text-align:center;border:1px solid var(--line);color:var(--tx)!important;opacity:.35;transition:opacity .4s} #mtp-llmd1 .tiers div.lit{opacity:1} #mtp-llmd1 .bars .row{margin-bottom:12px} #mtp-llmd1 .bars .lb{display:flex;justify-content:space-between;font-size:13px;margin-bottom:5px;color:var(--tx)!important} #mtp-llmd1 .bars .lb em{font-style:normal;color:var(--mut)!important;font-size:11px} #mtp-llmd1 .bars .tr{height:12px;background:var(--card);border-radius:6px;overflow:hidden} #mtp-llmd1 .bars .fl{height:100%;width:0;border-radius:6px;transition:width 1.1s cubic-bezier(.2,.8,.2,1)} #mtp-llmd1 .seg{display:flex;gap:6px;margin-bottom:14px} #mtp-llmd1 .note{font-size:11px;color:var(--mut)!important;margin-top:8px} #mtp-llmd1 .grid{display:grid;grid-template-columns:repeat(3,1fr);gap:10px} #mtp-llmd1 .card{background:var(--card)!important;border:1px solid var(--line)!important;border-radius:12px;padding:14px;cursor:pointer;transition:border-color .2s,transform .2s} #mtp-llmd1 .card:hover{border-color:var(--blue)!important;transform:translateY(-2px)} #mtp-llmd1 .card b{font-size:14px;color:var(--tx)!important;display:block} #mtp-llmd1 .card p{margin:6px 0 0!important;font-size:12.5px;line-height:1.5;color:var(--mut)!important;max-height:0;overflow:hidden;transition:max-height .35s} #mtp-llmd1 .card.open p{max-height:140px} #mtp-llmd1 .card .k{font-size:11px;color:var(--blue)!important} #mtp-llmd1 table{width:100%;border-collapse:collapse;font-size:13px} #mtp-llmd1 th,#mtp-llmd1 td{padding:9px 8px;border-bottom:1px solid var(--line)!important;text-align:left;color:var(--tx)!important;background:transparent!important;vertical-align:top} #mtp-llmd1 th{color:var(--mut)!important;font-weight:600;font-size:12px} #mtp-llmd1 th.us{color:var(--blue)!important} #mtp-llmd1 .ft{display:flex;justify-content:space-between;gap:10px;padding:12px 24px;border-top:1px solid var(--line);font-size:11px;color:var(--mut)!important;flex-wrap:wrap} #mtp-llmd1 .ft a{color:#76B900!important;text-decoration:none;font-weight:600} @media (max-width:640px){ #mtp-llmd1 .flow{grid-template-columns:repeat(2,1fr)} #mtp-llmd1 .grid{grid-template-columns:1fr} #mtp-llmd1 h3{font-size:20px!important} #mtp-llmd1 .pn{padding:16px} #mtp-llmd1 .tw{overflow-x:auto} #mtp-llmd1 table{min-width:520px} } <div id="mtp-llmd1"> <div class="hd"> <span class="tag"><span class="dots"><i style="background:#4285F4"></i><i style="background:#EA4335"></i><i style="background:#FBBC04"></i><i style="background:#34A853"></i></span>Google Cloud · Red Hat · IBM · Oct 20, 2026</span> <h3>How llm-d 1.0 serves agents at fleet scale</h3> <p class="sub">Open-source, Kubernetes-native distributed inference for vLLM and SGLang on TPUs and GPUs.</p> </div> <div class="tabs" role="tablist"> <button class="on" data-t="0">How it works</button> <button data-t="1">Production results</button> <button data-t="2">New in 1.0</button> <button data-t="3">vs Dynamo and AIBrix</button> </div> <div class="pn on"> <div class="flow"> <div class="node act"><span class="ic"></span><b>Agent request</b><small>1 endpoint</small></div> <div class="node"><span class="ic"></span><b>llm-d router</b><small>prefix + load aware</small></div> <div class="node"><span class="ic"></span><b>Prefill / decode</b><small>separate pools</small></div> <div class="node"><span class="ic"></span><b>Tiered KV cache</b><small>HBM to Lustre</small></div> </div> <div class="pkt"><span></span></div> <div class="exp" id="mtp-exp"></div> <div class="tiers" id="mtp-tiers"> <div>Accelerator HBM</div> <div>Host CPU RAM</div> <div>Local SSD</div> <div>Managed Lustre</div> </div> <div class="ctl"> <button class="btn gh" id="mtp-prev">Back</button> <button class="btn" id="mtp-next">Next step</button> <button class="btn gh" id="mtp-play">Auto-play</button> <span class="st" id="mtp-st">Step 1 of 4</span> </div> </div> <div class="pn"> <div class="seg"> <button class="btn" data-g="lat">Latency and cache</button> <button class="btn gh" data-g="thr">Throughput</button> </div> <div class="bars" id="mtp-bars"></div> <div class="note">All figures self-reported by Google, Snap and Zenteiq. Source: Google Cloud blog, Oct 20, 2026.</div> </div> <div class="pn"> <div class="grid"> <div class="card"><span class="k">Multi-tenant QoS</span><b>Flow control</b><p>Priority and fairness-aware routing keeps bursty agent loops from starving interactive users.</p></div> <div class="card"><span class="k">Upgrades</span><b>DisaggregatedSet</b><p>Built on LeaderWorkerSet. Rolls prefill and decode pools together on matching versions without dropping sessions.</p></div> <div class="card"><span class="k">Reliability</span><b>Router HA</b><p>Hot standby and rolling updates keep traffic flowing through upgrades and maintenance.</p></div> <div class="card"><span class="k">Agents</span><b>Session-aware routing</b><p>Turn-priority scheduling and native Responses API state help agents reuse context across turns.</p></div> <div class="card"><span class="k">Background jobs</span><b>Async processor</b><p>Pulls requests from Google Cloud Pub/Sub or Redis and runs them in slack capacity.</p></div> <div class="card"><span class="k">TPU</span><b>TorchTPU time-slicing</b><p>Interleaves inference into idle RL cycles. Duty cycle rises from about 40% to above 70%.</p></div> </div> <div class="note">Click a card to reveal details. Stable v1 APIs ship across all components.</div> </div> <div class="pn"><div class="tw"> <table> <tr><th></th><th class="us">llm-d 1.0</th><th>NVIDIA Dynamo</th><th>AIBrix</th></tr> <tr><td>Version</td><td>1.0 (Oct 2026)</td><td>v1.5.1 (Oct 2026)</td><td>v0.7.0 (Jun 2026)</td></tr> <tr><td>Governance</td><td>CNCF Sandbox</td><td>NVIDIA-led</td><td>vLLM project org</td></tr> <tr><td>Engines</td><td>vLLM, SGLang</td><td>SGLang, TRT-LLM, vLLM</td><td>vLLM, SGLang, TRT-LLM</td></tr> <tr><td>Accelerators</td><td>NVIDIA, AMD, Intel, TPU</td><td>NVIDIA GPUs</td><td>Mixed GPUs</td></tr> <tr><td>KV tiers</td><td>HBM, CPU, SSD, Lustre</td><td>GPU, CPU, SSD, remote</td><td>Distributed KV cache</td></tr> <tr><td>License</td><td>Apache 2.0</td><td>Apache 2.0</td><td>Apache 2.0</td></tr> <tr><td>GitHub stars</td><td>4.8k</td><td>8.3k</td><td>5.1k</td></tr> </table></div> <div class="note">Stars as of Oct 20, 2026. Sources: each project's GitHub repo and docs.</div> </div> <div class="ft"><span>Sources: Google Cloud, IBM Research, Red Hat, GitHub</span><a href=" target=" rel="noopener">© Marktechpost</a></div> </div> (function(){ var R=document.getElementById('mtp-llmd1'); var tabs=R.querySelectorAll('.tabs button'),pns=R.querySelectorAll('.pn'); tabs.forEach(function(b){b.onclick=function(){tabs.forEach(function(x){x.classList.remove('on')});pns.forEach(function(x){x.classList.remove('on')});b.classList.add('on');pns[+b.dataset.t].classList.add('on');if(+b.dataset.t===1)draw(cur);rs();};}); var steps=[ 'An agent sends one step of a multi-turn task to a single llm-d endpoint. TPU and GPU pools sit behind it, running vLLM or SGLang.', 'The router tracks which replicas already hold each cached prefix. It weighs load and predicted latency, then sends the step where its context lives.', 'Prefill (compute-heavy) and decode (memory-bound) run in separate pools. DisaggregatedSet upgrades both together without dropping sessions.', 'When accelerator memory fills, KV cache moves to host RAM, then SSD or Google Cloud Managed Lustre instead of being evicted.']; var s=0,nodes=R.querySelectorAll('.node'),pk=R.querySelector('.pkt span'),ex=document.getElementById('mtp-exp'),st=document.getElementById('mtp-st'),ti=R.querySelectorAll('#mtp-tiers div'),timer=null; function show(){nodes.forEach(function(n,i){n.classList.toggle('act',i===s)});pk.style.left=(s*25)+'%';ex.textContent=steps[s];st.textContent='Step '+(s+1)+' of 4'; ti.forEach(function(t,i){t.classList.remove('lit');if(s===3)setTimeout(function(){t.classList.add('lit')},i*350);else if(i===0)t.classList.add('lit')});} document.getElementById('mtp-next').onclick=function(){s=(s+1)%4;show()}; document.getElementById('mtp-prev').onclick=function(){s=(s+3)%4;show()}; document.getElementById('mtp-play').onclick=function(){var b=this;if(timer){clearInterval(timer);timer=null;b.textContent='Auto-play';return}b.textContent='Pause';timer=setInterval(function(){s=(s+1)%4;show()},2600)}; show(); var data={lat:[ ['Ironwood TPU: median TTFT cut','>85%',85,'#4285F4','agentic coding vs plain K8s Service'], ['Gemini: P95 TTFT cut (bursty chat)','>50%',50,'#34A853','GKE Inference Gateway'], ['A3 Mega 70B: TTFT improvement','~40%',40,'#FBBC04','Lustre + CPU offload, 50k-token prompts'], ['Gemini: TTFT cut (coding agents)','>35%',35,'#EA4335','production traffic'], ['Gemini: prefix cache hit rate','35% to 70%',70,'#4285F4','doubled']], thr:[ ['Zenteiq: aggregate throughput','>4x',4,'#34A853','1.47B to 5.97B tokens/day per 1,000 TPU v6e'], ['17,000-node GKE: throughput','2.9x',2.9,'#4285F4','3 clusters, 99.9% success'], ['Ironwood TPU: total throughput','2.4x',2.4,'#FBBC04','vs plain K8s Service']]}; var cur='lat'; function draw(g){cur=g;var box=document.getElementById('mtp-bars'),mx=g==='thr'?4:100;box.innerHTML=''; data[g].forEach(function(d){var r=document.createElement('div');r.className='row';r.innerHTML='<div class="lb"><span>'+d[0]+' <em>'+d[4]+'</em></span><b>'+d[1]+'</b></div><div class="tr"><div class="fl" style="background:&apos;+d[3]+&apos;"></div></div>';box.appendChild(r); var f=r.querySelector('.fl');setTimeout(function(){f.style.width=(d[2]/mx*100)+'%'},60);});rs();} R.querySelectorAll('[data-g]').forEach(function(b){b.onclick=function(){R.querySelectorAll('[data-g]').forEach(function(x){x.classList.add('gh')});b.classList.remove('gh');draw(b.dataset.g)}}); R.querySelectorAll('.card').forEach(function(c){c.onclick=function(){c.classList.toggle('open');setTimeout(rs,380)}}); function rs(){try{parent.postMessage({mtpH:R.offsetHeight+40,id:'mtp-llmd1'},'*')}catch(e){}} window.addEventListener('load',rs);window.addEventListener('resize',rs);setTimeout(rs,300); })(); " style="width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;" loading="lazy"> window.addEventListener('message',function(e){if(e.data&&e.data.id==='mtp-llmd1'&&e.data.mtpH){document.getElementById('mtp-llmd1-frame').style.height=e.data.mtpH+'px';}}); What do the llm-d 1.0 benchmarks show? All figures below are self-reported by Google and llm-d adopters. Ironwood TPU (TPU 7x): prefix-aware routing plus host memory offload cut median TTFT by more than 85%. Throughput rose 2.4x versus a plain Kubernetes Service. Gemini platform: prefix cache hit rate doubled from 35% to 70%. TTFT fell more than 35% for coding agents. P95 TTFT fell more than 50% under bursty chat. 17,000-node GKE benchmark: throughput scaled 2.9x across 3 clusters, with 99.9% success and under 1% overhead. A3 Mega, 70B model: Lustre plus CPU RAM offload improved TTFT by about 40% on 50k-token prompts. End-to-end latency fell 30% and GPU-hours fell nearly 60%. RL time-slicing: accelerator duty cycle rose from about 40% to above 70%. Adopters report similar patterns. Snap reports prefix cache hit rates of up to 75 to 80%. Zenteiq reports over 4x throughput, from 1.47 billion to 5.97 billion daily tokens per 1,000 TPU v6e chips. An IBM demo served 3,000 concurrent coding agents at 5 to 10 times below API pricing. How does llm-d compare with NVIDIA Dynamo and AIBrix? These are not strict rivals. CoreWeave runs llm-d alongside Dynamo, and NVIDIA contributes Dynamo components to llm-d. How do you deploy llm-d 1.0? On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service. It can also span clusters and regions with global failover. KEDA and HPA scale pools on queue depth and predicted latency. Kueue enforces per-model quotas. Verified Helm charts deploy TPU and GPU pools behind one endpoint. Google team suggests a staged path. Start with prefix-aware routing in front of an existing vLLM deployment. Then add flow control, tiered KV offload and disaggregation. Published runs can be compared on Prism. Key Takeaways llm-d 1.0 is the first stable release, with v1 APIs. New: flow control, router HA, DisaggregatedSet and agent-aware routing. Gemini doubled prefix cache hits from 35% to 70%. A 17,000-node GKE benchmark scaled throughput 2.9x. Check out the Google Cloud announcement, the IBM Research post, the GitHub repo and the GKE docs. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets appeared first on MarkTechPost.

AI

Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations

Elevenlabs' new Eleven Realtime speech-to-speech models combine speed and accuracy. The v1 Max variant scores 98.8 percent on the Big Bench Audio benchmark with a 0.57-second response time. Elevenlabs says no model on the Artificial Analysis leaderboard is both faster and more accurate. The article Elevenlabs releases two new speech-to-speech models designed for real-time AI conversations appeared first on The Decoder.

Advertisement
AI

Deepseek open-sources the inference engine that powers its fastest and cheapest AI model

Deepseek is open-sourcing DeepInfer, the inference framework behind its fast V4.1-Flash model. The vLLM-based system cuts compute costs and energy use when running AI models. The release adds more pricing pressure on Western providers like OpenAI and Anthropic. The article Deepseek open-sources the inference engine that powers its fastest and cheapest AI model appeared first on The Decoder.

AI

Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time

Mistral AI launched Voxtral 3, a new family of language models built for real-time voice agents. The system combines speech-to-text, text-to-speech, and a voice agent SDK, supports more than 50 languages, and lets agents tap into external tools during live conversations. It can run on your own infrastructure or through EU-hosted APIs to keep data under your control. The article Mistral AI launches Voxtral 3, a full voice agent stack that listens, thinks, and talks in real time appeared first on The Decoder.

AI

Sam Altman reportedly pitched three endgames for AI safety, one involving a single firm melting everyone else’s GPUs

OpenAI CEO Sam Altman laid out three paths to safe AI in an internal presentation: an international watchdog modeled after the IAEA, a merger of the major labs into one entity, or a single company so powerful it could shut down everyone else's compute. He called the last option "terrible." The article Sam Altman reportedly pitched three endgames for AI safety, one involving a single firm melting everyone else's GPUs appeared first on The Decoder.

Advertisement
AI

OpenAI says GPT-6 is much better with legacy programming languages such as COBOL and Fortran

OpenAI improved its latest models for low-resource programming languages like COBOL, Fortran, and Visual Basic. On the new MultiLangBench, GPT-6 Astra hits an 87.4 percent test pass rate across 20 languages, up from about 38 percent for GPT-5.6 Terra. For COBOL migration specifically, GPT-6.1 Sol reaches up to 97.9 percent accuracy, a result that could matter for modernizing critical financial infrastructure. The article OpenAI says GPT-6 is much better with legacy programming languages such as COBOL and Fortran appeared first on The Decoder.