Meta FAIR Self-Improving Memory Hits 56.0 on Continual Learning Bench
AIvia MarkTechPost
Quick Read
Meta FAIR has introduced Self-Improving Memory, a method for AI agents that manage their own memory. The model first discovers its own memory instructions. It then trains those skills into its weights. Co-authors include researchers from the University of Washington and Johns Hopkins University. TL;DR Method: The base is Kimi-K2.6, a 1T-parameter MoE with 32B active and 256K context. Distillation adds a LoRA adapter (rank 64) on attention projections. Runs on: Training hardware is not disclosed. Kimi-K2.6 serving needs a multi-GPU node; Vast.ai recommends 8x H200. Native INT4 weights exist. Performance: Self-found memory prompts average 56.0 on 3 tasks, versus 44.6 for the best human-written ones. Best: Codebase, 64.3 with optimized prompts, +20.6 over the best hand-written harness. Worst: Spectrum distillation adds only 2.9 points (46.5 vs 43.6 for SFT on generic runs). What is Self-Improving Memory? Self-Improving Memory is a 2-stage method from Meta FAIR that teaches an agent model to manage file-based memory. The agent works through a stream of subtasks. Each subtask starts with an empty context. Only a persistent memory directory, edited with file tools, carries over. Two prompts control that memory. OFFLOAD updates memory after a subtask. LOAD retrieves notes before the next one. Generic versions leave the strategy to the model. A pilot study in the research post found that no single hand-written strategy wins every task. How does harness optimization work? In stage 1, a meta agent rewrites the LOAD and OFFLOAD prompts. The search starts from 4 human-written seeds: Summary, Retrieval, Skill and Hypothesis. Each candidate runs the full task stream 5 times. The top 4 by mean score become parents for revision. For each parent, the meta agent reads trajectories, scores and final memory directories. It then proposes revised prompts. Selection uses a holdout set the meta agent cannot see. The discovered prompts are concrete. One Database prompt says to read GUARDRAILS.md, then SCHEMA.md, before answering. The best Codebase design uses a quickstart file, an indexed fix registry and separate fix files. How does harness distillation move the gains into weights? Stage 2 follows a Discover-Demonstrate-Distill pipeline. The task agent runs train subtasks with the optimized prompts. Every model call is logged. The research team then swaps the optimized prompts back to the generic ones in that history. The swap works because the behavior stays valid under generic instructions. Examples include keeping QUICKSTART.md, REGISTRY.md and SCHEMA.md files, and verifying join keys before aggregation. The model is fine-tuned with LoRA, rank 64, on attention projection layers. The loss covers only assistant tokens, including reasoning and tool calls. Each task gets equal weight. The meta agent and task agent are the same Kimi-K2.6 model. So the model is trained on memory skills it discovered itself. What do the benchmarks show? The tests use 3 Continual Learning Bench tasks: Database (SQL), Codebase (repo fixes) and Spectrum (radio scans). Test subtasks follow a designed phase shift. So results measure generalization to new instances of the same tasks. Source: Figures 4 and 5, Self-Improving Memory blog. Kimi-K2.6 for all rows. Optimized harnesses beat the best hand-written seed on all 3 tasks. Codebase gained the most, at 20.6 points. Search was noisy: 37 of 44 generated revisions failed to beat the previous best. Database and Spectrum peaked in the first generation. Codebase kept improving through generation 4. After distillation, every model runs the generic harness. The distilled model averages 43.1 across 10 matched seeds. SFT on generic trajectories reaches 37.6. That is a 5.5-point gain, or 14.6%. #mtp-sim-embed{background:transparent!important;border:0!important} #mtp-sim-embed iframe{width:100%!important;border:0!important;display:block!important;background:transparent!important} #mtp-sim-embed hr,#mtp-sim-embed p:empty,#mtp-sim-embed del,#mtp-sim-embed s{display:none!important} @media (max-width:640px){#mtp-sim-embed{margin:16px auto!important}} (function(){var f=document.getElementById('mtp-sim-frame');window.addEventListener('message',function(e){if(e.source===f.contentWindow&&e.data&&e.data.mtpSimH){f.style.height=e.data.mtpSimH+'px'}});})(); How does it compare with ALMA, MemEvolve and Harness-Zero? Several recent methods search over agent memory or harnesses. Self-Improving Memory searches memory prompts, then distills them into weights. Benchmarks differ, so headline numbers are not directly comparable. Harness-Zero is the closest match, but it uses a separate agent to correct the student. Self-Improving Memory skips that step. Key Takeaways Kimi-K2.6 writes memory prompts that beat human ones: 56.0 vs 44.6. Codebase sees the biggest jump: 64.3, up 20.6 points. Distillation keeps part of the gain: 43.1 vs 37.6 with generic prompts. 37 of 44 meta-agent revisions failed, so evaluation and selection matter. Check out the blog post, the X thread and the Continual Learning Bench paper. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meta FAIR Self-Improving Memory Hits 56.0 on Continual Learning Bench appeared first on MarkTechPost.
Meta FAIR has introduced Self-Improving Memory, a method for AI agents that manage their own memory. The model first discovers its own memory instructions. It then trains those skills into its weights. Co-authors include researchers from the University of Washington and Johns Hopkins University.
TL;DR
Method: The base is Kimi-K2.6, a 1T-parameter MoE with 32B active and 256K context. Distillation adds a LoRA adapter (rank 64) on attention projections.
Runs on: Training hardware is not disclosed. Kimi-K2.6 serving needs a multi-GPU node; Vast.ai recommends 8x H200. Native INT4 weights exist.
Performance: Self-found memory prompts average 56.0 on 3 tasks, versus 44.6 for the best human-written ones.
Best: Codebase, 64.3 with optimized prompts, +20.6 over the best hand-written harness.
Worst: Spectrum distillation adds only 2.9 points (46.5 vs 43.6 for SFT on generic runs).
What is Self-Improving Memory?
Self-Improving Memory is a 2-stage method from Meta FAIR that teaches an agent model to manage file-based memory. The agent works through a stream of subtasks. Each subtask starts with an empty context. Only a persistent memory directory, edited with file tools, carries over.
Two prompts control that memory. OFFLOAD updates memory after a subtask. LOAD retrieves notes before the next one. Generic versions leave the strategy to the model. A pilot study in the research post found that no single hand-written strategy wins every task.
How does harness optimization work?
In stage 1, a meta agent rewrites the LOAD and OFFLOAD prompts. The search starts from 4 human-written seeds: Summary, Retrieval, Skill and Hypothesis. Each candidate runs the full task stream 5 times. The top 4 by mean score become parents for revision.
For each parent, the meta agent reads trajectories, scores and final memory directories. It then proposes revised prompts. Selection uses a holdout set the meta agent cannot see.
The discovered prompts are concrete. One Database prompt says to read GUARDRAILS.md, then SCHEMA.md, before answering. The best Codebase design uses a quickstart file, an indexed fix registry and separate fix files.
How does harness distillation move the gains into weights?
Stage 2 follows a Discover-Demonstrate-Distill pipeline. The task agent runs train subtasks with the optimized prompts. Every model call is logged. The research team then swaps the optimized prompts back to the generic ones in that history.
The swap works because the behavior stays valid under generic instructions. Examples include keeping QUICKSTART.md, REGISTRY.md and SCHEMA.md files, and verifying join keys before aggregation. The model is fine-tuned with LoRA, rank 64, on attention projection layers. The loss covers only assistant tokens, including reasoning and tool calls. Each task gets equal weight.
The meta agent and task agent are the same Kimi-K2.6 model. So the model is trained on memory skills it discovered itself.
What do the benchmarks show?
The tests use 3 Continual Learning Bench tasks: Database (SQL), Codebase (repo fixes) and Spectrum (radio scans). Test subtasks follow a designed phase shift. So results measure generalization to new instances of the same tasks.
Source: Figures 4 and 5, Self-Improving Memory blog. Kimi-K2.6 for all rows.
Optimized harnesses beat the best hand-written seed on all 3 tasks. Codebase gained the most, at 20.6 points. Search was noisy: 37 of 44 generated revisions failed to beat the previous best. Database and Spectrum peaked in the first generation. Codebase kept improving through generation 4.
After distillation, every model runs the generic harness. The distilled model averages 43.1 across 10 matched seeds. SFT on generic trajectories reaches 37.6. That is a 5.5-point gain, or 14.6%.
How does it compare with ALMA, MemEvolve and Harness-Zero?
Several recent methods search over agent memory or harnesses. Self-Improving Memory searches memory prompts, then distills them into weights. Benchmarks differ, so headline numbers are not directly comparable.
Harness-Zero is the closest match, but it uses a separate agent to correct the student. Self-Improving Memory skips that step.
Key Takeaways
Kimi-K2.6 writes memory prompts that beat human ones: 56.0 vs 44.6.
Codebase sees the biggest jump: 64.3, up 20.6 points.
Distillation keeps part of the gain: 43.1 vs 37.6 with generic prompts.
37 of 44 meta-agent revisions failed, so evaluation and selection matter.
Check out the blog post, the X thread and the Continual Learning Bench paper. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.
Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Meta FAIR Self-Improving Memory Hits 56.0 on Continual Learning Bench appeared first on MarkTechPost.
NVIDIA has released Nemotron-Labs-Sigma, a collection of 14 open checkpoints at 3B and 8B scale. Nemotron-Labs-Sigma is a continuous diffusion language model family that denoises text in a 16-D embedding space. Most diffusion LLMs, such as LLaDA and Dream, work on discrete tokens. Sigma works in a smooth latent space instead. NVIDIA team states that this makes few-step distillation and trajectory steering practical for text. TL;DR Size: 3B and 8B parameters, 14 checkpoints. Context is 4,096 tokens (8,192 for Sigma-8B-instruct). Runs on: 1 NVIDIA GPU in bfloat16 (Ampere, Lovelace, Hopper, Blackwell). About 120 to 140 tokens/s on 1 H100. No CPU support, no quantized variants listed. Performance: Sigma-8B base averages 71.16 on NVIDIA’s 6-task suite, ahead of NLD-8B (70.22). Best: 91.96 on GSM8K (Sigma-8B-instruct, 256 DDPM steps). Worst: 17.00 on AIME25 (same model and sampler). Bottom line: Best: the 8-step distilled 8B keeps a 70.34 average. Worst: the instruct average (59.14) trails NLD-8B (59.92). What is Nemotron-Labs-Sigma? The research team calls Sigma the first large-scale family of continuous diffusion language models. The collection has 4 variant types at each size: Base: Sigma-3B and Sigma-8B. Instruct: Sigma-3B-instruct and Sigma-8B-instruct. Few-step distilled: distill-3B and distill-8B at NFE 2, 4, 8 and 16 (8 checkpoints). Self-speculative: distill-spec-3B and distill-spec-8B. How does continuous diffusion for language work? Each token maps to a point in a learned 16-D embedding space. Training adds Gaussian noise to these embeddings and teaches the model to remove it. The embedding geometry is learned jointly with the denoiser. Generation runs blockwise. Each denoised block is committed to a KV cache before the next starts. Blocks default to 64 tokens. The backbone is a Ministral-style decoder-only Transformer with grouped-query attention, SwiGLU and rotary embeddings. Training starts from an autoregressive warm start of about 1T tokens. About 0.3T tokens of diffusion training follow. At inference, classifier-free guidance and score temperature matter most. The model card page recommends guidance 3 and temperature 0.3 at 32 DDIM steps. Why does a continuous space matter? Discrete diffusion jumps between token states, so its trajectories are hard to steer. Sigma follows smooth ODE/SDE paths instead. Per the research paper, these degrade gracefully at low step counts and distill efficiently. Embedding-space steering also tunes the quality-diversity trade-off for pass@k sampling. How does Sigma perform on benchmarks? On the research’s 6-task base suite, Sigma-8B with 256 DDPM steps averages 71.16. NLD-8B in diffusion mode scores 70.22, Dream-7B 58.74 and LLaDA-8B 41.78. Qwen3-8B in AR mode scores 68.91. A cheaper 32-step DDIM sampler still gives Sigma-8B a 70.18 average. Coding is the standout. Sigma-8B scores 65.12 on HumanEval versus 62.80 for NLD-8B. MBPP is weaker, at 65.74 versus 67.00. After SFT, Sigma-8B-instruct scores 91.96 on GSM8K, 78.40 on MATH-500 and 82.50 on HumanEval. Hard reasoning stays limited, at 23.33 on AIME24 and 17.00 on AIME25. Its 10-task average of 59.14 sits just below NLD-8B at 59.92. All scores are NVIDIA’s own evaluations. <iframe id="mtp-sigma-frame" title="Nemotron-Labs-Sigma interactive explainer" style="width:100%;height:600px;border:0;display:block;background:transparent;" loading="lazy" srcdoc=" <title>Nemotron-Labs-Sigma Explainer | Marktechpost</title> #mtp-sigma{--g:#76B900;--g2:#a4e000;--bg:#0f0f0f;--card:#171717;--line:#2a2a2a;--tx:#e8e8e8;--mut:#9a9a9a;max-width:880px;margin:0 auto;background:var(--bg)!important;color:var(--tx)!important;font-family:Inter,system-ui,-apple-system,Segoe UI,Roboto,sans-serif;border:1px solid var(--line)!important;border-radius:14px;overflow:hidden;box-sizing:border-box} #mtp-sigma *{box-sizing:border-box} body{margin:0;background:transparent} #mtp-sigma .hd{padding:20px 22px 12px;border-bottom:1px solid var(--line)} #mtp-sigma .tag{display:inline-block;font-size:11px;letter-spacing:.08em;text-transform:uppercase;color:#111!important;background:var(--g)!important;padding:3px 8px;border-radius:4px;font-weight:700} #mtp-sigma h2{margin:10px 0 4px;font-size:22px;line-height:1.25;color:#fff!important} #mtp-sigma .sub{margin:0;color:var(--mut)!important;font-size:14px} #mtp-sigma .tabs{display:flex;gap:4px;padding:10px 14px 0;flex-wrap:wrap;border-bottom:1px solid var(--line)} #mtp-sigma .tab{background:transparent!important;color:var(--mut)!important;border:0!important;border-bottom:2px solid transparent!important;padding:9px 12px;font:600 13px Inter,system-ui,sans-serif;cursor:pointer;transition:color .2s} #mtp-sigma .tab:hover{color:#fff!important} #mtp-sigma .tab.on{color:var(--g)!important;border-bottom-color:var(--g)!important} #mtp-sigma .pn{display:none;padding:18px 22px 20px;animation:fi .35s ease} #mtp-sigma .pn.on{display:block} @keyframes fi{from{opacity:0;transform:translateY(6px)}to{opacity:1;transform:none}} #mtp-sigma .steps{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin-bottom:14px} #mtp-sigma .st{background:var(--card)!important;border:1px solid var(--line)!important;border-radius:10px;padding:10px;cursor:pointer;transition:all .25s;color:var(--tx)!important} #mtp-sigma .st b{display:block;color:var(--g)!important;font-size:12px;margin-bottom:3px} #mtp-sigma .st span{font-size:12.5px;line-height:1.35} #mtp-sigma .st.on{border-color:var(--g)!important;box-shadow:0 0 0 1px var(--g),0 0 18px rgba(118,185,0,.25)} #mtp-sigma .stage{background:var(--card)!important;border:1px solid var(--line)!important;border-radius:10px;padding:12px;display:grid;grid-template-columns:260px 1fr;gap:16px;align-items:center} #mtp-sigma svg{display:block;width:100%;height:auto} #mtp-sigma .desc h3{margin:0 0 6px;font-size:16px;color:#fff!important} #mtp-sigma .desc p{margin:0 0 8px;font-size:13.5px;line-height:1.5;color:var(--tx)!important} #mtp-sigma .ctl{display:flex;gap:8px;margin-top:12px;flex-wrap:wrap} #mtp-sigma button.b{background:#1f1f1f!important;color:#fff!important;border:1px solid #3a3a3a!important;border-radius:8px;padding:7px 12px;font:600 12.5px Inter,system-ui,sans-serif;cursor:pointer} #mtp-sigma button.b:hover,#mtp-sigma button.b.on{border-color:var(--g)!important;color:var(--g)!important} #mtp-sigma .bars{display:flex;flex-direction:column;gap:9px;margin-top:12px} #mtp-sigma .row{display:grid;grid-template-columns:120px 1fr 56px;gap:10px;align-items:center;font-size:13px} #mtp-sigma .tr{background:#222!important;height:18px;border-radius:5px;overflow:hidden} #mtp-sigma .fl{height:100%;width:0;background:#555!important;border-radius:5px;transition:width .9s cubic-bezier(.2,.8,.2,1)} #mtp-sigma .fl.s{background:linear-gradient(90deg,#4f7c00,var(--g))!important;box-shadow:0 0 12px rgba(118,185,0,.45)} #mtp-sigma .v{text-align:right;font-variant-numeric:tabular-nums;font-weight:700} #mtp-sigma .note{font-size:11.5px;color:var(--mut)!important;margin:12px 0 0;line-height:1.45} #mtp-sigma .grid2{display:grid;grid-template-columns:1fr 1fr;gap:14px} #mtp-sigma .box{background:var(--card)!important;border:1px solid var(--line)!important;border-radius:10px;padding:14px} #mtp-sigma .big{font-size:40px;font-weight:800;color:var(--g)!important;line-height:1} #mtp-sigma .lbl{font-size:12px;color:var(--mut)!important;margin-top:4px} #mtp-sigma .blk{display:flex;gap:3px;flex-wrap:wrap;margin-top:10px} #mtp-sigma .tk{width:16px;height:16px;border-radius:3px;background:#2a2a2a!important;transition:background .3s} #mtp-sigma .tk.lit{background:var(--g)!important} #mtp-sigma table{width:100%;border-collapse:collapse;font-size:12.5px;margin-top:6px} #mtp-sigma th,#mtp-sigma td{border-bottom:1px solid var(--line)!important;padding:6px 4px;text-align:left;color:var(--tx)!important;background:transparent!important} #mtp-sigma th{color:var(--mut)!important;font-weight:600} #mtp-sigma .chips{display:grid;grid-template-columns:repeat(2,1fr);gap:8px;margin-top:12px} #mtp-sigma a.chip{display:flex;justify-content:space-between;gap:8px;background:var(--card)!important;border:1px solid var(--line)!important;border-radius:8px;padding:9px 11px;color:var(--tx)!important;text-decoration:none!important;font-size:12.5px;transition:border-color .2s,transform .2s} #mtp-sigma a.chip:hover{border-color:var(--g)!important;transform:translateY(-1px)} #mtp-sigma a.chip em{font-style:normal;color:var(--g)!important;font-weight:700;white-space:nowrap} #mtp-sigma .ft{display:flex;justify-content:space-between;gap:10px;flex-wrap:wrap;padding:11px 22px;border-top:1px solid var(--line);font-size:11.5px;color:var(--mut)!important} #mtp-sigma .ft b{color:#76B900!important} #mtp-sigma .dot{transition:cx .9s ease,cy .9s ease,opacity .5s,fill .5s} #mtp-sigma hr,#mtp-sigma p:empty,#mtp-sigma del,#mtp-sigma s{display:none!important} @media(max-width:640px){ #mtp-sigma .steps{grid-template-columns:repeat(2,1fr)} #mtp-sigma .stage{grid-template-columns:1fr} #mtp-sigma .grid2{grid-template-columns:1fr} #mtp-sigma .chips{grid-template-columns:1fr} #mtp-sigma .row{grid-template-columns:92px 1fr 48px;font-size:12px} #mtp-sigma h2{font-size:19px} #mtp-sigma .tab{padding:8px 8px;font-size:12px} } <div id="mtp-sigma"> <div class="hd"><span class="tag">NVIDIA · Open weights</span> <h2>Nemotron-Labs-Sigma: continuous diffusion for language</h2> <p class="sub">14 checkpoints at 3B and 8B. Text is denoised in a learned 16-D embedding space.</p></div> <div class="tabs"> <button class="tab on" data-t="how">How it works</button> <button class="tab" data-t="bench">Benchmarks</button> <button class="tab" data-t="few">Few-step</button> <button class="tab" data-t="speed">Speculative speed</button> <button class="tab" data-t="ckpt">14 checkpoints</button> </div> <div class="pn on" id="p-how"> <div class="steps"> <div class="st on" data-s="0"><b>01</b><span>AR warm start</span></div> <div class="st" data-s="1"><b>02</b><span>Noise 16-D embeddings</span></div> <div class="st" data-s="2"><b>03</b><span>Denoise block by block</span></div> <div class="st" data-s="3"><b>04</b><span>Steer with CFG + temp</span></div> </div> <div class="stage"> illustrative 2-D view of a 16-D space <div class="desc" id="sg-desc"></div> </div> <div class="ctl"><button class="b" id="sg-prev"> Back</button><button class="b on" id="sg-next">Next step </button><button class="b" id="sg-play">Auto-play</button></div> </div> <div class="pn" id="p-bench"> <div class="ctl"> <button class="b on" data-m="avg">Base 6-task avg</button> <button class="b" data-m="gsm">GSM8K</button> <button class="b" data-m="he">HumanEval</button> <button class="b" data-m="mbpp">MBPP</button> <button class="b" data-m="ins">Instruct 10-task avg</button> </div> <div class="bars" id="bn-bars"></div> <p class="note" id="bn-note"></p> </div> <div class="pn" id="p-few"> <div class="grid2"> <div class="box"> <div class="lbl">Distilled checkpoint (native NFE, 32-token block)</div> <div class="ctl" id="nf-btns"><button class="b" data-n="2">nfe2</button><button class="b" data-n="4">nfe4</button><button class="b on" data-n="8">nfe8</button><button class="b" data-n="16">nfe16</button></div> <div style="display:flex;gap:22px;margin-top:14px"> <div><div class="big" id="nf-tpf">4</div><div class="lbl">tokens per forward</div></div> <div><div class="big" id="nf-n" style="color:#fff">8</div><div class="lbl">denoiser calls / block (+1 readout)</div></div> </div> <div class="blk" id="nf-blk"></div> <p class="note" id="nf-note"></p> </div> <div class="box"> <div class="lbl">Base model sampler settings (DDIM, 32-token blocks)</div> <table><tr><th>Steps</th><th>Guidance</th><th>Score temp</th></tr> <tr><td>32</td><td>3</td><td>0.3</td></tr><tr><td>16</td><td>3</td><td>0.4</td></tr><tr><td>8</td><td>3</td><td>0.6</td></tr><tr><td>4</td><td>2</td><td>0.8</td></tr></table> <p class="note">Base quality drops noticeably below about 16 steps. NVIDIA recommends the distilled checkpoints for few-step generation.</p> </div> </div> </div> <div class="pn" id="p-speed"> <p class="desc" style="margin:0;font-size:13.5px;line-height:1.5">A diffusion head drafts a 32-token block. The model's own AR head verifies it, so output exactly matches greedy AR decoding.</p> <div class="bars" id="sp-bars"></div> <div class="box" style="margin-top:14px"> <div class="lbl">Acceptance-length gain over NLD-8B base (Sigma-distill-spec-8B)</div> <div class="bars" id="al-bars"></div> </div> <p class="note">1 NVIDIA GB200, BF16, SGLang, 1 request at a time. Self-reported by NVIDIA (arXiv:2610.02665).</p> </div> <div class="pn" id="p-ckpt"> <div class="ctl"><button class="b on" data-f="all">All 14</button><button class="b" data-f="3B">3B</button><button class="b" data-f="8B">8B</button></div> <div class="chips" id="ck-list"></div> <p class="note">Base context 4,096 tokens (8B-instruct: 8,192). NVIDIA Nemotron Open Model License. CUDA GPU in bfloat16 required, no CPU.</p> </div> <div class="ft"><span>Sources: arXiv:2610.02665, Hugging Face model cards. Benchmarks self-reported by NVIDIA.</span><b>Built by Marktechpost</b></div> </div> (function(){ var R=document.getElementById('mtp-sigma'); function q(s){return R.querySelectorAll(s)} function rs(){try{parent.postMessage({mtpSigmaH:R.offsetHeight+40},'*')}catch(e){}} q('.tab').forEach(function(t){t.addEventListener('click',function(){q('.tab').forEach(function(x){x.classList.remove('on')});q('.pn').forEach(function(x){x.classList.remove('on')});t.classList.add('on');document.getElementById('p-'+t.dataset.t).classList.add('on');if(t.dataset.t==='bench')bench(cur);if(t.dataset.t==='speed')speed();setTimeout(rs,60);setTimeout(rs,600)})}); /* how it works */ var D=[ {h:'1. Autoregressive warm start',p:['Sigma starts from autoregressive weights trained on about 1T tokens.','The backbone is a Ministral-style decoder: grouped-query attention, SwiGLU, rotary embeddings.']}, {h:'2. Corrupt embeddings with Gaussian noise',p:['Each token maps to a point in a learned 16-D embedding space.','Training adds Gaussian noise and learns to remove it, about 0.3T tokens of diffusion training.']}, {h:'3. Denoise one block at a time',p:['Blocks default to 64 tokens. Finished blocks go into a KV cache.','Trajectories are smooth ODE/SDE paths, so they distill to few steps.']}, {h:'4. Steer at inference',p:['Classifier-free guidance and score temperature control quality and diversity.','Recommended: guidance 3, temperature 0.3 at 32 DDIM steps.']}]; var N=18,pts=[],S=0,timer=null; for(var i=0;i<N;i++){var cx=20+(i%6)*44,cy=30+Math.floor(i/6)*55;pts.push({cx:cx,cy:cy,nx:10+Math.random()*240,ny:10+Math.random()*150})} var g=document.getElementById('sg-dots'),ns=''; pts.forEach(function(p){var c=document.createElementNS(ns,'circle');c.setAttribute('r',6);c.setAttribute('class','dot');g.appendChild(c);p.el=c}); function draw(){ q('.st').forEach(function(x){x.classList.toggle('on',+x.dataset.s===S)}); var d=D[S];document.getElementById('sg-desc').innerHTML='<h3>'+d.h+'</h3><p>'+d.p.join('</p><p>')+'</p>'; pts.forEach(function(p,i){var x,y,f='#3a3a3a',o=1; if(S===0){x=p.cx;y=p.cy;f='#555'} else if(S===1){x=p.nx;y=p.ny;f='#8a8a8a'} else if(S===2){var done=i<6;x=done?p.cx:p.nx;y=done?p.cy:p.ny;f=done?'#76B900':'#8a8a8a'} else{x=p.cx+(i%2?3:-3);y=p.cy;f='#76B900'} p.el.setAttribute('cx',x);p.el.setAttribute('cy',y);p.el.setAttribute('fill',f);p.el.style.opacity=o}); setTimeout(rs,50)} q('.st').forEach(function(x){x.addEventListener('click',function(){S=+x.dataset.s;draw()})}); document.getElementById('sg-next').onclick=function(){S=(S+1)%4;draw()}; document.getElementById('sg-prev').onclick=function(){S=(S+3)%4;draw()}; document.getElementById('sg-play').onclick=function(){var b=this;if(timer){clearInterval(timer);timer=null;b.textContent='Auto-play';b.classList.remove('on')}else{timer=setInterval(function(){S=(S+1)%4;draw()},1800);b.textContent='Pause';b.classList.add('on')}}; draw(); /* benchmarks */ var M=['Sigma-8B','NLD-8B','Qwen3-8B (AR)','Dream-7B','LLaDA-8B']; var B={avg:[71.16,70.22,68.91,58.74,41.78],gsm:[88.57,87.26,86.73,77.18,70.96],he:[65.12,62.80,64.63,54.88,32.32],mbpp:[65.74,67.00,69.40,56.80,40.80],ins:[59.14,59.92,null,40.54,37.24]}; var NT={avg:'Base models, pass@1. Sigma-8B uses 256 DDPM steps. NLD, Dream and LLaDA in diffusion mode. Paper Table 2.',gsm:'GSM8K pass@1, base models. Paper Table 2.',he:'HumanEval pass@1, base models. Paper Table 2.',mbpp:'MBPP pass@1, base models. Sigma-8B trails NLD-8B and Qwen3-8B here. Paper Table 2.',ins:'Instruct models, 10-task average (GSM8K, MATH-500, AIME24/25, HumanEval, MBPP, LCB-CPP, MMLU, GPQA, IFEval). Diffusion mode. Qwen3-8B not in this row set. Paper Table 5.'}; var cur='avg'; function bench(k){cur=k;var box=document.getElementById('bn-bars');box.innerHTML=''; M.forEach(function(m,i){var v=B[k][i];if(v===null)return;var r=document.createElement('div');r.className='row';r.innerHTML='<span>'+m+'</span><div class="tr"><div class="fl'+(i===0?' s':'')+'"></div></div><span class="v">'+v.toFixed(2)+'</span>';box.appendChild(r);var f=r.querySelector('.fl');requestAnimationFrame(function(){requestAnimationFrame(function(){f.style.width=v+'%'})})}); document.getElementById('bn-note').textContent=NT[k]+' Self-reported by NVIDIA.'; q('#p-bench .b').forEach(function(b){b.classList.toggle('on',b.dataset.m===k)});setTimeout(rs,60)} q('#p-bench .b').forEach(function(b){b.addEventListener('click',function(){bench(b.dataset.m)})}); bench('avg'); /* few-step */ function few(n){document.getElementById('nf-tpf').textContent=32/n;document.getElementById('nf-n').textContent=n; var bk=document.getElementById('nf-blk');bk.innerHTML='';for(var i=0;i=n)return;for(var j=k*step;j8?90:220)})(); document.getElementById('nf-note').textContent=n===8?'Sigma-distill-8B-nfe8 card: 6-task mean 70.34, GSM8K 88.44, HumanEval 65.18, MBPP 62.60 (10 seeds).':'Tokens per forward = 32-token block / NFE. Per-checkpoint scores for nfe'+n+' are on its model card.'; q('#nf-btns .b').forEach(function(b){b.classList.toggle('on',+b.dataset.n===n)});setTimeout(rs,60)} q('#nf-btns .b').forEach(function(b){b.addEventListener('click',function(){few(+b.dataset.n)})}); few(8); /* speed */ function bars(id,rows,max,suf){var box=document.getElementById(id);box.innerHTML='';rows.forEach(function(r){var e=document.createElement('div');e.className='row';e.innerHTML='<span>'+r[0]+'</span><div class="tr"><div class="fl'+(r[2]?' s':'')+'"></div></div><span class="v">'+r[1].toFixed(r[3])+suf+'</span>';box.appendChild(e);var f=e.querySelector('.fl');requestAnimationFrame(function(){requestAnimationFrame(function(){f.style.width=(r[1]/max*100)+'%'})})})} function speed(){bars('sp-bars',[['Sigma-8B avg',3.46,1,2],['NLD-8B avg',3.04,0,2],['Sigma-8B peak',4.20,1,2],['Sigma-3B peak',3.18,1,2],['AR decoding',1.00,0,2]],4.5,'x'); bars('al-bars',[['GSM8K',33.0,1,1],['MBPP',22.7,1,1],['HumanEval',21.0,1,1]],40,'%');setTimeout(rs,60)} /* checkpoints */ var C=[['Nemotron-Labs-Sigma-3B','Base'],['Nemotron-Labs-Sigma-3B-instruct','Instruct'],['Nemotron-Labs-Sigma-8B','Base'],['Nemotron-Labs-Sigma-8B-instruct','Instruct'],['Nemotron-Labs-Sigma-distill-3B-nfe2','2 NFE'],['Nemotron-Labs-Sigma-distill-3B-nfe4','4 NFE'],['Nemotron-Labs-Sigma-distill-3B-nfe8','8 NFE'],['Nemotron-Labs-Sigma-distill-3B-nfe16','16 NFE'],['Nemotron-Labs-Sigma-distill-spec-3B','Self-spec'],['Nemotron-Labs-Sigma-distill-8B-nfe2','2 NFE'],['Nemotron-Labs-Sigma-distill-8B-nfe4','4 NFE'],['Nemotron-Labs-Sigma-distill-8B-nfe8','8 NFE'],['Nemotron-Labs-Sigma-distill-8B-nfe16','16 NFE'],['Nemotron-Labs-Sigma-distill-spec-8B','Self-spec']]; function ck(f){var l=document.getElementById('ck-list');l.innerHTML='';C.forEach(function(c){if(f!=='all'&&c[0].indexOf(f)<0)return;var a=document.createElement('a');a.className='chip';a.href=''+c[0];a.target='_blank';a.rel='noopener';a.innerHTML='<span>'+c[0].replace('Nemotron-Labs-','')+'</span><em>'+c[1]+'</em>';l.appendChild(a)}); q('#p-ckpt .b').forEach(function(b){b.classList.toggle('on',b.dataset.f===f)});setTimeout(rs,60)} q('#p-ckpt .b').forEach(function(b){b.addEventListener('click',function(){ck(b.dataset.f)})}); ck('all'); window.addEventListener('load',rs);window.addEventListener('resize',rs);setTimeout(rs,300); })(); "> window.addEventListener('message',function(ev){if(ev.data&&ev.data.mtpSigmaH){var f=document.getElementById('mtp-sigma-frame');if(f&&ev.source===f.contentWindow){f.style.height=ev.data.mtpSigmaH+'px';}}}); How fast are the distilled and speculative checkpoints? Sigma-distill-8B-nfe8 decodes each 32-token block in 8 denoiser calls plus 1 readout. Its card reports a 6-task mean of 70.34, with 88.44 on GSM8K and 65.18 on HumanEval. That is close to the 256-step teacher. NVIDIA’s launch thread frames the gain as 8 tokens per forward on math and 4 on code. The distill-spec checkpoints add self-speculative decoding. A diffusion head drafts a 32-token block, and the model’s own AR head verifies it. Output exactly matches greedy AR decoding. On 1 GB200 with SGLang, the 8B spec model runs 3.46x faster than AR decoding on average. NLD-8B reaches 3.04x on the same setup. Peak speedups reach 4.20x at 8B and 3.18x at 3B. How do you deploy Nemotron-Labs-Sigma? All 14 checkpoints are on Hugging Face under the NVIDIA Nemotron Open Model License. The cards state the models are ready for commercial use. You need a CUDA GPU in bfloat16: Ampere, Lovelace, Hopper or Blackwell. CPU inference is not supported. Loading needs PyTorch 2.4+, Transformers 4.55+ and trust_remote_code. On 1 H100, the base model generates about 120 to 140 tokens/s. The distilled card lists vLLM, SGLang and Docker Model Runner for serving. NVIDIA hosts no API. How does Sigma compare with other 8B diffusion and AR models? Benchmark rows use NVIDIA’s evaluations from the Sigma paper (Table 2 base, Table 5 instruct). Specs come from each model’s own card. Key Takeaways Sigma moves language diffusion into a learned 16-D continuous embedding space. 14 open checkpoints: 3B and 8B base, instruct, distilled and speculative. Sigma-8B base averages 71.16, ahead of NLD-8B at 70.22. Self-speculative 8B decodes 3.46x faster than AR on average. Weak spots: 17.00 on AIME25, 4,096-token base context, GPU only. Check out the Paper, Model Collection and Project Page. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model with 14 checkpoints at 3B/8B appeared first on MarkTechPost.
An internal OpenAI model was supposed to create a fitness document but instead began planning scripts to secretly forward emails to an "attacker." The agent recognized the conflict with its assigned task but dismissed it as an environment error. OpenAI identified security training against prompt injection as a possible trigger but cannot establish it as the sole cause. The article An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool appeared first on The Decoder.
Token prices for AI models fell 21 percent in six weeks while usage volume hit a record high, according to the Ramp AI Index. Anthropic dominates with nearly 59 percent of token spending. Data from Ramp, Stripe and OpenRouter independently show the same pattern: the AI industry is still growing, but revenue momentum is cooling. The article AI industry growth remains strong, but the pace is slowing appeared first on The Decoder.