NCM vs LLMs: How Next-Concept Modeling Writes Text Differently

Editorial disclosure: Recribel is an independent, reader-supported tech publication. If you buy software or services through links on this site, we may earn an affiliate commission at no extra cost to you.

Next-Concept Modeling: writes the sketch first. Illustration of text growing from a few short lines to a full paragraph
SpotlightAbout a 6 minute readSource: Hazy Research, Oct 12, 2026

Executive briefing

  • Most LLMs write one token at a time, with each token depending on the ones before it.
  • Next-Concept Modeling (NCM), from Stanford’s Hazy Research lab and Together AI, writes a rough outline first and adds detail in a few parallel steps.
  • The authors report 10 to 31 times faster generation than strong diffusion language models, with similar or better quality and 4 to 5 times fewer training tokens.
  • It is early research. The comparisons are against diffusion models, and the largest model tested has 725M parameters.
10–31×faster than strong diffusion language models
4–5×fewer training tokens than those baselines
86.1%of hard 30-clue Sudoku boards solved
725Mparameters in the largest model tested
Standard LLMDiffusion LMNCM
How it writesOne token at a timeRefines the whole sequence over many stepsOne level at a time, all codes in a level in parallel
Main costMany sequential stepsMany steps, full sequence each timeA handful of steps, with short early levels
Good atGeneral text at frontier scaleParallel refinement, global planningSpeed and data efficiency in the tested tasks
MaturityFrontier scaleResearchEarly research
[ DISPATCH 01 // THE PROBLEM ]

Typing one word at a time has a cost

Autoregressive models write like someone typing a sentence: pick the next word, then the next. It works remarkably well, but long outputs mean many sequential steps, and running these models has become a dominant computing workload.

Diffusion language models are a leading alternative. They refine many tokens at once, which can speed things up and may help with tasks that need planning. The catch is that they still need many sampling steps, process the full sequence at every step, and often need many passes over the training data.

Autoregressive one token per step, 12 steps for 12 tokens

NCM one level per step, 4 steps in this picture

level 1level 2level 3level 4

Illustration of the idea only, not real model output. The real hierarchy in the team’s translation demo has 11 levels.

[ DISPATCH 02 // THE IDEA ]

Sketch first, then add detail

NCM starts from a different question: can quality text be generated in just a handful of parallel steps? Its answer is to represent text as a hierarchy. The first levels are short and hold a few discrete codes that capture the broad structure. Later levels are longer and add finer detail.

At generation time the model begins with a high-level idea of what it wants to say, then progressively adds detail. The first decoded level is barely readable by itself, which is the point: it only needs to carry the outline.

Level 1: a rough ideaLevel 2: structure appearsLevel 3: details sharpenFinal level: clean text

However, there is concern that small and medium-sized enterprises will remain vulnerable to hacking and surveillance.

The sentence is the final decoded output from the team’s German-to-English example (scored at BLEU 77.8). The blur is only a visual metaphor for refinement, not what the model literally outputs at each level.

few codes more codes longer still full detail each level is longer than the one above it
Why shorter early levels matter: early steps work on only a few codes, so they are cheap. That is where much of the speed comes from.
[ DISPATCH 03 // HOW IT WORKS ]

Two training stages

Stage 1 · TokenizerLearns to turn text into a coarse-to-fine hierarchy of discrete codes, and back.
→
Stage 2 · GeneratorWith the tokenizer frozen, a transformer learns to predict each level from the earlier ones.
→
InferenceGenerates one level at a time, all codes in a level in parallel, then the decoder writes the text.
Stage 1 in more detail: three changes for text

The tokenizer builds on VAR, an approach to images, and makes several changes for text:

A codebook for every level. Each level has its own set of discrete codes.

Nested dropout. This pushes earlier levels to capture meaningful information without relying on later ones, so the coarse levels really are a usable outline.

A token-level reconstruction objective. This encourages accurate recovery of the original text.

Stage 2 in more detail: training and inference

The transformer predicts each level from the combined reconstruction of the levels before it. All levels are trained in a single forward pass. A fraction of the conditioning codes is deliberately corrupted during training, which makes the model more robust to its own mistakes at generation time.

At inference it produces one level at a time, predicting every code within the level in parallel. The tokenizer’s decoder then converts the finished hierarchy into text.

[ DISPATCH 04 // THE NUMBERS ]

What the authors report

The team tested language modeling (TinyStories, LM1B and OpenWebText), translation (WMT14 German to English), extreme summarization (XSum) and Sudoku. Speeds were measured as wall-clock time on a single NVIDIA H100, because counting function evaluations alone hides differences in the cost of each one.

Speed-up over the strongest diffusion baseline (baseline = 1×)

Diffusion baseline
NCM · OpenWebText
10×
NCM · LM1B
21×
NCM · XSum
31×

Completely solved Sudoku boards (%)

Easy · 40 clues
AR
14.6
MDLM
92.0
DUO
96.3
FMLM+
97.7
NCM
99.5
Medium · 35 clues
AR
5.1
MDLM
77.1
DUO
84.7
FMLM+
92.8
NCM
96.5
Hard · 30 clues
AR
1.0
MDLM
30.2
DUO
58.4
FMLM+
81.3
NCM
86.1

Sudoku numbers are read from the chart in the Hazy Research post. The authors say NCM degrades the least as puzzles get harder, and that it reaches this with fewer function evaluations (12 for the hard boards). Baseline names are as labeled in the chart, so please check them against the paper.

Fewer function evaluationsOn TinyStories and LM1B, NCM matched or beat the baselines’ quality with about 11× fewer function evaluations.
Less training dataAbout 4.9× fewer tokens on LM1B and 4× fewer on OpenWebText, counting both tokenizer and generator training.
ScalingEarly results across 97M, 317M and 725M parameters show perplexity improving as models grow.
Test-time scalingBecause sampling is fast, NCM can make more candidates in the same time. Choosing among them with minimum Bayes risk (MBR) decoding improved BLEU on WMT14 and ROUGE on XSum.
Why Sudoku?The authors suspect coarse-to-fine generation helps with global planning, since the overall structure is set before details are filled in. Sudoku needs many constraints to hold at once.
Read this before comparing to LLMs. The comparisons are mostly against diffusion language models, not today’s large autoregressive LLMs. No comparison to serving tricks like speculative decoding is reported, so “NCM vs LLM” is a fair question but not yet a settled one.
[ DISPATCH 05 // DON’T CONFUSE THE NAMES ]

Three “concept” projects, three different ideas

NCM (Stanford + Together AI)Learned discrete codes at different levels of detail. Text is generated level by level, in parallel within each level.
Large Concept Models (Meta)Treat each sentence as a concept and predict the next sentence in an existing sentence-embedding space (SONAR), December 2024.
ConceptLMAdds a Next Concept Prediction objective in a discrete latent space on top of ordinary next-token prediction. It is a separate project with a similar name.
[ DISPATCH 06 // WHAT’S NEXT ]

What the authors want to try

Scale it upHow does NCM behave with more parameters, compute and training tokens?
Train jointlyLearning the tokenizer and generator together could give representations better suited to generation.
Adapt to difficultyInstead of a fixed hierarchy for every example, the model could learn when to stop or spend more computation.
Want to try it? The code is on GitHub under an MIT license (HazyResearch/ncm), and the weights and tokenized datasets are on Hugging Face. Each released model is a frozen tokenizer plus a next-concept generator.

Quick glossary

Diffusion language model (DLM)

A model that generates text by repeatedly refining many tokens at once, instead of writing left to right.

Function evaluation (NFE)

The number of times the model is run to produce one output. Fewer is usually faster, but each run can cost different amounts, which is why wall-clock time is also reported.

Generative perplexity

A score for how plausible generated text is according to a reference model. Lower is better. The authors note that it does not fully capture generation quality, so they add other tasks.

BLEU and ROUGE-L

Scores that compare generated text with a reference, used here for translation (BLEU) and summarization (ROUGE-L). Higher is better.

MBR decoding

Generate several candidates, then pick the one that agrees most with the others. It trades extra compute for quality, which suits a fast sampler.

Common questions

Is NCM faster than ChatGPT-style models?

That has not been shown. The reported speed-ups are against diffusion language models on a single H100, not against large autoregressive chat models.

Will it replace LLMs?

Too early to say. The results are promising, but the largest model tested has 725M parameters, and the authors list further scaling as an open question.

Who made it?

Hermann Kumbong and Ishan S. Khare (equal contribution), with Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, and advisors Simran Arora, Azalia Mirhoseini and Christopher Ré, from Stanford’s Hazy Research lab and Together AI.

Bottom line

NCM is a promising sign that token-by-token generation is not the only workable design, and its speed and data-efficiency results against diffusion models are strong. Whether it holds up at frontier scale is the open question, and the authors list further scaling as their next step. It is worth watching, though it is not yet a replacement for the LLMs people use today.