NCM vs LLMs: How Next-Concept Modeling Writes Text Differently
Editorial disclosure: Recribel is an independent, reader-supported tech publication. If you buy software or services through links on this site, we may earn an affiliate commission at no extra cost to you.
Executive briefing
- Most LLMs write one token at a time, with each token depending on the ones before it.
- Next-Concept Modeling (NCM), from Stanford’s Hazy Research lab and Together AI, writes a rough outline first and adds detail in a few parallel steps.
- The authors report 10 to 31 times faster generation than strong diffusion language models, with similar or better quality and 4 to 5 times fewer training tokens.
- It is early research. The comparisons are against diffusion models, and the largest model tested has 725M parameters.
| Standard LLM | Diffusion LM | NCM | |
|---|---|---|---|
| How it writes | One token at a time | Refines the whole sequence over many steps | One level at a time, all codes in a level in parallel |
| Main cost | Many sequential steps | Many steps, full sequence each time | A handful of steps, with short early levels |
| Good at | General text at frontier scale | Parallel refinement, global planning | Speed and data efficiency in the tested tasks |
| Maturity | Frontier scale | Research | Early research |
Typing one word at a time has a cost
Autoregressive models write like someone typing a sentence: pick the next word, then the next. It works remarkably well, but long outputs mean many sequential steps, and running these models has become a dominant computing workload.
Diffusion language models are a leading alternative. They refine many tokens at once, which can speed things up and may help with tasks that need planning. The catch is that they still need many sampling steps, process the full sequence at every step, and often need many passes over the training data.
Autoregressive one token per step, 12 steps for 12 tokens
NCM one level per step, 4 steps in this picture
Illustration of the idea only, not real model output. The real hierarchy in the team’s translation demo has 11 levels.
Sketch first, then add detail
NCM starts from a different question: can quality text be generated in just a handful of parallel steps? Its answer is to represent text as a hierarchy. The first levels are short and hold a few discrete codes that capture the broad structure. Later levels are longer and add finer detail.
At generation time the model begins with a high-level idea of what it wants to say, then progressively adds detail. The first decoded level is barely readable by itself, which is the point: it only needs to carry the outline.
However, there is concern that small and medium-sized enterprises will remain vulnerable to hacking and surveillance.
The sentence is the final decoded output from the team’s German-to-English example (scored at BLEU 77.8). The blur is only a visual metaphor for refinement, not what the model literally outputs at each level.
Two training stages
Stage 1 in more detail: three changes for text
The tokenizer builds on VAR, an approach to images, and makes several changes for text:
A codebook for every level. Each level has its own set of discrete codes.
Nested dropout. This pushes earlier levels to capture meaningful information without relying on later ones, so the coarse levels really are a usable outline.
A token-level reconstruction objective. This encourages accurate recovery of the original text.
Stage 2 in more detail: training and inference
The transformer predicts each level from the combined reconstruction of the levels before it. All levels are trained in a single forward pass. A fraction of the conditioning codes is deliberately corrupted during training, which makes the model more robust to its own mistakes at generation time.
At inference it produces one level at a time, predicting every code within the level in parallel. The tokenizer’s decoder then converts the finished hierarchy into text.
What the authors report
The team tested language modeling (TinyStories, LM1B and OpenWebText), translation (WMT14 German to English), extreme summarization (XSum) and Sudoku. Speeds were measured as wall-clock time on a single NVIDIA H100, because counting function evaluations alone hides differences in the cost of each one.
Sudoku numbers are read from the chart in the Hazy Research post. The authors say NCM degrades the least as puzzles get harder, and that it reaches this with fewer function evaluations (12 for the hard boards). Baseline names are as labeled in the chart, so please check them against the paper.
Three “concept” projects, three different ideas
What the authors want to try
Quick glossary
Diffusion language model (DLM)
A model that generates text by repeatedly refining many tokens at once, instead of writing left to right.
Function evaluation (NFE)
The number of times the model is run to produce one output. Fewer is usually faster, but each run can cost different amounts, which is why wall-clock time is also reported.
Generative perplexity
A score for how plausible generated text is according to a reference model. Lower is better. The authors note that it does not fully capture generation quality, so they add other tasks.
BLEU and ROUGE-L
Scores that compare generated text with a reference, used here for translation (BLEU) and summarization (ROUGE-L). Higher is better.
MBR decoding
Generate several candidates, then pick the one that agrees most with the others. It trades extra compute for quality, which suits a fast sampler.
Common questions
Is NCM faster than ChatGPT-style models?
That has not been shown. The reported speed-ups are against diffusion language models on a single H100, not against large autoregressive chat models.
Will it replace LLMs?
Too early to say. The results are promising, but the largest model tested has 725M parameters, and the authors list further scaling as an open question.
Who made it?
Hermann Kumbong and Ishan S. Khare (equal contribution), with Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, and advisors Simran Arora, Azalia Mirhoseini and Christopher Ré, from Stanford’s Hazy Research lab and Together AI.
Bottom line
NCM is a promising sign that token-by-token generation is not the only workable design, and its speed and data-efficiency results against diffusion models are strong. Whether it holds up at frontier scale is the open question, and the authors list further scaling as their next step. It is worth watching, though it is not yet a replacement for the LLMs people use today.