Thinking Without Words

Jun 13, 2025

Reinforcement learning can teach language models to spend more tokens on problems with verifiable answers. DeepSeek-R1, for example, reports improvements on math and code tasks alongside longer reasoning traces. Those extra tokens provide more computation, but generating them takes time.

Continuous latent reasoning keeps intermediate computation in hidden-state vectors, avoiding a vocabulary choice at every reasoning step. The computation still costs time, and removing the text trace makes the intermediate states harder to inspect. The question is whether a model can use those states well enough to justify the cost.

Illustration of the core idea of latent reasoning, where a model's internal reasoning process is represented as a high-dimensional vector space with no natural interpretation.

Image inspired by the one from the paper Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning.

From text traces to latent reasoning

Early foundations (2022-2024)

Start with explicit reasoning. The Self-Taught Reasoner (STaR), introduced by Zelikman et al. (2022), generates rationales and fine-tunes on those that lead to correct answers. When an answer is wrong, it also tries generating a rationale with the correct answer supplied. Repeating that loop improves performance, but the reasoning traces are still text.

Diagram illustrating the Self-Taught Reasoner (STaR) methodology, where a model iteratively improves by learning from its own generated rationales.

Deng et al. (2023) take a different route with Implicit Chain-of-Thought (ICoT). A student first learns to answer using selected hidden states from a teacher’s explicit reasoning. An emulator learns to predict those states from the question; the emulator and student are then combined and fine-tuned together. This moves part of the sequential text computation into the model’s layers. It reduces inference time in the reported experiments, though accuracy can fall below explicit CoT.

Comparison of no CoT, explicit CoT, and implicit CoT. Implicit CoT uses reasoning supervision during training but produces the answer without verbalizing intermediate steps.

A model that answers directly may still perform intermediate reasoning in its hidden states. Yang et al. (2024) tested two-hop factual queries. Evidence for recalling the intermediate entity was substantial; evidence that the model then used it for the second hop was moderate on average and varied with the relation being queried.

Illustration of a latent multi-hop reasoning probe. A diagram shows how changing an input prompt is used to measure a model's internal recall for multi-step inferences.

Extra computation with discrete tokens (2023-2024)

Another way to give a model more computation is to delay its answer. Goyal et al. (2023) append a predetermined number of learnable pause tokens to the input prefix and delay reading the output until the last pause. Their gains depend on training the model with pauses, including during pretraining and downstream fine-tuning. The tokens provide extra computation positions; they don’t explicitly name reasoning states.

Comparison diagram: standard inference versus 'pause-inference'. The latter uses pause tokens to enable extra computation before output, illustrated with new computational paths.

Do the extra tokens need to mean anything? In “Let’s Think Dot by Dot,” Pfau et al. (2024) train transformers on two synthetic algorithmic tasks using meaningless filler tokens such as dots. In one harder 3SUM setting, models with filler tokens reached 100% accuracy, compared with roughly 66% without them.

The result shows that intermediate tokens can support useful computation without describing it. It also required training supervision that taught the models to use filler tokens. It doesn’t establish that adding dots to an arbitrary pretrained model improves its reasoning.

Comparison of three reasoning approaches: chain-of-thought (explicit reasoning), filler tokens (dots for computation), and immediate answer, showing filler tokens achieve similar performance to CoT.

Quiet-STaR, developed by Zelikman et al. (2024), trains a model to generate textual rationales at positions throughout an input sequence. Learnable start and end tokens delimit the rationales, and the training signal rewards thoughts that help predict subsequent text. Continued pretraining this way improved zero-shot results on GSM8K and CommonsenseQA without fine-tuning on those tasks. The thoughts are hidden from the ordinary text stream, but they are still discrete language tokens.

Diagram of the Quiet-STaR algorithm, showing its 'think, talk, learn' phases. The model generates internal thoughts for each token and uses reinforcement learning to improve predictions.

Continuous latent reasoning (2024-2025)

COCONUT (Chain of Continuous Thought), introduced by Hao et al. (2024), bypasses discrete tokens during a latent reasoning span. It takes the final hidden state at the last processed position and uses that vector as the next position’s input embedding. After the latent span, it returns to ordinary token generation for the answer.

Diagram comparing Chain-of-Thought (CoT) with Chain of Continuous Thought (COCONUT). CoT uses discrete text tokens, while COCONUT uses continuous hidden states for reasoning.

The loop works like this:

Process the input question normally through the transformer.
Take the final hidden state at the last position, whose size is the model’s hidden dimension.
Feed this vector back as the “next token” embedding, without projecting to vocabulary and sampling.
Repeat for multiple latent reasoning steps.
End the latent span and generate the answer with the normal vocabulary projection.

A hidden state gives the next computation a vector instead of a selected token ID. That preserves a different kind of representation, but it doesn’t tell us how much useful reasoning information the vector contains. Storage size, predictive uncertainty, and semantic content are different quantities; comparing their bit counts doesn’t establish a reasoning-capacity advantage.

Compressed Chain-of-Thought (CCoT), introduced by Cheng and Van Durme (2024), separates contemplation from answer generation. Its CCOT module (parameterized by φ) learns to predict a selected subset of hidden states from full reasoning chains; its DECODE module (parameterized by ψ) generates the answer from the question and those continuous representations. The contemplation tokens are generated autoregressively. Compressing the chain doesn’t make those steps parallel.

Illustration comparing Chain of Thought (CoT) and Compressed Chain of Thought (CCoT). CoT uses a long sequence of text, while CCoT uses a short sequence of continuous embeddings.

Hidden Chain-of-Thought (HCoT), proposed by Liu et al. (2024), replaces a thought segment with a special [CoT] representation. When the main model reaches that marker, an auxiliary model uses the preceding context to produce the hidden representation, which is fed back into the main model. Several such segments can be interleaved with ordinary content. A whole response need not collapse into a single token.

Examples of Hidden Chain-of-Thought (HCoT), where internal reasoning steps are shown as compressed thoughts with blue strikethrough text in response to user queries.

Token Assorted (Su et al., 2025) uses a VQ-VAE to encode an initial portion of the reasoning trace into discrete codebook tokens, leaving subsequent steps in text. These are latent tokens, but they aren’t continuous hidden states in the COCONUT sense. Across the paper’s mathematical benchmarks and model configurations, generated responses were about 17% shorter on average than the CoT baseline. Retaining some text does not by itself make the hidden steps interpretable or safe.

Illustration of the Token Assorted hybrid approach. A sequence of text-based CoT tokens is partially compressed into shorter, discrete latent tokens.

Architectures for latent reasoning (2025)

Huginn, from Geiping et al. (2025), builds latent reasoning into the architecture itself. It uses RNN-like iteration to adapt the amount of computation, with three parts:

Prelude layers that encode the input into a latent state.
A recurrent core of Transformer blocks applied repeatedly.
A coda that decodes the final answer.

Reusing the core separates computation depth from parameter count. Huginn has 3.5B parameters, and its benchmark scores improve as it runs more recurrent iterations. The paper’s comparison to a 50B model concerns an equivalent computation budget, not equivalent accuracy. It describes performance as roughly comparable to the first OLMo-7B generation on many metrics, with substantial variation across tasks. Geiping et al., §5.

Huginn benchmark accuracy versus recurrent iterations. The upper axis expresses repeated computation as materialized parameter count; it is not the size of a separately trained comparison model.

Diagram of the Huginn recurrent architecture. It shows three main blocks: a Prelude for encoding, a shared recurrent core for processing, and a Coda for decoding.

Yu et al. (2025) study length generalization on synthetic arithmetic, edit-distance, and longest-increasing-subsequence tasks. Their RELAY method (REasoning through Loop Alignment iterativelY) trains a looped transformer with intermediate supervision that aligns loop iterations with explicit CoT steps.

That model generates reasoning chains for longer problems, and those textual chains become fine-tuning data for an autoregressive model. The reported improvement is in those controlled tasks. It isn’t evidence that looped transformers generally replace language models or that the final model reasons without text.

Visualization contrasting a standard auto-regressive CoT model with a Looped Transformer. As problem complexity increases, the CoT model's reasoning token sequence grows, while the looped model increases its number of internal loop iterations.

Chen et al. (2025) proposed the Inner Thinking Transformer (ITT), which repeats computation within selected layers. Residual connections combine the iterations, thinking-step encodings distinguish them, and a routing network scores tokens to decide which receive further computation. The router predicts importance, not an intermediate answer.

Comparison of a standard transformer and ITT, where selected layers perform multiple inner iterations and combine their outputs through residual connections.

Mechanisms of continuous latent reasoning

Reusing hidden states

The distinction is easiest to see in a schematic version of COCONUT’s update. Let $E_t$ contain all input representations through position $t$ , including the question, and let $F(E_t)_t$ be the transformer’s final hidden state at that position. Ordinary token generation chooses from the vocabulary and appends the selected token’s embedding:

\begin{aligned} h_t &= F(E_t)_t \\ y_{t+1} &\sim \operatorname{softmax}(W_{\mathrm{vocab}}h_t) \\ E_{t+1} &= [E_t;\operatorname{embed}(y_{t+1})] \end{aligned}

Inside a COCONUT latent span, the next input is the hidden state itself:

\begin{aligned} E_{t+1} &= [E_t;h_t] \\ h_{t+1} &= F(E_{t+1})_{t+1} \end{aligned}

These equations suppress positional encoding, attention masks, and the equivalent KV-cache implementation. They retain the prompt and preceding positions: COCONUT adds a position for each latent step. Its matching hidden and embedding dimensions permit direct feedback. COCONUT, §3.

Huginn uses recurrence within model depth rather than appending one sequence position per inner iteration. Both methods compute with continuous states, but that shared description doesn’t give them the same context or memory behavior.

Training methods

These methods solve different training problems. Some start with explicit reasoning traces and remove or compress them; others train recurrent computation on ordinary text or use reward signals. Each approach has its own supervision requirements.

Learning from explicit reasoning

COCONUT begins with full CoT supervision, then replaces leading reasoning steps with continuous thoughts in stages. Those thoughts are produced by the model’s current computation, not copied from the hidden states of the deleted words. The loss applies to the remaining text and answer. The paper uses fixed stage lengths, with different schedules for GSM8K and the logical tasks.

Stepwise Internalisation (Deng et al., 2024) gradually removes leading CoT tokens without feeding replacement vectors into the sequence. Its removal schedule and smoothing procedure help the model adapt to shorter reasoning prefixes, until it can produce the answer directly. This distinction matters: learning to answer without a textual rationale is not necessarily the same mechanism as generating a sequence of continuous thoughts.

The earlier ICoT distillation method uses teacher states, an emulator, and a student. Its speed/accuracy trade-off is substantial in some settings: the GPT-2 Small experiment reaches 20% accuracy on GSM8K versus 41% for explicit CoT, while inference throughput at batch size one is about eight times higher. Those numbers describe that experiment, not a general property of distillation.

CODI (Shen et al., 2025) shares weights between explicit-CoT and latent-reasoning passes. It combines their language-model losses with a distillation loss on activations across layers at the boundary before the answer. Its GPT-2 Small experiment reaches 43.7% on GSM8K versus 44.1% for the CoT baseline, but the Llama-3.2-1B experiment retains a larger gap, 55.6% versus 61.6%. The reported 3.1× compression concerns reasoning length, not the entire context.

Training compact representations

CCoT trains its contemplation module to match selected hidden states, then trains the answer decoder to use its outputs. The paper tests compression ratios of 0.05 and 0.10. The decoder learns to produce the answer from the question and compressed states.

HCoT trains its auxiliary model with a language-model loss and a contrastive loss aligning each special-token representation with the corresponding thought segment. The main model then trains with the auxiliary model frozen. The auxiliary model can also generate the explicit thought text, although reconstruction alone doesn’t establish that every step is faithful.

Token Assorted trains a VQ-VAE to reconstruct text from discrete codes, then trains on traces with varying amounts of initial text replaced by those codes. This teaches the model to use both parts of the vocabulary.

Training recurrent computation

Huginn uses ordinary next-token prediction with randomly sampled recurrent depths. Its depth distribution has a long tail, and backpropagation is truncated to the last eight recurrent iterations.

RELAY supervises the looped model’s intermediate outputs, then uses its generated textual reasoning chains to train the autoregressive model. ITT instead learns token routing and repeated layer computation during language-model training.

Hybrid latent reinforcement learning

Hybrid Latent Reasoning via Reinforcement Learning (Yue et al., 2025) introduces Hybrid Reasoning Policy Optimization (HRPO). During the reasoning span, learned vector gates mix a sampled token embedding with a transformed hidden representation. Outside that span, the model uses ordinary token embeddings.

HRPO gives a reward of one for a correct final answer and zero otherwise. Its policy update uses reward normalization across several rollouts for the same question and KL regularization against a reference policy.

This is related to Group Relative Policy Optimization (GRPO), which estimates advantages from groups of outputs to the same question and avoids a separate value model. Removing that model saves resources, with the size of the savings depending on the configuration.

Adaptive Length Penalties (Xiang et al., 2025) penalizes long traces more strongly when a problem has a high sampled solve rate, and less strongly when the problem is difficult. With DeepScaleR-1.5B, the paper reports about 50% fewer generated tokens on average without a significant average performance decrease on the tested math tasks.

AdaRFT (Shi et al., 2025) changes which problems the model trains on. It samples near a target difficulty, then moves that target according to recent batch-level rewards. Both studies concern textual reasoning. Applying their ideas to continuous steps would require separate experiments.

Representational dynamics

COCONUT’s authors interpret its continuous thoughts as a form of latent search, with several possible next steps represented before the model commits to one. To investigate this, they train a variant that mixes curriculum stages, allowing it to return to language after a chosen number of latent steps. They then inspect the probabilities of the concepts it could generate next.

In the ProsQA example below, explicit CoT invents an edge in the graph. COCONUT reaches a wrong target after one continuous thought, but finds a correct path after two. The accompanying probabilities spread across several candidate concepts after the first thought and concentrate on a useful next concept after the second. This is evidence for the authors’ search interpretation, although the probe does not recover the full hidden computation.

ProsQA graph example: explicit CoT invents an edge, COCONUT with one latent thought reaches a wrong target, and COCONUT with two thoughts finds a correct path.

Probabilities of candidate concepts when COCONUT returns to text after one or two latent thoughts, illustrated on the ProsQA graph.

The paper treats each concept’s generation probability as an implicit value estimate for that node. Across test cases, the top candidate tends to account for more probability after the second thought than after the first, leaving less probability on alternatives. The authors interpret this concentration as a search that increasingly favors promising paths.

Cumulative probabilities of the top three candidate concepts across test cases. The top candidate generally accounts for more probability after the second latent thought.

Engineering constraints

The cost depends on where the recurrence happens. COCONUT adds latent positions to a sequence; Huginn repeats computation within depth. Treating them as one architecture hides the constraints that determine whether either is useful.

Learning latent steps can require a curriculum

In COCONUT’s GSM8K ablation, removing the curriculum lowers accuracy from 34.1% to 14.4%, below the no-CoT baseline’s 16.5%. That establishes the curriculum’s value in this setup. It does not make a curriculum mandatory for every latent method: Huginn trains recurrent computation on ordinary text, and HRPO uses final-answer rewards.

Sequential training passes limit parallelism

COCONUT’s training procedure needs $n+1$ forward passes for $n$ latent thoughts: each pass produces the next thought, and the final pass scores the remaining text. KV reuse avoids repeating earlier computation, but it cannot remove this dependency. Unlike teacher-forced textual CoT training, those latent inputs aren’t all available at the start. Inference is sequential for both ordinary autoregressive CoT and COCONUT; batch size still affects cost and throughput.

Cache cost depends on the architecture

COCONUT retains earlier latent positions in its attention history, so adding thoughts grows the KV cache. A shorter visible answer does not imply constant context length or memory. Huginn instead studies sharing and overwriting a bounded set of recurrent cache entries, so additional inner iterations need not require unbounded cache growth.

Quantization can reduce storage. SQuat (Wang et al., 2025), for example, evaluates two-bit KV caches in models generating text, including long reasoning traces. Those experiments do not measure COCONUT’s latent loops, so their results cannot establish its memory requirements.

Choosing when to stop

COCONUT considers a binary stopping classifier and a fixed number of continuous thoughts, and uses the fixed count in its main experiments after finding comparable results. Huginn’s adaptive exit is different: it can stop when the predicted token distribution changes little between iterations, measured through KL divergence, without training a halting classifier.

A stopping criterion trades computation against output quality. A stable prediction is not necessarily a correct one, so the criterion still needs evaluation on the intended workload.

Training can collapse or ignore recurrence

Huginn’s early large-scale runs either collapsed token representations or learned to ignore the recurrent state. Normalization, initialization, input adaptation, and learning-rate choices determined whether extra iterations improved the result. Unrolling a shared block is easy to describe; training it to use those iterations is a separate problem.

Outcome metrics don’t explain the hidden process

The studies measure final-answer accuracy, generated positions, and runtime on familiar tasks such as GSM8K. Those measurements tell us whether a system succeeds and what it costs. They do not, by themselves, identify which latent transition caused a wrong answer. Probes and forced text decoding can help investigate that question, but they need their own controls and validation.

Interpretability limits

The “neuralese” problem

A discrete token maps to a vocabulary entry; a hidden-state vector has no comparable built-in reading. Its size depends on the model. I use “neuralese” here as an informal name for that interpretability problem, not as evidence that the model has developed a language of its own.

Lindsey et al. (2025) trace internal computations in Claude 3.5 Haiku, finding shared features across languages and examples of reasoning that aren’t fully explained by the visible text. Their case studies illustrate why internal computation can be hard to read.

Interpreting and steering hidden states

Zhang and Viteri compare residual activations under CoT prompting and direct-answer prompting. They average across token positions and a set of questions, then use the difference as a steering direction at a selected layer. In simplified notation:

reasoning_vector = mean_h_with_cot - mean_h_direct
h_layer = h_layer + alpha * reasoning_vector

This can encourage textual CoT behavior. It does not decode COCONUT’s latent thoughts or recover every computation responsible for an answer.

Zhang and Viteri's comparison of CoT-prompted and direct-answer activations, producing a vector that is injected into a selected model layer to steer reasoning behavior.

Probes can test whether a hidden state contains information about an intermediate answer. Text reconstruction and low-dimensional visualizations offer other ways to inspect representations. But recovering a correlated signal is weaker evidence than showing that the model used it causally. We need both the measurement and a reason to trust the interpretation.

Alignment considerations

These interpretability limits raise possible alignment problems. A model could reach acceptable-looking final answers through shortcuts or biases that a reviewer cannot see, and a filter applied only to final text cannot inspect the preceding latent computation. Readable CoT is not a guarantee of faithful reasoning either; the Claude case studies include examples where the visible explanation doesn’t reflect the traced mechanism.

Possible mitigations to test include probing latent states for specific failure modes, periodically decoding them into text, and comparing systems that retain both explicit and latent representations. Each needs validation. A readable reconstruction is useful evidence, but it isn’t proof that the underlying computation was safe or faithfully described.

Current applications and performance

COCONUT on mathematical and logical reasoning

COCONUT’s December 2024 experiments use a pretrained GPT-2 base. On GSM8K, it reaches 34.1% accuracy, compared with 16.5% for no CoT and 42.9% for explicit CoT. The reported generated-position count per question falls from 25.0 to 8.2, counting the answer and continuous thoughts, rather than just visible reasoning words.

On ProntoQA, COCONUT reaches 99.8% versus 98.8% for CoT; on ProsQA, it reaches 97.0% versus 77.5%. Those logical tasks show the stronger comparison with explicit CoT, though other baselines also perform well: stepwise iCoT reaches 98.2% on ProsQA. The evidence supports a useful trade-off in these experiments, not a general ranking of latent and textual reasoning.

Appendix B reports 0.15 seconds per ProsQA test case for COCONUT versus 0.47 seconds for CoT, at batch size one on an Nvidia A100. That is a measured speedup under stated conditions; token counts alone wouldn’t establish it.

Multimodal reasoning

Heima (Shen et al., 2025) compresses intermediate multimodal reasoning into hidden thinking-token representations. It can use far fewer generated tokens, but accuracy varies by benchmark. On MathVista, for example, the paper reports 13.8 generated tokens versus 216.3 for LLaVA-CoT, roughly 6%, while accuracy falls from 50.9% to 43.6%.

A trained decoder can reconstruct visual details from the hidden thinking states without receiving the original image. It also receives the question and an explanatory prompt, so this is contextual reconstruction, not evidence that an isolated vector reveals its full reasoning process.

Diagram showing the Heima model's multimodal reasoning process. It shows how the model can reason about images and text in a continuous latent space.

Code generation and planning

Program state and control flow are plausible targets for compact internal representations, but that is a design hypothesis rather than a demonstrated result of COCONUT. Its ProsQA experiment concerns graph-based logical planning. Huginn also reports MBPP and HumanEval results, which provide evidence about code generation without showing that the model maintains a faithful internal execution state.

Where latent reasoning might help

Efficiency and token overhead

For full-attention cached decoding, each new position attends to the retained history. Appending fewer positions can therefore reduce attention work and cache storage, but replacing words with latent positions does not remove those costs. COCONUT benefits when it can do useful work with fewer generated positions; a recurrent-depth model has a different cost structure because it reuses positions while repeating layer computation.

This is an architectural argument, not a universal speedup claim. Runtime still depends on the number of forward passes, the batch size, the cache design, and the hardware.

Representation without text

Latent steps can carry intermediate state without serializing it into tokens. That can make some kinds of search or planning more efficient, but it also makes intermediate reasoning harder to inspect.

An illustrative mental model is to mix several partial hypotheses in a hidden representation:

h_{\text{combined}} = 0.6 \cdot h_{\text{path1}} + 0.3 \cdot h_{\text{path2}} + 0.1 \cdot h_{\text{exploration}}

The coefficients are invented for the illustration. COCONUT’s probes suggest that multiple possible next steps can remain represented, but they do not establish this linear decomposition. The model’s use of continuous states also doesn’t mean that it runs gradient descent over them during inference. End-to-end differentiability provides a training route; an inference-time search procedure would have to be specified and tested.

Architecture and scaling notes

Ye et al. (2024) study controlled synthetic grade-school math problems. In their depth/width comparisons, deeper, narrower models generalize to longer reasoning chains better than some shallower, wider alternatives. This is evidence for depth under that task and training setup, not a universal scaling law for arbitrary reasoning workloads.

Reinforcement learning

A latent-reasoning objective could reward correct answers while penalizing computation. For example, a schematic objective might be:

R = \mathbf{1}[\text{answer correct}] - \lambda\,\text{compute cost}

Here the cost measure and $\lambda$ would be design choices. This is not HRPO’s published reward, which uses final-answer correctness. Whether a compute penalty improves a latent model’s speed/accuracy trade-off would need an experiment.

A speculative design sketch

One possible design would combine recurrent computation with a gate between token embeddings and hidden states, drawing on the two approaches above. A controller would also decide how many inner steps to run before emitting another token. I haven’t implemented this combination.

Architecture

A shared recurrent core that can run for $k$ inner iterations at inference time. Reusing its weights increases compute without duplicating that core’s parameters.
A gate that mixes the most recently emitted token’s embedding with a transformed hidden state. A simplified scalar version is:

$e_t = \gamma_t\,W_h\,h_{t-1} + (1-\gamma_t)\,\operatorname{embed}(y_{\mathrm{last}}),\qquad 0 \leq \gamma_t \leq 1$

Here $t$ indexes internal steps and $y_{\mathrm{last}}$ is the most recently emitted token. Training could start with the gate near 0 and gradually introduce latent inputs. That would be a curriculum to test, not a requirement established by the existing methods. The gate permits a return to token embeddings; differentiability alone doesn’t ensure the model learns to use that option well.
Optionally, a small latent memory to retain hypotheses across inner steps. Its update rule and effect on accuracy would need to be defined and measured.

What you’d need to validate

Training stability when mixing token and latent steps.
Evaluation that doesn’t rely on readable CoT traces.
Interpretability hooks (forced reveal steps, probes) that work at scale.

This remains a sketch, not a result about practicality, accuracy, or superiority. The controller’s stopping rule, the memory update, and the training objective are all unresolved.

The attraction is being able to do more intermediate computation in dense vectors without emitting long token traces. Getting further will likely require more stable training and clearer evaluation protocols, along with better ways to inspect latent steps. The difficulty of interpreting, debugging, and evaluating that computation still limits what we can do with it.