Press play to start listening
Distillation attacks have escalated into a primary security threat for AI model providers in 2026. The US government has coordinated federal agencies and national laboratories under its memorandum (PDF), “Protecting American AI Models from Adversary Distillations,” while major AI companies have described attempts to extract capabilities from their models.
Reports from OpenAI (PDF), Anthropic, and Google DeepMind describe efforts targeting reasoning, coding, tool use, and other valuable model capabilities.
Distillation itself is a legitimate machine-learning technique. It allows a smaller student model to learn from the outputs of a more capable teacher. It becomes adversarial when someone collects another provider’s outputs at scale without permission or contrary to its terms.
One common defense is to hide the model’s full chain-of-thought and return only a compact reasoning summary. That should prevent straightforward copying of the original reasoning trace, but does it remove enough information to stop effective distillation?
We decided to test that question.
Simulating a Distillation Attack
Our experiment used GLM-5.1 as the teacher model. Its full chain-of-thought was available to us, which meant we could compare reconstructed reasoning with the original traces.
For the student, we selected GLM-4.7-Flash, a 30-billion-parameter mixture-of-experts model. We first trained it on the teacher’s complete reasoning trajectories to create GLM-4.7-Flash-Coder.
The dataset contained 7,777 successful runs of GLM-5.1 on 3,837 agentic coding tasks from 408 public repositories in SWE-rebench-V2. In total, the training material contained 177 million tokens.
Training on the complete trajectories increased the student model’s score by 26%. This gave us a baseline against which reconstructed reasoning could be measured.
We then treated GLM-5.1 as though it were a proprietary model whose full reasoning was unavailable. Instead of using its original chain-of-thought, we compacted the reasoning and attempted to rebuild it from the resulting summaries.
Reconstructing the Missing Reasoning
Compacted summaries are poor training material when used directly. They tend to adopt the wrong writing style and omit technical details such as class names, file paths, and intermediate observations.
We therefore used a separate language model to reconstruct fuller reasoning from three inputs:
- The preceding conversation
- The tool call that followed the reasoning
- The compacted chain-of-thought summary
The conversation helped restore missing technical context. The tool call acted as a constraint, keeping the reconstruction consistent with the action that the original agent performed.
Once reconstructed, the reasoning was inserted into the original trajectory. We then trained the same base model on the resulting dataset.
This method does not reproduce the teacher’s original reasoning word for word. The question was whether it could recover enough useful structure to improve the student’s performance.
Why Self-Correction Matters
Reasoning traces often contain self-corrections signaled by words such as “wait,” “actually,” “hmm,” and “however.” These moments show that the model considered an approach and then rejected or modified it.
A tool call records only the final decision. It does not explain why another option was discarded.
This missing information matters because two approaches can initially appear equally valid. A model trained only on final actions may learn which action occurred in the training data without learning why competing options were rejected.
That form of memorization may fail when a similar task presents the wrong option more convincingly. Reasoning traces provide examples of evaluation, rejection, and correction that final actions alone cannot supply.
Single Summaries Versus Chunked Compaction
We tested two ways of compacting the original GLM-5.1 reasoning.
The first summarized the entire reasoning trace in a single operation. We refer to this as mono compaction.
The second divided the reasoning into 200-token portions and compacted each portion separately.
We reconstructed both versions using the same method and used an LLM-based evaluator to count the self-corrections retained in the recovered reasoning.
Reconstruction from chunked summaries preserved five times more self-corrections than reconstruction from a single summary. It retained 37% of the occurrences found in the original reasoning.
The result suggests that dividing reasoning into separate summaries preserves more of its internal structure. A later portion can still contain a correction to an idea presented earlier, even when each portion has been shortened.
Measuring Consistency
Reconstructed reasoning must satisfy two conditions. It should resemble reasoning that the model could naturally produce, and it should remain consistent with the teacher’s recorded action.
We measured both conditions using negative log-likelihood (NLL), which represents how surprising a sequence is to the model. Lower values indicate that the sequence is more probable.
A sentence that simply announces the correct tool call may align perfectly with the action, but it provides almost no reasoning. At the opposite extreme, newly generated reasoning may sound natural while leading toward a different action.
The chart compares the trade-off between the recovery methods. The x-axis measures whether the reconstructed reasoning resembles text the model would naturally produce in the same context. The y-axis measures whether that reasoning remains consistent with the tool call made by the teacher. For each assistant turn, we report the 95th-percentile NLL, focusing on the least probable 5% of tokens. Lower values are better on both axes.
In the chart, the blue dots represent reconstructed reasoning. These traces were less probable than the original chain-of-thought but much more probable than the compacted summaries. They also remained closely aligned with the teacher’s original tool calls.
NLL remains an indirect measurement. It says more about probability and stylistic consistency than whether a dataset will improve a model. The final test must therefore involve training and evaluating the student.
Training Results
We trained GLM-4.7-Flash using three datasets:
- The original full-reasoning trajectories
- Reasoning reconstructed from chunked summaries
- Reasoning reconstructed from mono-compacted summaries
We evaluated the resulting models on 500 agentic coding tasks taken from SWE-rebench-V2 repositories that did not appear in the training set. The evaluation used the same PI agent harness across all three configurations.
Training on the chunked reconstruction increased pass@1 accuracy from 33.15% to 40.4%, a relative improvement of approximately 22%. This retained 85% of the improvement achieved through training on the complete reasoning traces. Pass@4, which measures whether the model solved a task at least once across four attempts, increased from 46.2% to 53%.
Results from mono-compacted reasoning were much weaker. That version retained only 25% of the improvement achieved through full-reasoning training, while its pass@4 score decreased by 0.8 percentage points.
The self-correction measurements followed a similar pattern. Reasoning reconstructed from mono compaction contained 0.07 measured occurrences per response, compared with 0.17 for the base model, 0.37 for the chunked reconstruction, and 1.77 for the model trained on complete trajectories. The full dataset is available on Hugging Face, while the training configuration and evaluation pipeline are described in the model card.
What Fable-5’s Chunking May Reveal
We also examined the reasoning summaries returned by Fable-5.
In our measurements, 30% of its reasoning responses arrived in multiple chunks. Among the second chunks, 2% began with “Actually,” indicating that some changes of direction remained visible after compaction.
Responses could be divided into as many as 70 chunks, with summaries arriving approximately every 650 reasoning tokens on average. The first 150 tokens were not divided, which is consistent with chunking being used mainly for longer responses that require live progress updates.
Response length alone did not determine the number of chunks. A 6,000-token reasoning trace could arrive in 7 to 12 portions.
Our working explanation is infrastructural. Separate processing resources may compact portions independently, with load and scheduling affecting chunk sizes. This has not been confirmed by Anthropic, and other technical explanations remain possible.
Regardless of its cause, chunking can preserve relationships between earlier ideas and later corrections. Our GLM-5.1 experiment indicates that these relationships can provide useful training signals after reconstruction.
What the Experiment Does and Does Not Show
Our experiment does not establish that Fable-5 can be copied using its reasoning summaries. We trained the student using GLM-5.1 and examined Fable-5’s chunking separately.
The work is an engineering experiment rather than a peer-reviewed study. It also uses one teacher model, one student model, and an agentic coding benchmark. Results may differ with other models, tasks, and compaction systems.
What the experiment does show is that reasoning summaries can retain useful information even when the original chain-of-thought is hidden. The amount retained depends partly on how those summaries are produced and divided.
Hiding raw chain-of-thought still blocks the simplest form of reasoning extraction. It does not, by itself, prevent an attacker from using conversations, summaries, and subsequent tool calls to reconstruct useful training data.
Providers therefore need additional controls, including detection of coordinated API activity, restrictions on automated and fraudulently created accounts, cross-account analysis, and testing of how much information their reasoning summaries disclose.
The distillation problem is not limited to whether raw chain-of-thought is visible. It also concerns what can be reconstructed from everything the model returns around it.
(Photo by Werclive on Unsplash)
