Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova | Amazon Web Services

When tailoring advanced large language models (LLMs) like the Amazon Nova 2 family for specific domains through Supervised Fine-Tuning (SFT), a significant hurdle has been the practical and financial burden of generating high-quality chain-of-thought (CoT) reasoning traces. These detailed step-by-step explanations are crucial for models to exhibit robust reasoning capabilities, particularly in complex tasks such as coding and mathematical problem-solving. Traditionally, the expense and time involved in creating these "golden" reasoning traces have led many to bypass them during SFT, focusing solely on input-output pairs. This omission, however, can lead to a degradation of the model’s inherent reasoning abilities, a phenomenon known as catastrophic forgetting, and consequently, suboptimal performance on challenging tasks.

Recognizing this critical gap, Amazon has introduced a novel technique called Self-Distilled Reasoning (SDR). This innovative approach aims to equip LLMs with enhanced reasoning capabilities without the prohibitive cost of manual annotation. SDR leverages the existing reasoning patterns of the base Amazon Nova 2 Lite model as a substitute for missing reasoning traces in SFT datasets. This allows for the seamless integration of complex reasoning into customized models, significantly boosting their predictive performance and addressing the persistent challenge of catastrophic forgetting.

The implications of SDR are far-reaching, particularly for organizations seeking to deploy highly capable LLMs in specialized fields. By enabling models to "think" through problems using their own internally generated reasoning, SDR promises to unlock higher levels of accuracy and reliability, especially in domains demanding intricate logical deduction.

The Reasoning Suppression Problem and the Quest for Performance

At the core of the challenge lies the "reasoning suppression problem." During standard SFT on datasets lacking explicit reasoning steps, the model’s training objective often leads it to prioritize direct input-output mapping over generating coherent thought processes. This occurs because the loss function penalizes tokens that do not directly contribute to the final output. Without explicit supervision for intermediate reasoning tokens, the model learns to bypass its sophisticated reasoning mechanisms, a behavior that exemplifies "shortcut learning." This is akin to a student learning to arrive at the correct answer through rote memorization rather than understanding the underlying principles.

The importance of reasoning in LLM performance has been well-documented. Amazon’s own research indicates that models equipped with reasoning capabilities, such as the Amazon Nova 2 family, demonstrate significantly enhanced prediction accuracy, particularly on hard problems. For instance, enabling reasoning during both training and inference can lead to substantial performance gains. Experiments on the LLaVA CoT dataset, as illustrated in preliminary data, show a dramatic increase in target performance when reasoning is active, with gains of over 17.3% observed at a full merge weight. This underscores that reasoning is not merely an add-on but a fundamental driver of advanced AI capabilities.

Model Merging: A Partial Solution with Trade-offs

Historically, when faced with the loss of general reasoning abilities due to domain-specific fine-tuning, a common strategy has been "model merging." This technique involves taking a fine-tuned SFT checkpoint and blending it with the original base model. The idea is to reintroduce the general capabilities that might have been diminished during the specialization process. However, model merging often comes with a significant drawback: it can compromise the gains achieved during SFT, leading to a trade-off between specialized performance and general capabilities. This phenomenon, where advancements in one area lead to degradation in another, is a hallmark of catastrophic forgetting.

The problem is further compounded when considering the nuances of SFT. If a dataset for SFT contains only input-output pairs, and reasoning is turned off during training, the model’s ability to reason can completely vanish. This was observed in experiments where vanilla SFT caused math performance to plummet from an average of 70% in the base model down to a mere 6%. While model merging can help recover some of these lost skills, it often requires careful tuning of the merge weight to strike a balance between retaining domain-specific accuracy and preserving general knowledge. This balancing act can be complex and resource-intensive.

Introducing Self-Distilled Reasoning (SDR)

To address these limitations, Amazon has developed Self-Distilled Reasoning (SDR). This innovative method fundamentally rethinks how reasoning is incorporated into the SFT process. Instead of relying on expensive human-annotated reasoning traces or accepting the trade-offs of model merging, SDR utilizes the base Amazon Nova 2 Lite model itself to generate reasoning chains for the training data.

The SDR process involves three key stages:

1. Generation of Reasoning Traces: The base Amazon Nova 2 Lite model is used to generate chain-of-thought reasoning for each example in the SFT dataset. This can be achieved through "basic reasoning" (natural forward thinking) or "guided reasoning" (reconstructing reasoning given a known answer).
2. Data Augmentation: The generated reasoning traces are then prepended to the original input-output pairs, effectively augmenting the training dataset.
3. Fine-Tuning: The model is fine-tuned with reasoning mode enabled, receiving supervision signals for both the generated reasoning steps and the final output.

This approach offers several compelling advantages. Firstly, it is a zero-annotation cost solution, as it leverages the model’s own capabilities, eliminating the need for human effort in creating reasoning data. Secondly, the generated reasoning traces are inherently consistent with the problem-solving strategies of the base model, providing a strong inductive bias for the fine-tuned model. Thirdly, SDR is highly scalable and can be applied to existing SFT datasets across various domains.

Experimental Validation: SDR Outperforms and Preserves

Extensive experiments have been conducted to validate the efficacy of SDR. Across three diverse benchmarks—MedMCQA (medical question answering), CoCoHD (long-context structured extraction), and Invoice-OCR (multimodal document understanding)—SDR has demonstrated significant improvements in target performance while simultaneously mitigating catastrophic forgetting.

A key finding from these experiments is that vanilla SFT without reasoning traces leads to a drastic decline in general capabilities, as measured by a math control benchmark. For instance, on the MedMCQA dataset, when SFT was performed without reasoning, math performance dropped from 70% to a negligible 0%. With SDR, this performance was almost entirely recovered, remaining at a robust 67.9%. This preservation of general reasoning skills is critical for models intended for real-world applications where versatility is paramount.

Furthermore, SDR consistently matches or exceeds the target performance achieved by pure SFT (without any model merging). Crucially, it does so while retaining a significant portion of the base model’s general capabilities. This is a notable departure from model merging, which often forces a compromise between specialized and general performance. In one comparison, self-distillation yielded nearly identical math performance (70% compared to 68%) while improving target performance by over 6.5% on average, aligning with recent research advocating for self-distillation as a powerful tool against catastrophic forgetting.

The practical implications are substantial. Organizations can now customize powerful models like Amazon Nova 2 without the fear of their general reasoning abilities deteriorating. This means that a model fine-tuned for a specific legal domain, for example, would still retain its proficiency in mathematical reasoning or coding, which might be required for ancillary tasks.

Constructing Reasoning Traces: Basic vs. Guided Approaches

The process of generating reasoning traces for SDR involves selecting an appropriate method. Amazon outlines two primary approaches:

• Basic Reasoning: This method involves prompting the base model to naturally generate its reasoning process in a forward direction, mimicking how it would typically approach a problem during inference. The prompt template guides the model to think through the question and format its response with <thinking> and <answer> tags. This approach is straightforward and leverages the model’s inherent reasoning style.

• Guided Reasoning: This technique utilizes the availability of ground-truth answers to steer the reasoning generation process. By providing both the question and the correct answer, the model is tasked with reconstructing plausible reasoning steps that lead to that answer. This "hindsight learning" approach can be particularly effective when the desired reasoning path is complex or not immediately obvious to the base model. A crucial secondary filtering step is employed to ensure that the generated reasoning does not inadvertently reveal the answer, thereby introducing noise into the training data.

The implementation pipeline for constructing these traces often involves leveraging services like Amazon Bedrock. For instance, a Python script utilizing the boto3 library can be employed to interact with Amazon Bedrock’s Converse API. This script would read the SFT training data, invoke the model to generate CoT reasoning, and then assemble the final dataset with the reasoning content prepended. This automated pipeline ensures scalability and efficiency in preparing the augmented training data.

Ablation Studies: The Fragility of Reasoning and SDR’s Resilience

To further understand the behavior of SFT and the impact of SDR, Amazon conducted ablation studies, particularly focusing on scenarios where reasoning data is partial or entirely missing. These studies revealed a stark contrast:

• Without SDR: As the proportion of reasoning data in the SFT dataset decreases, general capabilities—measured by a math control benchmark—experience catastrophic forgetting. For example, math accuracy plummeted from 66.6% to 21.7%, then to 3.3%, and finally to 2.5% as reasoning data dropped from 75% to 0%. This highlights the extreme fragility of general reasoning abilities when not adequately supported during fine-tuning.

  

• With SDR: The introduction of SDR pre-filling completely prevents this collapse. Math accuracy remained stable within the 65%–72% range across all percentages of reasoning data, even when no explicit reasoning data was present in the training set. This demonstrates SDR’s power in acting as a consistent inductive bias, safeguarding general knowledge.

Moreover, the impact on reasoning behavior at inference time is dramatic. Models trained with less than 25% reasoning data and no SDR prefill produced zero reasoning tokens, indicating a complete loss of step-by-step reasoning ability. In contrast, SDR prefill enabled models to generate a substantial number of reasoning tokens (median of 859), even with 0% reasoning data in training. This signifies that SDR effectively teaches the model to reason, not just to memorize outputs.

The target performance on the MathVista benchmark also showed significant gains when reasoning was enabled at inference, roughly doubling accuracy. The best configuration identified in these studies involved 50% reasoning data combined with SDR prefill, achieving 45.2% target accuracy on MathVista and maintaining 70.4% on the math control, thereby avoiding forgetting and maximizing performance. Notably, even when reasoning was disabled at inference, SDR prefill improved evaluation performance, suggesting that the learned reasoning patterns have a beneficial effect on task execution.

Broader Implications for Model Customization and Continual Learning

The findings from SDR have profound implications for how organizations approach LLM customization. Firstly, maintaining consistent reasoning modes during training and inference is critical for optimal performance. Secondly, domain-specific SFT, without careful intervention, can significantly erode general reasoning skills. Thirdly, while model merging offers a post-hoc solution, it necessitates careful tuning and introduces a trade-off.

SDR, by contrast, offers a training-time solution that preserves reasoning capability without requiring post-hoc interventions or manual annotation. This makes it an attractive option for practitioners looking to enhance their LLM customization workflows. The technique is particularly valuable in the context of continual learning, where models are updated over time with new data and skills, and it is essential to retain previously acquired knowledge. SDR acts as a robust inductive bias, helping to prevent the degradation of existing capabilities as new ones are learned.

The research also offers guidance on the choice of "teacher" model for generating reasoning traces. While a stronger teacher model might seem intuitively better, it can sometimes lead to a larger "student-teacher gap," which can hinder distillation and negatively impact general performance. For Amazon Nova 2 Lite users, utilizing traces generated by the same model family (Nova 2 Lite) often provides the most balanced outcome, optimizing both target and general performance.

Practitioner Decision Guide: Navigating the Options

To assist practitioners in making informed decisions, a clear guide has been developed:

• Default SFT with Regression: For latency-sensitive applications, model merging can recover general performance. For those with accommodating latency constraints, SDR with Nova 2 Lite traces offers the best balance of target performance and general capability preservation, often surpassing model merging.
• Weak Base Model on Target Domain: Guided reasoning can be employed to provide additional signal during trace generation, potentially improving performance in specialized areas.
• Partial Reasoning Data (≥ 50%): Training with reasoning turned on is generally sufficient to preserve general skills.
• Limited Reasoning Data (< 50%): Pre-filling missing traces with Nova 2 Lite reasoning (offline) is highly recommended to prevent general performance collapse.
• No Need for General Capabilities: Standard SFT without reasoning data may suffice if only target performance is critical.
• Continual Learning: Combining SDR with data mixing and a low or zero merge weight is suggested for optimal retention of learned skills.
• Continued Use of Model Merging: When using SDR, a smaller merge weight is recommended, as SDR already provides regularization that merging was previously compensating for.

This structured approach empowers developers and researchers to select the most appropriate customization strategy based on their specific data composition, performance requirements, and deployment constraints. The advent of Self-Distilled Reasoning represents a significant stride forward in making advanced LLM customization more accessible, effective, and robust, paving the way for more intelligent and reliable AI applications across a multitude of industries.