offline RL approaches such as DPO (Rafailov et al., 2023). Chen et al. (2025b) and Chen et al. (2025a) built specific rewards using retrieval tools to provide measures of factuality for RL training.

Reasoning and RL. Standard pretraining already gives reasoning capabilities, including chain-of-thought emergence (Kojima et al., 2022). These traits are further amplified via post-training, particularly through reinforcement learning on verifiable rewards (RLVR) (DeepSeek-AI, 2025). The success of improving reasoning at post-training time has encouraged researchers to try to move post-training techniques further upstream to either mid-training or pretraining. Recent works have augmented pretraining with thinking tokens (Wang et al., 2025; Fujii et al., 2025), and incorporated RL for optimizing thoughts for the next token (Dong et al., 2025; Hatamizadeh et al., 2025) or the next set of tokens (Yu et al., 2024; Li et al., 2025; Team et al., 2025).

5 Conclusion

Our work re-envisions pretraining by using a strong post-trained model to provide superior supervision signals. This works in two ways: (i) by providing rewrites on the original streaming pretrain data; and (ii) by acting as a judge. We showed that such a self-improving setup can improve the factuality, safety and overall generation quality of pretrained models.

6 Discussion

Here we discuss some common questions about our approach.

Isn’t this slower than next token prediction pretraining? Self-Improving Pretraining is indeed slower than standard next token prediction, especially when using rollouts. However, using rewrites and suffixes only, which can work at the start of pretraining, might not be that much slower. Nevertheless, our thinking follows that of Chung (2023): training methods should be designed to exploit future increases in compute, favoring incentive-based objectives over explicit skill instruction. Hence, using strong post-trained models as judges may prove to be a winner in the long run, especially as pretraining hits a “data wall” where increased compute with next token prediction does not offer gains, in the case that we have “run out of data”.

Is making models safe always a good idea? We showed how our approach can make models safer, but indeed there may be cases where safe generations are not the goal. An example is generating a movie script with dialogue from bad actors, which would necessitate the ability to generate unsafe text. During training, one way to get around this is the use of control tokens, or some other method of fine-grained control of safety, i.e. to train for both safe and unsafe cases, given the control token which can be switched on/off at inference time. We believe this might actually be a better choice than simply removing capabilities (Yi et al., 2025). As mentioned earlier, Korbak et al. (2023) is an early work incorporating safety into pretraining, which reported success with control tokens which incorporate human preferences.

What else can this framework do? How do you generalize it? We showed that safety, factuality and general quality can be optimized in our framework, e.g. simply by providing different LLM-as-judge prompts. An obvious approach to combine all three methods at the same time is to sum the rewards from the prompts, or potentially combine them into a single prompt. We already showed that combining quality and safety or quality and factuality works, so we believe this should not be difficult. Ideally we would prefer a more generic judge prompt that can capture all these skills well at the same time. Going further, there are other aspects of a powerful model one may wish for pretraining to also capture, i.e. other skills! – an obvious one being stronger reasoning ability. Training chain-of-thought can also fit fairly well into our framework, i.e. switching between rewrites from a strong post-trained model earlier in pretraining (in this case, to rewrite the original suffix to contain chain-of-thought), and then switching to improving rollouts later in training. See section 4 for existing related work in the area of chain-of-thought augmentation and reinforcement learning.