A Rank-One Representation Edit Recovers Much of RMU’s WMDP Accuracy Loss

1. Introduction

Machine unlearning asks whether the influence of selected training data can be removed from a trained model without repeating the entire training process. In its strongest form, unlearning aims to produce the same distribution over models that would have resulted from training without the deleted data. Most work on large language models uses a weaker, empirical notion: edit a trained model so that it performs poorly on a forget benchmark while preserving performance on retain tasks. The distinction matters because a behavioral score can change even when relevant information remains recoverable.

The WMDP benchmark was introduced as a public proxy for hazardous knowledge in biology, cybersecurity, and chemistry [Li et al., 2024]. The same work proposed Representation Misdirection Unlearning (RMU), which pushes intermediate representations of forget-set examples toward random target vectors while constraining representations on retain data. RMU substantially reduces WMDP accuracy, but this observation alone does not determine whether the model lost the tested information or learned a mechanism that suppresses its expression.

Several results motivate a stronger evaluation. Unlearning benchmarks can be sensitive to prompt construction, answer extraction, and dependencies between forget and retain data [Lynch et al., 2024] [Thaker et al., 2025]. Feng et al. further show that conclusions about WMDP recovery can change across multiple-choice and open-ended evaluation protocols [Feng et al., 2025]. For RMU specifically, white-box activation edits and small fine-tuning attacks can restore much of the suppressed performance [Łucki et al., 2025]. These findings do not prove that every low score is superficial, but they make recovery under a minimally informative intervention an important diagnostic.

This paper studies one such diagnostic in Gemma-2-2B. The analysis adapts difference-in-means direction extraction and rank-one orthogonalization from work on refusal mechanisms [Arditi et al., 2024]. The empirical question is deliberately narrow:

Does a low-dimensional direction account for a substantial part of the WMDP accuracy reduction observed in this RMU checkpoint?

The main findings are:

1. RMU-minus-base activation differences extracted independently on WMDP-Bio and WMDP-Cyber are strongly aligned across a band of middle layers.
2. A shared direction selected at block 9 and token position $-5$ supports a global rank-one edit that recovers 75.4% of the displayed WMDP-Bio accuracy gap and 66.1% of the WMDP-Cyber gap.
3. A same-norm random orthogonal control does not recover performance, making the result direction-specific rather than a generic consequence of perturbing the model.
4. The edit also recovers MMLU performance. The direction therefore carries collateral capability degradation as well as behavior on the forget benchmarks.
5. Sparse-autoencoder analysis identifies topic-related candidate features, but the evidence is exploratory and does not establish that those features form a necessary or sufficient circuit.

The intervention is causal in the limited experimental sense that editing the model changes its outputs. The recovery ratio is not a causal mediation estimand, nor is it a percentage of retained knowledge. It is a descriptive statistic for one checkpoint, intervention, and evaluation protocol.

2. Background and Related Work

2.1 Representation Misdirection Unlearning

Let $M_\theta$ be a causal language model and let $\phi_L(x;\theta)$ denote its activation tensor at transformer block $L$ for tokenized input $x$. RMU starts with a frozen base model $\theta$ and an editable copy $\hat\theta$. Given forget data $\mathcal D_f$, retain data $\mathcal D_r$, a unit random direction $\mathbf u$, and steering magnitude $c$, its losses are

and

The optimized objective is $\mathcal L_{\mathrm{forget}}+\alpha\mathcal L_{\mathrm{retain}}$. The forget term disrupts representations associated with the targeted corpus; the retain term limits collateral change. Later analysis of representation steering connects the random target to reduced token confidence and incoherent generations [Huu-Tien et al., 2025].

2.2 Recoverability and evaluation

The original WMDP paper presents its tasks as proxies, not certificates of deletion or safety. Subsequent work has exposed two complementary problems. First, a model may score poorly because a method disrupts a narrow evaluation interface. Second, a recovery attack may itself inject information and overstate what the unlearned model retained. A sound interpretation must therefore describe both the intervention and the downstream task rather than treating any single recovery score as proof of erasure or non-erasure [Feng et al., 2025].

Łucki et al. recover RMU performance through fine-tuning and activation-space orthogonalization [Łucki et al., 2025]. Arditi and Chughtai independently reported a closely related directional result in an informal research note [Arditi et al., 2024]. More recent work describes inhibitory units that hide rather than erase target behavior, although its neuron-level attribution and training setup differ from the residual-stream analysis here [Yang et al., 2026]. Together, these studies motivate mechanistic evaluation without making low-dimensional suppression a universal property of unlearning.

2.3 Difference-in-means directions and sparse features

Arditi et al. show that a difference-in-means direction can be both predictive of refusal and actionable through activation addition or rank-one weight edits [Arditi et al., 2024]. This paper transfers the method, not the conclusion: incoherent RMU output is different from refusal, and a direction found in one model need not generalize to another.

Sparse autoencoders (SAEs) provide an overcomplete feature basis for model activations. We use the layer-5 16k SAE released with Gemma Scope [Lieberum et al., 2024]. Feature-level interventions have also been explored for WMDP unlearning [Farrell et al., 2024]. Here, an SAE is used only to generate hypotheses about which layer-5 features align with the gradient of the selected direction. No claim of complete feature recovery or circuit identification is made.

3. Experimental Setup

3.1 Models and RMU checkpoint

The base model is google/gemma-2-2b. The RMU checkpoint was trained on biological and cybersecurity forget corpora with WikiText retain data. The recorded training command uses block 9, steering coefficient 350 for each forget domain, retain coefficient 300, learning rate $5\times10^{-5}$, batch size 4, 250 batches, one epoch, and seed 42. The implementation updates a selected parameter tensor in blocks 7–9. The evaluated checkpoint is published as lenguyen1807/gemma-2-2b-RMU.

This is a case study of one training run. There are no repeated RMU seeds, alternative model sizes, or independent checkpoints from which to estimate training variance.

3.2 Benchmarks

WMDP contains 1,273 Bio questions, 1,987 Cyber questions, and 408 Chemistry questions [Li et al., 2024]. The primary forget evaluations use zero-shot WMDP-Bio and WMDP-Cyber accuracy. MMLU accuracy measures broad utility [Hendrycks et al., 2021]. All three are multiple-choice evaluations run through the LM Evaluation Harness with seed 42.

The intervention-recovery result uses the complete WMDP-Bio and WMDP-Cyber benchmark evaluations recorded for the base, RMU, edited, and control models. It does not include open-ended answer grading, paraphrases, or adversarial prompts.

3.3 Direction-extraction data

Direction extraction starts after the first 500 WMDP examples reserved by the earlier probe pipeline. For each of WMDP-Bio and WMDP-Cyber, questions beginning with “which” are filtered out, questions longer than 1,024 characters are excluded, and 300 examples are sampled with seed 42. Each domain is split 80/20 into 240 extraction and 60 validation questions.

This filtering was introduced for computational and prompt-format convenience. It changes the distribution of extraction prompts and should not be interpreted as a principled property of WMDP.

4. Methods

4.1 An exploratory probe for incoherent output

RMU generations on forget-domain prompts often become repetitive, contradictory, or incomplete. We call this behavior junk output without assuming that it corresponds exactly to forgotten information. A 2,000-prompt dataset combines 1,000 WMDP and 1,000 Alpaca prompts [Taori et al., 2023]. Gemini-2.5-Flash labels generated outputs as junk or non-junk. The resulting class distribution is imbalanced:

For each transformer block, a linear probe is trained on the final-token residual stream. Features are standardized with training-set statistics, and weighted resampling balances the training batches. Block 12 obtains test ROC-AUC 0.708 and is used as a scalar junk score in direction selection.

The probe is an instrument for candidate search, not evidence that a distinct “junk feature” exists. Its labels are subjective, the model used for labeling may introduce systematic error, and selecting the best block on the test set leaks test information into the reported score.

4.2 Difference-in-means directions

Let $M_{\mathrm{base}}$ and $M_{\mathrm{RMU}}$ denote the base and RMU models. For prompt $x$, zero-indexed block $\ell$, and token position $i$, let

be the residual-stream vector entering block $\ell$. For domain $k\in\{\mathrm{bio},\mathrm{cyber}\}$, the candidate direction is

Candidates are computed at every transformer block and at the final five token positions, $i\in\{-5,-4,-3,-2,-1\}$. The shared candidate is

Bio and Cyber candidates have cosine similarity above 0.8 across several middle blocks. This alignment motivates a shared candidate, but it does not by itself show that the two domains use the same causal mechanism.

4.3 Candidate selection

Candidates are scored on the combined Bio and Cyber validation prompts using three diagnostics:

1. Removal: Globally project the candidate direction out of the RMU model and measure the mean block-12 junk score. Lower is better.
2. Addition: Add the unnormalized candidate at its source block in the base model and measure the mean junk score. Higher is better.
3. Specificity: Apply the removal edit to the base model and measure the KL divergence between the original and edited next-token distributions. Lower is better.

Candidates are discarded when the base-model KL divergence exceeds 0.1, addition fails to raise the probe score above zero, or the source block lies in the final 20% of the network. Among the remaining candidates, selection minimizes the removal score. The recorded shared direction comes from block 9 and token position $-5$, with removal score $-0.29$, addition score $0.54$, and KL divergence $0.08$.

The selection pipeline establishes a reproducible rule but remains exploratory. Addition uses each raw direction’s native norm, so the score can reflect magnitude as well as orientation. The same probe influenced both candidate selection and the qualitative interpretation of the selected direction.

4.4 Rank-one orthogonalization

The benchmark intervention is a persistent rank-one weight edit, not an ablation confined to block 9. For normalized direction

define the orthogonal projector

The edit projects the token-embedding matrix and the output maps of every attention and MLP block so that these components cannot write in direction $\widehat{\mathbf d}$. Equivalently, each affected residual contribution $\mathbf v$ is replaced by $P_{\mathbf d}^{\perp}\mathbf v$. This follows the rank-one intervention used to remove refusal directions [Arditi et al., 2024].

For the primary control, sample a Gaussian vector, remove its projection onto $\mathbf d$, and rescale the remainder to $\lVert\mathbf d\rVert_2$. Applying the same edit with this random orthogonal direction tests whether recovery is specific to $\mathbf d$ rather than a generic effect of rank-one model damage.

4.5 Accuracy-gap recovery

For benchmark $B$, define

$R_B$ measures the fraction of the displayed base-to-RMU accuracy gap reversed by the edit. It is not a natural direct or indirect effect: there is no identified mediation model, no intervention that holds other causal paths fixed, and the edited network is outside the original training process. It is also not the fraction of facts or information recovered.

4.6 Exploratory SAE attribution

Following the relative-gradient analysis introduced by Lee, Breck, and Arditi for tracing features upstream of a refusal direction [Lee et al., 2025], we adapt the target metric from refusal to the RMU-derived shared direction. Let $J(x)$ be the projection of the block-9, position-$-5$ residual stream onto that direction. For each prompt, differentiate $J(x)$ with respect to the earlier block-5 residual stream and average the gradients across prompts and token positions. The resulting vector $\mathbf g_5$ is compared by cosine similarity with decoder vectors from the Gemma Scope layer-5 16k SAE:

This computation belongs to the broader family of gradient-informed SAE attribution methods; GradSAE later formalized the use of output-side gradients to rank influential latents [Shu et al., 2025]. Our score is not GradSAE’s complete algorithm: it uses cosine alignment with an intermediate-direction gradient and does not multiply by each latent’s observed activation. It is therefore best understood as a local sensitivity analysis. A high score means an SAE decoder direction aligns with the gradient under the evaluated prompts; it does not show that the feature activated, uniquely generated the shared direction, or is causally necessary.

5. Results

5.1 The shared edit recovers WMDP accuracy

Using the values in the accuracy table, the shared edit recovers

and

The random control changes Bio and Cyber accuracy by $-0.39$ and $-0.10$ percentage points relative to RMU. The large recovery is therefore specific to the extracted direction under this control.

The edit also raises MMLU from 43.35% to 47.34%, recovering 62.5% of RMU’s displayed MMLU deficit. This is important counterevidence to a domain-specific interpretation: the direction carries a substantial portion of RMU’s general capability degradation.

5.2 Other extracted directions

The domain-specific directions recover most strongly on their corresponding WMDP subsets, while the direction extracted from general harmful prompts is much weaker. These comparisons are descriptive. One analysis notebook reconstructs a Cyber artifact using Bio-selected indices, and the repository does not include the serialized direction artifacts needed to resolve which saved checkpoint produced every row. The shared-direction result is therefore the primary result; the remaining rows should be treated as supporting exploratory evidence.

5.3 Exploratory SAE follow-up

Gradient–SAE alignment ranks several topic-related decoder directions, but it does not identify a causal circuit. After the shared-direction edit, a follow-up edit removes the average of decoder vectors 8085 and 16213. WMDP-Bio rises from 50.59% to 51.68%, WMDP-Cyber from 32.16% to 33.71%, and MMLU from 47.34% to 48.08%.

These small gains occur on both forget and utility benchmarks. Without random-feature controls, repeated seeds, or correction for selecting features from the same analysis, they do not establish that the two features encode a residual unlearning mechanism. Appendix C preserves the plots, the interactive SAE dashboard, and the full evidential interpretation.

6. Discussion

6.1 What the intervention establishes

The experiment supplies a causal perturbation result: applying a specified rank-one edit to the RMU checkpoint changes its measured behavior, while a same-norm random orthogonal edit does not. A direction derived from RMU-minus-base activation differences is therefore functionally important to the evaluated checkpoint.

The result is stronger than a correlational probe because the direction is manipulated. It is weaker than a causal mediation analysis because the intervention is global, alters the model at every layer, and does not isolate a path through block 9. Block 9 is the source from which the direction was extracted; it is not the only site at which the benchmark intervention acts.

6.2 What it does not establish

Recovery of multiple-choice accuracy does not measure how many facts remain in the weights. The edit might restore answer selection without restoring open-ended explanation, or it might remove a broadly disruptive component that had obscured capabilities already present. Conversely, failure to recover the entire gap would not prove that the remainder was deeply erased. It could reflect additional low-rank directions, nonlinear suppression, damage unrelated to the forget set, or limitations of the attack.

This is why the earlier decomposition into “total,” “indirect,” and “direct” causal effects is not retained. The unrecovered 24.6% of the Bio gap and 33.9% of the Cyber gap are simply portions not recovered by this intervention.

6.3 Implications for RMU evaluation

The shared edit recovers both WMDP and MMLU performance. This pattern is consistent with an RMU checkpoint that relies partly on a low-dimensional disruptive mechanism, but it complicates the label “unlearning direction.” A more precise name is RMU-induced recovery direction: it identifies a direction whose removal reverses observed effects without presupposing what information the direction represents.

Evaluation should therefore include multiple output formats, minimally informative recovery attacks, retain-task behavior, and controls matched in intervention strength. Recent evaluation work reaches the same general conclusion from different experiments: no single benchmark interface reliably distinguishes deletion from suppression [Feng et al., 2025].

7. Limitations and Reproducibility

This study has several limitations that materially constrain its conclusions:

• One checkpoint: all central results use one Gemma-2-2B RMU run with one training seed.
• One evaluation interface: recovery is measured with zero-shot multiple-choice accuracy. There is no open-ended, paraphrase, or cross-format evaluation.
• No uncertainty for the paired edit: the public repository does not contain the per-example predictions needed for paired bootstrap confidence intervals or exact comparison tests.
• Probe leakage: the best probe block was selected using test AUC, so the displayed ROC-AUC is exploratory.
• Global intervention: orthogonalization acts across embeddings and all transformer blocks, preventing layer-local causal attribution.
• Artifact gaps: notebooks refer to uncommitted files under /home/ubuntu/thesis/..., including datasets, learned directions, probe weights, and evaluation outputs. The public code describes the pipeline but does not reproduce every table from a clean checkout.
• Notebook inconsistency: one reconstruction cell indexes the Cyber direction with Bio-selected indices. The central shared-direction result does not depend on interpreting that cell, but domain-specific comparisons remain less secure.
• SAE validity: the Gemma Scope SAE was trained on the base-model distribution and is applied to edited-model gradients without reporting reconstruction error. Feature descriptions are post-hoc interpretations, and the two-feature edit lacks matched controls.

The most valuable follow-up is not a more elaborate claim from the current artifacts. It is a clean rerun that publishes per-example predictions, direction tensors, normalized candidate-selection scores, configuration snapshots, and multiple RMU seeds across model sizes.

8. Future Work

The original thesis proposed adversarially training RMU against whichever low-dimensional direction most strongly induces high-entropy output. Appendix D develops that idea as an untested research proposal and identifies a central problem: entropy maximization need not find the direction that makes unlearning recoverable. A defensible experiment would compare the method with Adaptive RMU [Huu-Tien et al., 2025], random-perturbation RMU [Huu-Tien et al., 2025], latent-adversarial-training RMU [Sheshadri et al., 2025], and other relearning-resistant objectives under a fixed attack budget.

The current result also motivates replacing a single direction with a learned low-rank subspace and testing rank–recovery curves. If recovery saturates at small rank across seeds and models, that would support a broader claim of low-dimensional suppression. If it does not, this checkpoint should remain a specific counterexample rather than a general theory of RMU.

9. Conclusion

In one Gemma-2-2B checkpoint, a shared direction extracted from RMU-induced activation changes supports a global rank-one edit that recovers 75.4% of the displayed WMDP-Bio accuracy gap and 66.1% of the WMDP-Cyber gap. A matched random direction does not recover performance. This makes the selected direction functionally important to the checkpoint’s behavior.

The same edit recovers broad MMLU performance, and the experiment is limited to multiple-choice evaluation. The result should therefore be read as evidence that much of this checkpoint’s measured degradation is low-rank and reversible—not as a measurement of retained knowledge, a causal mediation decomposition, or proof that the remaining gap represents deeper erasure.

Acknowledgements

We thank Andy Arditi for discussions about high-entropy output distributions and refusal directions in RMU, Long Phan for data support and the baseline RMU implementation, and Professor Le Hoai Bac for guidance throughout the original thesis. Special thanks go to the author’s parents for the material and emotional support that made the work possible.

Citation

1
@misc{nguyen2025rankone,
2
  author = {Nguyen, Le},
3
  title = {A Rank-One Representation Edit Recovers Much of RMU's WMDP Accuracy Loss},
4
  year = {2025},
5
  url = {https://lenguyen.vercel.app/research/rmu-improv}
6
}

References

Appendices

Appendix A. The exploratory probe and why it was not a result

The probe was an early attempt to turn the visible failure mode of RMU generations into a scalar search signal. It was useful operationally: instead of evaluating every candidate direction with a full benchmark run, the pipeline could ask whether adding a candidate to the base model induced output resembling the RMU failure mode and whether removing it from the RMU model reduced that resemblance. This role explains why the probe remains in the method even though it is not part of the paper’s central evidence.

Several properties prevent a stronger interpretation:

1. The target is an annotation convention. “Junk” combines repetition, incoherence, contradiction, and incompleteness. Those behaviors can have different causes, and none is logically equivalent to forgetting.
2. The labels are model-generated. Gemini-2.5-Flash supplied the labels. No independent human annotation, inter-rater agreement study, or sensitivity analysis establishes how often a different judge would agree.
3. The data are imbalanced. Of 2,000 outputs, 1,598 are labeled junk and 402 non-junk. Weighted resampling changes the training distribution, while ROC-AUC summarizes ranking rather than calibration at the operating point used in candidate selection.
4. Block selection uses the test split. The 0.708 test ROC-AUC at block 12 is the maximum found by comparing blocks on the nominal test set. It is therefore a post-selection descriptive statistic, not an unbiased estimate on untouched data.
5. The probe is checkpoint-specific. It would need retraining and revalidation for another model, unlearning run, decoding configuration, or failure mode.
6. The proxy and endpoint differ. Candidate selection optimizes a linear score on residual activations; the reported result is multiple-choice accuracy after a global weight edit. Success on the latter validates the chosen edit empirically, but it does not retroactively validate the probe as a semantic detector.

The probe should consequently be read as a fallible search heuristic. A cleaner rerun would preregister the labeling rubric, use multiple blinded human or model judges, reserve a final untouched test set, report calibration and uncertainty, and test whether candidate rankings remain stable when the probe architecture or label source changes.

Appendix B. Alternative directions, negative results, and notebook artifacts

The direction search included Bio-only, Cyber-only, shared, and out-of-distribution (OOD) harmful-prompt candidates. The OOD pool combined prompts from AdvBench [Zou et al., 2023], MaliciousInstruct [Huang et al., 2023], and HarmBench [Mazeika et al., 2024]. The selected candidates had the following recorded diagnostics:

The OOD direction is the clearest negative result. It transfers weakly to WMDP, produces a much smaller addition score, and has substantially higher base-model KL divergence. General harmfulness is therefore not an adequate substitute for directions extracted from the RMU target domains in this experiment. This could reflect domain mismatch, prompt-format mismatch, direction magnitude, or an overly restrictive rank-one model; the experiment does not distinguish among them.

The domain-specific rows are also less clean than the shared result. Their benchmark pattern is intuitively plausible—Bio performs best on Bio and Cyber performs best on Cyber—but one notebook reconstructs the Cyber candidate using indices selected for Bio. In a separate exploratory notebook, the expression bio_diff + cyber_diff / 2 appears where the intended average is (bio_diff + cyber_diff) / 2. The primary shared-direction notebook uses the parenthesized average, but these discrepancies mean that exploratory cells should not be treated as a canonical executable record.

Matched random-direction edits provide the most useful negative control:

The controls show that an arbitrary orthogonal rank-one edit does not reproduce the recovery. They do not measure variability across random controls because the record contains one matched draw per candidate. A stronger analysis would evaluate a distribution of random directions, normalize all candidate directions before the addition diagnostic, preserve per-example predictions, and rerun every row from immutable serialized artifacts.

Appendix C. What the SAE visualization shows

The SAE analysis asks a narrower question than “which feature stores the forgotten knowledge?” Adapting the relative-gradient procedure of Lee, Breck, and Arditi [Lee et al., 2025], it differentiates the selected block-9 shared-direction projection with respect to the block-5 residual stream. It then compares the average gradient with decoder directions from the Gemma Scope layer-5 16k sparse autoencoder [Lieberum et al., 2024]. The resulting cosine similarity is a geometric alignment score between a local sensitivity direction and an SAE dictionary vector.

Feature 3484 can be inspected in the embedded Neuronpedia page below. This is the HTML figure from the original article: an external interactive dashboard rather than an image generated by this experiment. It exposes information such as highly activating text examples, activation plots, and an automatically generated feature explanation. Those views help a reader audit why a feature received a human-readable topic label. They do not establish that the label is complete, that the feature fires in the present prompts, or that the feature causes the RMU behavior.

Open feature 3484 directly on Neuronpedia if the embedded dashboard does not load.

After the shared rank-one edit, the ranking changes and features 8085 and 16213 appear among the remaining biology-related candidates.

The follow-up intervention averages the decoder vectors for both features,

and applies another global orthogonalization after the shared-direction edit. This detail matters because the earlier prose described feature 16213 alone, while the saved code combines two decoder vectors. The resulting gains—1.09 points on Bio, 1.55 on Cyber, and 0.74 on MMLU—are compatible with a weak residual recovery effect, generic model repair, or selection noise. No random-feature baseline, resampled feature selection, repeated model seed, or edited-model SAE reconstruction measurement separates these explanations.

Appendix D. Proposed attacker–defender training mechanism

D.1 Original proposal

The motivating hypothesis is that RMU can satisfy its forget objective through a “lazy” low-dimensional mechanism: concentrate much of the visible disruption in one direction, leaving behavior recoverable when that direction is removed. An inner attacker searches for such a direction, and an outer defender discourages the current RMU model from writing forget-set activations into it.

Let $\phi_L(x_{1:t};\hat\theta)$ be the residual representation at block $L$ and token $t$ in the current model, and let $g_L$ denote the remainder of the network from that representation to the vocabulary logits. For a unit vector $\mathbf r$ and intervention strength $\alpha$, define

The proposed attacker finds a direction that makes the output distribution close to uniform over the $K$-token vocabulary:

Minimizing this KL divergence is equivalent to maximizing token entropy up to the constant $\log K$. The motivation follows evidence that RMU’s random-target steering can reduce token confidence and induce incoherent output [Yuan et al., 2025].

Given the attacker’s direction, the proposed defender penalty is

with the modified forget objective

The original RMU loss remains necessary: $\mathcal L_{\mathrm{def}}$ only says where forget activations should not lie and supplies no positive target for unlearning.

D.2 Why the original objective may fail

The most important unresolved issue is that the attacker’s objective and the desired robustness property are different. High output entropy locates a direction that flattens a token distribution. It need not locate a direction whose removal restores WMDP performance, nor the direction through which RMU concentrates its effect. An entropy attacker can therefore win its inner objective while training the defender against the wrong vulnerability.

There is also a possible direct conflict with RMU. RMU explicitly pushes forget activations toward a fixed random target $c\mathbf u$. If the learned $\mathbf r^*$ aligns with $\mathbf u$, the defender penalizes the same projection that the forget loss is trying to create. Depending on $\lambda_{mathrm{def}}$, optimization may weaken unlearning rather than force a more distributed mechanism.

Even when the losses do not conflict, orthogonality to one direction is easy to evade. The model may rotate disruption into another direction, spread it over a small subspace, change activation norms, or implement a nonlinear downstream gate. Raw residual-stream orthogonality also does not imply functional independence because later layers can reconstruct or amplify related components.

Finally, the proposal is a bilevel optimization problem. The attacker must be refreshed as $\hat\theta$ changes. Stopping gradients through $\mathbf r^*$ gives a practical alternating algorithm but ignores how the optimizer’s solution changes with model parameters; differentiating through the inner solve is more faithful and substantially more expensive. Multiple random restarts are needed because the attacker’s landscape is non-convex.

D.3 A stronger experimental version

A first implementation could alternate the following steps:

1. Sample forget and retain minibatches.
2. Starting from several random unit vectors, take $K$ projected-gradient steps on the attacker objective and renormalize each vector after every step.
3. Select the strongest attacker and freeze it for the outer update.
4. Update $\hat\theta$ on the RMU retain and forget losses plus $\lambda_{mathrm{def}}\mathcal L_{\mathrm{def}}$.
5. Periodically discard stale attackers and restart the inner search against the current checkpoint.

A closer match to the paper’s failure mode would replace the entropy attacker with a recovery attacker. Rather than adding a direction that produces incoherence, the inner problem would search for a direction or rank-$k$ subspace whose removal most improves a held-out recovery objective after the current RMU update. For an orthonormal basis $R\in\mathbb R^{d\times k}$ with $R^{\mathsf T}R=I$, the intervention projector is

The defender would minimize the worst recovery attainable under a bounded family of such edits while retaining the original unlearning and utility constraints. This formulation targets low-rank recoverability directly, but it is computationally expensive and risks leaking benchmark-answer information if WMDP labels drive the attacker. A separate proxy corpus and a strictly held-out final evaluation would be required.

D.4 Falsifiable evaluation plan

The proposal should count as successful only if it improves robustness without merely lowering every capability score. At minimum, the evaluation should report:

• WMDP Bio and Cyber accuracy before and after rank-$1$ through rank-$k$ recovery attacks;
• open-ended, paraphrased, and cross-format hazardous-knowledge evaluations, not only multiple-choice questions;
• MMLU and additional retain tasks before and after each attack;
• distributions over random-direction and random-subspace controls;
• fine-tuning and relearning attacks under a fixed compute and data budget;
• several RMU training seeds, attacker restarts, model sizes, and confidence intervals from per-example predictions;
• comparison with standard RMU, Adaptive RMU, random-perturbation RMU, and other robustness-oriented objectives.

The key falsifier is straightforward: if a new low-rank direction can be extracted after each defender update and achieves similar recovery, the mechanism has rotated the vulnerability rather than removed it. If low-rank recovery falls while open-ended hazardous capability and retain performance remain unchanged, the proposal would provide evidence—not proof—that the learned suppression mechanism has become harder to reverse.