Self-alignment: how much safety to teach a language model without making it useless

First instalment of Cluster X-ray, the series that goes line by line through what really runs on the ANTS group’s cluster . We start with the first of three lines on language model safety.

The question #

Fine-tuning a language model so that it refuses harmful requests has a cost: if it is taught too thoroughly to say no, it starts refusing harmless things too, and loses usefulness. The line studies that balance: what proportion of safety data has to be mixed with the utility data, and whether the model itself can generate its own refusal examples and regulate the dose on its own.

How it is approached #

The model is Llama 3.2 3B Instruct, fine-tuned with SFT and LoRA. Utility comes from Alpaca (or MetaMathQA when the domain is mathematical); safety from public sets such as BeaverTails, HH-RLHF harmless and OR-Bench, mixed in at 15% or 5% with several seeds. Each training run saves some twenty checkpoints, and each one is evaluated separately to trace the full curve, not just the endpoint.

The experiment families follow the logic of the question:

• Fixed mixtures. Alpaca alone; Alpaca with 15% of each safety set; MetaMath as a different domain.
• Canned responses. Same harmful prompts, but the response is a fixed refusal or a paraphrase from a pool of fifteen. It isolates whether what matters is the content of the response or whether it is enough for it to be a refusal.
• Self-alignment. The refusal pool is generated by Llama itself with best-of-4 and a pairwise judge: some 14,000 prompts yield 10,300 examples. A variant adds 25% of helpful responses generated on OR-Bench, to fight over-refusal from the data side.
• Dynamic regime. The safety proportion is not fixed: it is adjusted in rounds according to the attack success rate measured on a held-out set of 256 prompts. Three controllers (PID, bandit and dead band) are compared with the equivalent fixed mixture.

Evaluation combines HarmBench (attack success rate), XSTest (over-refusal), perplexity or GSM8K (utility) and, at the final checkpoint, IFEval, ARC and MMLU. The responses from each evaluation are stored separately from the adapters, and that made it possible, at the end, to re-judge them with an LLM judge and three different prompts without retraining anything.

What is learned #

The line builds a utility-safety Pareto front per experiment, with the unadapted base model as anchor. The questions it answers are the ones you read between the lines of the scripts: whether a refusal pool generated by the model itself is worth as much as a hand-annotated one; whether a controller that reacts to the attack rate achieves the same safety with less data; and how much the judge weighs, because the same checkpoint changes its score depending on who judges it. On the last day of the campaign, the principle that closes that drift was written down: “sensor↔test judge consistency… a single source of truth”.

On the cluster #

Each training run fires off an array of evaluations, one per checkpoint, and when those finish, an aggregation. That is why the line is second in number of jobs with relatively few GPU-hours: the 3B with LoRA trains in little over an hour on an L40S, and each evaluation takes a few minutes.

The second instalment is the other half of the same question: attacking the model instead of defending it.

Figures from Slurm accounting (reserved capacity, not measured usage) and the archived sbatch files. Anonymised post: no identifiable users, paths, emails or project names. Quotations are from the script comments.