· 4 min read · Gaia Lab
A council of small models against a 120B one: automated jailbreaking and who judges it
Several small open models deliberate to build prompts that make gpt-oss-120b give in. The line measures how much deliberating adds over attacking alone, and devotes as much effort to the uncomfortable question of who judges success as to the attack itself.
Second instalment of Cluster X-ray. The previous instalment fine-tuned a model so it would refuse what is harmful. This line does the opposite: it automatically attacks a large model and counts how many times it gives in.
The question #
Automated jailbreak attacks usually use an attacker model that refines a prompt turn by turn. The line asks whether a group of small models that deliberate attacks better than any of them on its own, and, with equal weight, how to measure success without fooling yourself, because the same attack can look successful or not depending on who judges it.
How it is approached #
On top of the public ARES framework a council is built: Qwen3 14B, Gemma 3 12B and DeepSeek-R1 32B propose and critique prompts, a chair (Qwen2.5 32B) aggregates the proposals by Borda count, and an in-the-loop judge scores the harm of the response from 0 to 5. The main target is gpt-oss 120B; gpt-oss 20B, Qwen3 8B and Llama 3.1 70B are also evaluated. Each target gets up to twenty turns.
The study is organised into ablations and controls:
- What each piece adds. Full council versus council without chair, without Borda or with pure Borda, over fifty representative HarmBench behaviours, and then over the full pool of 271 copyright-free behaviours.
- Deliberating versus attacking alone. Each member and the chair attack solo with the same surface, without aggregation. It is the control that says whether the council is worth what it costs.
- Baselines from the literature: Crescendo, encodings, direct request, human jailbreaks, FITD and PAPILLON.
And the measurement part. The judge guiding the attack is gpt-oss itself, which hides a circularity: target and judge are the same model. To break it, all the saved responses are re-scored with the official HarmBench classifier (a 13B Llama 2 trained for the purpose) as an independent test judge, and then with a third judge from another family (Mistral). The comment that sums it up: “J_search = 0-5 judge (guides the attack), J_test = independent HarmBench classifier”. A baseline that had failed because of an infrastructure error was rerun in full before comparing.
What is learned #
The scripts record the figure that organises the study: the council reaches a success rate of around 73% over the 271 behaviours with four attempts, and the solo controls exist precisely to know what part of that is deliberation and what part is simply insisting for twenty turns. The other lesson is methodological and appears written as a reply to a reviewer: the choice of judge moves the success rate, so a rate without saying who judges means nothing. Hence the three judges.
On the cluster #
Target, council, chair and judge are served at the same time with Ollama on the same card, some 102 GB of models, and that is why the line lives on the 141 GB H200 NVL: the 94 GB H100 cannot fit the 120B together with the council. The later judgements, with the 13B classifier, fit on any free GPU.
In the third instalment , the line that reserves the most GPU-hours: measuring quantized models.
Figures from Slurm accounting (reserved capacity, not measured usage) and the archived sbatch files. Anonymised post: no identifiable users, paths, emails or project names. Quotations are from the script comments.