What really runs on our cluster, read from the scripts themselves

A cluster’s job history is usually a wall of six-digit identifiers and cryptic names —mcpfw_cc, eval_dynamic_pid, boltz2— that say everything to whoever launched them and nothing to anyone else. But if you keep the scripts, the wall turns into a document: people write down why they do things, especially when a batch costs money and a night of waiting.

We have gone through four months of the compute cluster of the ANTS group, from the Department of Information and Communications Engineering of the Faculty of Computer Science at the University of Murcia —around 7,700 jobs from 19 users, reconstructed from the sbatch files themselves— and we tell the story anonymised: no names, no internal paths, looking at patterns rather than people. This is what a small research cluster looks like when you read it backwards.

Two meters, two stories #

If it had to be summed up in one sentence: the cluster today is mostly about language models, and mostly about testing them, not training them from scratch. Of the seven lines of work that share the machine, most touch LLMs: safety fine-tuning, code and function-calling benchmarks, automated attacks, a model for phones and inference serving. Of serious pretraining, nothing; the closest thing is LoRA fine-tunes and a great deal of evaluation. That is what a university GPU cluster looks like in 2026: not a training furnace but a measurement laboratory for models trained elsewhere.

But the counters tell two stories depending on which one you look at:

• GPU-hours (~3,000): dominated by the benchmarking of quantized models (~37%); then proteins (~15%), self-alignment, forests and attacks (10 to 12% each) and the model for phones (~9%). The GPUs belong to the LLMs, but above all to measuring them.
• CPU-hours (~30,000): split differently: LiDAR forests (~27%) and benchmarking (~25%) in front, followed by the mobile classifier (~19%) and the attacks (~12%). Whoever reserves the most GPU is not whoever asks for the most CPU.

The GPUs: defending and attacking the same model #

The GPU side is almost a self-contained debate on LLM safety, with different users in different roles.

The defence. An SFT + LoRA study on “self-alignment”, evaluating each checkpoint with HarmBench, the standard benchmark of harmful behaviours. All wired up as a dependency graph: train, and on finishing fire off an array of evaluations with --dependency=afterok, saving the outputs separately so they can be re-judged later without retraining.

The attack. Much smaller, but striking: an automated, multi-agent jailbreak framework that attacks a 120B open model across dozens of targets and turns, with the official HarmBench classifier as judge. The whole set —target, agents and judge— is stacked onto a single 141 GB H200, because each GPU node has exactly one card.

The measurement. Serving quantized GGUF models with llama.cpp and scoring them on BFCL (function calling), BigCodeBench and RULER (long context). Client and server on the same node, after discarding SSH tunnels because they kept failing.

So one user fine-tunes models to make them safer and scores them with HarmBench; another automatically attacks a large model and scores the attacks with the same HarmBench classifier. The two halves of the same question, on the same nodes.

The long tail: proteins, trees, teeth and lagoons #

The remaining lines are smaller, but each one is a clean, single-topic tenant:

• Protein folding: Boltz-2 , a diffusion model from the AlphaFold family, over ~2,500 protein pairs. Only a handful of jobs, but huge and resumable: a five-day reservation that skips the pairs already done.
• Forests from point clouds: above-ground biomass estimation from LiDAR with Pointcept (PTv3, OACNN), segmenting individual trees.
• Medical imaging: segmentation with UNet++, DeepLabV3+ and SegFormer, including a 100-seed array to measure run-to-run variance.
• A model for phones: a risk classifier for minors’ conversations, a small LLM exported to LiteRT-LM int4/int8 for phones, plus a “main” model on an H100.
• And, around the edges, spatiotemporal GNNs predicting chlorophyll in a lagoon from satellite imagery, an AutoML for intrusion detection pilot, and single-cell genomics.

What the scripts teach about sharing a machine #

Beyond what gets computed, the scripts are a quiet guide to being a good tenant. Several patterns repeat across users who have nothing to do with each other:

• Ask for less, start sooner. A comment dated the day someone discovered it: lowering the request from 4 CPUs / 8 GB to 1 CPU / 2 GB took a job from “wait for days” to “start in seconds”, because the small one fitted into a backfill gap on a node with 28 of 30 cores busy.
• Stage on local scratch and sync home at the end, inside a cleanup trap, so as not to hammer the NFS during the run and so that a dead job still collects its results.
• No preemption: people use a high --nice and honest walltimes. As one script sums it up, what really frees a GPU for your neighbour is not priority, it is finishing early.
• Notifications via ntfy and email: you enqueue and forget.

And my favourite, a security decision hidden in a benchmarking script. The team measuring code models needs the model to write code; their scripts are blunt: the cluster only generates text, it never executes the code the model writes. Grading —which does run that code— happens afterwards, in an isolated container on a local machine. One comment even corrects an earlier comment that had accepted the unsafe route, warning that misreading it “would have put the isolation in the wrong place”. Thinking clearly about where untrusted code is allowed to run, on a shared machine, is exactly the right instinct.

Conclusion #

A cluster’s job history, read calmly, is a portrait of what a community cares about. That of the ANTS group, in mid-2026, cares about language models —making them safer, breaking them on purpose, measuring what they can do and shrinking them down to a phone—, and proteins, forests and teeth around the edges.

None of that shows in the job names. It is in the scripts. Keep the scripts.

The series, line by line #

1. Self-alignment of a 3B LLM with LoRA and HarmBench : 2,600 jobs to fine-tune and measure safety.
2. A council of small models against a 120B one : automated jailbreaking on a single H200.
3. Measuring quantized LLMs : 300 GGUFs, three benchmarks and one security rule. Part of the results is published at Yardstick .
4. 4,400 protein complexes with Boltz-2 : fifteen jobs and a factor of ten.
5. Trees one by one from LiDAR point clouds : three architectures, ten seeds and two nodes.
6. Dental segmentation and 100 seeds : measuring variance before comparing.
7. A risk classifier for minors that fits in a phone : synthetic data, LoRA and LiteRT-LM. Code on GitHub .
8. GNNs for a lagoon’s chlorophyll , and an AutoML pilot.
9. Serving models and looking after the cluster : shared inference, burn-in and probes.

Figures reconstructed from Slurm accounting and the recovered sbatch files; GPU/CPU hours reflect reserved capacity (elapsed × allocated), not measured usage, and the intent of each job is inferred from the script bodies and their comments. One orphan record that appeared as RUNNING since May and inflated the GPU total by some 2,500 hours is excluded, and for five jobs that hung some 18 hours past their limit (Slurm closed them all at once the next day) the reservation is counted up to the requested limit, not to the close; the figures in the first version of this post included both. At the request of one of the group’s members, their line of work and all their jobs have been removed from this post, from the figures and from the charts. Anonymised post: no identifiable users, paths, emails or project names.