hardware-failure on Antonio Space

Predicting Disk Failures with a Small LLM, Part 2: Fixing the Evaluation

Fri, 16 Oct 2026 20:00:00 +0800

In Part 1 I fine-tuned Qwen2.5-1.5B-Instruct with 4-bit QLoRA on my RTX 3060 to answer one question: given seven days of SMART telemetry from a Toshiba MG07ACA14TA, will this drive fail in the next seven days? Training worked, the loss dropped, and the model returned clean JSON. It also predicted “no failure” for all 168 test drives and missed every one of the 18 failures.

I left one question open. My evaluation script rejected every answer from the original model, so I had no real baseline to compare against. Before training anything new, I wanted to close that gap. Changing the training data without a trustworthy evaluation would just give me a second number I couldn’t interpret.

The parser bug

The original model did not return bare JSON. It wrapped its answer in a Markdown code block:

```json
{
  "failure_within_7_days": false
}
```

My parser called json.loads on the raw string, failed, and recorded predicted: null. All 168 base-model answers were scored as invalid, which made the original model look like it had no classification ability at all. That was a measurement artifact, not a property of the model.

The fix is small: strip an optional code fence, then parse. The decision it changes is not small. “The base model can’t follow the format” and “the base model follows the format but always says no” lead to very different next steps.

import json, re

FENCE = re.compile(r"^```(?:json)?\s*(.*?)\s*```$", re.DOTALL)

def parse(raw: str):
    text = raw.strip()
    m = FENCE.match(text)
    if m:
        text = m.group(1)
    try:
        return bool(json.loads(text)["failure_within_7_days"])
    except (json.JSONDecodeError, KeyError, TypeError):
        return None

I re-scored the saved predictions in output/base-predictions.json without calling the model again.

Corrected results

With the fence stripped, all 168 base-model answers parse. They are also all identical: the same fenced JSON saying false.

Model	Valid answers	TN	FN	Accuracy	Failure recall
Qwen2.5-1.5B-Instruct (original)	168 / 168 (fenced)	150	18	89.3%	0%
Fine-tuned v1 (QLoRA)	168 / 168 (bare)	150	18	89.3%	0%
Always “no failure”	n/a	150	18	89.3%	0%

The three rows are the same. On this test set, fine-tuning changed the format of the answer and nothing about the prediction. Precision is undefined for all three, because none of them ever raises an alert.

Accuracy is the misleading number here. 150 of the 168 test examples are healthy drives, so a model that never alerts scores 89.3% without learning anything. And because I sampled healthy drives when building the dataset, about 11% of test examples are failures, far more than in the real fleet. Accuracy on the real fleet would look even better for a model that never alerts, and mean even less.

What the training log says

The v1 loss curve fits that picture. Loss was 1.08 at step 5, 0.61 at step 10 and 0.024 by step 20, then stayed flat around 0.02 for the remaining 68 of 88 steps. With 700 healthy and 79 failed drives in train.jsonl, answering false is right about 90% of the time. The output is also a short, fixed JSON string, so with response-only loss there are only a few tokens to learn. Low loss tells me the model learned the answer format and the majority class quickly. It does not tell me the model learned anything about SMART counters.

A note on the second run

I had already prepared a balanced training set, data/train-balanced.jsonl, with 79 failed and 79 healthy drives. My first attempt to train on it never started. Its log is a single line: the shell could not find the soup binary, because the command used an old workspace path that pointed to an empty copy left over from a migration. No v2 adapter exists yet, so there are no v2 results in this post.

What comes next

Balanced training (v2). A fresh adapter from the original base model, trained for three epochs on the balanced set. Balancing removes the shortcut of always answering no, but it is not guaranteed to make the model learn real signal. It may simply trade missed failures for false alerts.
Validation, not test. I’ll compare v2 on val.jsonl and leave test.jsonl alone until I’ve picked a model. Re-scoring every idea on the test set would slowly turn it into a training signal.
Precision, recall and false alerts. For an operator, the useful questions are how many failing drives the model catches and how many healthy drives it flags per week. A model that alerts on everything has perfect recall and is just as useless as one that never alerts.
A tabular baseline. Seven days of seven SMART counters is a small table. XGBoost or LightGBM on the same features and splits will show whether a language model adds anything here.
A future period. My splits are random by drive, not chronological. Before drawing operational conclusions, I need to train on one period and evaluate on a later one, such as Q2 2025.

The lesson from Part 1 still holds, now with a corrected baseline behind it: the pipeline runs on consumer hardware, but a working pipeline is not yet a working predictor.

Predicting Disk Failures with a Small LLM, Part 1: A Working Pipeline and a Useless Model

Thu, 15 Oct 2026 20:00:00 +0800

Most hardware I have operated tells me only part of what is happening inside it. A disk reports a handful of counters. A switch reports interface errors. A GPU reports temperature and a few error codes. The interesting physics, firmware decisions and wear sit underneath, and I mostly see their side effects.

I wanted to know how far those side effects can take you. Can a model look at a week of a drive’s self-reported telemetry and say whether it will fail soon? And can a small language model, fine-tuned on my own GPU, learn to do that?

This post covers the first attempt. The pipeline works end to end. The model it produced does not predict anything useful. Both results are worth writing down.

Why disk failure is hard to predict

A drive’s failure depends on things the telemetry only partly captures:

Workload. The same drive model behaves differently under a backup archive than under a busy database.
Hardware design and firmware. Vendors decide which events to count, when to remap a sector, and what each SMART attribute means. The same attribute number can carry different semantics across vendors.
Age. Wear accumulates, but early-life failures also exist.
Temperature and operating conditions. Heat, vibration, power events and the chassis a drive sits in all matter, and only some of it is recorded.

It also matters what kind of device we are talking about.

Mechanical hard drives have spinning platters, moving heads and motors. Many failures develop gradually: the surface degrades, sectors become hard to read, and the drive remaps them. Counters such as reallocated and pending sectors can rise before the drive dies. Some failures, such as a head crash or a motor or electronics fault, are sudden and may leave little trace in SMART beforehand.

Semiconductor devices such as SSDs have no moving parts. NAND flash wears with program/erase cycles, which drives expose through wear and spare-block indicators. But controller or firmware faults can take a device down abruptly, and the attributes that matter are different and often vendor-specific. A model trained on hard drive counters says nothing about SSDs.

This series stays with mechanical drives, because that is where the public data is deepest.

The data: Backblaze Drive Stats

Backblaze publishes daily snapshots of every drive in its data centers: model, serial number, capacity, a failure flag, and raw and normalized SMART values. I used Q1 2025, which is 90 daily CSV files in one archive (data_Q1_2025.zip, about 1 GB).

Mixing drive models would mix vendors’ SMART semantics, so I picked one model: the Toshiba MG07ACA14TA, a 14 TB drive with a large population in the fleet. Filtering the quarter to that model gave:

3,385,931 daily observations
133 unique drives that failed during the quarter

From each daily row I kept seven raw SMART fields:

SMART ID	Common meaning
5	Reallocated sectors count
9	Power-on hours
194	Temperature
196	Reallocation event count
197	Current pending sector count
198	Offline uncorrectable sectors
199	UDMA CRC error count

Most of these describe the media surface, plus age, heat and cabling errors. That is a narrow window into the drive, and it is deliberate. I wanted to start with the counters people usually watch.

Framing the prediction task

The question I gave the model:

Given seven consecutive daily observations of one drive, will it fail during the following seven days?

The answer is a single JSON object:

{"failure_within_7_days": true}

How the examples are built matters more than the model, so I was strict about it:

One example per drive. A drive that contributes many overlapping windows would dominate training and could leak between splits.
Positive examples end one to seven days before the recorded failure. The history never includes the failure day itself, so the model cannot see the event it is supposed to predict.
Negative examples need a complete seven-day follow-up without failure. A drive that disappears from the data three days later is not a confirmed healthy example.
Histories with missing selected SMART values are rejected. I would rather have fewer examples than teach the model to read gaps.
Serial numbers and failure flags are excluded from the input. Either would let the model cheat.
Splits are by drive, stratified, and random. No drive appears in more than one split. The split is not chronological, which I will come back to.
Healthy drives are sampled. Using every healthy drive would make failures vanishingly rare. The price is that my test set has far more failures than the real fleet, so its accuracy numbers do not transfer to production.

The examples are stored as JSONL in Alpaca format, with instruction, input and output fields.

File	Examples	Failures
`train.jsonl`	779	79
`val.jsonl`	167	17
`test.jsonl`	168	18

That is 114 usable failed drives out of 133. The rest were lost to the missing-value and window rules.

The local setup

Everything ran on my Linux box:

NVIDIA RTX 3060 with 12 GB VRAM (11.63 GiB usable), 16 GB RAM
Python 3.12.3, PyTorch 2.14.1+cu130
Soup 0.75.2 (the soup-cli package), which wraps Transformers, PEFT and TRL behind one YAML config
Workspace and Hugging Face cache on /srv/storage, so model downloads don’t fill the system disk:

export HF_HOME=/srv/storage/backblaze-soup/models/huggingface
soup doctor

soup doctor checks the Python, CUDA and library versions before you spend time on a run. It is worth running after every environment change. I learned that later, the hard way.

Fine-tuning Qwen2.5-1.5B with QLoRA

I chose Qwen2.5-1.5B-Instruct: small enough to train comfortably on a 3060, and already good at following a “reply with JSON” instruction.

Full fine-tuning of 1.5 billion parameters does not fit in 12 GB. QLoRA makes it fit. The base weights are loaded in 4-bit and frozen, and only small low-rank adapter matrices are trained. The adapter is a few megabytes, and the original model stays untouched.

This is the whole configuration:

base: Qwen/Qwen2.5-1.5B-Instruct
task: sft
backend: transformers
data:
  train: /srv/storage/backblaze-soup/data/train.jsonl
  format: alpaca
  val_split: 0.1
  max_length: 2048
  train_on_responses_only: true
training:
  epochs: 1
  lr: 0.0001
  batch_size: 1
  gradient_accumulation_steps: 8
  gradient_checkpointing: true
  quantization: 4bit
  seed: 42
  data_seed: 42
  logging_steps: 5
  save_steps: 100
lora:
  r: 16
  alpha: 32
  dropout: 0.05
  target_modules: auto
output: /srv/storage/backblaze-soup/output/qwen-backblaze-v1

A few choices worth explaining:

Batch size 1 with gradient accumulation 8 gives an effective batch of 8 without holding eight long sequences in memory at once.
Gradient checkpointing recomputes activations during the backward pass instead of storing them. It is slower, but it saves memory.
train_on_responses_only computes loss only on the answer, not on the long SMART history in the prompt. I want the model graded on its prediction, not on copying numbers.
val_split: 0.1 means Soup held back 10% of train.jsonl for its own validation. My separate val.jsonl was not used during training.

soup train --config soup.yaml

The run took 88 optimizer steps and about six minutes, with GPU memory peaking around 3.5 GB. I did not enable layer streaming. The reported loss fell from 1.0847 to 0.0198.

To see what the GPU was doing during that run, I later reran exactly the same configuration into a separate output folder and watched it in Grafana, with nvidia-smi sampling once per second alongside. The rerun matched closely: 88 steps, 408 seconds of training (about eight minutes including model and data loading), and loss from 1.0812 to 0.0198. The first logged loss differs slightly from the original 1.0847, a reminder that a fixed seed does not guarantee bit-identical GPU runs.

Two things stand out. Soup reported a 3.5 GB peak, but the GPU showed up to 8.1 GiB in use. My reading is that Soup reports memory PyTorch allocated for tensors, while the driver also counts the CUDA context and memory PyTorch’s caching allocator has reserved but not handed out. Both numbers are real; they answer different questions. The card was also busy without being maxed out: once training started, utilization averaged about 92%, power peaked at 133 W against a 170 W limit, and temperature rose from 59 °C to 71 °C. For a 1.5B model in 4-bit with batch size 1, the 3060 has headroom.

On paper, that looks like success.

Evaluating it

I ran the 168 test examples through the fine-tuned adapter and through the original model, then compared the answers with the labels.

The fine-tuned model returned valid, bare JSON for all 168 cases. It also answered false for all 168:

	Predicted failure	Predicted healthy
Actually failed	0	18
Actually healthy	0	150

Accuracy was 89.3%. Failure recall was 0%. A model that always says “no failure” gets exactly the same score, because 150 of the 168 test drives are healthy. The fine-tuned model learned the answer format and the majority class, and missed every drive that actually failed.

A bug in my own evaluation

The original model at first looked even worse: my script marked all of its answers invalid. When I inspected the first five, each was perfectly reasonable JSON, wrapped in a Markdown code block:

```json
{
  "failure_within_7_days": false
}
```

My parser expected bare JSON and rejected the fences. So “0% valid” described my evaluation code, not the model. Those five answers all predicted false too, but five answers are not a baseline, and the full corrected comparison is still pending.

What I take from this

Local fine-tuning is accessible. A consumer GPU, a 1.5B model and QLoRA got me from raw CSVs to a trained adapter on one machine, and the training run itself took six minutes.
A low training loss is not a useful predictor. With 90% healthy examples and a tiny fixed answer, the cheapest way to reduce loss is to always say no.
Valid JSON is not a useful predictor either. Format compliance is easy to measure and easy to mistake for progress.
Accuracy is the wrong headline metric for rare events. I’ll report failure recall, precision and false alerts from now on.

I’ve built a balanced training set with 79 failed and 79 healthy drives, so “always no” stops being a free win. The plan is to train a fresh adapter from the original base model for three epochs on it, and to compare it on the validation set rather than repeatedly on the test set. I also want a tabular baseline, such as XGBoost or LightGBM on the same features, to see whether a language model adds anything at all. Last, I want an evaluation on a later time period before drawing any operational conclusions.

That second training run has not produced a result yet.