In Part 1 I fine-tuned Qwen2.5-1.5B-Instruct with 4-bit QLoRA on my RTX 3060 to answer one question: given seven days of SMART telemetry from a Toshiba MG07ACA14TA, will this drive fail in the next seven days? Training worked, the loss dropped, and the model returned clean JSON. It also predicted “no failure” for all 168 test drives and missed every one of the 18 failures.

I left one question open. My evaluation script rejected every answer from the original model, so I had no real baseline to compare against. Before training anything new, I wanted to close that gap. Changing the training data without a trustworthy evaluation would just give me a second number I couldn’t interpret.

The parser bug⌗

The original model did not return bare JSON. It wrapped its answer in a Markdown code block:

```json
{
  "failure_within_7_days": false
}
```

My parser called json.loads on the raw string, failed, and recorded predicted: null. All 168 base-model answers were scored as invalid, which made the original model look like it had no classification ability at all. That was a measurement artifact, not a property of the model.

The fix is small: strip an optional code fence, then parse. The decision it changes is not small. “The base model can’t follow the format” and “the base model follows the format but always says no” lead to very different next steps.

import json, re

FENCE = re.compile(r"^```(?:json)?\s*(.*?)\s*```$", re.DOTALL)

def parse(raw: str):
    text = raw.strip()
    m = FENCE.match(text)
    if m:
        text = m.group(1)
    try:
        return bool(json.loads(text)["failure_within_7_days"])
    except (json.JSONDecodeError, KeyError, TypeError):
        return None

I re-scored the saved predictions in output/base-predictions.json without calling the model again.

Corrected results⌗

With the fence stripped, all 168 base-model answers parse. They are also all identical: the same fenced JSON saying false.

Model Valid answers TP TN FP FN Accuracy Failure recall
Qwen2.5-1.5B-Instruct (original) 168 / 168 (fenced) 0 150 0 18 89.3% 0%
Fine-tuned v1 (QLoRA) 168 / 168 (bare) 0 150 0 18 89.3% 0%
Always “no failure” n/a 0 150 0 18 89.3% 0%

The three rows are the same. On this test set, fine-tuning changed the format of the answer and nothing about the prediction. Precision is undefined for all three, because none of them ever raises an alert.

Accuracy is the misleading number here. 150 of the 168 test examples are healthy drives, so a model that never alerts scores 89.3% without learning anything. And because I sampled healthy drives when building the dataset, about 11% of test examples are failures, far more than in the real fleet. Accuracy on the real fleet would look even better for a model that never alerts, and mean even less.

What the training log says⌗

The v1 loss curve fits that picture. Loss was 1.08 at step 5, 0.61 at step 10 and 0.024 by step 20, then stayed flat around 0.02 for the remaining 68 of 88 steps. With 700 healthy and 79 failed drives in train.jsonl, answering false is right about 90% of the time. The output is also a short, fixed JSON string, so with response-only loss there are only a few tokens to learn. Low loss tells me the model learned the answer format and the majority class quickly. It does not tell me the model learned anything about SMART counters.

A note on the second run⌗

I had already prepared a balanced training set, data/train-balanced.jsonl, with 79 failed and 79 healthy drives. My first attempt to train on it never started. Its log is a single line: the shell could not find the soup binary, because the command used an old workspace path that pointed to an empty copy left over from a migration. No v2 adapter exists yet, so there are no v2 results in this post.

What comes next⌗

  • Balanced training (v2). A fresh adapter from the original base model, trained for three epochs on the balanced set. Balancing removes the shortcut of always answering no, but it is not guaranteed to make the model learn real signal. It may simply trade missed failures for false alerts.
  • Validation, not test. I’ll compare v2 on val.jsonl and leave test.jsonl alone until I’ve picked a model. Re-scoring every idea on the test set would slowly turn it into a training signal.
  • Precision, recall and false alerts. For an operator, the useful questions are how many failing drives the model catches and how many healthy drives it flags per week. A model that alerts on everything has perfect recall and is just as useless as one that never alerts.
  • A tabular baseline. Seven days of seven SMART counters is a small table. XGBoost or LightGBM on the same features and splits will show whether a language model adds anything here.
  • A future period. My splits are random by drive, not chronological. Before drawing operational conclusions, I need to train on one period and evaluate on a later one, such as Q2 2025.

The lesson from Part 1 still holds, now with a corrected baseline behind it: the pipeline runs on consumer hardware, but a working pipeline is not yet a working predictor.