Most people who fine-tune LLMs waste the majority of their compute on avoidable mistakes. After fine-tuning models across NLP, code generation, and domain-specific applications, I see the same three patterns over and over.

Mistake #1: Training on Too Much Data with Too Little Quality

More data ≠ better model. A 1,000-example dataset of perfectly curated instruction-response pairs will consistently outperform a 50,000-example dataset scraped from the internet.

The fix: Before scaling your dataset, train on 500 hand-verified examples first. Evaluate. If the model isn't learning the task structure from 500 clean examples, 50,000 noisy ones won't save you.

# Quick quality check before committing to a full training run
from datasets import load_dataset
import random

ds = load_dataset("your_dataset")
sample = random.sample(range(len(ds["train"])), 20)

for idx in sample:
    example = ds["train"][idx]
    print(f"INPUT: {example['instruction'][:200]}")
    print(f"OUTPUT: {example['response'][:200]}")
    print("---")
    # Manually verify: Is the output actually correct?
    # Is the instruction clear and unambiguous?

Mistake #2: Ignoring LoRA Rank Selection

Everyone uses r=16 because that's what the tutorial said. But rank selection should depend on your task complexity and base model size.

Rule of thumb:

The fix: Start with r=8, evaluate, then double it. If doubling rank doesn't improve eval metrics by >2%, you've found your ceiling. Stop burning compute.

Mistake #3: Not Evaluating on What Actually Matters

BLEU, ROUGE, perplexity — these are proxies. They don't tell you if your model actually does the thing you need it to do.

The fix: Build a task-specific eval set of 50–100 examples with human-written gold outputs. Score them on the dimensions that matter for your use case:

• Factual accuracy — did it get the facts right?

• Instruction following — did it do what was asked?

• Format compliance — is the output structured correctly?

# Simple LLM-as-judge evaluation
def evaluate_response(instruction, model_output, gold_output, judge_model):
    prompt = f"""Rate the following model output on a scale of 1-5 for:
1. Accuracy (does it match the gold standard?)
2. Instruction following (did it do what was asked?)
3. Format (is it properly structured?)

Instruction: {instruction}
Gold output: {gold_output}
Model output: {model_output}

Return JSON: {{"accuracy": X, "following": X, "format": X}}"""
    
    return judge_model(prompt)

---

These three fixes alone will save you hundreds of dollars in compute and days of wasted iteration cycles.

I cover all of this (and much more) in depth in the LLM Fine-Tuning Masterclass — including full training pipelines, QLoRA implementation, DPO alignment, and production deployment with vLLM.