AI Systems

Fine-tuning — Specializing the Model

Fine-tuning adapts a pre-trained model's weights using task-specific demonstrations. Where prompting and RAG inject knowledge at inference time, fine-tuning bakes behavior directly into the weights: consistent style, structured output adherence, domain-specific reasoning patterns, and shorter prompts since learned behavior replaces lengthy instructions. The cost is curated data, compute, evaluation overhead, and an extra model artifact to version and serve.

Quick reference

What it is: Updating a pre-trained model's weights on task-specific examples, instead of only steering it through the prompt.
Mental model: Prompting is giving someone instructions for one task; fine-tuning is actually training them — slower and more expensive up front, but they don't need the instructions repeated every time afterward.
Key numbers: LoRA typically updates well under 1% of total parameters (rank 8–64); a few hundred to low thousands of high-quality examples routinely beats tens of thousands of noisy ones.
Reach for it when: A quantified prompt-engineering and RAG baseline has already been tried and measurably falls short — never as the first tool reached for.

A checkpoint that fails the held-out gate never ships — the gate is not optional QA, it's the difference between "fine-tuned" and "fine-tuned and verified."

Full fine-tuning

Every parameter is updated on the task-specific dataset. This gives maximum expressiveness and suits a target domain that diverges significantly from the base model's pretraining distribution — turning a general-purpose model into a specialized biomedical or legal one, for instance. The costs are real: full-parameter gradients and optimizer states need several times the model's parameter count in GPU memory, catastrophic forgetting of general capabilities is a genuine risk without careful data mixing, and the result is a full model checkpoint to serve and version on its own. Proposing full fine-tuning without justifying why PEFT is insufficient is a weak system-design answer, not a strong one.

Parameter-efficient fine-tuning (PEFT)

LoRA injects trainable low-rank decomposition matrices (rank r, typically 8 to 64) into the attention projection layers while freezing the base weights — updating well under 1 percent of total parameters. QLoRA adds 4-bit quantization of the frozen base model, making it possible to fine-tune a 70B-parameter model on a single high-memory GPU. Because the base weights never move, one shared base model can serve many task-specific adapters, hot-swapped per request — the standard production pattern for multi-tenant fine-tuning, and the default answer unless full fine-tuning is explicitly justified.

Instruction and preference tuning

RLHF trains a separate reward model on human preference pairs, then optimizes the policy model against that reward using PPO. DPO (Direct Preference Optimization) reformulates the same objective as a single supervised loss directly on preference pairs, removing the reward model and the RL loop entirely — which is why it has largely displaced RLHF for practical fine-tuning pipelines outside frontier labs. These methods tune for preferred behavior — helpfulness, tone, refusal calibration — rather than narrow task accuracy, and are a distinct lever from task-specific supervised fine-tuning.

Productionizing this pattern

What changes past a demo

Always run a prompt-engineering and RAG baseline first and quantify the gap — fine-tuning has the highest fixed cost of any pattern in this domain, and skipping the baseline is the most common mistake.
Data quality dominates data quantity for PEFT — a few hundred to low thousands of high-quality, diverse examples routinely outperform tens of thousands of noisy ones. Deduplication and manual review of a sample is worth the time.
Hold out an eval set that never touches training, and gate every fine-tuned checkpoint behind it before it reaches production.
Plan for adapter lifecycle management: versioning, A/B testing new adapters against the incumbent on live traffic, and a fast rollback path.
Catastrophic forgetting is real even with PEFT at high rank — include a slice of general-capability examples in the training mix and evaluate on both the target task and a general benchmark.

Real-world applications

Domain-specialized assistants — biomedical, legal, or financial models tuned on domain-specific reasoning patterns a general model handles poorly.
Structured output adherence — forcing consistent JSON/format output that prompting alone produces inconsistently.
Style/tone consistency — a brand voice or a support persona baked into weights instead of re-specified in every prompt.
Multi-tenant SaaS — one shared base model serving many customers' individually fine-tuned LoRA adapters, hot-swapped per request.

Embedded applications

On-device model specialization — a small model running on an NPU-equipped edge board (the same class of hardware the inference-snap pattern targets) is far more likely to be a distilled/quantized model fine-tuned tightly for one task than a large general-purpose one — the compute and memory budget simply won't support a general model at acceptable latency.
Quantization as a shared discipline — QLoRA's 4-bit quantization during fine-tuning and the int8/int4 quantization used to fit a model onto an embedded NPU are the same underlying technique applied at different points in the lifecycle — understanding one transfers directly to the other.
Adapter hot-swapping over OTA — shipping a new LoRA adapter to a fleet of devices is a much smaller, faster OTA payload than a full model replacement — the same atomic-update-and-rollback discipline covered in Ubuntu Core and kernel patching applies directly: version the adapter, gate it behind an eval, and keep a rollback path if it regresses on-device.
Field-collected fine-tuning data — corrections and edge-case failures captured from real deployed devices (see User Feedback) are exactly the "few hundred high-quality examples" PEFT wants — often more valuable than a much larger synthetic dataset, because they're already in-distribution for the actual hardware and actual failure modes.

Gotchas

Field notes

Fine-tuning fixes behavior, not knowledge gaps — a model that's confidently wrong about a fact will still be confidently wrong after fine-tuning unless the training data specifically corrects that fact; that's usually a RAG problem, not a fine-tuning one.
An eval gate that only checks the target task, not general capability, will happily ship a checkpoint that got better at the task and quietly worse at everything else.
"It works better on my test examples" without a held-out set that never touched training is not evidence — it's the exact overfitting failure the held-out gate exists to catch.