Fine-tuning & Model Customization Cheat Sheet
By Syncfusion®, Inc. · Last updated
Every idea from Fine-tuning & Model Customization on one page: the gap types and ladder, training terms and curve shapes, data cleaning, methods and settings, before/after evals, distillation, maintenance and the decision record.
Fine-tuning Cheat Sheet
This page puts every idea from the course in one place.
Use it before you agree to fine-tune, while you build the data, and after you ship.
Each section names the lesson that teaches it, so you can go back for the animations and exercises.
Simple lesson: fine-tuning changes behaviour. RAG supplies knowledge. Most teams should not fine-tune yet.
Should You Fine-tune?
From Should You Fine-tune at All? and Exhaust the Cheaper Rungs First.
Write the eval first. Climb one rung at a time, and only when the eval says the current rung stopped helping.
| Gap | Looks like | Start with |
|---|---|---|
| Knowledge | Facts are missing or change | RAG |
| Behaviour | Facts right, format or tone wrong | Prompt, few-shot, then LoRA |
| Cost or speed | Quality fine, too slow or dear at volume | Distillation |
| Eval | Nobody can say what "better" means | Write the eval |
- Try structured output (an enforced schema) before training a format.
- Prove a plateau: several prompt versions, same eval, last changes add almost nothing.
- Two gaps, two tools: tune the interface, retrieve the content.
- Count the yearly cost of owning the choice, not only the build.
How Training Works
From How Fine-tuning Works and Reading a Training Run.
| Word | Meaning |
|---|---|
| Weights | The numbers inside the model |
| Loss | How wrong the guesses are; lower is better |
| Epoch | One pass over the training data |
| Learning rate | Size of each nudge to the weights |
| Validation set | Examples never trained on, to spot overfitting |
| Checkpoint | Saved weights at one step; keep the best |
| LoRA rank | How much room the adapter has to learn |
| Curve shape | Fix |
|---|---|
| Both fall and level off | Healthy: run the full board |
| Validation turns up | Overfitting: keep the earlier checkpoint |
| Both still high | Train longer |
| Jumping | Lower the learning rate |
| Suspiciously low | Check for leaks |
Data
From Dataset Design, Data Quality and Synthetic Data.
Split by customer, before you train. The eval set is only for the final score.
- One example per line, chat JSONL, same system message everywhere.
- One labelling guide. Count examples per intent.
- Clean: exact and near-duplicates, eval leaks, wrong labels, one dominant source. Log every removal.
- Real exports: consent first, HTML to text, cut quoted replies and signatures, redact PII with placeholders, keep order numbers.
- Generated data: fill a measured gap, run five filters, read a sample, keep it under about a quarter.
Formatting Traps
From Chat Templates, Masking and Stop Tokens and When the Fine-tune Misbehaves.
| Bug | Symptom | Loss shows it? | Fix |
|---|---|---|---|
| No end token | Keeps going after the JSON | No | End-of-turn token after every answer |
| No masking | Wandering answers | Yes | Train on the completion only |
| Template mismatch | Garbled answers, template tokens | No | apply_chat_template for training and serving |
| Length too short | Answers cut off | Yes | Measure tokens, raise the limit |
| Adapter not loaded | Base-model prose | No | Log adapter version per answer |
The Starter Kit
From Run a Real Fine-tune.
python -m unittest test_kit.py
python prepare_data.py fixtures/tickets-export.json --out data
python check_template.py
python generate.py --base-only --out preds/base.jsonl
python train_qlora.py
python generate.py --adapter runs/ticket-adapter-v1/best --out preds/tuned.jsonl
python score.py --eval data/eval.jsonl --base preds/base.jsonl --tuned preds/tuned.jsonlThe Build-along Repo
From Fine-tune with a Hosted API and Ship It. Download ticket-tuner.zip. The app side is in TypeScript (ts/) and C# (csharp/), with the same tests.
| Task | TypeScript | C# |
|---|---|---|
| Tests | npm test | dotnet run --project TicketTuner.Tests |
| Check a training file | npm run data:check -- train.jsonl | ... TicketTuner.Cli -- data-check train.jsonl |
| Start a hosted job | npm run job:start -- train.jsonl val.jsonl | ... TicketTuner.Cli -- start-job train.jsonl val.jsonl |
| Release check | npm run release:check | ... TicketTuner.Cli -- release-check |
| Canary decision | npm run rollout:check -- stats.json 5 | ... TicketTuner.Cli -- rollout-check stats.json 5 |
Routing: ROUTE_MODE is off, shadow, canary or full. CANARY_PERCENT steps through 1, 5, 25, 50 and 100. Any bad fine-tune answer falls back to the baseline.
Methods and Settings
From Choosing the Method.
| Method | Use it when |
|---|---|
| LoRA / QLoRA | Behaviour gap. QLoRA for a smaller GPU |
| DPO | Easier to compare answers than to write them (after SFT) |
| GRPO-style | A program can check answers (after SFT) |
| Distillation | Quality fine, cost or speed not |
| Full fine-tune | Rarely, with evidence LoRA is not enough |
Before, After and Beyond
From Evaluate Before and After, Distillation, and Serving and Maintaining.
- Score base and fine-tune on the same cases, by slice, with 95% intervals.
- Ship rule: task slices win, no general or safety regression.
- Distillation: ship the cheapest student inside quality, cost and p95.
- Log base + adapter + dataset versions. Monthly eval with an alert line.
- Re-train on measured triggers. Upgrade with board → shadow → canary → all. Keep a rollback.
The Decision Record
From Capstone.
- Gap type.
- Cheaper rungs' scores on the same eval.
- Target, stated first.
- One-year cost of owning it.
- Before/after board with general and safety slices.
- Leak check of the training data.
- Re-training triggers and an owner.
- Data sign-off: consent, redaction, location.
- Run notes: config and versions.