Decide and Prepare in Fine-tuning & Model Customization

Decide and Prepare is a chapter of the free Fine-tuning & Model Customization tutorial with 7 lessons: Should You Fine-tune at All?, Exhaust the Cheaper Rungs First, How Fine-tuning Works, Dataset Design, From Tickets to Training Data, Data Quality: Duplicates, Leakage, Label Errors, Bias and Synthetic Data from a Stronger Model. Plan on about 260 minutes. It has 27 exercises you run and check in your browser.

What each lesson covers:

  • Should You Fine-tune at All?: Four requests that all sound like fine-tuning. Only two need it. Learn to name the gap, climb the customization ladder from the cheapest rung, and count what a fine-tune costs to own.
  • Exhaust the Cheaper Rungs First: Push prompting step by step, prove a plateau on the same eval, count what few-shot examples cost, and learn when RAG or both tools are the answer.
  • How Fine-tuning Works: Weights, loss, epochs, learning rate, validation sets, overfitting and LoRA, explained for developers with no machine learning background.
  • Dataset Design: What one training example looks like in chat JSONL, why answers must follow one labelling guide, how to check coverage, and how to split the eval by customer.
  • From Tickets to Training Data: Turn a real helpdesk export into training lines: check consent, convert HTML, cut quoted replies and signatures, redact personal data and split by customer.
  • Data Quality: Duplicates, Leakage, Label Errors, Bias: Clean a 600-row export: exact and near-duplicates, eval leaks, wrong labels and one dominant customer. Write a near-duplicate check and see how leaks inflate the eval.
  • Synthetic Data from a Stronger Model: Fill rare intents with tickets a teacher model writes, filter them five ways, choose a safe mix, and export the v1 dataset kit with a dataset card.

When you finish, continue with Train and Operate.

Lessons in this section

  1. Should You Fine-tune at All?

    Four requests that all sound like fine-tuning. Only two need it. Learn to name the gap, climb the customization ladder from the cheapest rung, and count what a fine-tune costs to own.

    • Say what fine-tuning changes inside a model, and what it does not
    • Sort a request into a knowledge, behaviour, cost or eval gap
    • Climb the customization ladder and stop at the cheapest rung that meets the target
    • Count what a fine-tune costs to own for a year
  2. Exhaust the Cheaper Rungs First

    Push prompting step by step, prove a plateau on the same eval, count what few-shot examples cost, and learn when RAG or both tools are the answer.

    • Improve a prompt one change at a time and measure each version
    • Show a plateau with scores on the same eval
    • Count what few-shot examples cost per month
    • Tell a format plateau from a facts plateau
  3. How Fine-tuning Works

    Weights, loss, epochs, learning rate, validation sets, overfitting and LoRA, explained for developers with no machine learning background.

    • Explain weights, loss and one training step
    • Say what epochs and the learning rate do
    • Spot overfitting from training and validation loss
    • Pick the best checkpoint
  4. Dataset Design

    What one training example looks like in chat JSONL, why answers must follow one labelling guide, how to check coverage, and how to split the eval by customer.

    • Write a training example in the chat JSONL format
    • Keep every answer consistent with a labelling guide
    • Check coverage per kind of input
    • Keep an eval split apart, split by customer
  5. From Tickets to Training Data

    Turn a real helpdesk export into training lines: check consent, convert HTML, cut quoted replies and signatures, redact personal data and split by customer.

    • Check consent and sign-off before exporting
    • Turn HTML emails into clean text
    • Cut quoted replies, signatures and greetings
    • Redact personal data while keeping order numbers
  6. Data Quality: Duplicates, Leakage, Label Errors, Bias

    Clean a 600-row export: exact and near-duplicates, eval leaks, wrong labels and one dominant customer. Write a near-duplicate check and see how leaks inflate the eval.

    • Find exact and near-duplicates after normalising text
    • Check training data against the eval set for leaks
    • Choose a similarity threshold by looking at what it removes
    • Flag wrong labels for human review
  7. Synthetic Data from a Stronger Model

    Fill rare intents with tickets a teacher model writes, filter them five ways, choose a safe mix, and export the v1 dataset kit with a dataset card.

    • Decide when synthetic data helps
    • Prompt a teacher model for varied examples
    • Filter generated rows five ways
    • Choose how much generated data to mix in