Exhaust the Cheaper Rungs First

By Syncfusion®, Inc. · Last updated November 2, 2026

Push prompting step by step, prove a plateau on the same eval, count what few-shot examples cost, and learn when RAG or both tools are the answer.

Exhaust the Cheaper Rungs First

The JSON ticket team from Should You Fine-tune at All? wants to fine-tune. Their manager asks one question: "Show me that prompting has stopped working."

They cannot. They tried two prompts, saw 70%, and decided it was "not good enough".

So they go back and do it properly. Six prompt versions, each one measured on the same 200-ticket eval.

The score climbs from 62% to 79%, then stops. The last version reaches 80%.

Meanwhile the prompt grew from 60 tokens to 2,420 tokens, and every single call pays for those tokens.

Simple lesson: "prompting is not good enough" is a feeling. A plateau on the same eval is evidence.

A Real-World Example

Think about tuning a guitar by ear.

At first, each turn of the peg makes it sound much better. Then each turn changes almost nothing. You have reached the limit of what that peg can do.

If it still sounds wrong, the problem is not the peg. Maybe the string is old. You change the string, not the peg again.

A prompt is the peg. When turning it stops helping, you move to a different tool.

Simple lesson: stop when the gains stop, not when you get bored.

What You Will Learn

In this lesson, you will learn:

1. How to push a prompt step by step, measuring each version.
2. What a plateau is, and how to prove you have reached one.
3. Why few-shot examples help the format but cost money on every call.
4. When RAG is the next step instead of fine-tuning.
5. Why some requests need both: tune the format, retrieve the facts.

TL;DR

Improve the prompt one change at a time. Measure every version on the same eval.

Few-shot = putting example inputs and answers inside the prompt. It helps a lot with format, but every example is sent on every call.

A plateau is when the last changes barely move the score. Only then is it time to climb.

If the problem is facts, add RAG. If it is format and facts, use both: fine-tune the format, retrieve the facts.

Improve the Prompt, One Change at a Time

Here is what the team changed, in order:

1. p1: a plain instruction: "Reply as a ticket in JSON."
2. p2: spell out every field and its allowed values.
3. p3: add 3 example tickets.
4. p4: add 8 example tickets.
5. p5: add 16 example tickets.
6. p6: turn on the provider's JSON mode.
7. Schema: enforce the exact schema (structured output). The next section looks at it.

JSON mode = a setting some model providers offer that forces the reply to be valid JSON. It does not force your field names or your tone.

The rule is simple: change one thing, then measure. If you change three things at once, you cannot tell which one helped.

Here is the same check in code. It flags a plateau when the last two changes together gained less than 2 points.

// Scores on the same 200-ticket eval, one per prompt version.
const scores = [62, 70, 76, 79, 79, 80];

// A plateau: the last 2 changes together gained less than 2 points.
function plateaued(history: number[], window = 2, minGain = 2): boolean {
  if (history.length < window + 1) return false;
  const recent = history.slice(-window - 1);
  return recent[recent.length - 1] - recent[0] < minGain;
}

for (let n = 1; n <= scores.length; n++) {
  const sofar = scores.slice(0, n);
  const gain = n > 1 ? sofar[n - 1] - sofar[n - 2] : 0;
  console.log(`p${n}: ${sofar[n - 1]}%  gain ${gain}  plateau? ${plateaued(sofar)}`);
}

// Scores on the same 200-ticket eval, one per prompt version.
int[] scores = [62, 70, 76, 79, 79, 80];

// A plateau: the last 2 changes together gained less than 2 points.
bool Plateaued(int[] history, int window = 2, int minGain = 2)
{
    if (history.Length < window + 1) return false;
    var recent = history[^(window + 1)..];
    return recent[^1] - recent[0] < minGain;
}

for (var n = 1; n <= scores.Length; n++)
{
    var sofar = scores[..n];
    var gain = n > 1 ? sofar[n - 1] - sofar[n - 2] : 0;
    Console.WriteLine($"p{n}: {sofar[n - 1]}%  gain {gain}  plateau? {Plateaued(sofar).ToString().ToLower()}");
}

Expected output:

p1: 62%  gain 0  plateau? false
p2: 70%  gain 8  plateau? false
p3: 76%  gain 6  plateau? false
p4: 79%  gain 3  plateau? false
p5: 79%  gain 0  plateau? false
p6: 80%  gain 1  plateau? true

At p6 the check says true. Now the team has evidence: six versions, the same eval, and the last changes added almost nothing.

Try this: change the last score from 80 to 84 and run it again. Is it still a plateau?

Enforce the Schema Before You Train

Many model providers and serving tools offer structured output: you give a JSON schema, and the decoder is only allowed to produce tokens that fit it. That is stronger than JSON mode, which only promises valid JSON of any shape.

JSON schema = a description of the fields a JSON object must have, their types, and their allowed values.

With the schema enforced, every reply parses and has every field. The overall score rises to 88%. What remains is content: the right priority for a cancellation, and the house tone in the summary. A schema cannot choose the right value for you.

What Few-shot Examples Cost

Examples are the strongest prompting tool for format. But a prompt is sent with every request.

Token = a small piece of text, often part of a word. Models are paid for by the token.

// Every example in a few-shot prompt is sent on EVERY call.
const tokensPerExample = 140;
const basePrompt = 180;           // instructions and field list
const callsPerMonth = 300_000;
const dollarsPer1kTokens = 0.0005;

for (const examples of [0, 3, 8, 16]) {
  const tokens = basePrompt + examples * tokensPerExample;
  const monthly = (tokens / 1000) * dollarsPer1kTokens * callsPerMonth;
  console.log(`${String(examples).padStart(2)} examples: ${tokens} tokens per call, $${monthly.toFixed(0)} a month`);
}

// Every example in a few-shot prompt is sent on EVERY call.
var tokensPerExample = 140;
var basePrompt = 180;           // instructions and field list
var callsPerMonth = 300_000;
var dollarsPer1kTokens = 0.0005;

foreach (var examples in new[] { 0, 3, 8, 16 })
{
    var tokens = basePrompt + examples * tokensPerExample;
    var monthly = (tokens / 1000.0) * dollarsPer1kTokens * callsPerMonth;
    Console.WriteLine($"{examples,2} examples: {tokens} tokens per call, ${monthly:F0} a month");
}

Expected output:

 0 examples: 180 tokens per call, $27 a month
 3 examples: 600 tokens per call, $90 a month
 8 examples: 1300 tokens per call, $195 a month
16 examples: 2420 tokens per call, $363 a month

Sixteen examples cost about $363 a month in prompt tokens alone, at 300,000 calls. A fine-tuned model does not need those examples in the prompt, because it has learned the format.

That does not mean "fine-tune to save money" every time. A fine-tune has its own costs, as Should You Fine-tune at All? showed. It means the cost of a long prompt belongs in the comparison.

Simple lesson: few-shot examples are cheap to write and expensive to send.

When RAG Is the Next Step

Remember the refund policy request from Should You Fine-tune at All?. Prompting scored 41%. Examples scored 46%.

That is not a plateau in format. The model simply does not know this month's policy. No prompt can fix a missing fact.

RAG (retrieval-augmented generation) = look up the right documents when the question arrives, and put them in the prompt.

RAG is the next rung for a knowledge gap. It is usually a week of work, and the answer updates the moment the document changes.

Tune the Interface, Retrieve the Content

Some requests have two gaps at once.

The support team now wants tickets in the exact JSON format and the summary must quote this month's refund policy.

The fine-tune alone gets the format right but quotes old policy facts: only 61% right. RAG alone gets the facts but the format slips.

Together, each tool does what it is good at. This pattern has a short name: tune the interface, retrieve the content.

Interface = the shape of the answer: its format, fields, tone and length.

Content = the facts inside the answer.

Simple lesson: one tool per gap. Two gaps, two tools.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Improve the prompt one change at a time, and measure each version on the same eval.
• A plateau is evidence: the last changes barely moved the score.
• Structured output guarantees the shape; values and tone still need examples or training.
• Few-shot examples help format but are paid for on every call.
• A knowledge gap goes to RAG, not fine-tuning.
• Two gaps, two tools: tune the interface, retrieve the content.

Next: How Fine-tuning Works, what really happens inside a model when you train it, explained without the maths.