# Self-Consistency Prompting

Course: [Prompt Engineering](https://www.devtutorials.ai/learn/prompt-engineering) · Section: Advanced Prompting
Source: https://www.devtutorials.ai/learn/prompt-engineering/advanced-prompting/self-consistency-prompting

> Ask several times and trust the most common answer: sampling with temperature, extracting answers, majority votes, agreement as confidence, accuracy versus cost.

Ask a model the same hard question five times, with a little randomness turned on. You might get 69, 69, 70, 69, 84. Which answer should you trust?

The one that came up most often. A model can reach the right answer through many different ways of thinking, but wrong answers tend to be scattered. So if you take several replies and choose the answer most of them gave (a **majority vote**), you are more likely to get the right answer. This is called **self-consistency**.

**The main idea:** ask the same question several times, take the final answer from each reply, and keep the most common one. How much the replies agree also tells you how much to trust the result.

## A Real-World Example

Think about asking for directions in a city you don't know.

- You ask one person, and they point left. Are they right? You can't tell.
- You ask five people. Four point left, and one points right. You go left, and you have a good reason to.
- If they had pointed in five different directions, you would know that nobody is sure, and you would check a map.

Self-consistency asks the model several times. How much the answers agree shows you the likely answer, and how sure you can be about it.

## What You Will Learn

In this lesson, you will learn:

1. What self-consistency is, and why it works.
2. Why it needs a temperature above 0.
3. How to take out the answers and hold a majority vote.
4. How to use the level of agreement as a sign of confidence.
5. How the answers get more accurate when you ask more times.
6. What it costs, and when to use it.

The Prompt sandbox has a **Vote** tab made for this lesson. It asks the same question several times, takes out each answer, and shows the vote. The "Try it" card below opens it.

## How Self-Consistency Works

1. Write the question using **chain of thought**, with a fixed answer line.
2. Run it **N times**, with a temperature above 0, for example 0.7.
3. **Take out** the final answer from each reply.
4. **Count** the answers, and choose the most common one.

In short, it is chain of thought, run several times, followed by a vote.

## Why It Needs Randomness

At temperature 0, the model always chooses its most likely token. So five runs give five identical replies. A vote between five copies tells you nothing new.

With a temperature of about 0.7, each run can take a slightly different path. When different paths reach the **same** answer, that is good evidence that the answer is right.

```js runnable title="Votes at temperature 0 and 0.7"
function vote(answers) {
  const counts = {};
  for (const a of answers) counts[a] = (counts[a] || 0) + 1;
  return Object.entries(counts).sort((x, y) => y[1] - x[1]);
}

const atZero = ["70", "70", "70", "70", "70"];
const atSeven = ["69", "70", "69", "69", "84"];

console.log("T = 0:  ", JSON.stringify(vote(atZero)));
console.log("T = 0.7:", JSON.stringify(vote(atSeven)));
```

Expected output:

```text
T = 0:   [["70",5]]
T = 0.7: [["69",3],["70",1],["84",1]]
```

These answers are samples written for this example. At temperature 0, the same wrong answer comes up 5 times, and it looks certain. At 0.7, the paths spread out, and the right answer, 69, wins the vote.

## Pull Out the Answers and Vote

Each reply is a full chain of thought. Take the final answer line out of each reply, and then hold the vote. If a reply has no answer line, count it as "none" instead of guessing.

```js runnable title="Extract answers, then vote"
const replies = [
  "7 x 12 = 84. 84 - 15 = 69.\nAnswer: 69",
  "12 x 7 = 84 pens. Minus 15 gives 69.\nAnswer: 69",
  "7 + 12 = 19. 19 x 15 is too big.\nAnswer: 285",
  "84 pens in total. She gives 15 away.\nAnswer: 69",
  "She has some pens left.",
];

const answers = replies.map((r) => {
  const m = r.match(/Answer:\s*(-?\d+)/);
  return m ? m[1] : "none";
});

const counts = {};
for (const a of answers) counts[a] = (counts[a] || 0) + 1;
const ranked = Object.entries(counts).sort((x, y) => y[1] - x[1]);

console.log("Answers:", answers.join(", "));
console.log("Winner:", ranked[0][0], "with", ranked[0][1], "of", answers.length, "votes");
```

Expected output:

```text
Answers: 69, 69, 285, 69, none
Winner: 69 with 3 of 5 votes
```

One reply made a wrong step, and one forgot the answer line. But the vote still finds 69.

Now try it with the real model. The Vote tab runs the question 5 times at temperature 0.7, with a different seed for each run.

**Easy Way to Remember:** self-consistency means **ask several times, count the answers, and trust the crowd**. It only works if each reply ends with an answer that your code can find.

## Agreement as Confidence

The share of the votes that the winning answer gets is called the **agreement**. It is a useful sign of how confident you can be:

- **High agreement** (like 5 out of 5) means many different paths reached the same answer.
- **Low agreement** (like 2 out of 5) means the model is unsure. Mark the answer for a person to check, or use a bigger model.

```js runnable title="Flag answers with low agreement"
function decide(answers, threshold) {
  const counts = {};
  for (const a of answers) counts[a] = (counts[a] || 0) + 1;
  const [winner, votes] = Object.entries(counts).sort((x, y) => y[1] - x[1])[0];
  const agreement = votes / answers.length;
  return winner + " (" + Math.round(agreement * 100) + "% agree) -> " + (agreement >= threshold ? "use it" : "check by hand");
}

console.log(decide(["69", "69", "69", "69", "70"], 0.6));
console.log(decide(["69", "70", "84", "69", "12"], 0.6));
```

Expected output:

```text
69 (80% agree) -> use it
69 (40% agree) -> check by hand
```

Both votes choose 69. But in the second case, only 2 of the 5 replies agree, so you should not trust the answer without checking it.

## More Samples, More Accuracy

How much does voting help? The next box runs a **simulation**, not a real model. It pretends that a model gets the right answer 60% of the time, and that its mistakes are spread over three wrong answers. Then it measures how often the majority vote is right, for different numbers of replies. A random number generator with a fixed seed makes the result the same every time.

```js runnable title="Simulate how voting improves accuracy"
function seededRandom(seed) {
  let a = seed >>> 0;
  return function () {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

const rand = seededRandom(7);
function sampleAnswer() {
  const r = rand();
  if (r < 0.6) return "right";
  return r < 0.75 ? "wrong A" : r < 0.9 ? "wrong B" : "wrong C";
}

for (const n of [1, 3, 5, 9]) {
  let wins = 0;
  for (let trial = 0; trial < 2000; trial++) {
    const counts = {};
    for (let i = 0; i < n; i++) {
      const a = sampleAnswer();
      counts[a] = (counts[a] || 0) + 1;
    }
    const top = Object.entries(counts).sort((x, y) => y[1] - x[1])[0][0];
    if (top === "right") wins++;
  }
  console.log(n + " sample(s): right " + Math.round((wins / 2000) * 100) + "% of the time");
}
```

Expected output:

```text
1 sample(s): right 61% of the time
3 sample(s): right 72% of the time
5 sample(s): right 82% of the time
9 sample(s): right 91% of the time
```

With 5 replies, voting turns a model that is right 60% of the time into one that is right about 82% of the time. But each extra reply helps a little less than the one before. This only works when the model gives the right answer **more often than any one wrong answer**. If the model usually makes the same mistake, voting makes that wrong answer look certain.

## Self-Consistency in One Chat Prompt

It is easy to run the same question many times in code or in the Vote tab, but it is harder in a simple chat app. A version that uses one prompt gets some of the benefit. Ask the model to **solve the problem in two different ways**, compare the results, and give an answer **only if they agree**.

This is weaker than running the question many times, because both ways are written in the same reply, and one can copy the other's mistake. But it often shows when the thinking is weak. The "only if they agree" rule also gives the model a clear way to say that it is unsure.

## What It Costs

Every reply is a full answer with its thinking. Five replies cost about five times as much and take about five times as long, unless your app can run them all at the same time.

```js runnable title="Estimate the cost of N samples"
const tokensPerReply = 60;
const tokensPerSecond = 8;

for (const n of [1, 3, 5, 9]) {
  const tokens = n * tokensPerReply;
  console.log(n + " samples: " + tokens + " tokens, about " + Math.round(tokens / tokensPerSecond) + " s in your browser");
}
```

Expected output:

```text
1 samples: 60 tokens, about 8 s in your browser
3 samples: 180 tokens, about 23 s in your browser
5 samples: 300 tokens, about 38 s in your browser
9 samples: 540 tokens, about 68 s in your browser
```

Use self-consistency when:

- The answer is **short and easy to compare**, like a number, a label, or yes or no.
- A wrong answer would **cause serious problems**, and you can afford the extra calls.
- You want a **sign of how confident to be**, not just an answer.

Don't use it for free writing, like an email or a story. There is no single answer to vote on.

A majority is not proof. If the model has the same blind spot every time, all the replies can agree on the same wrong answer. Even when the agreement is high, you still need to check a small model's answers on anything important.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Voting at temperature 0 | "Five runs means five opinions." | Use about 0.7, so the paths are different. | Five copies of one answer look certain when they shouldn't. |
| No answer line | "I'll find the number in the text." | Ask for a fixed final answer line. | Code counts numbers from the middle of the thinking. |
| Ignoring the agreement | "The winning answer is always right." | Check the answer yourself when agreement is low. | Answers the model was unsure about get used as if they were certain. |
| Voting on essays | "Self-consistency improves any reply." | Use it for short answers that you can compare. | No two essays are the same, so there is nothing to vote on. |
| Too many replies | "More is always better." | Each extra reply helps less. 5 is often enough. | You wait a long time for very little gain. |
| The same blind spot every time | "If all the replies agree, the answer is right." | They can all agree and still be wrong. Check important answers. | A wrong answer that sounds sure gets through. |

## Exercises

In each exercise, the last line should give back a value. Press **Check** to compare your value with the expected answer.

## What to Remember

- **Self-consistency** asks the same question several times, and keeps the **most common answer**.
- Use it with **chain of thought**, a **fixed answer line**, and a temperature of about **0.7**.
- The **agreement** (the winning answer's share of the votes) is a sign of how confident you can be.
- Voting makes answers more accurate when the model is **right more often than it gives any one wrong answer**.
- It costs about **N times** as much. Start with 5 replies.
- In a simple chat, ask for **two different solutions**, and an answer **only if they agree**.
- A majority is **not proof**. Check important answers.

This is the end of Chapter 3. Next: **ReAct Prompting**, where the model thinks and uses tools, like a calculator, to find answers.