Self-Consistency Prompting: Check Claude's Answers

You ask Claude to calculate something, a shipping cost split three ways, a tax deduction, a percentage change. It answers fast and sounds sure of itself. You paste the answer into the email you're about to send, then some instinct makes you ask the exact same question again. This time the number is different.

Nothing crashed. Nothing threw an error. The model just landed on a different answer the second time, and now you don't know which one to believe. That gap, between "sounds confident" and "is actually right," is exactly what self-consistency prompting and LLM-as-judge verification are for.

Key Takeaways

• Claude's answers are generated probabilistically, so re-asking the same question can produce a genuinely different result, this is normal, not a glitch
• Self-consistency means asking the same question a few times and going with the answer that shows up most, like polling a small panel instead of trusting one vote
• Self-critique means asking Claude to review its own answer, useful, but weakened by self-preference bias, a model tends to approve what it already wrote
• LLM-as-judge uses a separate prompt to score or compare outputs, which is a more reliable check than self-review because the judge isn't defending its own work

---

Why One Answer From an LLM Isn't Always the Right One

Claude doesn't look up answers in a table the way a calculator does. It predicts the next word, then the next, based on probabilities learned during training. For most everyday questions those probabilities are lopsided enough that you get the same answer every time. That's not true for anything with a few steps of reasoning: math, logic puzzles, multi-part comparisons. There, the probabilities can be close enough that a slightly different path through the words produces a different final answer.

This is the same idea covered in Chain-of-Thought prompting research: a single reasoning path is one sample from many possible paths, not a guaranteed correct one. If you already broke a task into smaller prompts using decomposition and prompt chaining, you've seen this too. Each step in a chain is a chance for the model to drift, and drift compounds.

I still ask Claude the same question twice on anything I'm about to send to someone else, purely out of habit at this point. It's caught real mistakes often enough that the ten extra seconds feels cheap by comparison.

Skip this and you'll treat the first answer as gospel, including on the one question where the model happened to pick the wrong path. That's the failure mode the rest of this lesson fixes.

Self-Consistency Prompting: Ask the Same Question Multiple Times

Self-consistency prompting is the simplest fix: instead of trusting one answer, generate a few and see where they agree. Think of it like asking three friends the same trivia question independently and going with whatever two of them say, rather than trusting whichever friend answered first.

Here's a rewritten version of a classic self-consistency example. Ask Claude this exact prompt three separate times, in three separate messages:

Most people rush past "all but 9" and answer 15 (subtracting instead of reading the phrase correctly). The correct answer is 9, that's the number left after all the others sold. If you run this three times, you'll likely see at least two runs land on 9 and possibly one land on 15. The majority answer, 9, is the one to trust.

That's the whole technique: sample multiple times, take the answer that shows up most. Skip this on a task with real stakes and you're betting the outcome on whichever single path the model happened to sample first, correct or not.

Self-Critique: Asking Claude to Check Its Own Answer

The second layer is cheaper than running a prompt three times: just ask Claude to review what it already wrote. A simple version looks like "Review your answer above. Check the math and flag anything that might be wrong." This catches real mistakes, arithmetic slips, a missed constraint, a contradiction between two paragraphs.

Here's the catch: a model reviewing its own work is a bit like grading its own homework. Researchers call this self-preference bias. It shows up as Claude saying "looks correct" about a paragraph it just wrote, even when a fresh set of eyes would catch the issue immediately. This isn't Claude being dishonest, it's a structural weak point in asking anything to evaluate the thing it just produced. Self-critique is worth doing, it just isn't the final word.

What Is LLM-as-Judge?

LLM-as-judge is a technique where a separate prompt, sometimes a separate model, scores or compares two AI outputs against criteria you set, instead of the model reviewing its own answer. This separation avoids self-preference bias: the judge has nothing invested in either output, so its verdict is more reliable than self-critique.

LLM-as-judge fixes the grading-your-own-homework problem by splitting the roles. One prompt generates an answer, or two competing answers. A separate prompt, with no attachment to either output, scores or compares them against criteria you set. This separation is what makes LLM-as-judge more reliable than self-review, the judge has nothing invested in either answer.

There are two common setups:

Pointwise vs pairwise LLM-as-judge

Here's what pairwise looks like in practice: write two different drafts of an email, then start a fresh conversation and paste both in with this prompt: "Which of these two emails is clearer and more persuasive? Explain your reasoning before picking one." That fresh conversation is the judge. It never wrote either draft, so it has no reason to defend one over the other.

Skip this step and you're often stuck choosing between two versions of something with no real way to tell which one is stronger, other than your own gut, which is exactly the situation a second opinion is supposed to fix.

When This Is Worth the Extra Cost

None of this is free. Running a prompt three times costs roughly three times the tokens and time of running it once, and adding a judge step is another full prompt on top. For a quick summary or a first-draft tweet, that overhead isn't worth it, one answer is fine.

Reach for self-consistency prompting, self-critique, or an LLM-as-judge pass when the output has real consequences: a number that goes into a report, code you're about to run, a factual claim you'll repeat to a client. The cost of a second check is a few extra seconds and a bit of API spend. The cost of shipping the wrong number is usually much higher.

That combination, self-consistency and verification, is the difference between an answer that sounds right and one you've actually checked.

FAQ

Common questions

• Why does Claude give me a different answer when I ask the same question twice?

  Claude doesn't retrieve a fixed answer, it generates one word at a time based on probability, so small differences can compound into a different final answer. This is normal, not a bug. It's exactly why self-consistency, asking the same question a few times and comparing, is worth doing on anything that matters.

• Can I trust Claude when it says its own answer is correct?

  Not fully. A model reviewing its own work tends to favor what it already wrote, a pattern called self-preference bias. Self-critique catches real mistakes, but treat a self-graded "looks good" as a first pass, not a verdict, especially on code or math.

• Is LLM-as-judge the same thing as asking Claude to double-check its work?

  They're related but not identical. Self-critique means the same model reviews its own answer. LLM-as-judge means a separate prompt, sometimes even a separate model, compares two or more outputs and picks the stronger one. The separation is what makes judging less biased than self-review.

• Do I need to do this for every prompt I write?

  No. Save it for anything with real stakes: numbers you'll act on, code you'll run, claims you'll repeat to someone else. For a first draft of a tweet or a quick summary, one answer is fine. Match the checking to what's actually riding on the answer.