Chain of Thought Prompting: The Rules Just Changed

You had a prompt that worked. Every time you added "let's think step by step" to the end of it, the model's math checked out, its logic held together, and the answer got noticeably better. Then you pointed the exact same trick at a newer model, one built specifically to reason. The answer came back no better, sometimes worse, after a longer wait and a bigger bill.

That's not a fluke, and it's not a bug in your prompt. Chain-of-thought prompting, asking a model to reason step by step before it answers, still works. But the rules for when and how changed the moment reasoning models showed up, and almost nothing written about this technique mentions the flip. This lesson covers both halves: the technique itself, and the twist nobody warns you about.

Key Takeaways

• Chain-of-thought (CoT) prompting means asking a model to reason step by step before it answers, either with a trigger phrase or worked examples
• On standard models, CoT reliably improves multi-step tasks like math, logic, and planning, but adds little to simple factual questions
• Reasoning models like OpenAI's o-series and Claude's extended thinking already reason internally, so bolting on "think step by step" is often redundant and can hurt accuracy
• A scratchpad, an explicit space for the model to plan before answering, is chain-of-thought's more structured cousin, useful for agentic and multi-step tool-use tasks
• Check which kind of model you're talking to before choosing a technique: the same instruction helps one and wastes tokens on the other

What Is Chain-of-Thought Prompting?

Chain-of-thought (CoT) prompting, also written without hyphens as chain of thought prompting, means asking a model to work through a problem in visible, intermediate steps before it states a final answer, instead of jumping straight to a conclusion. Instead of "what's 17% of 240, plus 12?" and hoping for the best, you get the model to show its arithmetic first, then land on the number.

The idea comes from a 2022 research paper that tested this on large language models. Researchers found that models told to reason step by step got dramatically better at multi-step math and logic problems than models asked for a direct answer. But the effect mostly showed up once models were large enough to reason coherently in the first place. According to the original chain-of-thought research (arXiv), this reasoning ability "emerges naturally" in sufficiently large models. Smaller models sometimes produced reasoning that looked plausible but was actually wrong, which made their final answers worse, not better.

That distinction matters more than most beginner guides let on: CoT isn't a universal upgrade. It's a technique that trades extra tokens and time for a shot at better reasoning, and that trade only pays off under specific conditions covered later in this lesson. Get the conditions wrong and you either pay for reasoning a simple question never needed, or ship a confident, fluent, wrong answer with no visible reasoning to catch the mistake before it reaches someone.

You've already used pieces of this puzzle without naming them. The six parts of a prompt named the task as one ingredient, but never said how much room the model gets to work through that task before answering. Controlling the output in the last lesson was about shaping the answer; this lesson is about shaping what happens right before it.

Zero-Shot vs. Few-Shot: Two Ways to Ask for Reasoning

There are two ways to trigger chain-of-thought reasoning, and they differ in how much work you do up front.

Zero-shot CoT is the simplest version: you add a short instruction, like "let's think step by step," and let the model generate its own reasoning structure from scratch. No examples required.

Question: A store had 84 shirts. It sold 3/7 of them on Monday
and half of what remained on Tuesday. How many shirts are left?

Let's think step by step.

Few-shot CoT goes further: you show the model one or two worked examples, each with the reasoning spelled out, before asking your real question. The model imitates that pattern.

Question: A bakery had 60 muffins and sold 1/4 of them in the morning.
How many are left?
Reasoning: 60 muffins × 1/4 = 15 sold. 60 − 15 = 45 remaining.
Answer: 45

Question: A store had 84 shirts. It sold 3/7 of them on Monday
and half of what remained on Tuesday. How many shirts are left?
Reasoning:

Few-shot CoT tends to produce more consistent, better-structured reasoning than zero-shot. That's the same principle behind showing the model examples in the last few-shot lesson: the model has a concrete pattern to follow instead of inventing its own. The tradeoff is a longer prompt and more setup work on your end. For most day-to-day use, the one-line "let's think step by step" trigger gets you most of the benefit for a fraction of the effort. Skip it entirely on a genuinely multi-step problem, though, and you'll end up with an answer that sounds sure of itself and simply isn't.

The Scratchpad Prompting Technique: Giving the Model a Place to Plan

Scratchpad prompting is chain-of-thought's more structured cousin. Instead of letting the model narrate reasoning inline with its answer, you give it an explicit, clearly marked space to plan first. Then it gets a separate space for the real response.

Before answering, write your plan inside <scratchpad> tags.
Then write your final answer inside <answer> tags.

<scratchpad>
[model's planning happens here]
</scratchpad>

<answer>
[final response only]
</answer>

This structure earns its keep on multi-step or agentic tasks: several tool calls, a plan that might need revising mid-task, or a sequence of decisions that build on each other. Anthropic's own engineering team tested this directly, giving Claude a dedicated "think" step during a multi-step airline-booking task. Task completion rose from roughly 0.37 to 0.57 on an internal benchmark, according to Anthropic's engineering blog. That's a real, measured gain, but notice the setting: sequential decisions with real stakes for getting them wrong, not a one-shot factual question.

That's also a different job than the one this technique does for a chatbot. In an agent, that same reasoning step happens before a tool call, so it's shaping which action gets taken, not just which sentence gets written. The agentic version of this walks through exactly where that reasoning earns its tokens and where it doesn't.

The Modern Twist: Chain of Thought Prompting Meets Reasoning Models

Here's the part that trips up almost everyone who learned prompting before 2025: reasoning models change the advice.

Models like OpenAI's o1 and o3, DeepSeek R1, and Claude with extended thinking enabled aren't just larger versions of the models chain-of-thought prompting was built for. They're built to reason internally before producing a visible answer, generating their own hidden reasoning steps whether you ask for them or not. According to Anthropic's own documentation, extended thinking creates genuine internal reasoning tokens that Claude uses to work through a problem. That's not just text that narrates thinking after the fact, the way a standard model's chain-of-thought output does.

That difference is exactly why bolting "think step by step" onto a reasoning-model prompt often does nothing useful, and sometimes makes things worse. OpenAI's reasoning best practices documentation is direct about it: since these models perform reasoning internally, prompting them to "think step by step" or "explain your reasoning" is unnecessary. It recommends keeping prompts simple, direct, and specific about the end goal instead of prescribing how to get there.

I've watched this trip up prompt writers who are anything but beginners, not just people new to the field. They built solid chain-of-thought habits on older models, moved to a reasoning model without noticing, and kept doing the thing that used to help. The habit is the trap, not the lack of skill.

The honest takeaway isn't "chain-of-thought is dead." It's that the technique's designers built it to compensate for something standard models don't do on their own: pause and reason before answering. Reasoning models don't have that gap, so the same fix has nothing left to fix. That's a direct consequence of how a model generates tokens in the first place. A standard model predicts its next token immediately. A reasoning model spends hidden tokens deliberating first.

When to Use Which

The choice comes down to two questions: what kind of model are you using, and how many reasoning steps does the task actually need.

Chain-of-thought decision guide