Prompting Techniques for Reasoning: CoT, Self-Consistency, ToT

A maths teacher at a coaching centre in Vijay Nagar sets one question: "A bus goes from Indore to Ujjain at 40 km/h and returns at 60 km/h. What is its average speed?" She does not trust one student's answer. She asks five students to solve it on their own sheets.

Three write 48 km/h, one writes 50 and one writes 45. She goes with 48, because independent solvers rarely make the same slip. Her topper works differently: she sketches two or three possible approaches in the margin, crosses out the ones that lead nowhere and only finishes the most promising one.

Both habits work for language models too.

What are reasoning prompting techniques?

Reasoning prompting techniques are ways of structuring an LLM's prompt, and sometimes several calls to it, so that the model works through intermediate steps before answering, which improves accuracy on problems that need more than one step.

Mapping the classroom: one student writing out working is chain-of-thought (CoT). Five students voting is self-consistency. The topper exploring and crossing out approaches is Tree of Thoughts (ToT). The teacher's final pick is the aggregation step that your code performs.

The prompt engineering guide introduced "think step by step". This tutorial covers sampling, voting and searching, and when the extra cost pays off.

Chain-of-thought: zero-shot vs few-shot

An LLM produces one token at a time, and every token it writes becomes context for the next one (see how decoding works). If it must jump straight to "48", all the arithmetic has to happen in one prediction. Writing "time out = d/40, time back = d/60" first gives later tokens something to lean on.

There are two common forms:

• Zero-shot CoT adds an instruction such as "Reason step by step, then write the final answer on a line starting with ANSWER:".
• Few-shot CoT shows two or three solved examples whose answers include the working. The model copies the style of reasoning as well as the format.

Few-shot CoT suits problems that share a pattern, such as GST calculations, because the examples teach the method. Zero-shot CoT is the default when problems vary.

How does self-consistency work?

Self-consistency, proposed by Google researchers in 2022, sends the same CoT prompt N times with a temperature above zero, so each run takes a different path, then keeps the most common answer.

Different paths, one vote. The working is thrown away; only the extracted final answers are counted.

Wrong paths tend to end in different wrong answers, while correct paths converge. At temperature zero every sample is identical, so voting adds nothing; the generation parameters guide explains how temperature creates variety.

A simulation shows the effect. Each path is right with some probability, otherwise it lands on a slip. In the last row most mistakes are the same slip, 50 km/h, the tempting arithmetic mean:

import random
from collections import Counter

random.seed(7)

def sample_path(p_correct, slips, slip_weights):
    if random.random() < p_correct:
        return "48"
    return random.choices(slips, weights=slip_weights)[0]

def majority(n, p, slips, weights):
    votes = Counter(sample_path(p, slips, weights) for _ in range(n))
    return votes.most_common(1)[0][0]

scenarios = {
    "scattered slips, p=0.40": (0.40, ["40", "45", "50", "54", "60"], [1, 1, 1, 1, 1]),
    "scattered slips, p=0.60": (0.60, ["40", "45", "50", "54", "60"], [1, 1, 1, 1, 1]),
    "one common slip, p=0.40": (0.40, ["50", "45"], [9, 1]),
}
trials = 2000
for name, (p, slips, w) in scenarios.items():
    results = []
    for n in (1, 5, 9):
        hits = sum(majority(n, p, slips, w) == "48" for _ in range(trials))
        results.append(f"n={n}: {hits / trials:5.0%}")
    print(f"{name:<26}", " | ".join(results))

With scattered slips, a model right 40 percent of the time climbs to about 70 percent with 9 votes. With one dominant slip, voting makes things worse. That is the key caveat of self-consistency.

Turning votes into a confidence score

If 9 of 9 paths agree, you can trust the answer more than a 4, 3 and 2 split. Production systems use the vote share to abstain: below a threshold, the question goes to a human or a stronger model.

Answers also need normalising before counting, because "48 km/h", "48 kmph" and "48" are the same vote:

import re
from collections import Counter

paths = [
    "Time out = d/40, time back = d/60 ... ANSWER: 48 km/h",
    "Average of 40 and 60 is 50. ANSWER: 50 km/h",
    "Total distance 2d over total time d/24 ... ANSWER: 48 kmph",
    "Harmonic mean 2*40*60/100 = 48. ANSWER: 48",
    "I am not sure about this one.",
]

def extract(text):
    match = re.search(r"ANSWER:\s*([-\d.]+)", text)
    return match.group(1) if match else None

def vote(texts, min_share=0.6):
    answers = [a for a in (extract(t) for t in texts) if a is not None]
    if not answers:
        return None, 0.0
    best, count = Counter(answers).most_common(1)[0]
    share = count / len(texts)               # unparseable paths count against confidence
    return (best if share >= min_share else "ABSTAIN"), share

print("extracted:", [extract(p) for p in paths])
print("decision, share:", vote(paths))
print("stricter threshold:", vote(paths, min_share=0.8))

Tree of Thoughts: search instead of a single path

Self-consistency compares only the endings. Tree of Thoughts, introduced by Princeton and Google DeepMind researchers in 2023, works on steps: the model proposes several next moves, an evaluator (often the same model asked to "rate this partial solution") scores them, and only the best few branches are expanded.

ToT is a search algorithm. The LLM proposes and scores steps; your code decides which branches to keep.

It shines where an early wrong move dooms the attempt, such as timetables or the "make 24 from four numbers" game. In this toy, a cheap scoring function stands in for the LLM evaluator, and the beam width sets how many branches survive each level:

from fractions import Fraction as F
from itertools import combinations

def next_states(nums):
    for i, j in combinations(range(len(nums)), 2):
        a, b = nums[i], nums[j]
        rest = [n for k, n in enumerate(nums) if k not in (i, j)]
        options = {a + b, a * b, a - b, b - a}
        options |= ({a / b} if b else set()) | ({b / a} if a else set())
        for value in options:
            yield tuple(sorted(rest + [value]))

def score(state):                    # cheap, imperfect judge: "does this look close to 24?"
    if len(state) == 1:
        return 100 if state[0] == 24 else -100
    pair_hits = sum(24 in {a + b, a * b, abs(a - b)} for a, b in combinations(state, 2))
    whole = all(n.denominator == 1 for n in state)
    return 10 * pair_hits + 3 * whole - min(abs(n - 24) for n in state) / 4

def tree_of_thoughts(start, beam):
    frontier, explored = [tuple(map(F, start))], 0
    while len(frontier[0]) > 1:
        children = {s for state in frontier for s in next_states(list(state))}
        explored += len(children)
        frontier = sorted(children, key=score, reverse=True)[:beam]
    return frontier[0][0] == 24, explored

for beam in (1, 3, 10):
    solved, explored = tree_of_thoughts([4, 7, 8, 8], beam)
    print(f"beam={beam:<2}  solved={solved!s:<5}  states scored={explored}")

With a beam of 1 the search is a single greedy chain of thought and hits a dead end. Wider beams find (7 - 8/8) × 4 = 24, but each scored state would be an LLM call in a real system, which is why ToT is kept for a few hard problems.

Structured reasoning and diversity prompts

Two more techniques solve problems that free-form CoT does not.

Structured reasoning checklists. In long, rule-heavy chats such as a refund bot, free-form reasoning drifts and forgets rules stated 20 turns ago. A fix, sometimes called attentive reasoning queries, makes the model first answer fixed questions as JSON: "Which policy applies?", "Is a refund allowed under it?".

Each answer restates a rule just before the reply, and code can check it. The structured output tutorial shows how to enforce the schema.

Verbalized sampling for variety. Preference-tuned models tend to repeat one "safe" answer, a side effect of how LLMs are trained. Asking "Give 5 different answers with a probability for each" draws out more diverse options. It is a creativity tool, not an accuracy tool.

Which reasoning technique should you use?

Follow these steps to pick one:

1. Build a test set of 30 to 100 questions with known answers.
2. Measure a direct answer and zero-shot CoT.
3. Try self-consistency with 5 samples if wrong answers vary.
4. Use ToT only for search-shaped problems.
5. Compare accuracy per rupee.

What about reasoning models?

Models trained to reason with reinforcement learning (see SFT vs RLHF vs GRPO) already write a long chain of thought, so "think step by step" adds little. Self-consistency still works, but each sample is expensive.

Self-consistency with a real API

This version calls a local model through Ollama's OpenAI-compatible endpoint, so you can try it on a laptop (see running LLMs locally). Change base_url, api_key and MODEL for a hosted provider:

import re
from collections import Counter
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # local Ollama server
MODEL = "llama3.2"   # any chat model you have pulled

PROMPT = ("A bus goes from Indore to Ujjain at 40 km/h and returns at 60 km/h. "
          "What is the average speed for the round trip? Reason step by step, "
          "then give the final number on a last line starting with ANSWER:")

answers = []
for _ in range(7):
    reply = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": PROMPT}],
        temperature=0.8,          # variety between paths is the whole point
        max_tokens=400,
    )
    match = re.search(r"ANSWER:\s*([-\d.]+)", reply.choices[0].message.content)
    answers.append(match.group(1) if match else None)

votes = Counter(a for a in answers if a)
print(answers)
print(votes.most_common(3))

Some APIs accept an n parameter that returns several completions in one call, saving input tokens.

Common mistakes with reasoning prompts

• Voting at temperature 0. Every sample is identical, so you pay N times for one answer.
• Counting the working instead of the answer. Vote only on a normalised final answer.
• Voting on open-ended text. Summaries have no single answer; use a rubric or judge model.
• Expecting votes to fix systematic errors. A misread question makes every path wrong.
• Ignoring cost. Ten 400-token samples cost about ten times one answer.

Key takeaways

• Chain-of-thought gives the model room to compute; always end with a fixed answer marker.
• Self-consistency samples several paths and votes; the vote share doubles as a confidence score.
• Voting helps only when errors are scattered. Correlated errors win votes too.
• Tree of Thoughts searches over scored partial steps and is worth its cost only on search-shaped problems.
• Choose by measuring accuracy and cost on your own test set.