Tokens & Context
Claude tokens: the definitive guide
A token is the unit everything else on this site is measured in — your bill, your limits, your context window. Understanding it properly is what turns cost from something that happens to you into something you control.
What a token is
A token is a chunk of text, usually a word fragment. Common words are one token; longer or unusual words split into several; punctuation and whitespace count too. For English prose a useful approximation is about 0.75 words per token, or roughly four characters. Code is denser — indentation, braces and identifiers all consume tokens, so a file costs more than its word count suggests.
Models do not read characters or words. They read tokens, and every price, limit and context window is expressed in them.
Input and output are not the same thing
| Direction | What it is | Relative cost | Who controls it |
|---|---|---|---|
| Input | Everything you send, including history | Cheaper per token | You, almost entirely |
| Output | Everything Claude writes back | Several times input | You, via instructions |
Output costs more per token, but input almost always dominates the total, because input includes the entire conversation so far. This is the single most misunderstood fact about LLM cost.
The context window, and why it fills up
The context window is how much the model can hold at once: your system prompt, the whole conversation, any attached files, and the answer being generated. It is not memory that persists between conversations — it is a working surface rebuilt on every single turn.
That rebuild is the crux. Models are stateless. When you send message twenty, the model does not remember messages one through nineteen; they are re-sent with your new question. You pay for all of them again, every turn.
Worked example: the quadratic problem
Suppose each exchange adds 1,000 tokens of conversation. On turn one you send 1,000 tokens. On turn two, 2,000. On turn twenty, 20,000. Across the whole twenty-turn conversation you have sent around 210,000 input tokens to produce 20,000 tokens of actual new content.
Attach a 15,000-token file at the start and every one of those twenty turns carries it. The file alone costs 300,000 tokens across the session, for a document that was sent once and was relevant to perhaps three questions.
Where tokens actually go
| Source | Typical share | Avoidable? |
|---|---|---|
| Re-sent conversation history | 50–70% | Mostly — restart threads |
| Attached files and documents | 10–30% | Mostly — send excerpts |
| Restated standing instructions | 5–15% | Entirely — use project instructions |
| Model output | 10–20% | Partly — ask for diffs, cap length |
| Genuinely new input | 5–15% | No |
The habits that waste the most
- The immortal thread. One conversation used for unrelated tasks all day. Costs more than any other single habit.
- Whole-file pasting. Sending 900 lines to change 10.
- Restating the rules. “Remember, TypeScript only” in every message.
- Uncapped output. Asking for a full rewrite when a diff would do.
- Politeness padding. Harmless individually, measurable at volume.
Counting tokens without a tokenizer
- English prose: words ÷ 0.75.
- Code: characters ÷ 3.5 is closer than ÷ 4.
- A typical page of text: roughly 500 tokens.
- A 300-line source file: roughly 4,000–6,000 tokens.
Where to go next
If you want the numbers: API pricing and the cost calculator. If you want to stop being cut off: usage limits. If you want to change how you work: the optimization playbook.