CodingSoftware & developer tools
Simon Willison releases ttok 1.0, making o200k_base the default tokenizer and applying it to GPT-6
Version 1.0 of the ttok command-line token counter makes OpenAI's o200k_base encoding the default, which is a breaking change. Its author applies it to GPT-6 based on one community test with matching input token counts. OpenAI's tiktoken still has no GPT-6 mappings.
AT Pulse research deskOctober 9, 2026 · 4 min read · AI-assisted analysisttok 1.0 replaces the old GPT-3.5/GPT-4 default (cl100k_base) with o200k_base. Because this is a breaking change, it was released as 1.0 rather than 0.5. Scripts that rely on the default will now return different counts unless they pass -m gpt-3.5-turbo or pin the version. [2] [6] [7]
The claim that GPT-6 uses the GPT-5 tokenizer comes from one community experiment. In a single run, seven GPT-5.x and GPT-6 models returned an identical 44,794 input tokens across 31 fixture files, counted through the providers' own counting endpoints. Token IDs were never compared, and the experiment itself says the result doesn't prove the tokenizers are identical. [1] [3]
seven models gave identical input counts on 31 files. Implication: ttok's GPT-6 cost and context estimates rest on inference, not OpenAI confirmation.
Read the full assessment
Billing-critical work should check counts against provider counting endpoints, and pipelines should pin versions or pass -m.
Executive brief
OpenAI has not said which tokenizer GPT-6 uses, and its official tiktoken library still has no entry for GPT-6. On 9 October 2026 Simon Willison shipped ttok 1.0 anyway, making o200k_base (the GPT-5 tokenizer) the default and saying it covers GPT-6 too (Willison). His evidence is one community experiment. In it, seven GPT-5.x and GPT-6 models returned identical input token counts on 31 test files (Liu commit). The experiment's own write-up says this does not prove the tokenizers are the same. Teams estimating GPT-6 costs or context limits are relying on that inference.
What changed and event timeline
Tiktoken issue flags missing GPT-6 support
A user opened an issue saying tiktoken 0.14.0 doesn't support GPT-6, about 20 days after gpt-6-astra shipped. The thread shows no reply from OpenAI maintainers ().
Community token-count comparison
William Liu counted tokens for 14 models using the providers' own counting endpoints. All seven GPT models returned 44,794 tokens on 31 fixtures, with no provider errors ().
Ttok 0.4 ships
This was the first release in several years. It fixed a Click warning, updated CI and added
--list-models().Default-tokenizer issue filed and closed
Willison opened an issue noting the default was still the GPT-4 tokenizer. Because switching it was a breaking change, he called for 1.0 rather than 0.5 ().
Ttok 1.0 released
The default is now
o200k_base, and--list-modelsshows how prefixes likegpt-5*resolve (;).
Capabilities and access
- Tool:
ttok1.0, a Python command-line tool built on OpenAI'stiktoken. Install with pip, uv, Homebrew oruvx(README; PyPI). - Functions: count tokens, truncate to N tokens (
-t), and print token IDs or convert them back to text (--encode/--decode) (README). - Default:
o200k_base. To get the old GPT-3.5/GPT-4 tokenizer (cl100k_base), pass-m gpt-3.5-turbo(README).
Read the full section
- Tool:
ttok1.0, a Python command-line tool built on OpenAI'stiktoken. Install with pip, uv, Homebrew oruvx(README; PyPI). - Functions: count tokens, truncate to N tokens (
-t), and print token IDs or convert them back to text (--encode/--decode) (README). - Default:
o200k_base. To get the old GPT-3.5/GPT-4 tokenizer (cl100k_base), pass-m gpt-3.5-turbo(README). - Models covered by the evidence: GPT-5.5, GPT-5.6 Sol/Terra/Luna and GPT-6 Astra/Sol/Luna (Liu commit).
Technical analysis for researchers and developers
- How tiktoken picks a tokenizer: it matches model names against prefixes. There is no
gpt-6entry (tiktoken model.py). - How Liu measured: he called OpenAI's and Anthropic's official count endpoints directly rather than tokenizing locally.
- What this means in practice: matching counts are consistent with a shared tokenizer, but token IDs were never compared.
Read the full section
- How tiktoken picks a tokenizer: it matches model names against prefixes.
gpt-5maps too200k_base,gpt-4o-too200k_baseandgpt-4-tocl100k_base. There is nogpt-6entry (tiktoken model.py). A lookup by a GPT-6 model name therefore won't resolve, so callers need a fallback. - How Liu measured: he called OpenAI's and Anthropic's official count endpoints directly rather than tokenizing locally. The 31 fixture files had the same hashes across runs, and ten models measured earlier reproduced their previous totals exactly. The study covers input tokens only, with no check against generation or billed usage (Liu commit).
- What this means in practice: matching counts are consistent with a shared tokenizer, but token IDs were never compared.
Claims and evidence
- "GPT-6 uses the GPT-5 tokenizer." This is Willison's inference (post). OpenAI has not confirmed it (tiktoken #608). No second independent replication was found.
- "Seven models, 44,794 tokens, all 31 fixtures match." This comes from a single run on 2026-09-26 (Liu commit).
- "1.0 changes the default to
o200k_base." The release notes and the closing commit for issue #25 document this (release; ttok #25).
Read the full section
- "GPT-6 uses the GPT-5 tokenizer." This is Willison's inference (post). It rests on one community experiment, and that experiment explicitly says it does not establish that the tokenizers are identical (Liu commit). OpenAI has not confirmed it (tiktoken #608). No second independent replication was found.
- "Seven models, 44,794 tokens, all 31 fixtures match." This comes from a single run on 2026-09-26 (Liu commit).
- "1.0 changes the default to
o200k_base." The release notes and the closing commit for issue #25 document this (release; ttok #25).
Context and prior work
ttok dates from Willison's 2023 set of command-line tools for working with LLMs (2023 post). He also wrote an earlier explainer on how GPT tokenizers work (2023 explainer). OpenAI's cookbook recommends tiktoken for counting tokens (OpenAI cookbook). o200k_base already served the GPT-4o and GPT-5 lines (tiktoken model.py).
Limitations, safety and contested findings
- Liu's write-up warns that the totals don't imply a universal multiplier for other content (Liu commit).
- The person who opened the tiktoken issue pushed back: functional equivalence doesn't fix delayed official updates, which they called a recurring pattern (tiktoken #608).
- Willison acknowledged that 0.4 shipped what amounted to a breaking change (ttok #25). Any scripts that rely on the default will now return different counts.
Read the full section
- Liu's write-up warns that the totals don't imply a universal multiplier for other content (Liu commit).
- The person who opened the tiktoken issue pushed back: functional equivalence doesn't fix delayed official updates, which they called a recurring pattern (tiktoken #608).
- Willison acknowledged that 0.4 shipped what amounted to a breaking change (ttok #25). Any scripts that rely on the default will now return different counts.
Business and practitioner implications
- Budget and context limits: counts for GPT-6 inputs from 1.0 rest on inferred tokenizer equivalence, not vendor confirmation (tiktoken #608).
- Pipelines: pin
ttokversions, or pass-mexplicitly, so counts don't change silently between releases (README). - Dependency risk:
tiktokenstill lacked GPT-6 mappings 20+ days after release, so tools built on it need their own fallbacks (tiktoken model.py).
Read the full section
- Budget and context limits: counts for GPT-6 inputs from 1.0 rest on inferred tokenizer equivalence, not vendor confirmation (tiktoken #608). For billing-critical work, check counts against the provider's counting endpoint, as Liu's method does (Liu commit).
- Pipelines: pin
ttokversions, or pass-mexplicitly, so counts don't change silently between releases (README). - Dependency risk:
tiktokenstill lacked GPT-6 mappings 20+ days after release, so tools built on it need their own fallbacks (tiktoken model.py).