CodingSoftware & developer tools
Simon Willison releases ttok 0.4, then ships ttok 1.0 a day later with o200k_base as the default tokenizer
Simon Willison's ttok, a command-line wrapper around OpenAI's tiktoken for counting and truncating text by tokens, got its first update in about two years with version 0.4. A day later, version 1.0 switched the default tokenizer to o200k_base.
Reach: not measured yet
AT Pulse research deskPublished · 4 min read · AI-assisted analysisttok 0.4 (8 October 2026) was mainly a maintenance release. It fixed a Click warning on --tokens, made --allow-special work for counting and truncation, added a --list-models command, required Python 3.10 or later and moved packaging to pyproject.toml. The tool can also be run without installing it, through uvx. [1] [4] [5] [6]
The evidence is the author's release notes and tiktoken's source code: the default changed, and tiktoken maps model names to tokenizers.
Read the full assessment
The implication is that scripts relying on default counts or truncation limits may shift after upgrading, so they should pin the tokenizer explicitly.
Executive brief
The ttok 0.4 release that Simon Willison posted on 8 October 2026 was replaced the next day. ttok 1.0 (9 October) changed the default tokenizer to o200k_base, which the release notes tie to the GPT-5 and GPT-6 families. ttok is a small command-line tool for counting and truncating text by tokens. Version 0.4 was its first release in about two years and was mostly maintenance work. If you upgrade, the default token counts change too. ttok only counts tokens accurately for OpenAI tokenizers. No independent coverage or testing of the release exists.
Read the full section
The ttok 0.4 release that Simon Willison posted on 8 October 2026 was replaced the next day. ttok 1.0 (9 October) changed the default tokenizer to o200k_base, which the release notes tie to the GPT-5 and GPT-6 families. ttok is a small command-line tool for counting and truncating text by tokens. It wraps OpenAI's open-source tiktoken library. Version 0.4 was its first release in about two years and was mostly maintenance work. If you upgrade, the default token counts change too. ttok only counts tokens accurately for OpenAI tokenizers. No independent coverage or testing of the release exists.
What changed and event timeline
Ttok launches
Willison built it alongside
strip-tagsas a thin wrapper over tiktoken, so token counts and truncation could be used in Unix pipelines that feed LLMs ().Ttok 0.4 released
Fixes Click warnings on
--tokens, makes--allow-specialwork for counting and truncation, adds--list-models, requires Python 3.10+ and moves packaging to pyproject.toml ().The blog post
Willison describes the update as a fix for a Click warning, updated CI and the new
--list-modelscommand. He also shows that the tool runs without installation throughuvx().Ttok 1.0 released
The default tokenizer becomes
o200k_base, and--list-modelsnow shows how model-name prefixes such asgpt-5*are resolved ().
Capabilities and access
- Versions: 0.4 (8 Oct 2026) and 1.0 (9 Oct 2026) (releases).
- Features: count tokens, truncate with
-t, show token IDs with--encode, turn IDs back into text with--decode, show a readable breakdown with--tokens, choose a tokenizer with-m, list models with--list-models, and accept special tokens with--allow-special(README). - Install: pip, Homebrew (
simonw/llm/ttok) oruvx ttok(README; PyPI).
Read the full section
- Versions: 0.4 (8 Oct 2026) and 1.0 (9 Oct 2026) (releases).
- Features: count tokens, truncate with
-t, show token IDs with--encode, turn IDs back into text with--decode, show a readable breakdown with--tokens, choose a tokenizer with-m, list models with--list-models, and accept special tokens with--allow-special(README). - Install: pip, Homebrew (
simonw/llm/ttok) oruvx ttok(README; PyPI).
Technical analysis for researchers and developers
- ttok does no tokenization itself.
- tiktoken picks a tokenizer from the model name in two ways. An unrecognised name raises a
KeyError(tiktoken model.py). - The new
--list-modelsoutput exposes this lookup, so you can see which tokenizer a model name will actually use.
Read the full section
- ttok does no tokenization itself. It passes the work to tiktoken, OpenAI's byte-pair-encoding tokenizer, which is MIT-licensed and turns text into tokens and back without losing anything (tiktoken).
- tiktoken picks a tokenizer from the model name in two ways. Exact names like
gpt-5map too200k_base. Name prefixes likeo1-,o3-andgpt-4o-also map too200k_base, whilegpt-4-andgpt-3.5-turbo-map tocl100k_base. An unrecognised name raises aKeyError(tiktoken model.py). - The new
--list-modelsoutput exposes this lookup, so you can see which tokenizer a model name will actually use. - If you need counts to be reproducible, pin both the ttok version and the tokenizer explicitly. The default changed between 0.4 and 1.0.
Claims and evidence
- The tool had gone unmaintained for about two years: stated by the author (blog).
- tiktoken is 3–6x faster than a comparable open-source tokenizer: this is OpenAI's own figure, measured on 1 GB of text against older
tokenizers/transformersversions (tiktoken). - The default model is unclear across sources. Older package listings say the default is the GPT-3.5/GPT-4 tokenizer (Datasette tools listing).
Read the full section
- The tool had gone unmaintained for about two years: stated by the author (blog). The release history is consistent with this: 0.3 is the previous release (releases).
- tiktoken is 3–6x faster than a comparable open-source tokenizer: this is OpenAI's own figure, measured on 1 GB of text against older
tokenizers/transformersversions (tiktoken). It has not been independently re-run. - The default model is unclear across sources. Older package listings say the default is the GPT-3.5/GPT-4 tokenizer (Datasette tools listing). The current README and the 1.0 notes say
o200k_base(README). Which default you get depends on the version installed.
Context and prior work
- ttok was part of Willison's 2023 Unix-pipe toolkit. In his New York Times homepage example, raw HTML came to 210,544 tokens, stripping tags cut it to 115,117, and filtering with CSS selectors cut it to 2,165 (Willison, 2023).
- Other projects also put a command-line interface on tiktoken, for example tiktoken-cli on PyPI and the .NET Tiktoken.Cli.
Read the full section
- ttok was part of Willison's 2023 Unix-pipe toolkit. In his New York Times homepage example, raw HTML came to 210,544 tokens, stripping tags cut it to 115,117, and filtering with CSS selectors cut it to 2,165 (Willison, 2023).
- Other projects also put a command-line interface on tiktoken, for example tiktoken-cli on PyPI and the .NET Tiktoken.Cli.
Limitations, safety and contested findings
- ttok counts tokens the way OpenAI's tokenizers do. Willison's 2023 post notes it is calibrated for OpenAI models (Willison, 2023).
- Other vendors' tokenizers can give very different counts. It recommends counting tokens through its own API, whose counts are themselves estimates (Anthropic token counting docs).
- Scripts that relied on the old default tokenizer may report different numbers after upgrading to 1.0.
Read the full section
- ttok counts tokens the way OpenAI's tokenizers do. Willison's 2023 post notes it is calibrated for OpenAI models (Willison, 2023).
- Other vendors' tokenizers can give very different counts. Anthropic says Claude Opus 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text than earlier models. It recommends counting tokens through its own API, whose counts are themselves estimates (Anthropic token counting docs).
- Scripts that relied on the old default tokenizer may report different numbers after upgrading to 1.0.
Business and practitioner implications
- Budget checks: ttok is a cheap, offline way to estimate prompt size and cost for OpenAI models.
- Multi-vendor stacks: don't use tiktoken counts for budgeting or context-window checks on other vendors' models. Use each provider's own counter, such as Anthropic's free count_tokens endpoint.
- Upgrades: check any token-based limits or truncation after moving to 1.0, because the default tokenizer change will shift the numbers.
Read the full section
- Budget checks: ttok is a cheap, offline way to estimate prompt size and cost for OpenAI models. Running it through
uvxmeans it can go into CI or shell pipelines without a setup step. - Multi-vendor stacks: don't use tiktoken counts for budgeting or context-window checks on other vendors' models. Use each provider's own counter, such as Anthropic's free count_tokens endpoint.
- Upgrades: check any token-based limits or truncation after moving to 1.0, because the default tokenizer change will shift the numbers.