CodingSoftware & developer tools
llm-echo 0.5 adds system prompts, tool calls and reasoning to simulated token counts
Version 0.5 of llm-echo, a fake model plugin for Simon Willison's LLM command-line tool, now includes system prompts, tool calls and reasoning in its simulated token usage. Developers can test usage accounting deterministically without calling any real model.
From the Simon Willison article of October 18, 2026 and cited sources: llm-echo provides an 'echo' model that sends each prompt back as JSON. [1] [4] [5]
SOURCES[1] llm-echo 0.5 simonwillison.net[4] simonw/llm-echo GitHub releases github.com[5] Simon Willison: llm-echo 0.4 (Mar 31, 2026) simonwillison.net
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- Simon Willison article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 18, 2026 article published
- Covered
- Oct 19 edition
- Evidence
- 7 sources across 3 sites
- 1llm-echo 0.5simonwillison.net
- 2Simon Willison: new release of LLM (0.32, Aug 4, 2026)simonwillison.net
- 3Simon Willison: llm-echo 0.2 (May 4, 2025)simonwillison.net
- 4simonw/llm-echo GitHub releasesgithub.com
- 5Simon Willison: llm-echo 0.4 (Mar 31, 2026)simonwillison.net
- 6Simon Willison: llm-echo 0.5a0 (May 5, 2026)simonwillison.net
Reach: not measured yet
llm-echo provides an 'echo' model that sends each prompt back as JSON. The JSON includes the system prompt, attachments and conversation history. Tests run with no API calls, network access or model weights. Version 0.5 extends the simulated usage figures to cover system prompts, tool calls and reasoning tokens, so the token fields added in 0.4 now reflect more of what a real exchange contains. [1] [4] [5]
The plugin has tracked LLM's move to structured messages. LLM 0.32 turned prompts and responses into typed parts: text, reasoning, tool calls, tool results and attachments. The 0.5a0 pre-release added a simulated reasoning block for testing against that design. The final 0.5 release requires LLM 0.36 or later. [2] [4] [6]
Willison returned to the plugin while testing pull requests for an upcoming LLM release. Plugin maintainers should therefore expect more compatibility work. [1]
The echo model is deterministic, and its usage now covers reasoning and tool calls. Plugin authors and CI owners can therefore check token accounting for agent-style workflows cheaply and repeatably.
Read the full assessment
The counts are test fixtures, not estimates of what a real tokenizer would produce.
Executive brief
The most useful thing about llm-echo 0.5 is that it contains no model. The plugin adds an echo "model" to Simon Willison's LLM command-line tool that returns the prompt it receives as JSON, so tests can run without API calls. In version 0.5 the simulated token counts now include system prompts, tool calls and reasoning tokens (Willison). The release supports the structured-message redesign that arrived in LLM 0.32 (Willison, LLM 0.32). It is a small tool, but it matters to anyone writing plugins or tests that depend on usage accounting. No independent coverage of 0.5 was found.
What changed and event timeline
Llm-echo 0.2 ships
An early release of the debug plugin that offers a fake echo model; the release notes add example-option support (;).
Token fields arrive
Version 0.4 fills in
input_tokensandoutput_tokenson echo responses, so tests can check usage reporting ().Simulated thinking
The 0.5a0 pre-release adds
-o thinking 1, which writes a fake reasoning block for testing against LLM 0.32a0 and later ().LLM 0.32 restructures messages
Prompts and responses become typed parts (text, reasoning, tool calls, tool results and attachments), and reasoning traces now go to stderr. Willison calls it the project's most significant release since launch (;).
Llm-echo 0.5 final
Simulated token usage now counts tool calls, system prompts and reasoning. Willison revisited the plugin while testing pull requests for an upcoming LLM release ().
Capabilities and access
- Package: llm-echo 0.5, which requires LLM 0.36 or later according to the GitHub releases page.
- Usage:
uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage. - Other test models: earlier releases added mechanisms for testing tool calls and raw responses, plus an
echo-needs-keymodel for testing API-key logic (GitHub releases).
Read the full section
- Package: llm-echo 0.5, which requires LLM 0.36 or later according to the GitHub releases page.
- Usage:
uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage. This prints a canned reasoning line, then a JSON echo with the fieldsprompt,system,attachments,streamandprevious, then a usage line (2 input, 24 output tokens in Willison's example) (Willison). - Other test models: earlier releases added mechanisms for testing tool calls and raw responses, plus an
echo-needs-keymodel for testing API-key logic (GitHub releases). - Cost: free and open source. No network calls or model weights are involved.
Technical analysis for researchers and developers
- Why it exists: LLM 0.32 changed responses from plain string iterables to typed event streams, and model plugins had to upgrade to support it (Willison, LLM 0.32).
- Token counts are simulated: the reviewed sources do not document how the counts are computed.
- Reproducibility: output is deterministic, which suits CI snapshot tests. That is the main advantage over mocking a provider's SDK.
Read the full section
- Why it exists: LLM 0.32 changed responses from plain string iterables to typed event streams, and model plugins had to upgrade to support it (Willison, LLM 0.32). A deterministic echo model lets developers test that plumbing (system prompts, conversation history in
previous, attachments, tool calls and reasoning on stderr) without depending on a real provider. - Token counts are simulated: the reviewed sources do not document how the counts are computed. Treat them as fixtures for tests, not as estimates of what a real tokenizer would count.
- Reproducibility: output is deterministic, which suits CI snapshot tests. That is the main advantage over mocking a provider's SDK.
Claims and evidence
- The echo model never calls an LLM and only repeats its prompts. The author states this (Willison), and the open-source repository allows inspection (GitHub).
- 0.5 counts tool calls, system prompts and reasoning in usage. Both the blog post and the release notes say so, but both are the author's own claims.
- A new LLM release is in progress. This rests only on Willison's statement (Willison).
Read the full section
- The echo model never calls an LLM and only repeats its prompts. The author states this (Willison), and the open-source repository allows inspection (GitHub).
- 0.5 counts tool calls, system prompts and reasoning in usage. Both the blog post and the release notes say so, but both are the author's own claims.
- A new LLM release is in progress. This rests only on Willison's statement (Willison). No independent corroboration of 0.5 or of the upcoming LLM release was found.
Context and prior work
- llm-echo has followed LLM's feature changes step by step: JSON inspection of model input first, then tool-call testing, token fields in 0.4 and simulated reasoning in 0.5a0 (GitHub releases).
- The LLM 0.32 redesign added server-side tools from OpenAI and Anthropic, tool loops that can pause for approval and resume, and a
model.prompt(messages=[])API (Willison). Each of these makes deterministic test doubles more useful.
Read the full section
- llm-echo has followed LLM's feature changes step by step: JSON inspection of model input first, then tool-call testing, token fields in 0.4 and simulated reasoning in 0.5a0 (GitHub releases).
- The LLM 0.32 redesign added server-side tools from OpenAI and Anthropic, tool loops that can pause for approval and resume, and a
model.prompt(messages=[])API (Willison). Each of these makes deterministic test doubles more useful.
Limitations, safety and contested findings
- The token numbers are synthetic and cannot validate billing or context-window behaviour against real providers.
- 0.5 requires LLM 0.36 or later (GitHub releases), so it does not fit test suites pinned to older LLM versions.
- The release notes do not document the counting method, so assertions that depend on exact token values could break in later versions.
Read the full section
- The token numbers are synthetic and cannot validate billing or context-window behaviour against real providers.
- 0.5 requires LLM 0.36 or later (GitHub releases), so it does not fit test suites pinned to older LLM versions.
- The release notes do not document the counting method, so assertions that depend on exact token values could break in later versions.
- No contested findings or safety issues appear in the reviewed sources.
Business and practitioner implications
- Plugin authors and CI owners: echo-style models make tests free, fast and deterministic.
- Teams building on LLM: usage accounting that includes reasoning and tool calls is now testable, and that matters as agents with reasoning models drive token spend.
- Ecosystem watchers: the release signals that another LLM version is coming, so plugin maintainers should expect compatibility work.
Read the full section
- Plugin authors and CI owners: echo-style models make tests free, fast and deterministic. That fits the cost-control theme Willison raised in an October 3 post arguing for default hard budget caps (Willison).
- Teams building on LLM: usage accounting that includes reasoning and tool calls is now testable, and that matters as agents with reasoning models drive token spend.
- Ecosystem watchers: the release signals that another LLM version is coming, so plugin maintainers should expect compatibility work.
The source trail.
Sources (7)
llm-echo 0.5
Article text retrieved; extracted text may omit tables or interactive elements.
simonwillison.net