AgentsAutonomy & tool use
datasette-llm 0.1a10 fixes a filter-hook bug that let Datasette Agent's demo use excluded models
A new alpha of datasette-llm changes how multiple model-filter hooks combine, so each can only remove models. Previously a later hook, such as a pricing filter, could restore models that allowlists or blocklists had excluded, affecting the public Datasette Agent demo.
From the Simon Willison article of November 4, 2026 and cited sources: The bug came from how hooks were combined. [1] [2] [8]
SOURCES[1] Simon Willison's Weblog: datasette-llm 0.1a10 simonwillison.net[2] GitHub: PR #11, Preserve restrictions when combining model… github.com[8] PyPI: datasette-llm-accountant pypi.org
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- Simon Willison article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 4, 2026 article published
- Covered
- Nov 5 edition
- Evidence
- 9 sources across 4 sites
- 1Simon Willison's Weblog: datasette-llm 0.1a10simonwillison.net
- 2GitHub: PR #11, Preserve restrictions when combining model…github.com
- 3Datasette blog: Datasette Agentdatasette.io
- 4GitHub: datasette-llm release 0.1a10github.com
- 5GitHub: datasette-llm READMEgithub.com
- 6GitHub: datasette-llm releasesgithub.com
Reach: not measured yet
The bug came from how hooks were combined. The plugin passed the same starting model list to every llm_filter_models hook and kept only the last hook's output. A pricing filter from datasette-llm-accountant returned everything it received, which brought back models that the configuration had excluded. On agent.datasette.io, Datasette Agent then both offered and accepted those models. [1] [2] [8]
The fix keeps only the models that every hook allows, matched by model ID. A hook can now remove models but can never add them back. New regression tests cover sync and async filters, global and per-purpose settings, empty results and filters that change nothing. The pull request reports that all 75 tests pass. [2] [4]
Anyone using datasette-llm model allowlists should upgrade, especially alongside the accountant or spending-limit plugins. They should also check that a restricted model is actually rejected when requested, not just hidden from the list. The package is still in alpha, so pinning versions is prudent. [2] [5] [6] [9]
combining filters could quietly bypass configured model restrictions. Implication: teams that limit which models can be used to cap spending or risk should design filter hooks to only narrow results and should test enforcement on real requests.
Executive brief
A small alpha release of datasette-llm 0.1a10 fixed a bug that let the public agent.datasette.io demo offer LLM models its configuration should have excluded. The cause was in how plugin hooks were combined. When several llm_filter_models hooks ran, the last hook's output replaced the earlier ones instead of narrowing them. A cost-tracking plugin could therefore add back models that an allowlist or blocklist had removed, according to PR #11. Datasette Agent then both offered and accepted those restricted models. The fix now keeps only models that every filter allows. No independent coverage was found.
Read the full section
A small alpha release of datasette-llm 0.1a10 fixed a bug that let the public agent.datasette.io demo offer LLM models its configuration should have excluded. The cause was in how plugin hooks were combined. When several llm_filter_models hooks ran, the last hook's output replaced the earlier ones instead of narrowing them. A cost-tracking plugin could therefore add back models that an allowlist or blocklist had removed, according to PR #11. Datasette Agent then both offered and accepted those restricted models. The fix now keeps only models that every filter allows. Only maintainer materials document the bug. No independent coverage was found.
What changed and event timeline
Datasette Agent launches
An open-source AI assistant for SQLite databases that calls models through datasette-llm. Its public demo runs Gemini 3.1 Flash-Lite and requires GitHub sign-in.
PR #11 merged
"Preserve restrictions when combining model filter hooks" fixes cases where the pricing filter in datasette-llm-accountant brought back excluded models. It adds regression tests.
0.1a10 released
The release notes say the plugin had been overriding previously configured model allowlists. Willison connects the bug to his agent.datasette.io demo.
Capabilities and access
Read the full section
- Package: datasette-llm 0.1a10, an alpha "LLM integration plugin for other plugins to depend on" (post).
- What it provides: a shared way to choose models and manage API keys. It has
default_model, a globalmodelsallowlist,blocked_models, per-purpose overrides underpurposes.<name>, and arequire_keyssetting that defaults to true (README). - Python API:
llm.model(),llm.models()(filtered by actor and purpose) andllm.group()(README). - Release history: earlier alphas added purpose-specific API keys, per-model default options and default-model fallback (releases).
Technical analysis for researchers and developers
- Root cause:
LLM.models()gave everyllm_filter_modelshook the same starting list. It then replaced the running result with whatever each hook returned. - Fix: results are now intersected by model ID, so each hook can only remove models, never add them back.
- Tests: regression tests cover sync and async filters, global and per-purpose configuration, non-overlapping results, empty sets and a pricing filter that changes nothing.
Read the full section
- Root cause:
LLM.models()gave everyllm_filter_modelshook the same starting list. It then replaced the running result with whatever each hook returned. As a result, a hook that returned everything it was given could undo restrictions from earlier hooks (PR #11). - Fix: results are now intersected by model ID, so each hook can only remove models, never add them back. The composition rules are documented (PR #11).
- Tests: regression tests cover sync and async filters, global and per-purpose configuration, non-overlapping results, empty sets and a pricing filter that changes nothing. The PR reports 75 tests passing on Python 3.12 (PR #11).
- Lesson for developers: filter hooks in a plugin system should only ever narrow the result.
Claims and evidence
- Claim: the bug affected agent.datasette.io and let models outside the configuration through. Source: the maintainer's own report (blog).
- Claim: the trigger was combining datasette-llm-accountant's pricing filter with allowlists or blocklists (PR #11).
- Claim: the fix is validated by tests and integration checks. These results are self-reported in the PR, and no independent reproduction has been published.
Read the full section
- Claim: the bug affected agent.datasette.io and let models outside the configuration through. Source: the maintainer's own report (blog).
- Claim: the trigger was combining datasette-llm-accountant's pricing filter with allowlists or blocklists (PR #11).
- Claim: the fix is validated by tests and integration checks. These results are self-reported in the PR, and no independent reproduction has been published.
- Missing details: how long the demo was exposed, which models leaked through and whether any cost resulted are not documented.
Context and prior work
- Datasette Agent builds on Willison's LLM library and supports hundreds of tool-calling models.
- datasette-llm-accountant calculates cost from token usage with pricing from llm-prices.com.
- datasette-llm-limits uses the accountant to apply periodic spending caps.
Read the full section
- Datasette Agent builds on Willison's LLM library and supports hundreds of tool-calling models. Other plugins can add tools for it, such as charts, image generation and sandboxed code execution (datasette.io).
- datasette-llm-accountant calculates cost from token usage with pricing from llm-prices.com. It enforces budgets by reserving an estimated cost before each prompt and settling the actual cost afterwards.
- datasette-llm-limits uses the accountant to apply periodic spending caps.
Limitations, safety and contested findings
- Nature of the bug: it was an access-control failure.
- Scope: all evidence comes from the maintainer. The demo already required GitHub sign-in to limit abuse (datasette.io).
- Date conflict: the extracted GitHub release page shows November 4, 2025, while the blog post and PR show 2026 (release).
Read the full section
- Nature of the bug: it was an access-control failure. Models excluded by configuration could still be selected, which matters when model choice is used to cap cost or limit risk (PR #11).
- Scope: all evidence comes from the maintainer. The demo already required GitHub sign-in to limit abuse (datasette.io).
- Date conflict: the extracted GitHub release page shows November 4, 2025, while the blog post and PR show 2026 (release). The 2026 date fits the May 2026 Agent launch, but the discrepancy is unresolved.
Business and practitioner implications
- Upgrade: if you run Datasette Agent or other datasette-llm consumers with model allowlists, upgrade to 0.1a10, especially if you use the accountant plugins.
- Check enforcement: test that the model is actually rejected when a request is made, not just hidden from the list.
- Expect churn: the package is still in alpha.
Read the full section
- Upgrade: if you run Datasette Agent or other datasette-llm consumers with model allowlists, upgrade to 0.1a10, especially if you use the accountant plugins.
- Check enforcement: test that the model is actually rejected when a request is made, not just hidden from the list. The PR's integration check did exactly this (PR #11).
- Expect churn: the package is still in alpha. Pin versions and watch the release notes.