ResearchPapers, evidence & method
Google Research releases Agile Deliberation, a tool that turns user verdicts on borderline images into classifier prompts
Agile Deliberation, from Google Research, helps people build image classifiers for subjective concepts. Users first define the concept, then judge borderline images, and those labels automatically refine a Gemini classification prompt. The code and a demo notebook are open source.
From the Google Research article of November 4, 2026 and cited sources: The workflow has two stages. [1] [2]
SOURCES[1] Google Research blog: Agile Deliberation research.google[2] arXiv 2512.10821 (HTML) arxiv.org
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- Google Research article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 4, 2026 article published
- Covered
- Nov 5 edition
- Evidence
- 9 sources across 5 sites
Reach: not measured yet
The workflow has two stages. First, the user splits a concept such as "healthy food" into in-scope subcategories and explicit exclusions. Next, the system generates search queries for borderline cases, retrieves and clusters matching images, and picks which clusters to show using a bandit rule. After the user labels these images, the system writes several candidate prompts, scores each by F1 against the labels, and keeps the best one. [1] [2]
The study had nine Google-internal participants across 18 sessions, and all results come from Google's own authors. In that study, the paper reports 7.5% higher F1 than an automated decomposition baseline and more than 3% higher than manual deliberation. The paper doesn't make clear whether these are absolute or relative gains. Participants also rated effort lower (3.11 vs 4.67) and stress lower (1.67 vs 3.00) than with manual deliberation. [2] [6]
This is the third step in Google's work on subjective visual classifiers, after Agile Modeling and Modeling Collaborator. It changes one assumption: instead of treating the user's definition as fixed, it treats it as still taking shape. That approach came from interviews with five content-moderation experts. [2] [3] [4]
a small user study suggests that labeling borderline examples works better than hand-editing prompts for subjective image policies.
Read the full assessment
Implication: trust-and-safety, ad-review and curation teams could adopt this pattern, but should first test it on their own policies with several labelers.
Executive brief
Google Research has released Agile Deliberation, a tool for building image classifiers for subjective categories such as "healthy food" or misleading ad bait. Instead of asking people to write a perfect prompt, it shows them borderline images and turns their verdicts into a better prompt automatically. The headline numbers come from only 9 participants across 18 sessions, all recruited from inside Google. With that group, the paper reports 7.5% higher F1 than an automated decomposition baseline and more than 3% higher than manual deliberation (arXiv). Participants also reported lower effort and stress. The code is open source. No independent evaluation was found.
What changed and event timeline
Agile Modeling paper
Stretcu et al. introduced classifiers built with a user in the loop for subjective visual concepts. In a 14-person study, users trained models in under 30 minutes ().
Modeling Collaborator
An LLM breaks a concept into questions for a visual question-answering (VQA) model, needing about 100 labeled images. It appeared at CVPR 2024 and later served as Agile Deliberation's automated baseline ().
Agile Deliberation preprint
Wang, Stretcu and colleagues posted arXiv:2512.10821, which describes concept scoping, mining of borderline images, and automatic prompt optimization ().
CVPR 2026 acceptance
The paper was published in the CVPR 2026 open-access proceedings (;).
Blog post and code
Google Research published a summary, open-source code and a "Healthy Food" demo notebook. It credits a partnership with Google Ads (;).
Capabilities and access
- Two-stage workflow (blog):
- Scoping: the user splits a concept into in-scope subcategories and explicit exclusions.
- Iteration: the user labels borderline images, and those labels drive automatic prompt refinement.
Read the full section
- Two-stage workflow (blog):
- Scoping: the user splits a concept into in-scope subcategories and explicit exclusions.
- Iteration: the user labels borderline images, and those labels drive automatic prompt refinement.
- Models named in the preprint (arXiv):
- Gemini 2.5 Pro breaks the concept down.
- Gemini 2.5 Flash classifies images, writes queries for borderline cases and mines ambiguous examples.
- CLIP or ALIGN embeddings handle image retrieval.
- Access: open code in the google-research repository, plus a demo notebook. No hosted product has been announced.
Technical analysis for researchers and developers
- Retrieval: the system generates text queries for borderline cases, retrieves images by nearest-neighbour search, removes duplicates and clusters the results with dictionary learning.
- Prompt optimization: each round produces several candidate definitions, scores them by F1 against the labels the user has given so far, and keeps the best one (greedy selection).
- Evaluation: F1 is measured on 200 test images per concept, labeled by the participant.
Read the full section
- Retrieval: the system generates text queries for borderline cases, retrieves images by nearest-neighbour search, removes duplicates and clusters the results with dictionary learning. It then chooses which clusters to show using an upper-confidence-bound (UCB) bandit rule (arXiv).
- Prompt optimization: each round produces several candidate definitions, scores them by F1 against the labels the user has given so far, and keeps the best one (greedy selection).
- Evaluation: F1 is measured on 200 test images per concept, labeled by the participant. The paper says there is no fixed ground truth for these concepts. Each participant used different concepts in different conditions, so some comparisons are between participants rather than within the same person.
- Reproducibility: the code is public, but the study data comes from internal participants, and the results depend on Gemini APIs.
Claims and evidence
- Accuracy: +7.5% F1 over automated decomposition (Modeling Collaborator) and more than 3% over manual deliberation, per the abstract (arXiv).
- Workload (NASA-TLX survey): effort scored 3.11 vs 4.67 for manual, and stress 1.67 vs 3.00, both p<.05 on paired t-tests (arXiv HTML).
- Preference: every participant preferred Agile Deliberation (blog).
Read the full section
- Accuracy: +7.5% F1 over automated decomposition (Modeling Collaborator) and more than 3% over manual deliberation, per the abstract (arXiv). The detailed figures give +10.5% on average over zero-shot: 11% for "paid to play" and 10% for "healthy food" (arXiv HTML). The paper does not clearly say whether these are absolute points or relative gains.
- Workload (NASA-TLX survey): effort scored 3.11 vs 4.67 for manual, and stress 1.67 vs 3.00, both p<.05 on paired t-tests (arXiv HTML).
- Preference: every participant preferred Agile Deliberation (blog).
- All of these figures come from Google authors. No independent replication was found.
Context and prior work
This is the third step in a Google line of work on subjective visual classifiers: Agile Deliberation changes the assumption behind both. It treats the user's definition as something still taking shape, an approach drawn from interviews with five content-moderation experts (arXiv HTML). Google separately offers "agile classifiers" guidance for custom content-policy classifiers.
Read the full section
This is the third step in a Google line of work on subjective visual classifiers:
- Agile Modeling (ICCV 2023) used active learning with a user in the loop (ICCV paper).
- Modeling Collaborator (CVPR 2024) used LLM tool-use to cut down manual labeling (arXiv).
Agile Deliberation changes the assumption behind both. It treats the user's definition as something still taking shape, an approach drawn from interviews with five content-moderation experts (arXiv HTML). Google separately offers "agile classifiers" guidance for custom content-policy classifiers.
Limitations, safety and contested findings
- Small, internal sample: 9 participants recruited inside Google. The authors themselves note the small group and short sessions (arXiv HTML).
- Unstable scores: F1 moved up and down from one iteration to the next. The greedy optimizer favours quick responses over thorough refinement.
- It says manual deliberation "often" failed to improve or even hurt accuracy.
Read the full section
- Small, internal sample: 9 participants recruited inside Google. The authors themselves note the small group and short sessions (arXiv HTML).
- Unstable scores: F1 moved up and down from one iteration to the next. The greedy optimizer favours quick responses over thorough refinement.
- The blog overstates one result. It says manual deliberation "often" failed to improve or even hurt accuracy. The paper's own numbers show manual deliberation gained 8% on "paid to play" and lost 3% on "healthy food" (blog).
- Small-number sources disagree. The detailed results give about 7% over Modeling Collaborator, while the abstract says 7.5%.
- Circular measure: the same user both defines the concept and labels the test set, so F1 measures how well the classifier matches that person's intent, not general correctness.
Business and practitioner implications
- Trust and safety, ad review and curation teams: the pattern of showing borderline examples and optimizing the prompt automatically could replace hand-tuning prompts.
- Developers: the notebook is a quick way to test on your own data. Expect to depend on Gemini and on an image index you supply.
- Buyers: treat the accuracy gains as a promising signal from a study on Google's own staff, not as a benchmark.
Read the full section
- Trust and safety, ad review and curation teams: the pattern of showing borderline examples and optimizing the prompt automatically could replace hand-tuning prompts. It also addresses the concern participants raised that fixing one error might break something else (blog).
- Developers: the notebook is a quick way to test on your own data. Expect to depend on Gemini and on an image index you supply.
- Buyers: treat the accuracy gains as a promising signal from a study on Google's own staff, not as a benchmark. Validate on your own policies with several labelers before relying on it.