AgentsAutonomy & tool use
Midwest AI Summit talks on Practical AI: give LLMs narrow roles, build evals, and set agent autonomy by constraints
Speakers at the Midwest AI Summit, excerpted on Practical AI, argued that production AI workflows usually stall because of how work is designed, not model choice: give LLMs narrow, checked roles, build evals with frontline users, and set agent autonomy to real constraints.
From the Practical AI podcast episode of October 29, 2026 and cited sources: Heidi Porter's deployment examples give the LLM a narrow job inside a larger system. [2] [4] [8]
SOURCES[2] Transcript: Abundant Intelligence at the Midwest AI Summit share.transistor.fm[4] AWS: Syngenta develops a generative AI assistant using… aws.amazon.com[8] Intelligent defect-detection framework integrating… sciencedirect.com
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- Heidi Porter's deployment examples give the LLM a narrow job inside a larger system. [2] [4] [8]
- Original source
- Practical AI podcast episode, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2603.17234 (Mar 2026) [3] date from arXiv ID
- When
- October 29, 2026 podcast episode published
- Covered
- Oct 29 edition
- Evidence
- 8 sources across 7 sites; transcript sourced
- 1Practical AI: Abundant Intelligence at the Midwest AI…share.transistor.fm
- 2Transcript: Abundant Intelligence at the Midwest AI Summitshare.transistor.fm
- 3Deployment and Evaluation of an EHR-integrated…arxiv.org
- 4AWS: Syngenta develops a generative AI assistant using…aws.amazon.com
- 5DORA 2025 State of AI-assisted Software Development Reportresearch.google
- 6Stanford GSB: JAMA Network Open publication recordgsb.stanford.edu
Reach: not measured yet
Heidi Porter's deployment examples give the LLM a narrow job inside a larger system. In a rubber-hose line, a modified YOLO detector finds defects with no LLM involved. The LLM then drafts a report from retrieved knowledge-graph and maintenance evidence, and rule checks confirm the report stays within that evidence. Weak-evidence cases go to a human. Syngenta's Cropwise AI uses existing agronomy models to make predictions, while the LLM interprets questions, routes requests and explains results. The models can be swapped. [2] [4] [8]
Stanford Health Care's EHR-integrated triage tool applies a rubric the hospital owns to surgical cases, and a physician reviews every recommendation. A prospective study covered 6,193 cases. The tool had a sensitivity of 0.94 and a specificity of 0.74. The LLM was at fault in only 2 of 19 reviewed false negatives. Most misses came from gaps in the criteria, the workflow or physician variation. [3] [6]
Kevin Solorio argued that an agent without evals is only a proof of concept. His eval loop starts with an offline dataset. Live traces are then scored, failures go to an annotation queue, and those cases are added back to the dataset. Frontline workers help build the tool, often by correcting a deliberately rough first version, and they become its champions. [2]
SEP's Chris Shinkle and Andrew Dunlap said most teams do not need a fully autonomous 'dark factory' for coding. They suggested choosing a level of autonomy based on six constraints: token budget, organizational sponsorship, goal clarity, how much domain knowledge is written down, verifiability, and code-oversight requirements. Their premise is that AI amplifies a team's existing strengths and weaknesses, which matches DORA 2025. The six-fold reduction in effort on a credit-union app is SEP's own uncorroborated figure. [2] [5]
The Stanford study gives peer-reviewed support for the view that most misses came from criteria and workflow rather than the model. The other lessons are practitioner advice.
Read the full assessment
For teams, they point spending toward data, evals, reversible actions and swappable models over added autonomy.
Sources are checked; here is the dossier.
Executive brief
At Stanford Health Care, a large language model (LLM) triage tool screened 6,193 surgical cases. In a check of the cases it missed, the model itself was at fault in only 2 of 19. Heidi Porter (Moyer Group AI) argued that LLMs fail when "asked to do too much."
Read the full section
At Stanford Health Care, a large language model (LLM) triage tool screened 6,193 surgical cases. In a check of the cases it missed, the model itself was at fault in only 2 of 19. Most misses came from gaps in the clinical criteria, the hospital's workflow or physician variation (arXiv). That finding sums up this Practical AI episode of Midwest AI Summit talks: when AI workflows fail, the model is usually not the cause. Heidi Porter (Moyer Group AI) argued that LLMs fail when "asked to do too much." Kevin Solorio (Focus) called agents without evals proofs of concept. SEP's Chris Shinkle and Andrew Dunlap argued that most teams don't need fully autonomous "dark factory" coding.
What changed and event timeline
Syngenta details Cropwise AI
AWS describes a multi-agent seed-recommendation assistant built on Bedrock Agents. It was evaluated on 100 question-answer pairs from sales reps ().
DORA frames AI as an amplifier
Google's DORA report, based on about 5,000 respondents, finds that AI magnifies an organization's existing strengths and dysfunctions ().
Peer-reviewed publication
The Stanford study appears in JAMA Network Open, vol. 9, issue 8 ().
Stanford triage preprint
Prospective study of SCM Navigator from September 2025 to February 2026: sensitivity 0.94, specificity 0.74 ().
Midwest AI Summit
The listing places the event at The Crane Bay in Indianapolis. Host Daniel Whitenack (named only as "Daniel" in the transcript) was emcee (;).
Capabilities and access
- Cropwise AI: runs on Amazon Bedrock Agents, with sign-in through Cognito and retrieval through Bedrock Knowledge Bases on OpenSearch.
- SCM Navigator: built into the electronic health record (EHR), with a physician reviewing every recommendation (GSB). The paper's abstract does not name the model.
- Rubber-hose system: a dual-branch YOLO detector, a curated knowledge graph and a retrieval-augmented LLM (ScienceDirect).
Read the full section
- Cropwise AI: runs on Amazon Bedrock Agents, with sign-in through Cognito and retrieval through Bedrock Knowledge Bases on OpenSearch. It supports Claude 3.5 Haiku and Sonnet and Llama 3.1, and models can be switched without code changes (AWS).
- SCM Navigator: built into the electronic health record (EHR), with a physician reviewing every recommendation (GSB). Porter said Stanford picked o3-mini from five candidate models (transcript 15:57). The paper's abstract does not name the model.
- Rubber-hose system: a dual-branch YOLO detector, a curated knowledge graph and a retrieval-augmented LLM (ScienceDirect).
Technical analysis for researchers and developers
- In the hose paper, YOLO finds defects with no LLM involved.
- Prompts are not enforcement. Guardrails need code, gates or people, and any approval scheme has to account for reviewer fatigue (transcript 08:24).
- Build an offline dataset, score live traces online, send failures to an annotation queue, then add them back to the dataset (transcript 30:44).
Read the full section
- Narrow LLM roles. In the hose paper, YOLO finds defects with no LLM involved. A defect event pulls a section of the knowledge graph plus passages from procedures and maintenance logs. The LLM writes a schema-constrained report, rule checks confirm it stays within the evidence, and weak-evidence cases go to a human (ScienceDirect).
- Prompts are not enforcement. Guardrails need code, gates or people, and any approval scheme has to account for reviewer fatigue (transcript 08:24).
- Eval flywheel. Build an offline dataset, score live traces online, send failures to an annotation queue, then add them back to the dataset (transcript 30:44).
- Syngenta's evaluation: human review plus Ragas LLM-as-judge scores for relevance, conciseness and faithfulness (AWS).
Claims and evidence
- Stanford results: sensitivity 0.94 (95% CI 0.91–0.96), specificity 0.74. 25.5% of cases were flagged, and the LLM caused 2 of 19 false negatives.
- Stanford, talk-only figures: that manual triage previously missed half of eligible patients, that physicians agreed with 97% of "no" calls, and that about 4 in 10 flags were declined (transcript 14:56–17:13).
- Hose detector: 86.31% mean average precision (mAP) on edge hardware (ScienceDirect). The "52 weeks of monitoring" figure comes only from the talk.
Read the full section
- Stanford results: sensitivity 0.94 (95% CI 0.91–0.96), specificity 0.74. 25.5% of cases were flagged, and the LLM caused 2 of 19 false negatives. These come from a peer-reviewed study (arXiv; GSB).
- Stanford, talk-only figures: that manual triage previously missed half of eligible patients, that physicians agreed with 97% of "no" calls, and that about 4 in 10 flags were declined (transcript 14:56–17:13). None of these appear in the reviewed sources.
- Hose detector: 86.31% mean average precision (mAP) on edge hardware (ScienceDirect). The "52 weeks of monitoring" figure comes only from the talk.
- Syngenta: sales reps produce recommendations "5x faster," per the AWS/Syngenta blog; no independent measurement found (AWS).
- SEP's Forum credit union app: a "six x reduction of effort" claimed by SEP, the consultancy that did the work. No corroboration found (transcript 48:17).
Context and prior work
- SEP's main premise, that AI amplifies what a team already is, matches DORA 2025. DORA reports that 90% of respondents use AI at work and that the biggest gains come from platform quality and clear workflows (DORA).
- Solorio's "be wrong first" approach to gathering feedback and Shinkle's references to Toyota's andon cord come from established practice in user-centred design and lean manufacturing.
Read the full section
- SEP's main premise, that AI amplifies what a team already is, matches DORA 2025. DORA reports that 90% of respondents use AI at work and that the biggest gains come from platform quality and clear workflows (DORA).
- Solorio's "be wrong first" approach to gathering feedback and Shinkle's references to Toyota's andon cord come from established practice in user-centred design and lean manufacturing.
Limitations, safety and contested findings
- Stanford's flagging rate: one summary gives 23%, but 1,582 of 6,193 cases is 25.5% (arXiv).
- Specificity of 0.74 means many false positives. Porter described over-flagging as a deliberate design choice (transcript 16:47).
- Hose system: Porter called its results "good, but not perfect," and grounding still needs checking (transcript 11:25).
Read the full section
- Stanford's flagging rate: one summary gives 23%, but 1,582 of 6,193 cases is 25.5% (arXiv). The study was unblinded and run at a single site.
- Specificity of 0.74 means many false positives. Porter described over-flagging as a deliberate design choice (transcript 16:47).
- Hose system: Porter called its results "good, but not perfect," and grounding still needs checking (transcript 11:25).
- SEP's prototype: stalled when the token budget ran out within days and when outputs had to be checked across millions of inputs (transcript 43:21).
Business and practitioner implications
- Treat the model as the replaceable part. Stanford compared candidate models, and Syngenta can switch models without code changes (AWS).
- Spend on data quality, evals and ways to undo actions rather than on extra autonomy.
- Check six things: token budget, organizational sponsorship, goal clarity, how much domain knowledge is written down, verifiability, and code-oversight requirements.
Read the full section
- Treat the model as the replaceable part. Stanford compared candidate models, and Syngenta can switch models without code changes (AWS).
- Spend on data quality, evals and ways to undo actions rather than on extra autonomy.
- Match autonomy to constraints. Check six things: token budget, organizational sponsorship, goal clarity, how much domain knowledge is written down, verifiability, and code-oversight requirements. In regulated or safety-critical work (SEP's building-components example), a team may choose not to go past shared-context mode (transcript 42:24–47:19).
- Get frontline users to help build the tool, so they become its champions (transcript 32:10).
The source trail.
Sources (8)
Practical AI: Abundant Intelligence at the Midwest AI Summit (episode page)
share.transistor.fmTranscript: Abundant Intelligence at the Midwest AI Summit
published_transcript
share.transistor.fm