AgentsAutonomy & tool use
MIT Technology Review Insights survey finds 48% of AI-agent analytics insights lead to business improvements
A survey of companies using AI agents for analytics finds fewer than half of agent-generated insights lead to business improvements, with missing business context cited most often. Separate 2026 benchmarks show curated business definitions raise SQL accuracy more than switching models.
From the MIT Technology Review · AI article of November 2, 2026 and cited sources: Surveyed firms say 48% of insights from analytics agents lead to actions that improve the business. [1]
SOURCES[1] MIT Technology Review Insights: Raising the quality of… technologyreview.com
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- Surveyed firms say 48% of insights from analytics agents lead to actions that improve the business. [1]
- Original source
- MIT Technology Review · AI article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2604.25149 (Apr 2026) [5] date from arXiv ID
- When
- November 2, 2026 article published
- Covered
- Nov 4 edition
- Evidence
- 7 sources across 7 sites
- 1MIT Technology Review Insights: Raising the quality of…technologyreview.com
- 2The Star: Gartner on agentic AI cancellationsthestar.com.my
- 3Open Source For You: Open Semantic Interchangeopensourceforu.com
- 4dbt Labs: Semantic Layer vs. Text-to-SQL, 2026 benchmarkdocs.getdbt.com
- 5arXiv 2604.25149: Semantic Layers for Reliable LLM-Powered…arxiv.org
- 6Snowflake Docs: Cortex Analystdocs.snowflake.com
Reach: not measured yet
Surveyed firms say 48% of insights from analytics agents lead to actions that improve the business. Respondents recalled the figure as 37% a year earlier, so that comparison was not measured. Snowflake sponsored the survey and sells semantic-layer tools. Its MIT Technology Review Insights report also says self-identified 'insight leaders' exceed 70%, but the article does not define this group. [1]
Respondents most often named missing business context (55%) and fragmented data (45%) as the reasons insights fall short. Two benchmarks back the narrower technical point. In a paired arXiv study, adding a 4 KB business-semantics document raised accuracy for three frontier models from about 46–51% to about 68–69%. In dbt Labs' tests, querying through a semantic layer reached 98–100%, against 84–90% for direct text-to-SQL, and needed added data-modelling work to get there. [1] [4] [5]
Evaluation is the bottleneck. 52% of surveyed firms get more agent insights than they can review, and only 24% review all of them. Respondents rated prescriptive outputs lower than descriptive ones, and flawed insights that trigger wrong actions carry financial and reputational risk. [1]
two benchmarks found that business definitions improved query accuracy more than the choice of model. Implication: governed metric definitions, paired internal evaluations and human review of prescriptive outputs may pay off more than model upgrades.
Executive brief
Fewer than half (48%) of AI-agent analytics insights at surveyed companies lead to actions that improve the business. That figure comes from a Snowflake-sponsored survey by MIT Technology Review Insights, which is the publication's custom-content arm and not its newsroom (MIT Technology Review). Respondents most often blamed missing business context. Separate 2026 benchmarks point the same way: giving a model curated business definitions (a "semantic layer") raised SQL accuracy far more than switching models did. The survey's sample size and methodology are not in the article text.
What changed and event timeline
Open Semantic Interchange launched
Snowflake, Salesforce, dbt Labs, BlackRock and others announced a vendor-neutral YAML specification for sharing business definitions across AI and BI tools ().
Gartner forecasts agent cancellations
Gartner predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing costs, unclear value and weak risk controls ().
Dbt Labs updates its benchmark
On its ACME Insurance benchmark, querying through dbt's semantic layer reached 98–100% accuracy, versus 84–90% for models writing SQL directly from natural language (text-to-SQL) ().
arXiv paired benchmark posted
Rumiantsau and Fokeev found that adding a 4 KB business-semantics document raised accuracy by 17–23 percentage points for three frontier models ().
MIT Technology Review Insights publishes the survey
It reports that 48% of agent insights drive improvement, up from a recalled 37% a year earlier. The full report is a download ().
Capabilities and access
- The story is a survey report, not a product or model launch. The full report sits behind a download link.
- The sponsor's relevant stack: Cortex Analyst generates SQL from natural language using semantic models. Semantic Views are schema-level objects that define metrics and relationships, and both agents and BI tools can use them.
- The arXiv study tested Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4. dbt tested Claude Opus 4.6 and Sonnet 4.6, GPT-5.3-Codex and GPT-5.2.
Read the full section
- The story is a survey report, not a product or model launch. The full report sits behind a download link.
- The sponsor's relevant stack: Cortex Analyst generates SQL from natural language using semantic models. Semantic Views are schema-level objects that define metrics and relationships, and both agents and BI tools can use them.
- The arXiv study tested Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4. dbt tested Claude Opus 4.6 and Sonnet 4.6, GPT-5.3-Codex and GPT-5.2.
Technical analysis for researchers and developers
- arXiv study design: 100 questions on the Contoso Retail dataset in ClickHouse. Accuracy rose from 45.5–50.5% to 67.7–68.7%.
- dbt study design: 11 questions, 20 runs each. Code is public (dbt Labs).
- Failure modes differ: dbt reports that direct text-to-SQL returns answers that look plausible but are wrong, while semantic-layer failures surface as errors.
Read the full section
- arXiv study design: 100 questions on the Contoso Retail dataset in ClickHouse. Each model answered single-shot twice: once with the schema only, once with the schema plus the semantics document. Accuracy rose from 45.5–50.5% to 67.7–68.7%. Differences between the two conditions were significant at p<0.01; differences between models within a condition were not (arXiv).
- dbt study design: 11 questions, 20 runs each. Code is public (dbt Labs).
- Failure modes differ: dbt reports that direct text-to-SQL returns answers that look plausible but are wrong, while semantic-layer failures surface as errors.
- The context is not free: in dbt's tests, adding three dbt models raised semantic-layer accuracy from 72.7% to 100%. Data modelling effort drives results.
Claims and evidence
- 48% of insights drive improvement (37% a year ago): from the survey. It is sponsored by a vendor that sells semantic-layer tools.
- "Insight leaders" exceed 70%: the article gives neither the size of this group nor how it was defined.
- Missing context is the top cause (55%), fragmented data second (45%): these are respondent opinions.
Read the full section
- 48% of insights drive improvement (37% a year ago): from the survey. It is sponsored by a vendor that sells semantic-layer tools. The 37% figure is respondents' recollection, not a measurement from an earlier survey.
- "Insight leaders" exceed 70%: the article gives neither the size of this group nor how it was defined.
- Missing context is the top cause (55%), fragmented data second (45%): these are respondent opinions. The arXiv and dbt benchmarks support the narrower technical point that context improves SQL accuracy. dbt Labs also sells a semantic layer.
- No independent corroboration of the survey's percentages was found.
Context and prior work
- Interest in semantic layers has grown along with skepticism about agents in general (Gartner via The Star).
- Vendors are trying to standardize business definitions through OSI. Snowflake is a member and is also the survey's sponsor.
Limitations, safety and contested findings
- Absolute accuracy conflicts across benchmarks: about 68% with semantic context in the arXiv study versus 98–100% in dbt's.
- Benchmarks measure query correctness, not business value. Nothing in the reviewed sources directly links SQL accuracy to the survey's "insights lead to improvement" figure.
- Risk: the survey says flawed insights that trigger wrong actions can cause reputational or financial damage.
Read the full section
- Absolute accuracy conflicts across benchmarks: about 68% with semantic context in the arXiv study versus 98–100% in dbt's. Results depend heavily on the dataset and question set.
- Benchmarks measure query correctness, not business value. Nothing in the reviewed sources directly links SQL accuracy to the survey's "insights lead to improvement" figure.
- Risk: the survey says flawed insights that trigger wrong actions can cause reputational or financial damage. 52% of firms report more agent insights than they can evaluate, and only 24% evaluate all of them (MIT Technology Review).
Business and practitioner implications
- Invest in governed metric definitions before switching models. Both benchmarks found semantic context mattered more than model choice.
- Treat prescriptive outputs as higher risk than descriptive ones.
- Reviewing insights is still mostly manual, and most surveyed firms cannot keep up with the volume.
Read the full section
- Invest in governed metric definitions before switching models. Both benchmarks found semantic context mattered more than model choice.
- Treat prescriptive outputs as higher risk than descriptive ones. Respondents rated prescriptive insights lower, so put human review or approval gates before any automated action.
- Plan for evaluation capacity. Reviewing insights is still mostly manual, and most surveyed firms cannot keep up with the volume.
- Run your own paired evaluation (schema only versus schema plus semantics) on internal questions rather than relying on vendor or survey figures.