ResearchPapers, evidence & method
Microsoft builds synthetic company workspaces from public data to evaluate Copilot
Microsoft Research describes an internal platform that generates simulated pharma, capital-markets and IT/software workspaces from O*NET data, job postings and SEC filings, now serving as Copilot's first standardized evaluation baseline. It reports data-quality gains and savings but no Copilot performance results.
From the Microsoft Research article of October 30, 2026 and cited sources: The platform builds three synthetic tenants, each with hundreds of users and communication histories that change over time, including meetings, multi-week email threads and org changes. [1]
SOURCES[1] From synthetic enterprise data evaluations to real… microsoft.com
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- The platform builds three synthetic tenants, each with hundreds of users and communication histories that change over time, including meetings, multi-week email threads and org changes. [1]
- Original source
- Microsoft Research article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 30, 2026 article published
- Covered
- Oct 31 edition
- Evidence
- 5 sources across 2 sites
- 1From synthetic enterprise data evaluations to real…microsoft.com
- 2WorkBench: a Benchmark Dataset for Agents in a Realistic…arxiv.org
- 3Can LLMs Help You at Work? A Sandbox for Evaluating LLM…arxiv.org
- 4The Era by Eon Benchmark: A Generated Enterprise Estate…arxiv.org
- 5Who evaluates the evaluators? The data science behind…microsoft.com
Reach: not measured yet
The platform builds three synthetic tenants, each with hundreds of users and communication histories that change over time, including meetings, multi-week email threads and org changes. That lets Copilot be tested on time-dependent questions. Test queries come from a six-stage pipeline: a generic task library is filtered by industry, adapted to domain language, grounded in tenant data, rewritten to sound natural, and checked for accuracy and uniqueness. [1]
The gains are vendor-reported. Microsoft says a composite data-quality score rose 39% over its initial generation method, and some environments scored above 0.9 out of 1.0. It also estimates savings of about $100,000 per dataset and $300,000 for the project. The quality metric is internal and its definition is unpublished. It measures how realistic the synthetic data is, not how well Copilot performs. [1]
The approach follows academic work. WorkBench built simulated workplace sandboxes. EnterpriseBench grounded 500 tasks in O*NET, where the best model completed 41.8% of tasks against 70% human accuracy. Era by Eon generates entire companies so every answer is known exactly. Microsoft's difference is running this at production scale for one product, but unlike those benchmarks it publishes no task counts or scores. [2] [3] [4]
Realistic synthetic data does not prove results will carry over to real customers. Microsoft's own Copilot Studio team warns that narrow or unrealistic test data can produce confident but wrong metrics, and it adds human-labeled and production data. The Era by Eon authors flag the same transfer risk. [4] [5]
The post describes a repeatable way to test Copilot on industry scenarios without using customer data. For buyers, that is not yet a performance measure.
Read the full assessment
With no per-industry scores or scoring method published, the baseline cannot show how accurate Copilot is for real organizations.
Executive brief
Microsoft now tests Copilot against fake companies built from public data, not customer data. In a Microsoft Research blog post dated 30 October 2026, the company describes its "Industry Evaluation Data Generation Platform." It builds simulated company workspaces (Microsoft calls each one a "tenant") for pharma, capital markets and IT/software. Their content draws on O*NET occupational data, job postings and SEC filings. Microsoft says the platform is now Copilot's first standardized evaluation baseline. The post reports better data quality and cost savings, but it reports no Copilot performance results. The reviewed sources contain no independent coverage or evaluation of the platform.
What changed and event timeline
WorkBench sets a workplace sandbox baseline
This academic benchmark tests agents on realistic workplace tasks inside a simulated environment with databases. It is an early example of the synthetic-workplace evaluation approach ().
EnterpriseBench grounds tasks in O*NET
Fujitsu Research released 500 enterprise tasks with goal templates drawn from O*NET 29.2. The best model completed 41.8% of tasks, compared with 70% accuracy for humans ().
Era by Eon generates an entire enterprise
This benchmark generates a full synthetic company, including org hierarchy, sales pipeline, tickets, calls and documents. Because every fact is generated, the correct answer to each question is known exactly ().
Microsoft describes its Copilot evaluation platform
The post covers synthetic tenants, a six-stage query pipeline and quality scoring. Microsoft says the platform is published to an internal catalog as Copilot's standard baseline ().
Capabilities and access
- Product tested: Microsoft Copilot. The post names no model or version.
- Environments: three synthetic tenants, each with "hundreds of users" and full communication histories.
- The post links no paper, code or dataset.
Read the full section
- Product tested: Microsoft Copilot. The post names no model or version.
- Environments: three synthetic tenants, each with "hundreds of users" and full communication histories. The data changes over time (meetings, multi-week email threads, org changes), so time-dependent questions can be tested (Microsoft Research).
- Access: internal to Microsoft only. The post links no paper, code or dataset.
- Related public tool: Copilot Studio offers customers synthetic test-set generation (Copilot Studio blog).
Technical analysis for researchers and developers
- Query pipeline (six stages): generic task library → filter by industry → adapt domain language → generate queries grounded in the tenant data → rewrite to sound natural → check accuracy and uniqueness (Microsoft Research).
- Quality checks: the post says there are three dimensions but lists four: structural fidelity, contextual relevance, behavioral realism and content safety.
- Reproducibility: Microsoft says the same configuration regenerates identical datasets.
Read the full section
- Query pipeline (six stages): generic task library → filter by industry → adapt domain language → generate queries grounded in the tenant data → rewrite to sound natural → check accuracy and uniqueness (Microsoft Research).
- Quality checks: the post says there are three dimensions but lists four: structural fidelity, contextual relevance, behavioral realism and content safety.
- Reproducibility: Microsoft says the same configuration regenerates identical datasets. The post does not explain how that works with LLM generation (for example, seeding or caching).
- Not documented: how Copilot responses are scored, how ground truth is built, or whether any judge model is used.
Claims and evidence
- Data quality: a composite quality score rose 39% over Microsoft's initial generation method, and some environments scored above 0.9 out of 1.0 (Microsoft Research).
- Savings: about $100,000 per dataset and $300,000 for the project.
- No customer data at any stage, and public sources passed legal review. Both are vendor statements.
Read the full section
- Data quality: a composite quality score rose 39% over Microsoft's initial generation method, and some environments scored above 0.9 out of 1.0 (Microsoft Research). The metric is internal and its definition is not published.
- Savings: about $100,000 per dataset and $300,000 for the project. Microsoft labels these as estimates.
- No customer data at any stage, and public sources passed legal review. Both are vendor statements.
- Independent corroboration: none in the reviewed sources.
Context and prior work
Academic benchmarks already use this approach. EnterpriseBench also starts from O*NET and builds synthetic emails, CRM records and HR data. Era by Eon generates a whole company so that answers can be checked exactly. WorkBench came earlier. Microsoft's contribution is running this at production scale for one product. Unlike these benchmarks, it publishes no task counts, agent scores or artifacts.
Limitations, safety and contested findings
- Microsoft's own Copilot Studio team warns that narrow or unrealistic evaluation data yields metrics that "look confident but be wrong."
- EnterpriseBench's authors note that their pipeline still needs initial real enterprise data and human verification (arXiv).
- The Era by Eon paper raises a general concern for this kind of evaluation: results on generated companies may not carry over to real ones (arXiv).
Read the full section
- Microsoft's own Copilot Studio team warns that narrow or unrealistic evaluation data yields metrics that "look confident but be wrong." That team supplements synthetic data with human-labeled and production data (Copilot Studio blog).
- EnterpriseBench's authors note that their pipeline still needs initial real enterprise data and human verification (arXiv).
- The Era by Eon paper raises a general concern for this kind of evaluation: results on generated companies may not carry over to real ones (arXiv).
- Microsoft's quality metric scores how realistic the synthetic data is. It does not show that Copilot's results on that data predict its results for real customers.
Business and practitioner implications
- Buyers: Microsoft's statement that Copilot is evaluated on industry tenants is not a published performance number.
- Builders: the recipe can be copied with public data: O*NET roles, job-posting hierarchies, SEC-informed company scale, data that evolves over time, and validated query generation (Microsoft Research).
- Regulated sectors: fully synthetic data avoids privacy reviews for evaluation work. Teams should still check it against real usage, as Copilot Studio's own guidance recommends.
Read the full section
- Buyers: Microsoft's statement that Copilot is evaluated on industry tenants is not a published performance number. Buyers should ask for per-industry results and how answers are scored.
- Builders: the recipe can be copied with public data: O*NET roles, job-posting hierarchies, SEC-informed company scale, data that evolves over time, and validated query generation (Microsoft Research).
- Regulated sectors: fully synthetic data avoids privacy reviews for evaluation work. Teams should still check it against real usage, as Copilot Studio's own guidance recommends.