Oct 16 edition/Podcast
ResearchPolicyBusiness

ResearchPapers, evidence & method

Practical AI hosts say schools lag AI-era work, citing study where an unrestricted GPT-4 chatbot cut later exam scores 17%

On Practical AI, Chris Benson and Daniel Whitenack argue schools lag behind how AI is used at work. They cite research showing an unrestricted GPT-4 chatbot hurt later unassisted exam scores while guardrailed tutors did not, and point to Purdue's new AI requirement.

From the Practical AI podcast episode of October 15, 2026 and cited sources: How a tutor is built around the model mattered more than the model itself. [3] [5] [4]

SOURCES[3] Knowledge at Wharton: Without guardrails, generative AI… knowledge.wharton.upenn.edu[5] IDEAS/RePEc: PNAS record, Bastani et al. 2025 ideas.repec.org[4] Kestin et al., AI tutoring outperforms in-class active… pmc.ncbi.nlm.nih.gov

AI-assisted, not human-edited. Methodology Corrections Who owns this site

KEY FACTS
What happened
How a tutor is built around the model mattered more than the model itself. [3] [5] [4]
Original source
Practical AI podcast episode, independent perspective
Brief by
AT Pulse research desk AI-assisted; publisher and contact on the desk page
When
October 15, 2026 podcast episode published
Evidence
9 sources across 7 sites; transcript sourced
7/100QuietHow it’s measured

YouTube video · 32 views

Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.

Illustration from Practical AI: Practical AI hosts say schools lag AI-era work, citing study where an unrestricted GPT-4 chatbot cut later exam scores 17%
Image: Practical AI — Original article ↗
THE CORE IDEAS4 TAKEAWAYS
01

How a tutor is built around the model mattered more than the model itself. In a peer-reviewed field experiment with nearly 1,000 Turkish high-school math students, open GPT-4 access raised practice scores by 48% but lowered later no-AI exam scores by 17%. A tutor version that gave hints instead of answers avoided that harm. A separate randomized crossover trial with 194 Harvard physics students used a structured GPT-4 tutor built on expert-written prompts. It produced more than double the learning gains of an active-learning class across two lessons. Both studies show that scores measured while students have AI can mislead, and that a later test without AI is what reveals real learning. [3] [5] [4]

02

The hosts argue schools should treat AI as a permanent tool while keeping the useful struggle that builds critical thinking. Their alternatives to banning AI and running detection tools include having students critique AI-written essays and letting teachers experiment. Teachers have pushed back that classroom policy is set above them, so the hosts say change has to come through districts and state requirements. [2]

03

Purdue has made AI competency a requirement for every undergraduate degree. Trustees approved it in December 2025, and it starts with students entering in fall 2026, who can meet it through 22 courses of 1–4 credits each. Critics argue that relying on AI wears down core skills. [6] [8] [9] [7]

04

Whitenack says daily use of AI assistants is a basic expectation at his company, Prediction Guard. He worries that university AI-detection policies leave graduates reluctant to use AI at work. This is his own hiring and mentoring experience, not survey data. [2]

WHY IT MATTERS

The strongest evidence comes from controlled studies: chat without scaffolding can lift practice scores while weakening real learning. For edtech builders and schools, the takeaway is to design for hints and test without AI.

Read the full assessment

The hosts' claims about workplace readiness remain anecdotal.

Executive brief

The most useful point in this Practical AI episode is a finding the hosts cite from outside research. In a field experiment with nearly 1,000 Turkish high-school students, an unrestricted GPT-4 chatbot raised practice scores by 48% but cut later scores on an exam taken without AI by 17%. A tutor version with guardrails avoided that harm (Knowledge at Wharton). Hosts Chris Benson and Daniel Whitenack argue that schools lag behind how AI is used at work. They want to keep useful struggle in learning while letting students try AI openly, and they point to Purdue's new AI graduation requirement (episode transcript). The episode is commentary, not new evidence.

What changed and event timeline

  1. Wharton reports the guardrails experiment

    A working paper by Bastani et al. reports that unrestricted GPT-4 helped students during practice but hurt their later exam scores. The tutor version scored about the same as the control group on the exam ().

  2. PNAS publishes the Turkey study

    The peer-reviewed paper appears in PNAS vol. 122, issue 26 ().

  3. Harvard AI-tutor trial published

    A randomized crossover study of 194 physics students found more than double the learning gains from a structured GPT-4 tutor than from an in-class active-learning session ().

  4. Purdue trustees approve an AI competency requirement

    Purdue's board approves an "AI working competency" outcome for every undergraduate degree (). Critics respond with scorn on social media ().

  5. Requirement takes effect

    The requirement starts with the fall 2026 incoming class. Students can meet it through 22 courses of 1–4 credits each ().

  6. Practical AI episode airs

    The hosts argue that education is "falling short" of AI-era realities and cite the Turkey, Harvard and Purdue examples ().

Capabilities and access

  • Turkey study: "GPT Base" was a ChatGPT-4-like interface. "GPT Tutor" was GPT-4 with teacher input that gave hints instead of answers (Wharton).
  • Harvard trial: a GPT-4 tutor with expert-written prompts and pre-written step-by-step solutions to keep answers accurate (PMC).
  • Purdue: coursework built around five areas: learning with AI, learning about AI, research AI, using AI and partnering in AI (Purdue Engineering).
Read the full section
  • Turkey study: "GPT Base" was a ChatGPT-4-like interface. "GPT Tutor" was GPT-4 with teacher input that gave hints instead of answers (Wharton).
  • Harvard trial: a GPT-4 tutor with expert-written prompts and pre-written step-by-step solutions to keep answers accurate (PMC).
  • Purdue: coursework built around five areas: learning with AI, learning about AI, research AI, using AI and partnering in AI (Purdue Engineering).

Technical analysis for researchers and developers

  • What drives the difference is the design around the model. Both studies used GPT-4. Prompt and guardrail design is the variable that matters.
  • How the Harvard trial was measured: a crossover design, so each student served as their own control.
  • Implication for evaluation: both studies show that results while AI is available can mislead.
Read the full section
  • What drives the difference is the design around the model. Both studies used GPT-4. The open chat interface hurt later performance, while the scaffolded tutors helped or did no harm. Prompt and guardrail design is the variable that matters.
  • How the Harvard trial was measured: a crossover design, so each student served as their own control. It used pre- and post-tests and an effect size of 0.63 SD, or 0.73–1.3 SD after adjusting for ceiling effects. Median time was 49 minutes, and time on task did not correlate with scores (PMC).
  • Implication for evaluation: both studies show that results while AI is available can mislead. Testing again without AI is what separates real learning from leaning on the tool.

Claims and evidence

  • Unrestricted AI hurts later exam scores by 17%: a peer-reviewed field experiment (PNAS record; Wharton).
  • A structured AI tutor beats active learning: one randomized trial at one institution covering two lessons (PMC).
  • Purdue's requirement aims at "critical thinking, responsible use": Purdue's own description of its program (Purdue).
Read the full section
  • Unrestricted AI hurts later exam scores by 17%: a peer-reviewed field experiment (PNAS record; Wharton). The hosts cite it accurately (transcript, 29:24).
  • A structured AI tutor beats active learning: one randomized trial at one institution covering two lessons (PMC).
  • Purdue's requirement aims at "critical thinking, responsible use": Purdue's own description of its program (Purdue).
  • Graduates struggle with AI-centric workplaces: the hosts' personal impressions from mentoring and hiring, with no supporting data (transcript, 19:10; 31:29).

Context and prior work

The episode links back to the active-learning research that Whitenack saw as a Purdue teaching assistant (transcript, 29:24). The Harvard trial uses active learning as its comparison condition (PMC). Benson draws on his work advising universities and a book he is writing about life in a world where AI agents are everywhere (transcript, 03:37; 04:39).

Limitations, safety and contested findings

  • Harvard trial: the authors limit their claims to foundational material, not complex synthesis or higher-order thinking.
  • Turkey study: students who used AI showed overconfidence in their abilities, including high achievers (Wharton).
  • Purdue: critics call AI "a shortcut for incompetent people," and some academics worry it erodes skills (Yahoo News).
Read the full section
  • Harvard trial: the authors limit their claims to foundational material, not complex synthesis or higher-order thinking. The results depend on prompts written by experts (PMC).
  • Turkey study: students who used AI showed overconfidence in their abilities, including high achievers (Wharton).
  • Purdue: critics call AI "a shortcut for incompetent people," and some academics worry it erodes skills (Yahoo News).
  • The hosts' own proposals are untested: publishing the full curriculum at the start of the year, and giving each student an AI earpiece in class (transcript, 23:06; 28:36). Teachers have pushed back that they lack the authority to change policy (transcript, 20:13).

Business and practitioner implications

  • Edtech builders: design for hints and step-by-step help rather than handing out answers, and measure outcomes without the AI present.
  • Employers: Whitenack treats daily AI use as a basic expectation at Prediction Guard.
  • Institutions: Purdue built 22 courses and updated more than 300 plans of study within months, which shows a whole-university rollout is feasible (Purdue Engineering).
Read the full section
  • Edtech builders: design for hints and step-by-step help rather than handing out answers, and measure outcomes without the AI present. A plain chatbot can harm learning even when engagement metrics look good.
  • Employers: Whitenack treats daily AI use as a basic expectation at Prediction Guard. Hiring teams should expect graduates' habits to vary, because AI-detection policies at universities can leave students wary of using AI at all (transcript, 31:29).
  • Institutions: Purdue built 22 courses and updated more than 300 plans of study within months, which shows a whole-university rollout is feasible (Purdue Engineering).
FOLLOW THE EVIDENCE

The source trail.

Sources (9)
A LITTLE LESS NOISE. A LOT MORE CONTEXT.

Stay curious.
Follow the evidence.

Independent perspectives, the original sources, and room for the questions that don't have easy answers.

How we build the brief
Connect with us

Find us where you already read.