Every factual sentence carries assertions checked against the recorded evidence: references must resolve, figures and quotations must be in the cited passage, arithmetic must reproduce, and an independent reviewer must approve the wording. Automated validation says whether the narration is ready for human editorial review; it does not establish truth.
Open a sentence to see its assertions, their status and the passages behind them. Sentences without a marker make no factual claim.
verified s04.a In The Pragmatic Engineer episode on AI at Shopify, it was said that AI agents will help the best engineers more than mediocre ones.
supported (approved): no direct support, 5 contradicting, 2 qualifying, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s05.a Simon Willison said that combining his programming knowledge with these tools lets him run circles around someone who has never written code.
supported (approved): no direct support, 5 contradicting, 2 qualifying, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s07.a In the Copilot field experiments, short-tenure developers increased output by 27% to 39% while long-tenure developers gained 8% to 13%.
qualified (qualify): 1 supporting (1 independent, strongest 0.5), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s08.a The tenure estimates are noisy and not statistically significant at conventional levels, but the pattern is the same across all three main outcome measures.
qualified (qualify): 1 supporting (1 independent, strongest 0.5), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s09.a The authors say longer-tenure estimates may be attenuated by developers abandoning the tool after an initial trial, but with no difference in usage between junior and senior developers conditional on adoption, juniors still increased output more than seniors.
qualified (qualify): no direct support, 3 qualifying, 2 context only.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s10.a In Shan et al.'s moderation analysis, developers with fewer years of experience benefited in faster coding while coding quality remained unaffected.
qualified (qualify): 1 supporting (1 independent, strongest 0.45), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s11.a Peng et al.'s Copilot trial suggests less experienced programmers benefit more, which the authors say, if it persists in further studies, points to possibilities for job transitions into software development.
qualified (qualify): 1 supporting (1 independent, strongest 0.4), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s12.a In the code-ownership experiment, all participant categories (BSc, MSc, Researcher, Professional) had a significant increase in completeness with AI, and differences between categories were not related to their level of experience.
qualified (qualify): 1 supporting (1 independent, strongest 0.25), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s13.a Professional developers benefited more than BSc students, but each category got tasks suited to its level, with researchers and professionals given the most complex tasks.
qualified (qualify): 1 supporting (1 independent, strongest 0.25), 3 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites] Google RCT: Google RCT: more senior engineers and those codin…
verified s14.a METR's developers had on average 5 years and 1,500 commits of experience on their respective repositories.
disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.
warning needs-qualification: Contradicted by Google RCT: Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s15.a AI helped METR's developers less than in existing literature, with a baseline slowdown estimate, and METR presents moderate evidence that developers are slowed down more when they have high prior task exposure.
disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.
warning needs-qualification: Contradicted by Google RCT: Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s15.b Qualitatively, METR's developers note that AI is particularly helpful on unfamiliar issues and less helpful on familiar ones.
disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.
warning needs-qualification: Contradicted by Google RCT: Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s16.a One METR developer said that as the dedicated maintainer of a very specialized part of the codebase, agent mode could not do better than them.
disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.
warning needs-qualification: Contradicted by Google RCT: Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s17.a In the Google trial on a complex task, the AI x seniority interaction was negative, suggesting more senior participants were faster with AI, but not statistically significant.
qualified (qualify): no direct support, 3 qualifying.
warning needs-qualification: Google RCT: Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported. Add the qualification: the evidence supports a narrower or weaker statement.
verified s18.a The Google trial's authors believe this may be because AI is not yet able to close a skill gap on complex tasks like the one they tested.
qualified (qualify): no direct support, 3 qualifying, 2 context only.
warning needs-qualification: Google RCT: Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported. Add the qualification: the evidence supports a narrower or weaker statement.
verified s20.a Up to 50 hours of Cursor experience (self-reported prior hours plus hours in the study), more experience broadly did not reduce METR's slowdown effect.
qualified (qualify): no direct support, 7 qualifying.
warning needs-qualification: METR 2025 RCT: METR RCT: speedup did not improve over the first 50 hours of Cursor use, and the lone >50h developer's speedup (~25%) fell to ~0% once a second developer's misreported experience was corrected. The 'high skill ceiling' idea is plausible but underpowered. Add the qualification: the evidence supports a narrower or weaker statement.
verified s21.a METR saw a speedup for the one developer with more than 50 hours of Cursor experience, but after a second developer corrected their reported experience, the over-50-hours bucket moved from roughly 25% speedup to roughly 0%.
qualified (qualify): no direct support, 7 qualifying.
warning needs-qualification: METR 2025 RCT: METR RCT: speedup did not improve over the first 50 hours of Cursor use, and the lone >50h developer's speedup (~25%) fell to ~0% once a second developer's misreported experience was corrected. The 'high skill ceiling' idea is plausible but underpowered. Add the qualification: the evidence supports a narrower or weaker statement.
verified s22.a The Google trial, which sampled only full-time Googlers, found no statistically significant evidence that more frequent prior use of AI coding tools increased speed with AI; the authors list inadequate statistical power, a low usability floor (benefits accruing quickly), and Natural Language to Code requiring familiarity with LLMs and prompting, skills which might not be universal at this time, among possible reasons.
qualified (qualify): no direct support, 7 qualifying.
warning needs-qualification: Google RCT: Google RCT, 96 engineers: senior and heavy-coding developers were tentatively faster with AI and natural-language-to-code needs prompting skill, but how often people already used AI tools had no significant effect on their AI speed-up, so the authors point to a 'low usability floor'. Add the qualification: the evidence supports a narrower or weaker statement.
verified s23.a In the Echoes of AI trial, the more AI-skilled the Task 1 participants were, the stronger the effect of their AI use on Task 2 CodeHealth, but building on code from a proficient Java developer mattered even more.
qualified (qualify): no direct support, 7 qualifying.
warning needs-qualification: Echoes of AI maintainability RCT: Bayesian analysis: the more AI-skilled the original developer, the larger the downstream CodeHealth benefit (+0.10, about half an SD, for habitual users). The authors conclude developers vary in their ability to use AI well, though Java proficiency mattered even more. Add the qualification: the evidence supports a narrower or weaker statement.
verified s24.a None of the trials shows AI amplifying experienced engineers far more than novices: the Microsoft/Accenture, Shan and Peng trials found larger gains for less experienced developers, though the Microsoft/Accenture estimates are noisy and not statistically significant and Peng's result is conditional on persisting in further studies, and the Google trial's seniority interaction was not significant either.
qualified (qualify): 3 supporting (3 independent, strongest 0.5), 3 qualifying.
warning needs-qualification: Google RCT: Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported. Add the qualification: the evidence supports a narrower or weaker statement.
verified s25.a Gains were larger for short-tenure developers in the Copilot field experiments, and METR's developers found AI more helpful on unfamiliar issues and were slowed more on tasks with high prior exposure.
disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.
warning needs-qualification: Microsoft/Accenture field RCTs: Microsoft experiment: gains were larger for short-tenure (newer to the company and its code) and junior developers (27-39% vs 8-13%), but differences were not significant and the routine-code half of the claim was not tested. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s27.a When using Cursor, METR's developers accepted less than 44% of the generations.
qualified (qualify): 1 supporting (1 independent, strongest 0.6), 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit. Vibe-coding literature review: Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional v…
verified s28.a When METR's developers did not accept generations they reattempted with different prompts or gave up, and the low reliability qualitatively resulted in significant wasted time spent reviewing, testing or modifying AI code before rejecting it.
qualified (qualify): 1 supporting (1 independent, strongest 0.6), 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit. Vibe-coding literature review: Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional v…
verified s29.a Of METR's experienced developers, 75% report reading every line of AI generated code and 56% report often needing to make major changes to clean it up.
qualified (qualify): no direct support, 1 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users. "every" is stronger than the evidence (strongest support 0). Add the qualification: the evidence supports a narrower or weaker statement.
verified s30.a In the Anthropic study, an engineer said that for a task they expect to take less than 10 minutes they probably will not bother using Claude.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit. Add the qualification: the evidence supports a narrower or weaker statement.
verified s31.a Another engineer described a cold start problem: intrinsic information about their team's codebase that Claude does not have by default, so rather than iterate on the perfect prompt they do it themselves.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit. Add the qualification: the evidence supports a narrower or weaker statement.
verified s32.a In the migration report, 5,359 files were migrated with AI-generated changes; the bottleneck was the speed at which engineers could review them, and the number of changes generated each week was purposefully limited to avoid overwhelming reviewers.
qualified (qualify): 1 supporting (1 independent, strongest 0.55), 1 qualifying.
warning needs-qualification: Code-ownership experiment (Martin-Lopez): AI users produced more, larger code but understood it less. Their median answering time was 3 min longer (+27%) and they switched back to the code more than twice as often, though the time effect was not robust. The authors suggest time saved may be offset later in the lifecycle. [partial: the verdict note limits its own scope ("though")] Add the qualifica…
verified s33.a The vibe-coding literature review finds the central risk is a verification gap, a mismatch between the speed at which AI-assisted workflows generate code and the capacity of developers and processes to verify correctness, security and provenance.
qualified (qualify): 1 supporting (1 independent, strongest 0.4), 1 qualifying.
warning needs-qualification: Code-ownership experiment (Martin-Lopez): AI users produced more, larger code but understood it less. Their median answering time was 3 min longer (+27%) and they switched back to the code more than twice as often, though the time effect was not robust. The authors suggest time saved may be offset later in the lifecycle. [partial: the verdict note limits its own scope ("though")] Add the qualifica…
verified s34.a The review suggests that skill in verification, constraint specification and review may matter as much as authoring skill.
qualified (qualify): 1 supporting (1 independent, strongest 0.5), 1 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users. Add the qualification: the evidence supports a narrower or weaker statement.
verified s35.a The Anthropic study describes a paradox of supervision: effectively using Claude requires supervision, and supervising Claude requires the coding skills that may atrophy from AI overuse.
qualified (qualify): 1 supporting (1 independent, strongest 0.5), 1 qualifying.
warning needs-qualification: METR 2025 RCT: METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users. Add the qualification: the evidence supports a narrower or weaker statement.
verified s36.a METR lists low AI reliability, which results in significant wasted time reviewing, testing or modifying AI generated code before rejecting it, as one factor limiting AI performance for its developers.
qualified (qualify): 1 supporting (1 independent, strongest 0.6), 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit. Vibe-coding literature review: Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional v…
27 scenes, 7:47 at 150 words per minute including pauses. Narration is copied from the script; every chart value and on-screen citation is checked against a passage.
1. sc01 · 0:00 to 0:10 typography
Picture a pair of developers switching on the same AI coding assistant on the same morning: a veteran, and a newcomer just starting out.
Visual. Dark field; two words build left and right, 'veteran' then 'newcomer', as the narration names them.
On screen. A veteran and a newcomer
2. sc02 · 0:10 to 0:13 typography
Which of them gets more out of it?
Visual. The question types on in large type; 'more' held in accent colour.
On screen. Which of them gets more out of it?
3. sc03 · 0:13 to 0:21 typography
The confident answer online is the veteran, so let's put that answer next to the trials that actually measured it.
Visual. Title line; 'the veteran' lands first, then the rest builds.
On screen. The confident answer: the veteran
4. sc04 · 0:21 to 0:34 typography
On The Pragmatic Engineer, in an episode about AI at Shopify, it was offered as a maybe: "the AI agents are going to help the best engineers more than the mediocre engineer".
Visual. Quote card with the quoted words only; 'the best engineers' turns accent as spoken.
On screen. "the AI agents are going to help the best engineers more than the mediocre engineer"
Cite. The Pragmatic Engineer, episode on AI at Shopify
5. sc05 · 0:34 to 0:45 typography
On the same show, Simon Willison said that with these tools he will "run circles around somebody who's never written a code line of code in their life".
Visual. Quote card with the quoted words only; 'run circles' punches in.
On screen. "run circles around somebody who's never written a code line of code in their life"
Cite. Simon Willison on The Pragmatic Engineer
6. sc06 · 0:45 to 1:03 typography
Willison's yardstick is someone who has never coded, and the trials that follow compared developers with each other, so what they can test is the first claim: when researchers split their results by experience, did the best engineers gain more than the rest?
Visual. The testable question types on.
On screen. Did the best engineers gain more than the rest?
7. sc07 · 1:03 to 1:25 data-chart
In field experiments that gave developers the coding assistant Copilot, short-tenure developers raised their output by 27 to 39 percent, while long-tenure developers gained 8 to 13 percent. Those estimates are noisy and not statistically significant, but the pattern was the same on all three of the studies' main output measures.
Visual. Two stat tiles side by side: short-tenure '27% to 39%' left, long-tenure '8% to 13%' right; the second tile arrives as the narration reaches it, then a small 'not statistically significant' line fades in under both on s08.
On screen. Noisy, not statistically significant
| Short-tenure developers | 27% to 39% | 27 | percent | Microsoft/Accenture field RCTs, lines 758-L770 |
| Long-tenure developers | 8% to 13% | 8 | percent | Microsoft/Accenture field RCTs, lines 758-L770 |
8. sc08 · 1:25 to 1:39 document-excerpt
The authors note that longer-tenure developers abandoning the tool after a trial may shrink their estimate, yet junior developers, who used it as much as seniors once they had adopted it, still gained more than senior ones.
Visual. The cited paragraph from the paper; the phrase about equal usage after adoption highlights as the narration reaches it.
9. sc09 · 1:39 to 2:03 typography
A separate experiment found the same direction: developers with fewer years of experience got the speed benefit from generative AI, while the quality of their code was unaffected. Another Copilot trial found that less experienced programmers benefited more, and its authors called that promising for people moving into software development, if the result holds up in further studies.
Visual. List builds one line per sentence: first the separate experiment's finding, then the Copilot trial's.
On screen. Fewer years of experience: faster, quality unaffected / Less experienced programmers benefited more
Cite. Shan et al., generative AI experimentPeng et al., Copilot trial
10. sc10 · 2:03 to 2:27 typography
An experiment that ran from bachelor's students to professional developers found that every group got significantly further through its task with AI support, and its authors say the differences between groups were not related to experience. Professionals gained more there than bachelor's students, but each group had tasks suited to its level, and the professionals' were among the most complex.
Visual. List builds one line per sentence: every group improved; professionals gained more, on harder tasks.
On screen. Every group got further with AI / Professionals gained more, on the most complex tasks
Cite. Code-ownership experiment, BSc students to professionals
11. sc11 · 2:27 to 2:37 data-chart
Then there is a trial of developers who knew their projects deeply: 5 years and 1,500 commits on average in the repositories they worked on.
Visual. Two stat tiles: '5 years' and '1,500 commits', each landing as spoken.
| Years in the repository | 5 years | 5 | count | METR 2025 RCT, lines 960-L984 |
| Commits | 1,500 commits | 1500 | count | METR 2025 RCT, lines 960-L984 |
12. sc12 · 2:37 to 2:57 document-excerpt
With AI allowed, these developers were slower, not faster, and its researchers report moderate evidence that they were slowed down more on issues where they had high prior exposure to the task; the developers said AI was particularly helpful on unfamiliar issues and less helpful on familiar ones.
Visual. The cited METR passage; 'moderate evidence' and 'high prior task exposure' highlight as the narration says them.
13. sc13 · 2:57 to 3:09 typography
As one developer put it: "if I am the dedicated maintainer of a very specialized part of the codebase, there is no way agent mode can do better than me."
Visual. Quote card with the quoted words only.
On screen. "if I am the dedicated maintainer of a very specialized part of the codebase, there is no way agent mode can do better than me."
14. sc14 · 3:09 to 3:29 typography
One trial leaned the other way: on a complex task, more senior engineers were, if anything, faster with AI than less senior ones, though that difference was not statistically significant. Its authors suggest AI may not yet be able to close a skill gap on a task that complex.
Visual. List builds one line per sentence: the lean toward seniors with 'not statistically significant', then the authors' explanation.
On screen. Seniors faster with AI, not statistically significant / AI may not yet close a skill gap on complex tasks
15. sc15 · 3:29 to 3:39 typography
That leaves the other half of the belief: that using these tools well is a skill of its own, and the skilled pull ahead.
Visual. Question types on; 'skill' in accent.
On screen. Is using these tools well a skill of its own?
16. sc16 · 3:39 to 4:04 data-chart
Among those deeply experienced developers, the slowdown broadly did not shrink with up to 50 hours of Cursor experience, counting the hours they reported from before the study. One developer past 50 hours did see a speedup, but when a second developer corrected the experience they had reported, that group's estimate fell from roughly 25 percent faster to roughly 0 percent.
Visual. Two stat tiles for the over-50-hours group: 'roughly 25%' faster before the correction, then 'roughly 0%' after; the second tile arrives on 'roughly 0 percent' and the first dims.
| Before the correction | roughly 25% | about 25 | percent | METR 2025 RCT, lines 1217-L1234 |
| After the correction | roughly 0% | about 0 | percent | METR 2025 RCT, lines 1217-L1234 |
17. sc17 · 4:04 to 4:28 typography
In the trial that leaned toward seniors, run with full-time Googlers, how often people already used AI tools made no significant difference to their speed with AI; among the possible reasons its authors offer are too little statistical power, a low usability floor where the benefits come quickly, and one feature needing prompting skills that may not be universal yet.
Visual. List: the finding, then the possible reasons as the narration names them.
On screen. Prior AI use: no significant difference / Too little statistical power / A low usability floor / Prompting skills not universal yet
18. sc18 · 4:28 to 4:47 document-excerpt
The clearest sign that tool skill matters comes from a maintainability trial: the more practised the original developer was with AI, the better the code scored on a code-health measure once someone else built on it, though that developer's proficiency in Java mattered even more.
Visual. The cited paragraph; 'the more AI-skilled' highlights, then the closing sentence on the proficient Java developer.
19. sc19 · 4:47 to 5:21 typography
So across these trials, nothing shows AI multiplying experts far more than novices: several leaned toward the less experienced, though the field experiments' gaps were noisy and not significant and another trial's result awaits further studies, and the one hint the other way was not significant either. What runs through them is how much the developer already knew: the tool helped more for short-tenure developers in one trial and on unfamiliar issues in another, and helped less for maintainers on tasks they knew well.
Visual. List builds one line per sentence: the verdict, then the thread.
On screen. Nothing shows AI multiplying experts far more / It helped more where the developer knew less
Cite. Cui et al., Copilot field experimentsMETR 2025 randomized trial
20. sc20 · 5:21 to 5:29 typography
But how can a tool that writes code for you leave an expert slower on work they know best?
Visual. Question types on; 'slower' in accent.
On screen. How can it leave an expert slower on work they know best?
21. sc21 · 5:29 to 5:50 data-chart
Back with those maintainers, the answer starts with how often they took what the tool offered: they accepted fewer than 44 percent of Cursor's generations. When they turned one down, they reprompted or gave up, and the researchers saw significant time wasted reviewing, testing or modifying code before rejecting it.
Visual. One big stat tile: '<44%' of generations accepted; on s28 the tile shifts up as the wasted-time point is spoken.
| Generations accepted | <44% | lt 44 | percent | METR 2025 RCT, lines 1004-L1012 |
22. sc22 · 5:50 to 6:02 data-chart
Even accepted code cost time: 75 percent said they read every line the AI wrote, and 56 percent said they often had to make major changes to clean it up.
Visual. Two horizontal bars: 75% read every line, 56% often made major changes; each grows as spoken.
| Read every line the AI wrote | 75% | 75 | percent | METR 2025 RCT, lines 1015-L1021 |
| Often made major changes to clean it up | 56% | 56 | percent | METR 2025 RCT, lines 1015-L1021 |
23. sc23 · 6:02 to 6:27 typography
Engineers deciding when to hand work to Claude described the same trade; for a task under 10 minutes, one said, "I'm probably not going to bother using Claude." Another described a cold start: their team's codebase holds knowledge Claude does not have by default, so rather than iterate on the perfect prompt, "I'm just going to go and do it myself."
Visual. Two quote lines, one per sentence, quoted words only.
On screen. "I'm probably not going to bother using Claude." / "I'm just going to go and do it myself."
24. sc24 · 6:27 to 6:43 data-chart
The same limit shows up at larger scale: in one company's AI-assisted migration of 5,359 files, the bottleneck was how fast engineers could review the changes, so the team deliberately capped how many changes it generated each week.
Visual. One stat tile: '5,359 files' migrated; 'review' line fades under it as the bottleneck is named.
On screen. Bottleneck: how fast engineers could review
| Files migrated | 5,359 files | 5359 | count | Google migrations report, lines 417-L423 |
Cite. Google migrations report
25. sc25 · 6:43 to 7:04 typography
A review of what has been written about vibe coding names its central risk a verification gap: code generated faster than developers and their processes can verify its correctness, security and provenance. Its authors suggest that skill in verification, constraint specification and review may matter as much as skill in writing code.
Visual. List builds one line per sentence: the gap, then the skills.
On screen. A verification gap / Verification and review may matter as much as writing code
Cite. Vibe-coding literature reviewVibe-coding literature review
26. sc26 · 7:04 to 7:22 typography
And the study of engineers using Claude names the catch, a "paradox of supervision": using Claude effectively requires supervising it, and supervising it requires the very coding skills that may atrophy from overuse. That checking is part of the price those maintainers paid.
Visual. Quote card with the two quoted words only.
On screen. "paradox of supervision"
Cite. Anthropic study of its engineersMETR 2025 randomized trial
27. sc27 · 7:22 to 7:47 typography
So before you hand a task to an assistant, ask yourself how much of it you already know, and how long checking its answer will take. And if you find yourself deleting most of what it writes, point it at the parts that are new to you, and keep enough of the skill to check what it hands back.
Visual. List builds one line per sentence.
On screen. How much of it do you already know? / Keep enough of the skill to check what it hands back