Every factual sentence carries assertions checked against the recorded evidence: references must resolve, figures and quotations must be in the cited passage, arithmetic must reproduce, and an independent reviewer must approve the wording. Automated validation says whether the narration is ready for human editorial review; it does not establish truth.
Open a sentence to see its assertions, their status and the passages behind them. Sentences without a marker make no factual claim.
verified s04.a DHH, on the Lex Fridman Podcast, said that getting a 10X, 100X or in rare cases 1000X productivity boost requires interacting with the agents directly.
supported (approved): no direct support, 1 contradicting, 1 qualifying, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s05.a Several videos and posts report that experienced developers were 19% slower with AI tools.
supported (approved): no direct support, 5 source statements.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s05.b The channel Macro Lens said that when the developers were allowed AI tools they were 19% slower than when they weren't.
supported (approved): no direct support, 1 source statement.
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s07.a Observing the output of almost 5000 software developers at three companies in their regular jobs, partly over years, the authors estimate that a generative AI code suggestion tool raised developer productivity by 26.08%, about 26%.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. [partial: the verdict note limits its own scope ("rather than")] Shan et al. field study + RCT: Field DID at a 5,000-person IT services firm (201 devs, 27 w…
Calculation: 26.08 = 26
verified s08.a The field estimate is substantially smaller than the lab result of Peng et al.; the authors suggest some coding tasks are less readily assisted by the tool and coding is only part of a developer's job.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. [partial: the verdict note limits its own scope ("rather than")] Shan et al. field study + RCT: Field DID at a 5,000-person IT services firm (201 devs, 27 w…
verified s09.a The trial estimated a roughly 21% increase in development speed attributable to AI in its best-fit model, an effect that did not meet the usual significance threshold.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. [partial: the verdict note limits its own scope ("rather than")] Shan et al. field study + RCT: Field DID at a 5,000-person IT services firm (201 devs, 27 w…
verified s11.a METR's randomized controlled trial had 16 experienced open-source developers complete 246 tasks in mature projects on which they had an average of 5 years of prior experience, with each task randomly assigned to allow or disallow AI tools.
supported (approved): 2 supporting (2 independent, strongest 0.85).
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s12.a The trial tested AI tools at the February-June 2025 frontier; developers primarily used Cursor Pro with Claude 3.5/3.7 Sonnet.
supported (approved): 1 supporting (1 independent, strongest 0.85).
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s13.a Allowing AI increased task completion time by 19%.
supported (approved): 1 supporting (1 independent, strongest 0.85).
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s14.a METR hypothesizes that AI helped its developers less than in earlier RCTs because of the size and complexity of the repositories, which average about 10 years old and >1,100,000 lines of code, compared with the more greenfield projects in Peng et al. and other trials.
qualified (qualify): no direct support, 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. Add the qualification: the evidence supports a narrower or weaker statement.
verified s16.a Based on conversations with study participants, METR believes it is likely that developers are more sped up from AI tools in early 2026 than in its early 2025 estimates.
qualified (qualify): no direct support, 1 qualifying.
warning needs-qualification: METR 2026 update: METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes. Add the qualification: the evidence supports a narrower or weaker statement.
verified s17.a METR's later study estimates a speedup of -18% (confidence interval -38% to +9%) for returning developers and -4% (confidence interval -15% to +9%) for newly recruited developers; both intervals reach into longer task times.
qualified (qualify): no direct support, 1 qualifying.
warning needs-qualification: METR 2026 update: METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes. Add the qualification: the evidence supports a narrower or weaker statement.
verified s18.a Because of the selection effects in its new experiment, METR calls its data only very weak evidence for the size of the increase in speedup.
qualified (qualify): no direct support, 1 qualifying.
warning needs-qualification: METR 2026 update: METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes. Add the qualification: the evidence supports a narrower or weaker statement.
verified s18.b METR observed a significant increase in developers choosing not to participate in its new experiment because they do not wish to work without AI, which it says likely biases its estimate of AI-assisted speedup downwards.
supported (approved): 1 supporting (1 independent, strongest 0.7).
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s20.a Anthropic employees self-reported that they now use Claude in 59% of their work and get +50% productivity gains from it on average.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x. Add the qualification: the evidence supports a narrower or weaker statement.
verified s20.b Anthropic saw a 67% increase in merged pull requests per engineer per day when it adopted Claude Code across its engineering org.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x. Add the qualification: the evidence supports a narrower or weaker statement.
verified s21.b A self-reported +50% gain and 67% more merged pull requests are an order of magnitude below 10x.
qualified (qualify): no direct support, 2 qualifying.
warning needs-qualification: Anthropic internal study: Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x. Add the qualification: the evidence supports a narrower or weaker statement.
verified s22.a Field and trial estimates in workplaces with the assistants they tested (26.08% more output; a roughly 21% speed increase that missed significance) show modest average gains, far below headline claims.
qualified (qualify): 2 supporting (2 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. [partial: the verdict note limits its own scope ("rather than")] Shan et al. field study + RCT: Field DID at a 5,000-person IT services firm (201 devs, 27 w…
verified s23.a Early-2025 AI tools increased completion time for experienced open-source developers on mature projects they knew well.
supported (approved): 1 supporting (1 independent, strongest 0.85).
info wording: Wording differs from the canonical claim; the reviewer confirms the passages entail it.
verified s23.b METR's newer data is only very weak evidence for how much developers are sped up now, because of selection effects.
qualified (qualify): no direct support, 1 qualifying.
warning needs-qualification: METR 2026 update: METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes. Add the qualification: the evidence supports a narrower or weaker statement.
verified s26.a Before starting tasks, the developers forecast that allowing AI would reduce completion time by 24%.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s27.a After completing the study, the developers estimated that allowing AI had reduced completion time by 20%, while the trial found it increased completion time.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s28.a Experts in economics predicted 39% shorter completion times and experts in ML predicted 38% shorter.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s30.a In the Copilot trial, participants in both treated and control groups estimated a 35% increase in productivity on average, an underestimation compared with the 55.8% increase in their revealed productivity.
disputed (qualify): 1 supporting (1 independent, strongest 0.45), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by METR 2025 RCT: Developers forecast a 24% speedup and afterward estimated 20%, but were actually slowed 19%. Experts also overestimated, predicting 38-39% faster. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s30.b Peng et al.'s Copilot result came from a software engineering task in the lab, on a more greenfield project.
qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.
warning needs-qualification: METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. Add the qualification: the evidence supports a narrower or weaker statement.
verified s31.a METR's experienced developers on their mature projects overestimated their speedup, while participants in the Copilot lab trial underestimated theirs.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s32.a METR's later update says some developers self-report very high speedups, and that as its earlier study documented those estimates can be quite unreliable.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: METR 2026 update: METR Feb 2026: some developers self-report very high speedups, which METR notes can be 'quite unreliable' per its earlier study; it also says the true speedup may be higher among those who selected out, so no new measured overestimate. Present the disagreement: name the studies on each side instead of stating one side as fact.
verified s33.a In METR's trial developers' estimates were far above the measured effect, and in the Copilot trial participants' estimates were far below it.
disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.
warning needs-qualification: Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup. Present the disagreement: name the studies on each side instead of stating one side as fact.
27 scenes, 6:06 at 150 words per minute including pauses. Narration is copied from the script; every chart value and on-screen citation is checked against a passage.
1. sc01 · 0:00 to 0:05 typography
How much faster does AI actually make the people who write code?
Visual. Dark field; the question types on in large type, 'faster' held a beat in accent colour.
On screen. How much faster does AI make the people who write code?
2. sc02 · 0:05 to 0:21 typography
Ask around and the answers point in opposite directions, from enormous multiples to slower than before. So set the loudest answers aside, look at the studies that measured programmers at work, and ask what each one actually measured.
Visual. Two lines build one per sentence: first the spread of answers, then the plan.
On screen. From enormous multiples to slower than before / Ask what each one actually measured
3. sc03 · 0:21 to 0:36 typography
On the Lex Fridman Podcast, DHH described "that magical 10X, 100X, in a few rare cases, 1000X productivity boost", and said that to get it you have to work with the AI agents directly.
Visual. Quote card: the quoted words only, the multiples punch in one by one as spoken.
On screen. "that magical 10X, 100X, in a few rare cases, 1000X productivity boost"
Cite. DHH on the Lex Fridman Podcast
4. sc04 · 0:36 to 0:52 typography
From the other side, a run of videos and posts repeats one trial's result, in the words of the channel Macro Lens: when experienced developers were allowed AI tools, "they were 19% slower than when they weren't".
Visual. Quote card in a contrasting colour to sc03; '19% slower' emphasised.
On screen. "they were 19% slower than when they weren't"
Cite. Macro Lens, on the METR trial
5. sc05 · 0:52 to 0:54 typography
Start with ordinary workplaces.
Visual. Section title card.
On screen. Ordinary workplaces
6. sc06 · 0:54 to 1:25 data-chart
One set of field experiments followed almost five thousand software developers at three companies, in their regular jobs and partly over years, and estimated that an AI code suggestion tool raised their productivity, measured by their output, by about 26 percent. The authors note that this is substantially smaller than an earlier lab result, and suggest why: some kinds of coding are less readily helped by the tool, and coding is only part of a developer's job.
Visual. One big tile: 26.08% productivity increase, label beneath. Holds through the authors' caveat.
| Productivity increase with an AI code suggestion tool | 26.08% | 26.08 | percent | Microsoft/Accenture field RCTs, lines 774-L784 |
7. sc07 · 1:25 to 1:39 data-chart
A separate trial estimated a roughly 21 percent increase in development speed attributable to AI, but that estimate missed the usual bar for statistical significance, so it is less certain than it sounds.
Visual. One tile: roughly 21% increase in development speed, drawn dimmed with a 'not significant' tag.
| Increase in development speed attributable to AI | 21% | about 21 | percent | Google RCT, lines 519-L528 |
Cite. Google randomized trial
8. sc08 · 1:39 to 1:43 typography
Then there is the trial behind those slowdown claims.
Visual. Section title card.
On screen. The trial behind the slowdown claims
9. sc09 · 1:43 to 2:08 data-chart
METR ran a randomized controlled trial with 16 experienced open-source developers on 246 tasks in mature projects they had worked on for an average of 5 years, randomly allowing or disallowing AI tools task by task. The tools were the February to June 2025 frontier: mostly the Cursor Pro editor with Claude 3.5 and 3.7 Sonnet.
Visual. Three tiles build in order: 16 developers, 246 tasks, 5 years average prior experience.
| Experienced open-source developers | 16 | 16 | count | METR 2025 RCT, lines 11-L27 |
| Tasks in mature projects | 246 | 246 | count | METR 2025 RCT, lines 11-L27 |
| Years of prior experience on the project, average | 5 | 5 | count | METR 2025 RCT, lines 11-L27 |
10. sc10 · 2:08 to 2:13 data-chart
With AI allowed, tasks took 19 percent longer to complete.
Visual. One big tile in red: 19% longer completion time.
| Longer to complete a task | 19% | 19 | percent | METR 2025 RCT, lines 11-L27 |
11. sc11 · 2:13 to 2:29 data-chart
The researchers' hypothesis for why AI helped less than in earlier trials: their projects averaged about 10 years old and more than 1,100,000 lines of code, compared with the more greenfield projects in those trials.
Visual. Two tiles: about 10 years old, more than 1,100,000 lines of code.
| Years old | 10 | about 10 | count | METR 2025 RCT, lines 995-L1001 |
| Lines of code | 1,100,000 | gt 1100000 | count | METR 2025 RCT, lines 995-L1001 |
12. sc12 · 2:29 to 2:32 typography
But would newer tools change that?
Visual. Question types on.
On screen. Would newer tools change that?
13. sc13 · 2:32 to 2:44 document-excerpt
METR's own 2026 update says that, from talking to participants, it believes developers are likely more sped up in early 2026 than in early 2025.
Visual. The METR 2026 passage, the belief sentence highlighted.
Cite. METR, 2026 update
14. sc14 · 2:44 to 2:58 data-chart
Its new raw numbers lean the same way: tasks took an estimated 18 percent less time for returning developers and 4 percent less for new recruits, but both ranges still stretch to a slowdown.
Visual. Two tiles: 18% less task time for returning developers, 4% for new recruits, each with a small caption that the range reaches a slowdown.
| Returning developers | 18% | 18 | percent | METR 2026 update, lines 61-L62 |
| Newly recruited developers | 4% | 4 | percent | METR 2026 update, lines 61-L62 |
15. sc15 · 2:58 to 3:14 document-excerpt
And METR calls this only very weak evidence: more developers now decline to take part because they don't want to work without AI, which likely pushes its speedup estimate down, so the ones left are not a fair sample.
Visual. The METR passage, the selection-effect sentence highlighted.
Cite. METR, 2026 update
16. sc16 · 3:14 to 3:17 typography
And the multiples from the start?
Visual. Question types on.
On screen. And the multiples from the start?
17. sc17 · 3:17 to 3:48 data-chart
The largest figures so far come from Anthropic's own report on its staff: employees self-reported that they now get a 50 percent productivity gain from Claude on average, and merged pull requests per engineer per day rose 67 percent when its engineering org adopted Claude Code. A count of merged changes is not the same as time saved, but even taken at face value, both figures are an order of magnitude below ten times.
Visual. Two tiles: +50% self-reported productivity gain, 67% more merged pull requests per engineer per day.
| Self-reported productivity gain, average | +50% | 50 | percent | Anthropic internal study, lines 35-L35 |
| Merged pull requests per engineer per day | 67% | 67 | percent | Anthropic internal study, lines 35-L35 |
18. sc18 · 3:48 to 4:11 typography
So, how much faster? In the workplace studies, with the tools they tested, the measured gains were modest, nowhere near the multiples in the hype. For experienced developers in big, mature projects they knew well, early-2025 tools made them slower, and whether today's tools have reversed that, METR's own newer data is too weak to say.
Visual. Two lines build one per sentence.
On screen. Workplace studies: modest gains / Experienced developers, mature projects: slower
Cite. Cui, Demirer et al.; Google trialMETR, 2025 trial and 2026 update
19. sc19 · 4:11 to 4:18 transition
But that slowdown trial holds a stranger detail, and it is about what the developers believed was happening.
Visual. Split field wipes to a new colour for the second question.
20. sc20 · 4:18 to 4:26 typography
Take a second and make your own guess: how much time does AI save you?
Visual. Question types on, then holds for the viewer's guess.
On screen. How much time does AI save you?
21. sc21 · 4:26 to 4:53 data-chart
Before starting, the METR developers forecast that allowing AI would cut their completion time by 24 percent. After the study, with the work behind them, they estimated it had cut their time by 20 percent, in the same trial that measured them taking longer. Outside experts did no better: economists predicted 39 percent shorter completion times, and machine learning experts 38 percent.
Visual. Horizontal bars of predicted time saved, building one per group as named.
| Developers, before | 24% | 24 | percent | METR 2025 RCT, lines 17-L24 |
| Developers, after | 20% | 20 | percent | METR 2025 RCT, lines 17-L24 |
| Economists | 39% | 39 | percent | METR 2025 RCT, lines 17-L24 |
| Machine learning experts | 38% | 38 | percent | METR 2025 RCT, lines 17-L24 |
22. sc22 · 4:53 to 4:56 typography
So people overestimate the gain? Not always.
Visual. Question types on.
On screen. So people overestimate the gain?
23. sc23 · 4:56 to 5:13 data-chart
In an earlier randomized trial of Copilot, a lab task on a greenfield-style project, the measured productivity gain was 55.8 percent, yet participants in both the AI and the control group estimated 35 percent on average: they underestimated it.
Visual. Two bars: measured gain 55.8%, estimated gain 35%; the gap highlighted.
| Measured | 55.8% | 55.8 | percent | Copilot RCT (Peng), lines 184-L188 |
| Estimated by participants, average | 35% | 35 | percent | Copilot RCT (Peng), lines 184-L188 |
24. sc24 · 5:13 to 5:25 typography
So the same kind of guess missed in opposite directions: too high for experienced developers in big projects they knew well, too low on a fresh task in the lab.
Visual. Title card.
On screen. Missed in opposite directions
25. sc25 · 5:25 to 5:36 document-excerpt
METR's later update adds that some developers now self-report very high speedups, and repeats that, as its earlier study found, such estimates can be quite unreliable.
Visual. The METR 2026 passage with the unreliability clause highlighted.
Cite. METR, 2026 update
26. sc26 · 5:36 to 5:44 typography
So can developers tell? In both of these trials, not reliably: their guesses missed by a wide margin.
Visual. Title card.
On screen. Not reliably
27. sc27 · 5:44 to 6:06 typography
Every number in this video belongs to particular people, doing particular work, with particular tools. So when someone tells you how much faster AI makes them, ask what they measured, and if you want to know for yourself, time comparable tasks with the tool and without it rather than trusting the feeling.
Visual. Two lines build one per sentence.
On screen. Particular people, particular work, particular tools / Ask what they measured