AI can make a software team look dramatically more productive while leaving customers unable to see much difference.
The reason is straightforward: writing code is only one stage of delivering software. If AI increases the amount of code entering the system but does not improve review, testing, merging, deployment, maintenance, or adoption, the organisation may simply create a larger queue of unfinished work.
A study discussed in the MIT Sloan article AI boosts worker productivity — but does that translate to final outputs? examined data from more than 100,000 GitHub developers using different generations of AI coding tools. Developers using the tools increased coding activity by up to 180%. That increase translated into 50% more projects and 30% more actual releases than among developers who did not use the tools.
Those are meaningful gains. But the downstream results were more sobering: the study found no corresponding increase in app downloads or user reviews after the adoption of AI coding tools.
This is the productivity trap. A team can become faster at producing code without becoming faster at delivering software that customers use.
The 180% illusion: activity is not delivery
Coding activity is easy to count. Teams can track commits, pull requests, generated functions, or lines of code. These measures are visible and often improve quickly after an AI coding assistant is introduced.
But none of them proves that the business is shipping more useful software.
A release is closer to the outcome that matters. Even that is not the final measure: a release must be reliable, discoverable, usable, and relevant enough for customers to adopt. The MIT Sloan article’s findings show the gap clearly. A large increase in coding activity produced a smaller increase in releases, followed by no measurable increase in downloads or reviews.
That does not mean AI coding tools are ineffective. It means their benefits are constrained by the rest of the delivery system.
Consider a small product team building a customer portal. Before AI adoption, developers spend four days implementing a feature and two days waiting for review, integration, test fixes, and deployment. An AI assistant reduces implementation time to one day. If the review queue, test environment, release approval, and deployment process remain unchanged, the team has not removed the six-day delivery cycle. It has made the first day less demanding while leaving the rest of the system intact.
The practical implication for an engineering manager is important: do not evaluate an AI coding investment primarily through code volume. Ask instead:
- How long does a change take from approved idea to production?
- How long does a pull request wait for review?
- How often do merges create rework or conflicts?
- How long does it take to diagnose and fix a failed build?
- How frequently are completed features delayed by release approvals?
- After release, are customers adopting the functionality?
These questions connect engineering activity to business value.
Where the pipeline breaks after AI accelerates coding
When coding becomes faster, the limiting step often moves downstream. The exact bottleneck will differ by organisation, but the common pattern is a workflow designed for a slower rate of code production.
Code review queues
If developers can produce more changes but reviewers still have the same available hours, pull requests accumulate. Review quality may decline as people rush, or delivery may slow as work waits for attention.
A practical response is not simply to ask reviewers to work harder. First, measure review wait time separately from review time. If a pull request takes six hours to review but two days to be picked up, the problem is queue design, not reviewer capability.
Teams can then reduce the queue by narrowing pull requests, setting review ownership, adding automated checks for routine issues, and reserving regular review capacity. AI may help summarise a change or identify obvious inconsistencies, but it should not replace accountable human review for important behaviour, security, or architectural decisions.
Merging and integration
AI-assisted development can increase the number of parallel changes. That creates more opportunities for conflicts, duplicated work, incompatible assumptions, and integration failures.
A team should track how often completed work must be reworked because it does not fit the current branch or system. Shorter-lived branches, smaller changes, clearer interface contracts, and earlier integration can help. The goal is to make integration a frequent, low-risk activity rather than a large event at the end of a sprint.
Testing and environment access
More code creates more test cases, dependencies, and edge conditions. If test suites are slow, unreliable, or difficult to run locally, the additional code increases waiting and diagnosis time.
Before adding another coding assistant, examine the test workflow:
- Identify the tests that most often delay a release.
- Separate product failures from flaky infrastructure.
- Automate repeatable environment setup.
- Make test results understandable enough that a developer can act on them.
- Establish who owns failures that block the pipeline.
AI can assist with test generation or failure summaries, but generated tests are not automatically good tests. They can reproduce the same assumptions as the code they are intended to check.
Deployment and release approval
Some organisations can build software quickly but release it through a manual sequence of tickets, approvals, environment checks, and handoffs. In that situation, coding acceleration increases the amount of work waiting for release.
Map the path from a merged change to production. Record each handoff, approval, wait, and failure. If a routine release requires several people to confirm information that already exists in the system, the organisation may have an administrative bottleneck that is suitable for automation.
The right target is not to remove all controls. It is to distinguish high-risk changes from routine ones and make the routine path faster while preserving stronger checks where the consequences justify them.
Maintenance and customer feedback
A release is not the end of the work. Teams must monitor it, respond to defects, maintain dependencies, and learn whether customers are using the feature.
The article reports no corresponding increase in app downloads or user reviews after AI coding tools were adopted. That finding is a useful warning against treating release count as the final business metric. A team that ships more unused features may be more active without being more effective.
Why smaller teams may absorb AI output more easily
The article includes the view that smaller teams may be more nimble because they face fewer collaboration and communication challenges. This is a reasonable operational consideration, but it should not be mistaken for a universal rule.
A smaller team often has fewer handoffs between the person who understands the customer problem, the person writing the code, the reviewer, and the person releasing it. That can make it easier to coordinate a higher volume of changes.
For example, a four-person product team may agree in one conversation whether an AI-generated change is safe to merge and how it should be tested. A larger organisation may need coordination across product, engineering, security, quality assurance, platform operations, and release management. The coding assistant works in both environments; the surrounding decision system does not.
A larger organisation can still benefit, but it must make the workflow explicit. Useful measures include:
- ownership of each stage from requirement to release;
- service-level expectations for review and incident response;
- clear definitions of what can be automated and what requires approval;
- shared standards for generated code, tests, documentation, and dependencies;
- a feedback loop connecting product usage to engineering priorities.
The leadership decision is therefore less about choosing between AI tools and more about deciding whether the organisation is prepared to handle the additional output.
A practical audit for AI-enabled software delivery
An engineering leader can run a lightweight workflow audit before expanding AI access.
1. Choose an outcome metric
Start with a measure that represents delivery rather than activity. Examples include median time from approved work to production, percentage of releases completed without rollback, or time from a customer-reported defect to a verified fix.
Keep coding activity as a diagnostic measure, not the main success criterion.
2. Map the full path of a change
Write down every stage:
idea → specification → implementation → review → merge → test → release → monitoring → customer adoption
For each stage, record the average waiting time, active effort, failure points, and person responsible for the next decision. Waiting time is often more revealing than effort because it exposes queues and handoffs.
3. Find the constraint
If implementation takes one day but review and release take five, adding more coding capacity is unlikely to improve throughput. If testing is the constraint, focus there. If the team releases reliably but customers do not adopt features, the problem may be prioritisation, usability, onboarding, or distribution rather than engineering speed.
4. Apply AI to the constraint, not the most visible task
AI may be useful for meeting summaries, converting requirements into testable acceptance criteria, producing release notes, explaining build failures, or identifying repetitive administrative work. These uses can support the human workflow across the lifecycle rather than concentrating all investment in code generation.
The appropriate choice depends on the actual bottleneck. A team with a review queue needs better review flow; a team with unreliable deployments needs release engineering; a team with poor adoption needs stronger product discovery and feedback — not necessarily more generated code.
5. Review the result against customer evidence
After making a workflow change, compare delivery time, defect rates, release reliability, and user behaviour. A faster internal process is valuable only if it improves the organisation’s ability to deliver and sustain useful software.
Callout: The constraint determines the return
If AI makes the first stage twice as fast while the slowest stage remains unchanged, the return will be limited. Measure where work waits, then improve that constraint before increasing the flow of new work.
Limitations and failure modes
The findings should be applied carefully. The study analysed GitHub developers and specific categories of AI coding tools, so its results may not transfer directly to every enterprise environment, regulated team, legacy system, or internal development platform. Definitions of coding activity, projects, releases, downloads, and reviews also matter when interpreting the numbers.
There is another risk in turning the findings into a new simplistic rule. Releases are more meaningful than commits, but release count alone can become another vanity metric if teams ship low-value or unreliable features. The strongest measurement system connects engineering throughput with reliability and customer use.
AI tools may also evolve. Future systems could assist with testing, deployment, maintenance, and other downstream work. That possibility does not eliminate the current management task: organisations still need to identify their constraints and redesign processes deliberately rather than assuming a new tool will repair them automatically.
The central lesson is not that AI coding assistants fail. It is that productivity is a property of the whole workflow. Speeding up the easiest part of a constrained system can create a larger pileup at the finish line. Teams will see more value when they measure what reaches customers, find where completed work gets stuck, and direct automation and human attention toward that point.