AI Tools in Development: Real Productivity Data
PyPI data analysis reveals no overall productivity gain among developers since the adoption of AI. The number of new packages hasn’t increased, despite expectations of automation streamlining routine tasks. The trend of frequent package updates began in 2019 and is linked to CI/CD tools like GitHub Actions—not AI.
Separating packages into AI and non-AI categories uncovers key nuances: AI-related packages are updated twice as often, especially popular ones. This correlates with heavy investment in AI sectors, not a universal efficiency boost. Hype-driven resource allocation masks the lack of systemic improvement.
Google Cloud’s report "Impact of Generative AI in Software Development" captures subjective perceptions: 75% of developers report increased productivity from GenAI. Yet objective metrics tell a different story:
- Delivery throughput drops by 1.5% at 25% GenAI adoption.
- Delivery stability declines by 7.2%.
Developers feel they’re doing less valuable work while routine tasks grow—directly contradicting automation expectations.
GitHub and Enterprise Case Studies
GitHub Innovation Graph data shows no rise in repository counts. Revolutionary tools were expected to spark a surge in side projects—but that hasn’t happened. In enterprise settings, AI coder rollouts initiated by managers lead to reduced efficiency.
Take Notion: product margin dropped from 90% to 80% after adding AI features. Market leaders feel pressured to follow trends, but without user growth, this becomes unprofitable.
AI excels for prototyping, but novelty effects fade quickly. It’s similar to AI-generated images: initial excitement has given way to skepticism.
METR Study: The Gap Between Expectations and Reality
A respected METR study evaluated tools like Cursor and Claude. Experienced developers subjectively report a 20% speedup; non-developers claim up to 40%. Objectively:
- AI slows work down by 20%.
- Less manual coding, but more time spent on review, debugging, and waiting.
- Perceived gains diverge from actual metrics across multiple dimensions.
This confirms: AI doesn’t accelerate routine work—it adds overhead for verification.
What Matters
- Objective metrics (throughput, stability) show declining efficiency with GenAI adoption.
- Subjective feelings of productivity don’t align with real outcomes.
- Activity spikes appear only in AI-hyped segments, driven by funding, not broad gains.
- Enterprise AI adoption reduces margins without increasing users.
- For startups, the bottleneck isn’t code—AI doesn’t solve core challenges.
Conclusions for Business and Developers
Serious AI adoption in development isn’t inevitable without breakthroughs. Current tools are pleasant to use but hurt delivery performance. Businesses should focus on ROI: measure throughput and stability, not just feelings.
Developers spend more time post-generation: debugging, refactoring, integration. This explains the slowdown. For mid-to-senior engineers, the key is selective use: leverage AI for boilerplate, but always verify rigorously.
Future research is needed across other roles. For now, facts suggest: AI makes work more engaging—but not more productive. Measuring ‘enjoyment’ of the process is a separate challenge.
— Editorial Team
No comments yet.