The AI Productivity Puzzle: New Study Untangles the Complex Effects of AI on Knowledge Workers

TL;DR

Goal: Researchers from Harvard Business School, Wharton, MIT, Warwick Business School, and Boston Consulting Group conducted field experiments to assess the impact of AI on consultant productivity and output quality.

Methodology:

• 758 consultants from Boston Consulting Group participated in randomized controlled experiments using GPT-4.
• Consultants were assigned realistic tasks inside and outside the capabilities frontier of AI.
• Performance was measured on productivity (task completion rates) and output quality (expert evaluations).

Key findings:

• For tasks inside the frontier, consultants leveraging AI exhibited 12% higher task completion rates and over 40% higher output quality.
• Consultants in the bottom half of skills benefited more from AI (43% increase) versus top half (17% increase).
• For tasks outside the frontier, consultants using AI produced worse results, with 19 percentage points lower likelihood of correct solutions.
• Two collaboration approaches emerged: Centaurs (delegating between human and AI) and Cyborgs (intertwining efforts with AI).

Recommendations:

• Firms should evaluate AI integration at the task level based on the capabilities frontier rather than wholesale adoption.
• Training and oversight is crucial to ensure appropriate usage and validations for tasks near the frontier edge.
• Roles and workflows will need adjustment to enable fluid human-AI collaboration.
• Diversity of perspectives should be maintained as AI tends to decrease output variability.

Thinking Critically

Implications:

• Adoption of AI like LLMs for knowledge work tasks could substantially boost productivity, creativity, and quality (12-40% in the study), with potential to accelerate economic growth. However, lower-skilled workers may face displacement without reskilling opportunities.
• Organizations will need to redesign workflows, roles, training programs, and management practices to enable effective human-AI collaboration. Optimal collaboration may involve Centaur or Cyborg approaches.
• Overreliance on AI could lead to deskilling, erosion of critical thinking, and more homogenized outputs if variability is not actively maintained.

Alternative perspectives:

• The study was limited to a single consulting firm so findings may not generalize across other organizations and industries. More research is needed on different tasks.
• There may be longer-term risks of overreliance on AI like reduced innovation or issues with accountability that were not captured in a short-term study.
• The tasks tested may not fully represent the breadth and complexity of real-world knowledge work activities. More complex tasks may fall outside the capabilities frontier.

 

AI predictions:

• As AI capabilities expand, the share of tasks inside the frontier is likely to grow rapidly, accelerating integration. However, new challenges will arise as the frontier shifts.
• Effective human-AI collaboration will become a core competency, changing training priorities and potentially hiring criteria. Companies may need specialists in AI collaboration.
• Governance frameworks, monitoring, and controls will be imperative to ensure responsible and ethical AI usage and address risks as they emerge.

Glossary

• Jagged frontier: The uneven capabilities of AI where some tasks are easily automated while other tasks of similar complexity remain outside of AI’s current skills. The frontier shifts as AI advances.
• Centaur: A human-AI collaboration model where humans strategically switch between themselves and AI for subtasks best suited to each, fusing together strengths like the mythical half-human, half-horse creature.
• Cyborg: A human-AI collaboration model with intricate integration where humans and AI collaboratively work on subtasks together in tight tandem, like a cybernetic organism.
• LLMs: Large language models, a type of AI like ChatGPT capable of generating human-like text.
• Retainment: The degree subjects directly retained and copied AI-generated text into their own work.
• Semantic similarity: A measure of how similar two pieces of text are in their underlying meaning, not just word usage.
• USE: Google’s Universal Sentence Encoder, which encodes text into semantic vector representations.