AI reviewing its own code challenges software engineering norms

The AI code review landscape faces a philosophical dilemma as AI systems increasingly generate code at scales surpassing human contributions. The question of whether an AI should review its own code challenges traditional software development practices and reveals surprising insights about both human and machine abilities in code quality assessment.

The big picture: The discovery that an AI bot named “devin-ai-integration[bot]” opened more pull requests than any human user raises fundamental questions about AI code review practices and accountability.

• This observation came from analyzing the power law distribution of pull requests opened by Greptile users, where the AI bot appeared at the far left of the distribution.
• The phenomenon creates a unique scenario where the same underlying AI technology could potentially both generate and review code.

The core dilemma: Traditional software development practices dictate that code authors shouldn’t review their own work, but AI systems present unique considerations that challenge this rule.

• Having the same AI model both generate and review code seems counterintuitive when the purpose of code review is to get a fresh perspective.
• The stateless nature of LLMs means each inference is essentially a clean slate, potentially making an AI’s self-review more objective than a human’s self-review.

Key counterpoints: Several factors suggest AI code review shouldn’t be dismissed outright, even when reviewing code the same model generated.

• Different AI tools built on the same underlying model often use distinct “scaffolding” – specialized workflows and context management systems that effectively create different reviewer perspectives.
• Human reviewers, while physically different people, share similar neural architecture, training (as engineers), and company context – making them potentially less different from each other than two AI systems built on the same base model.

Why this matters: AI-generated code requires more thorough review processes, not less, as it introduces unique challenges and error patterns.

• Good engineers often produce lower quality code when using AI tools due to imperfect prompting and insufficient review of machine-generated solutions.
• The speed at which AI generates code eliminates the natural “review-as-you-go” process that occurs when humans write code themselves.

Behind the numbers: AI demonstrates surprising effectiveness at finding bugs that humans miss, despite producing code with more defects.

• Tests showed Anthropic’s Sonnet model correctly identified 32 out of 209 “hard” category bugs in a benchmark.
• Human engineers at Greptile could identify no more than 5-7 bugs from the same set, showing AI’s superior bug detection capabilities despite its limitations.

The paradox: AI introduces more bugs into code than humans would, yet outperforms humans at detecting bugs during review.

• The bugs that AI introduces are often different from those humans would create, making them harder for human reviewers to catch.
• Traditional PR review processes are already limited in catching bugs, focusing more on style and architectural considerations than defect detection.