AI Falls Short of Human Skill in Document Summarization Trial

AI falls short in document summarization: A government trial conducted by Amazon for Australia’s Securities and Investments Commission (ASIC) has revealed that artificial intelligence performs worse than humans in summarizing documents, potentially creating additional work for people.

• The trial tested AI models, with Meta’s Llama2-70B emerging as the most promising, against human staff in summarizing submissions from a parliamentary inquiry.
• Ten ASIC staff members of varying seniority levels were tasked with summarizing the same documents as the AI model.
• Blind reviewers assessed both AI and human-generated summaries, unaware of the involvement of AI in the exercise.

Human superiority across all criteria: The trial results demonstrated that human-generated summaries consistently outperformed AI-generated ones in every aspect of evaluation.

• Human summaries scored 81% on an internal rubric, compared to the AI’s 47%.
• Humans excelled particularly in identifying references to ASIC documents within lengthy texts, a task known to be challenging for AI.
• The superiority of human summaries extended across all evaluation criteria and for every submission analyzed.

AI shortcomings and reviewer feedback: Reviewers highlighted several deficiencies in the AI-generated summaries, raising concerns about their practical usefulness.

• AI summaries often missed crucial emphasis, nuance, and context in the original documents.
• Incorrect information was sometimes included, while relevant information was overlooked.
• The AI occasionally focused on auxiliary points or introduced irrelevant information.
• Three out of five reviewers correctly guessed that they were reviewing AI-generated content.

Potential counterproductivity of AI summaries: The overall feedback from reviewers suggested that AI-generated summaries might actually hinder rather than help the summarization process.

• Reviewers felt that using AI summaries could create additional work due to the need for fact-checking.
• The necessity to refer back to original submissions for clarity and conciseness was noted as a drawback.
• Human-generated summaries were found to communicate messages more effectively and concisely.

Study limitations and future prospects: The report acknowledges certain limitations of the trial and leaves room for potential improvements in AI summarization capabilities.

• The AI model used in the study has already been superseded by more advanced versions with enhanced capabilities.
• Amazon was able to improve the model’s performance through refined prompts and inputs, suggesting further optimization possibilities.
• The report expresses optimism that AI may eventually become competent at summarization tasks in the future.

Human analytical skills remain unmatched: Despite the potential for AI improvement, the trial underscores the current superiority of human cognitive abilities in critical information analysis.

• The report emphasizes that human ability to parse and critically analyze information remains unparalleled by AI.
• This finding supports the view that generative AI should be positioned as a tool to augment human tasks rather than replace them entirely.

Implications for AI integration: The trial’s results provide valuable insights into the current state of AI capabilities and their practical applications in document summarization.

• Organizations considering AI implementation for summarization tasks should carefully weigh the potential drawbacks and limitations highlighted by this study.
• The findings underscore the continued importance of human expertise in critical analysis and information synthesis.
• As AI technologies evolve, ongoing evaluation and comparison with human performance will be crucial in determining their appropriate roles and applications in various industries.