The argument against unfettered AI development

The AI revolution’s ethical dilemma: The rapid development of artificial general intelligence (AGI) by private companies raises significant questions about public consent and democratic oversight in technological innovation.

The ambitious goals of AI companies: Major tech firms are actively working to create AGI, a form of artificial intelligence that could surpass human capabilities.

• OpenAI’s CEO Sam Altman has described their goal as building “magic intelligence in the sky,” essentially aiming to create a godlike AI.
• Altman himself acknowledges that AGI could “break capitalism” and poses “probably the greatest threat to the continued existence of humanity.”
• This push for AGI goes far beyond narrow AI systems designed for specific tasks, aiming instead for a general-purpose reasoning machine.

The democratic deficit in AI development: The creation of potentially world-altering technology is occurring without explicit public consent or government oversight.

• Jack Clark, co-founder of AI company Anthropic, has expressed unease about the lack of government involvement in such a transformative project.
• Clark questions how much permission AI developers should seek from society before making irreversible changes.
• The tech industry’s “move fast and break things” philosophy is being applied to AI development, raising concerns about potential consequences.

Addressing common objections: Proponents of unfettered AI development often raise several arguments, each with significant counterpoints.

1. “Our use is our consent”:
   • While AI tools like ChatGPT have seen rapid adoption, usage doesn’t necessarily imply informed consent.
   • Many users may be unaware of the broader implications and costs associated with these systems.
   • Professional pressures often compel individuals to use technologies they might otherwise avoid.
   • Using narrow AI tools doesn’t equate to consenting to the development of AGI.
2. “The public is too ignorant to guide innovation”:
   • While technical expertise is crucial, the public should have a say in broad policy directions and societal goals.
   • Historical precedent exists for global oversight of potentially existential technologies, such as nuclear weapons.
   • Democratic input on AI development doesn’t mean the public dictates technical specifics, but rather guides overall policy directions.
3. “Innovation can’t be curtailed”:

• • This argument ignores historical examples of successfully restricted technologies, such as human cloning and certain space-based activities.
  • The 1967 Outer Space Treaty demonstrates the possibility of international agreements to limit potentially dangerous innovations.
  • Ethical considerations have led to moratoria on certain scientific experiments in the past, showing that innovation can be responsibly managed.

Public opinion and AGI: Polling indicates that most Americans do not support the development of AGI, highlighting the disconnect between tech companies’ goals and public sentiment.

Broader implications: The development of AGI represents a pivotal moment in human history, with potential consequences that could affect all of humanity.

• The ancient principle that “what touches all should be decided by all” applies as much to superintelligent AI as it does to other existential technologies.
• There is a pressing need for a broader societal discussion on the direction and limits of AI development, particularly concerning AGI.
• Balancing innovation with democratic oversight and ethical considerations will be crucial as AI technology continues to advance rapidly.