‘AI pimping’ and the rise of deepfake dating scams

The rapid proliferation of AI-generated influencer accounts on Instagram represents a growing challenge at the intersection of artificial intelligence, social media, and content creation, raising concerns about authenticity and revenue impacts on legitimate creators.

Current state of affairs: Instagram faces an unprecedented surge in AI-generated influencer accounts that appropriate content from real creators while using artificially generated faces to monetize stolen material.

• Over 1,000 AI-generated accounts have been identified engaging in this practice
• Content thieves are specifically targeting videos from legitimate models and adult content creators
• The accounts employ sophisticated AI tools to create hybrid personas that blend features from multiple celebrities

Technical implementation: A combination of AI tools and platforms enables the creation and monetization of these artificial personas.

• Leonardo AI serves as the primary tool for generating realistic facial features
• Face-swapping capabilities are provided through apps like HelloFace and Discord plugins such as InsightFace
• ChatGPT assists in developing comprehensive backstories and personalities for these AI personas
• Various image generation applications specifically permit NSFW content creation

Monetization strategies: These artificial accounts employ multiple revenue streams to capitalize on their stolen and AI-modified content.

• Accounts direct followers to dating websites and adult content platforms
• Many utilize Patreon and OnlyFans alternatives for subscription-based revenue
• Some promote AI applications and related tools
• Operators often manage multiple AI personas simultaneously to maximize earnings

Impact on legitimate creators: The proliferation of AI-generated accounts poses significant challenges for authentic content creators.

• Real models and adult content creators report declining earnings
• Original content is being repurposed without consent or compensation
• The flood of AI content makes it harder for authentic creators to maintain visibility
• Instagram’s content moderation appears insufficient to address the issue

Platform response: Instagram’s approach to managing this trend has been largely reactive and ineffective.

• The platform only removes accounts after specific reports
• No systematic approach to identifying or preventing AI-generated accounts exists
• The volume of new accounts appears to overwhelm current moderation capabilities
• Existing content policies have not adapted to address this emerging challenge

Future implications: The rising tide of AI-generated content presents complex challenges for social media platforms and content authenticity.

• The sophisticated nature of AI generation tools makes detection increasingly difficult
• Without effective countermeasures, authentic human creators may face growing competition from AI-generated accounts
• The situation highlights the need for more robust content authentication systems and platform policies