Adobe launches AI filmmaking tools with voice-powered sound generation

Adobe has launched new generative AI filmmaking tools that allow users to create sound effects using voice recordings of silly noises and control video generation with reference footage. The tools, available in beta on the Firefly app, provide filmmakers with more intuitive ways to generate custom audio and video content compared to traditional text-only prompts.

How the sound effects tool works: The Generate Sound Effects feature lets users record onomatopoeia-like sounds that are synchronized with video footage to create realistic audio effects.

• Users can play a video of a horse walking and simultaneously record “clip clop” noises in time with its hoof steps, alongside a text description like “hooves on concrete.”
• The tool then generates four sound effect options to choose from, displayed on a video editing timeline interface.
• It supports impact sounds like twigs snapping, footsteps, and zipper effects, plus atmospheric noises like nature sounds and city ambience.
• The feature doesn’t work for speech generation but builds on Adobe’s Project Super Sonic experiment shown at its Max event in October.

New video generation controls: Adobe is adding advanced features to its Firefly Text-to-Video generator that give users more precise control over output.

• Composition Reference allows users to upload a reference video alongside text prompts to mirror the composition in generated footage.
• Keyframe cropping lets users crop and upload images of first and last frames that Firefly uses to generate video between them.
• New style presets provide quick visual style selections including anime, vector art, and claymation options.

The competitive landscape: Adobe appears to be positioning itself as a creative software hub even as it lags behind rivals in generative AI model development.

• The style presets are currently only available with Adobe’s own Firefly video model, though results showed room for improvement in live demos.
• Alexandru Costin, Adobe’s Generative AI lead, indicated similar controls may become available for third-party AI models in the future.
• The company continues adding support for rival AI models within its tools, suggesting a strategy to maintain its creative software dominance as AI tools grow in popularity.