How to use AI to enhance software testing practices

AI-powered test generation revolutionizes software development: Large Language Models (LLMs) are transforming the way software engineers approach testing, significantly reducing the time and effort required to create comprehensive test suites.

• Assembled, a software company, has leveraged LLMs to save hundreds of engineering hours by automating the test writing process.
• The company’s engineers now complete tasks that previously took hours in just 5-10 minutes, allowing them to allocate more time to developing new features and refining existing ones.

The importance of robust testing: Comprehensive testing is crucial for maintaining software quality and enabling rapid development, but it is often overlooked due to time constraints or complexity.

• Martin Fowler, a renowned software developer, emphasizes that testing not only reduces production bugs but also instills confidence in making system changes.
• LLMs have made it significantly easier and faster to generate robust tests, addressing the common challenge of balancing quality with development speed.

Implementing LLM-powered test generation: Assembled’s approach to using LLMs for test generation involves crafting precise prompts and iterating on the results.

• Engineers use high-quality LLMs like OpenAI’s o1-preview or Anthropic’s Claude 3.5 Sonnet for code generation.
• A sample prompt includes the function to be tested, relevant struct definitions, and an example of a good unit test from the existing codebase.
• The generated tests are reviewed, refined, and adjusted to match the codebase conventions before integration.

Versatility of the approach: The LLM-powered test generation method can be adapted for various testing scenarios and programming languages.

• The technique can be applied to different programming languages by adjusting the prompt and providing language-specific examples.
• It can be extended to frontend component testing, including React components with user interactions and state changes.
• Integration testing with mocked services can also be generated using this approach.

Key considerations for effective implementation: While LLM-powered test generation has proven highly beneficial, there are several factors to consider for optimal results.

• Iterative refinement is often necessary to cover all edge cases and align with codebase standards.
• Engineers should double-check the logic of generated tests, as LLMs can occasionally produce incorrect output.
• Customizing prompts to specific contexts and providing high-quality examples significantly enhances the quality of generated tests.
• Using the most advanced LLM models generally yields better results, even if they have higher latency.

Impact on development practices: The adoption of LLM-powered test generation has had a significant positive impact on Assembled’s development process.

• The reduced “activation energy” for writing tests makes it less likely for engineers to skip testing due to time constraints.
• This approach has resulted in a cleaner, safer codebase that has increased overall development velocity.
• Engineers who previously wrote few tests have begun consistently writing them after utilizing LLMs for test generation.

Broader implications for software development: The success of LLM-powered test generation at Assembled suggests potential industry-wide implications for software development practices.

• This approach could become a standard tool in software engineering, potentially leading to higher-quality codebases across the industry.
• As LLM technology continues to advance, we may see even more sophisticated applications in software development, further enhancing productivity and code quality.