Case Study

Transforming SDLC with AI: Diffblue Cover's impact on unit test automation

Software developer coding on a desktop computer

Witnessing how artificial intelligence is revolutionizing software development, a global healthcare and insurance leader seized the opportunity to innovate by creating an Intelligent Automation unit dedicated to evaluating AI-powered tools.

70%

faster unit testing

24%

increase in software delivery speed

One standout solution was Diffblue Cover, which uses AI to automate unit test creation for Java codebases. By implementing Diffblue Cover, the organization accelerated development, enhanced software delivery predictability, and empowered developers to focus on high-value tasks, all while maintaining robust code coverage and reliability.

Answering the question: Can AI speed up unit testing and impact ROI?

Like many other firms relying on software development, this multinational healthcare enterprise recognized that, while necessary, manually writing unit test cases is an outdated and time-consuming step in the development process. Equally important, it’s a task most software developers don’t enjoy.

The Intelligent Automation division at this enterprise believed that automating the process with AI-enabled tools would free developers to focus on more meaningful, value-added coding tasks, all while maintaining or improving overall code coverage.

They hypothesized that incorporating AI-driven unit testing into the software development lifecycle (SDLC) would significantly increase sprint velocity and/or reduce defect rates enough to justify the tool’s cost and ultimately create a predictable ROI. However, they first needed to find the right tool for the pilot.

Criteria checklist for AI-unit test vendor selection

The unit’s AI vendor selection process was thorough and focused on four key areas: privacy, security, cost, and purpose. The following criteria guided their decision on which vendor solutions to test during the pilot program.

Privacy:

Will our company retain control over the code, data, and input used with the tool?
Are we entitled to ownership of the output produced by the AI tool?
Is there a risk that the tool could expose Personally Identifiable Information (e.g., names, addresses, social security numbers, email addresses)?
Could the tool potentially reveal Protected Health Information (e.g., HIPAA-compliant medical records, patient history, health insurance details)?
Is there a possibility that the tool could make confidential or competitive information vulnerable to breaches?

Security:

Does the tool require internet access?
Will it need access to our internal systems?
Is the tool hosted on-premises, in the cloud, through hybrid hosting, colocation, VPS, or another hosting model?
Could using this tool introduce vulnerabilities to our systems?

Cost:

Is the tool’s cost justifiable based on its value?
Does the pricing structure (e.g., licensing, per user, per lines of code, per transaction) remain reasonable as usage scales?
Does the tool require additional support or consultation services that may incur extra fees or contracts?

Purpose:

Is there a clear advantage to incorporating this tool into our processes?
Does it simply accelerate the value already achieved from our processes, or does it create new value?
Will using the tool provide a competitive edge for the organization?
What type of Artificial Intelligence and Machine Learning is used (e.g., generative, deterministic reinforcement, stochastic reinforcement, supervised, unsupervised)?

Why Diffblue Cover made the cut

The organization selected Diffblue Cover because it met a majority of the critical criteria outlined above.

First, the tool’s architecture demonstrated a strong commitment to privacy and security, offering an enterprise license with an air-gapped implementation that eliminates the need for internet access.

Additionally, Diffblue Cover does not send usage or diagnostic data back unless the user opts to enable telemetry.

The team also valued Diffblue’s use of deterministic reinforcement learning, which assured them that the generated test cases would provide repeatable and predictable outcomes for their codebase.

Finally, the tool’s cost was deemed reasonable, with the team confident that the investment would be justified by the efficiency gains from automating the time-consuming task of manually writing unit test cases.

Furthermore, the potential savings in human resources were considered significant, as automating manual testing can help attract and retain skilled developers. With manual testing often leading to coder burnout and a lack of engagement, the team recognized the importance of offering developers more challenging, meaningful assignments to maintain job satisfaction and reduce turnover.

Statistical analysis for intelligent automation of unit testing

Over six months, the unit conducted quantitative and qualitative analyses using Diffblue Cover. Leveraging JIRA data and statistical analytics, the Intelligent Automation team measured historical velocity from the past year, establishing significant baselines for meaningful comparison. To ensure accurate results, they captured stability metrics to confirm that the pilot test teams would be operating in equally stable environments.

Next, they selected a team of developers with expertise in Java-based code repositories. They trained them to integrate Diffblue Cover into their workflows and ensured the dev environment was fully stabilized before launching the pilot.

The pilot development team focused on three key assumptions that AI-powered unit testing needed:

It must improve sprint velocity.
It must strengthen software resiliency.
It must stabilize software predictability.

Diffblue exceeded expectations in both qualitative and quantitative measures

Diffblue Cover’s implementation resulted in substantial gains in both development speed and software quality, validating its effectiveness in optimizing the testing process. By leveraging AI to automate unit testing, the team reduced the time spent on manual testing, ultimately enabling developers to focus on high-priority tasks and driving greater business impact.

Quantitative results: Improved sprint velocity by 24%

The team hypothesized that increasing test coverage above 70% would result in an improved sprint velocity. By measuring effort using the number of story points per sprint as their baseline, they saw velocity increase from 56 to 69 points — a 24% improvement, far exceeding the 17-20% deemed statistically significant.

The following key factors collectively contributed to a more efficient development process, leading to improved sprint velocity.

Automating the generation of unit tests saves developers time and effort.
Eliminating manual, repetitive, and error-prone testing tasks frees developers to concentrate more on feature development.
Increasing code base test coverage enables earlier identification and resolution of defects, significantly reducing time spent on defect remediation.
Leveraging reinforcement learning to deliver deterministic results ensures the creation of high-quality test cases that significantly strengthen code reliability and stability.

Qualitative results: Driving developer innovation

4-5 rating

In an era of growing AI generated software concerns, qualitative outcomes were equally important to the Intelligent Automation task force. The team wanted to determine whether AI-powered unit testing could enhance productivity for developers and testers, ultimately achieving the following goals:

Lower risk through automation
Eliminate manual effort
Shift talent and focus to feature development
Drive quality engineering from start to finish

What we heard? Diffblue Cover demonstrated its ability to improve developer productivity, deliver high-quality outputs, and streamline the development process, driving innovation across the board. In fact, on a scale of 1-5 the developers gave Diffblue an overall rating of 4.8.

From proven pilot to enterprise-wide rollout

This global healthcare and insurance leader’s implementation of Diffblue Cover marks a significant step forward in leveraging AI to transform software development processes. With Diffblue’s AI-powered unit testing solution, the organization saw a transformative impact on both qualitative and quantitative metrics, demonstrating the value of automating a traditionally manual and error-prone task.

The team achieved over 70% faster testing by automating unit testing, resulting in a 24% increase in sprint velocity — far exceeded expectations. This efficiency gain allows developers to focus on high-value tasks, enabling strategic feature development and driving innovation across the organization.

What’s next for this enterprise? A carefully planned, staged rollout of Diffblue Cover, starting with one application team. This phased approach ensures seamless integration, allows for real-time feedback, and provides measurable results that can inform broader adoption strategies.

Ready to improve quality and accelerate your software delivery lifecycle?

Book a demo with us and ask about taking Diffblue Cover for a test drive.

Book a demo