ROI in Software Testing: How QA Automation Helps to Cut Costs

Let’s face it: manual testing can’t always prove that software is free of bugs and performance issues. Especially if you’re working on a high-quality product with complex functionality.

That’s why experienced QA teams apply the automated testing approach. But how cost-effective this is – to write code to check code?

In this guide, we’re sharing our QA automation experience, showing the best use cases, and defining test automation ROI.

Let's get started.

Software test automation is an investment that pays off with the right strategy and approach.

That’s why I suggest we start with defining what types of software tests should be automated. And how our team deals with test automation challenges and best practices.

We apply autotests when we’re developing large-scale projects with a high demand for quality – like medical or financial ones.

Why? When dealing with tons of data in, for example, financial reports, a human tester can make a mistake which may result in financial loss.

But autotests cannot: scripts repeat the same steps again and again; they don't get bored, tired nor distracted.

Next, I suggest checking upon which tests we apply automation testing practices.

Here’s a quick list of types of automated tests:

• regression testing
• stress / load testing
• end-to-end testing

Let’s take a closer look at them.

Mainly, we write autotests for regression testing. What's it used for? By running these checks, we approve that software's functionality works as it should after updates or code changes.

In the second case, the faster we complete regression testing and make sure everything is working correctly, the quicker we move forward to making those changes.

How much time regression testing takes? It’s simple: the bigger is the project, the more time it requires for testing. For example, on the first sprint testing may take about two hours, but on the third – more than ten hours.

As a result, when dealing with a large-scale product, a QA team may need more than a week just for regression testing.

Finally, when our QA team does manual checks, we have time just for running one regression test per one or two development sprints.

By applying autotests, we can run regression tests as often as needed.

And as we work on continuous delivery principles, we can start running autotests each time the code gets changed.

It all depends on how fast the QA team starts getting first results – in a few hours (manual testing) or a few minutes (automated testing).

Among the best examples of automation testing are load and stress tests. Load testing shows how the software works under normal conditions. By stress testing, the QA team defines whether the software will continue running under crunch situations.

Both these tests can’t be run manually. So if we want to test how software behaves during a heavy load, we emulate a huge number of users with the help of autotests.

We work with client-server apps, and most of our clients' business requirements are implemented on the backend.

To check whether everything works properly, we run end-to-end tests that reflect how a user is interacting with the software via a client.

On large-scale project backend automation is worth it. It's cheaper (one server is working with a few clients) and faster as the automated tests themselves take less time to complete.

Finally, there’s one more thing about how automated tests help us to build bug-free software products. It’s detailed reporting.

Automated tests prepare extensive reports. By checking them, the QA team sees what doesn’t work the right way and at what stage it all began.

Allure is one of the best examples of automation testing tools our QA team uses. This framework indicates what has been tested, at what time each test case began running, and how long it took. Graphs help us to estimate test results visually.

Here’s the list of tools we use for automated testing:

• Backend automation: Python + pytest + requests
• Web UI automation: Python + pytest + Selenium web driver
• Performance / Load testing: Python + Locust or Apache Jmeter

All these tools are open-source, provided, and distributed for free. But if during a product’s testing QA engineers need to use sophisticated tools, product owners should consider investing in these tools.

In the next part, we’re going to discuss why QA automation is cost-effective and what ROI means in automation testing.

Sure there’s a ROI for test automation. But how big the ROI is going to fully depend on how test automation is adopted and executed.

A human tester runs checks only during their eight-hour workday, and then the process stops.

So the cost of manual testing completely depends on human resources. If the testing takes, let's say, a week, there would be serious delays in the whole development process, which is not cost-effective at all.

If we automate the testing process, the cost of labor will be the same. It’s the number of tests that changes – up to 5x more tests per hour.

With testing automation, we make checks faster, cover larger parts of the product, and find more bugs before the beta testing and production stages.

And it costs much less to fix bugs found earlier in the development cycle. Defects in code found after the release cost 5x more to fix than those detected during unit testing.

Generally, test automation provides value in two ways:

Business value

• Better software quality
• Avoiding operational problems
• Decrease cost of fixing bugs

Tech value

• Simplifies routine tasks
• Run more tests simultaneously
• Increase the scope of coverage

Sure, autotests take more time and skills to write. Thus, you'll need a more qualified and highly paid expert for creating them.

So it’s essential to calculate automation ROI to figure out when to use automation testing and where to go for manual checks.

Experienced companies that provide software development services begin testing automation with calculating test automation ROI and choosing the most cost-effective testing strategy.

Besides, automation itself should be based on the company’s experience and ready-made framework libraries. That helps to save their clients’ time and thus money.

Finally, I’d like to share a couple of our success stories about how automated tests helped us reduce the time needed for QA testing.

Our case is the most extensive database containing more than 130.000 gyms around the world. It helps travelers always stay fit, train wherever they are, and make workouts painless during the trips.

Due to a flexible search bar with various filters, users can find a gym by name, city, or type of exercise they would like to choose. It also deals with payment processing (the payment history is also given) and appropriately protects private user info.

This functionality is hard to test manually, so we wrote automated tests to check its performance. Automated tests had adjusted the environment – and we made sure that TrainAway’s payment system worked correctly.

And that’s not the only TrainAway test automation case. We also applied automated tests to decrease the time required for regression testing.

Another project we wrote automated tests for is ZoSmart – a networking platform. As it was a large-scale project, by applying an automated testing approach, we managed to reduce the time needed for backend regression testing significantly

That’s how we managed to check a large amount of backend logic which the platform interacts with much faster.

Wrapping up, I'd like to say that QA automation is worth it when it exactly matches the project and technology the development team is using. After all, ROI in software testing is more than just software license plus human resource costs.

And don’t forget about the types of testing and the scale of your project – automate what needs to be automated, and you’ll save plenty of time and money.