Manual vs. Automation Testing: Which Approach Works Best?

Bugs and performance issues cost software owners’ reputation, users, and profit. So to test or not to test isn't a question at all. It's rather 'how to choose the right approach to testing'?

In this article, we’re comparing two approaches – manual vs. automation testing. Uncovering their benefits, drawbacks, and figuring out when to go for automation testing and when manual checks are just enough.

And, finally, deciding on how cost-effective this is – to write code to check code.

But first, let's find out what manual and automation testing approaches stand for. Starting from the manual testing.

Manual tests are done by hand: quality assurance team checks how the software works by following conditions written in test cases.

If you’re picking this type of QA testing services, you’re paying for the time engineers spend on running checks. This works great on a short-time perspective: if the team needs to make a few simple checks, manual testing is a fast, accurate, and cost-effective solution.

Besides, even if there’s a need of automation testing, full automation is simply impossible. You can't skip the manual testing stage. What’s more, QA engineers have to test the software manually to make sure automated testing is possible.

What is manual software testing applicable for?

• Adhoc testing
• Exploratory testing
• Usability testing

Here’s an example of a usability check. QA engineer clearly sees that, for instance, the contrast between a button and the background in an app is too light. So it will be hard for users to see the button and figure out what it’s used for.

Automated testing wouldn't find any issues here at all.

Finally, the QA team’s feedback is closer to what you’ll hear from customers. Which is really helpful if you’re aiming at user-friendliness and better customer experience.

This far, we know that manual testing can’t be fully skipped. But if not, why we should consider QA automation? And what is software test automation all about?

Some checks are impossible or too complex to make manually. They mostly include load testing, but there also are processes like image comparison, simultaneous testing of several parts of the product from different devices, and so on. All these tasks become much easier with automated scripts.

What QA automation testing is used for?

• checking the UI, especially if lots of platforms or devices are supported
• for the main testing flows
• rarely changing cases

There's little use of creating autotest for features that constantly change. Or functionality that may be potentially updated (like business logic) as with the change of the functionality, there’s a need to change the tests.

Thus, we can name what test automation works for:

• GUI, API tests
• Unit tests
• Integration tests
• Smoke tests
• Regression testing
• Acceptance testing
• Load and performance testing

What are the limitations of automated testing? Only stable functionality, which was tested manually and approved by the product owner, is worth automating. Simply put, if you know that this part of the software won't be changed soon, go for QA automation.

As our experience shows, manual regression testing takes about 10h hours per platform, if we're talking about small-size products with simple functionality. But when it comes to large-scale ones, testing may take up to a week per one platform.

To reduce the time spent on regression testing, we apply autotests.

Today, QA engineers have testing frameworks that support almost all existing languages. Why they still do manual checks?

Cause there are lots of advantages of manual testing over automation! But the automated approach offers benefits too.

In the next sections, we’re taking a detailed look at pros and cons of both manual and automated testing.

Time to check them all out.

So, what pros come along with manual testing approach? And why it can't be fully skipped on a project of any scale?

Let’s imagine you need to check one small change in the app. Wouldn’t it be faster and cheaper to test it manually on the fly? With no expenses on automation tools and processes.

Here’s another case: developers often get feedback from customers during the development process. What if clients don’t like how a feature works and want to change it before the release?

If you had software tests written, you'd have to rewrite them. But the time you’ll spend on updating the cases and autotests can be taken to update the feature.

This benefit of manual testing is especially true for mobile app development. Most apps are built in short sprints, and developers add new features on the fast track. And manual testing allows testers to give feedback as fast as possible.

Among the advantages of manual testing is that you’re getting the exact kind of feedback you'll hear from real users. If you can predict what they will or won't like in UI design (like a computer cannot), you can always make it better.

Automated testing won't let you check whether the new app's features are easy-to-use. Only QA engineers can do that.

In manual tests, you improvise, create new, crazy combinations of actions. They may or may not occur to app's users, but leave place for accidents.

New cases show up while testers ask themselves 'what-if' questions. That's how they find new ways of interacting with the app even when basic scenarios are missing them.

When QA engineers interact with software as users would, they'll quickly notice usability issues and UI glitches. Autotests can only tell whether the UI is working as expected.

Scripts won’t pick up any visual issues nor provide feedback about how UI feels like. Or how it affects user experience.

But a qualified QA engineer can, and their comments will be the closest to the actual opinion of users.

Sounds good, but what are the cons? Does manual testing show one hundred percent accurate results? How cost-efficient is it on large-scale projects?

We got it covered too!

The biggest manual testing con is the human factor. As manual testing fully depends on the engineers that perform it, there’s a higher chance of errors. After all, testing results can only be as good as the testers are.

Let’s face it: with manual inspection, there's always a possibility QA specialists missed out some bugs. Any QA engineer may get lost in the process and skip a potential bug for a variety of reasons – from the lack of experience to family troubles.

You can set automated tests and leave them running in the background. Manual testing doesn't work this way.

QA team needs lots of time and resources to make sure the software is bug-free and has no performance issues.

Besides, QA engineers can't reuse manual tests. Which means with every new update in the product, the team has to start all over again. And make sure new changes won’t break the build (regression testing) even if they are minor.

Some tests are hard to simulate manually, like stress testing. That's when the QA team tests an app under a heavy load to find its breaking points.

Mainly, stress testing requires hundreds or thousands of requests done in a short time frame.

For manual stress testing, you'd have to monitor a bunch of users connecting to the app at the same moment. For a single testing engineer, it's simply not possible to recreate this type of behavior manually.

When manual testing doesn't fit in, the automation approach comes to the rescue — time to check what advantages automated testing brings, and how testers benefit from it.

First, it's about finding bugs quickly. Here's a thing: the sooner you find the issue, the faster you fix it.

Among the main benefits of test automation over manual testing is that the latter allows developers to start checking code as soon as it's compiled. This way, engineers find and fix issues faster, cathing most of the issues at the Unit/API layers, long before the project moves into production.

Second, it's reducing the average cost of testing hours.

A QA engineer does manual checks for up to 8 hours per day. But a test automation specialists will spend twice less time on writing scripts. And then they leave the scripts running for the rest of the day, switching to other tasks.

Sure, it’s all individual and should be calculated according to project requirements, complexity, and so on. Besides, we should take into account the time needed for re-writing scripts and maintaining them.

With the product expansion, developers add new features on top of the existing functionality. That means more code and more work for the quality assurance team. Now they need to make sure that are no bugs in the new features that can break the software – this is the regression testing I’ve mentioned.

For large-scale projects, there are two ways out: leave some parts untested (and bear all the risks) or automate the process.

The importance of automation testing is that the QA team can launch many scripts and leave them running during off-work hours. Or while QA engineer is checking the part of the software not covered by autotests. These scripts go through the same steps as many times as needed; they don't get bored or tired.

In turn, the quality assurance team has enough time for in-depth manual testing, more challenging, and rewarding tasks.

As the project grows, so does time needed for its testing.

As mentioned, in some cases, manual regression testing and eliminating bugs take up to 5 days per platform. Benefits of automation testing are obvious here: autotests reduce the number of days to a few.

This is crucial for any startup: the latter they launch a product, the more marking opportunities will slip through fingers. So while a company spends weeks on testing the software, their competitors might have already released a similar product.

With moving to automated testing tools, you move to a fully new level of reporting.

Tools like Allure, UFT, Tricentis Tosca make full reports that show the results from all the tests processes in a single run. Any member of the QA team can check and identify the cause of an issue. There won't be any need to contact the test's creator. Which gives the quality assurance team a real-time picture of the app's stability.

For example, if testers wrote scripts to check software on Google Chrome, they might be able to adapt them to other browsers. With very little time and effort.

Pros don’t come without cons, and automation testing has a few. Let’s take a closer look at them – and find out why QA teams don't automate everything at once.

No autotest will give an unprejudiced mark on an app's UX or plan. It'll just check parts of the app against the parameters you set. To make sure your app has a decent look and feels right, you need to ask a QA engineer to check it out.

As I’ve said, autotests can't be applied everywhere – some tests can't be automated. Like app usability checks, layout verification, installation testing, and similar processes.

It may seem weird, but most bugs are found by accident or during exploratory testing. Probably because in each testing session, we check the app in different ways.

Automated testing always goes the set path, sometimes with the same sets of test data. This lowers the chance of finding new defects or bugs.

Among the drawbacks of automation testing is its cost. To calculate the exact sum, development teams must consider three cost areas:

• test writing
• environment maintenance
• test maintenance

The final one – test maintenance cost – often gets skipped. But in the long run, test automation needs lots of resources to keep its efficiency.

Autotests require tech experts to watch over them and the environments in which these test run. And test automation experts are highly sought-after and well-paid. That’s why hiring a team to keep autotest running will cost you some.

Experienced companies that provide software development services start the automation process only after calculating test automation ROI.

So, which works better – automated or manual testing? They’re both great.

You can't pick manual or automation testing and turn down the other option. Automating all tests is unprofitable. AIOps have not reached the desired level yet, so manual testing will stay in demand.

The best idea for project managers is to find the exact balance between these two approaches. Figure out when there are more advantages of automation testing over manual testing (or vice versa) and implement the best-fitting testing approach.