What is Automation Testing?
May 19, 2021
Automation testing is a software testing methodology to automatically test and compare the actual outcomes against predicted outcomes in software development. This can be achieved with automation testing tools or by writing test scripts. Test automation can automate many repetitive tasks in the test case suite or alleviate those tasks too difficult to perform manually. “Besides being critical to Continuous Delivery (CD) and Continuous Testing (CT), automation testing saves tons of manual testing hours, ultimately becoming a cost-effective regression testing method,” explains Maxim Ivanov, CEO & Co-Founder of Aimprosoft.
Seena Ahuja, Director of QA at Fueled explains, “Automation testing helps save the manual effort here by running the test suite (a collection of test scripts) periodically to ensure nothing has broken in the app thus eliminating the chance of human oversight/error.” In contrast, manual testing is necessary when testing software as 100% of each test case scenario cannot be automated. However, the goal is to aim for automating as much of the test case suite as possible to reduce human errors and produce superior quality software.
In this article, we will define the discipline thoroughly, discuss the various types of testing, point out common misconceptions, expand on the vast benefits, and share commentary from experts in the testing industry.
What are the types of automation testing?
Many practitioners are very familiar with the top four listed. However, there are more types of test automation principles in the software testing life cycle. We’ve detailed each below…
Unit testing is a method of testing smaller isolated portions (or units) of code. Unit tests are usually conducted with test automation scripts on the smallest testable portion of the software. Unit tests cannot have any outside dependencies on other systems, databases, or APIs. They should also be performed the most frequently in comparison to other testing techniques. Typically, at each code commit. This way, developers can quickly and easily catch any bugs before they are merged into the codebase.
Integration testing, sometimes abbreviated I&T (for Integration and Testing), is a software testing practice in which individual software modules are combined and then tested as a group. This stage of group testing typically occurs after unit testing. The main reason why this is done is that an individual software developer’s understanding and programming logic of the module may be different than another developer’s. Integration testing verifies the collective modules by testing them in unity.
APIs enable communication and data exchange between systems. API testing is a technique to test application programming interfaces (APIs.) It is a portion of integration testing and verifies the efficacy of functionality, reliability, performance, and security. APIs do not have a user interface and API testing happens at the business layer, between the presentation and the database layers.
UI testing is the methodology of testing the user interface presentation layer of software. UI testing is sometimes referred to as graphical user interface (GUI) testing. It tests that visual elements accurately meet functionality and performance specifications. These are typically performed by writing test cases.
Smoke testing, also has several nicknames such as confidence testing, build verification test (BVT), or build acceptance testing, which determines whether a software build is stable or not. If the latter, development is sent back for correction until it can pass to proceed through further testing stacks.
Acceptance testing, sometimes referred to as user acceptance testing (UAT,) is a testing technique of verifying that software works for the user. This is regardless of whether the software has met the required specifications. This testing methodology is performed after various systems tests aforementioned and before moving the software into production.
Performance testing, also known as stress testing, evaluates the responsiveness, stability, reliability, and resource usage of software under heavy workloads.
Accessibility testing evaluates software that is usable by people with disabilities such as those with visual and hearing impairment, color blindness, older aged individuals, and other disadvantaged groups. Accessibility testing is a subset of usability testing.
SheriByrne-Haber is an Accessibility Architect at VMware. She offers an interesting point of expertise as many software developers overlook this group. She says, “One of the biggest issues in accessibility testing is the inability to automate the majority of tests. This is because, in automation testing, you are measuring how the assistive technology that someone with a disability uses to interact with the software. About one-third of the WCAG 2.1 Level AA guidelines can be done through automation, the rest is manual. But many organizations top after the automation testing is clean, and they miss two-thirds of the issues.”
This supports the notion that test automation is great, but cannot be used for every test scenario. Manual testing is a necessity that can co-exist with automation testing. There are some tools, however, such as Autify’s AI which help reduce the need for so much reliance on manual testing.
What are the benefits of automation testing?
Experts in the field offer their commentary on the importance of test automation. They include time savings, execution at scale, test coverage, reducing human errors, and avoiding redundancy.
Vlad Anhelov is a Quality Assurance Engineer at HelpCrunch and he explains, “The main purpose of automation testing is to save time and resources. Let’s say you have a checklist consisting of 150 points. A QA engineer will spend 5 hours running it from top to bottom. Automation testing may take about 10 minutes.”
Sebastian Schaeffer, CTO & co-owner of Dofollow.io says, “Automation testing is crucial because it allows you to weed out any UX and UI issues that might affect the widespread adoption of your software and avoid releasing a product that isn’t yet ready. It is also an important part of any future updates and patches once you have released software or application because it shows you how these releases will play out once incorporated.”
Cindy Francês is the QA Manager at ArcTouch further elaborates, “When appropriate, we use test automation to help execute more test cases at scale and improve test coverage, but we always also include manual testing.”
As mentioned, manual testing will never be non-existent. However, minimizing reliance on the method and opting for more automation can have cost-effective benefits for software companies. Seena Ahuja of Fueled describes, “It would also make sense to invest the time and money on automation for projects that are long-running to avoid the redundancy of manual efforts.” Alex Dorokhin, CEO and Founder of HuskyJam explains, “It is worth remembering that even 100 percent coverage does not guarantee the absence of errors. Tests help save time, minimize the number of errors but do not guarantee complete lack [of errors].”
Common misconceptions about test automation
Now that we are clear on the definition, various testing types, and benefits of automation testing- we want to define some common misconceptions about the methodology.
Automation eliminates the need for manual testing
Many DevOps teams rely on automation to speed up testing as well as reduce human errors, save man-hours, and reduce redundancy. At many of these shops, automation accounts for the bulk of testing. Aditya Ramachandran, Quality Assurance Head at Testhouse supports the statement saying, “About 60% of our test cases are automated and the rest are manual. However, it varies to projects. For applications under development, the test cases are primarily manual and for matured products, the test cases are primarily automated.” Anton Malyy, Co-founder and CTO of TRIARE shares with us, “Roughly it is about 30% manual testing versus 70% automated testing.” Cindy Francês of ArcTouch offers, “Even though people tend to think automation is the solution for everything, it’s not always the case. Some app projects are well suited for automation, others aren’t.”
Automation is too expensive
It is true, the initial investment in test automation can be expensive. However, according to a survey, DevOps teams that make the investment improve product quality and increase time to market. In fact, most respondents indicate they invest between ten and forty-nine percent of their overall QA budget to test automation-related expenditures. Larger companies with bigger QA teams invest up to 75% of their budget into automation. It reveals that those companies investing less than ten perfect are far behind their peers.
Record-and-playback tools are not comprehensive enough
A few years ago, record-and-playback tools probably paled in comparison to tools like Selenium. Fast-forward to the present day and you will find that no code record-and-playback automation testing tools have advanced and disrupted the industry. These tools help the industry by removing the barrier for entry that only highly skilled test automation engineers can do, but now anyone can automate testing. They reduce test case authoring as there is no “writing” of test cases- just point and click. And maintenance is a breeze as some tools use artificial intelligence and machine learning to point out test anomalies.
Aditya Ramachandran of Testhouse explains, “For us, the adoption of ‘no code’ automated testing was critical, especially on projects where Agile and DevOps methodologies were implemented. We found tests a lot more accurate and could extend the scope of automation.” Ramachandran elaborates, “We had more projects during the pandemic, as it triggered the digital transformation like never before. The need for automated testing was quite high. We see that automated testing will be used for everything – performance, load, usability, security, etc. The drive to continuous integration and continuous deployment (CI/CD) will put pressure on automated testing. There will be [an] increase in adoption of AI/ML tools as well. DevOps will be a unifier and enabler.”
How do I start automation testing?
Maxim Ivanov of Aimprosoft states, “In the age of Agile [software development life cycle] SDLC, testing goes hand in hand with development almost from the beginning. Since the primitive stages of development, you should clearly understand when and what needs to be automated. But it is worth applying automation after the user interface is approved so that it won’t lead to scripts failing. We recommend applying automation to check newly added features, re-test them, and identify regression as a result.”
Studies indicate that regardless of company size, all organizations agree that the biggest struggle to start test automation is evaluating and choosing the right tools. Experts in the field share their best practices on getting started:
Seena Ahuja, a Director of QA, explains how to get started, “You could start automation testing by first picking a tool that you’d like to use for automation (Selenium, Ranorex, etc) and also pick a language you’d like to use for coding (Java, Python, C#, Ruby, etc).”
Igor Zaporozhets, CTO & Co-Founder at Aimprosoft explains the various tools his team uses citing, “Selenium WebDriver, Spring, Selenide, Serenity BDD, Cucumber, Appium, JUnit, TestNG, Selenium Grid, Katalon Studio, Protractor, Robot Framework, and many others.”
Are there easier tools to use?
All of the tools aforementioned are great and serve their respective purposes. However, studies reveal most teams are not happy with their tooling. In fact, half of DevOps teams are not satisfied with their testing tools, and 71% search for new tools several times per year.
Hence the reason why no code tools are disrupting the industry. For those shops pondering no code software integrations, we’ve compiled a helpful guide of some of the top no code testing tools.
Here’s a quick guide on how to integrate test automation. In brief:
- Starting with small tasks
- Ponder how to efficiently use test automation software in the test workflow
- Design test scenarios with data-driven test functions
- Collaborating knowledge with team members accelerated usage at scale after starting small
We offered a comprehensive explanation of what exactly automation testing is. This will help practitioners understand the various types of testing techniques and the value of each use case. We heard testimonials from a massive variety of experts in the industry ranging from QA engineers to CTOs. And many agree test automation is not an end all be all, but an absolutely necessary tactic that should make up the bulk of QA test expenditures. Furthermore, we tackled some common misconceptions including how record-and-playback no code tools are disrupting the testing industry. Finally, our experts offered advice on how to get started with popular tools.