Chaos Testing and Autonomous Testing: Triumphs of Digital Quality Engineering in 2022

Following the initial transition from traditional testing to modern quality engineering, numerous enterprises now view continuous innovation and evolution as the new battleground. According to many analyst firms, this transformation around AI/ML, blockchain, cloud computing, data, and cyber security can act like growth drivers for IT, with quality engineering expenses in 2022 reckoned to exceed 100 billion USD.

As predicted, this year saw the advent of several novel technologies that can help organisations transform their IT workforces across all major verticals. However, the quality engineering domain has several major service lines that are geared to accommodate and expedite comprehensive technology transformation.

Before we dive deep into two of the most sought-after emerging technologies, let’s briefly discuss digital quality engineering.

Digital Assurance and Quality Engineering: The What and Why?

The long-held definition of quality engineering is: “the discipline of engineering concerned with the principles and practise of product and service quality assurance and control.” (Source: Wikipedia) The field of digital quality engineering extends beyond software testing. 

Stress detection through stress testing is an essential component of quality assurance. Initially, manual quality assurance was used to support the mainframe era, which was followed by the introduction of automation through the World Wide Web (WWW), Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), and so on. Businesses have evolved into an “Intelligent Digital Mesh” (IDM), and the Cloud Computing revolution has ushered in the “digital quality assurance” method.

The three components of speed, quality, and intelligence combine in digital QA to establish a unified approach to accelerated Continuous Quality Improvement (CQI). As organisations scramble to attain a digital makeover, digital quality checkpoints shine through with enhanced risk mitigation and holistic customer-centric solutions. Many aspects of digital QA, such as cognitive QA, IoT testing, cloud testing, chatbot testing, and voice bot testing, aid in digital transformation.

As companies mature in addressing this new IT environment, Digital QA will evolve to address the end-to-end automation of test ecosystems, improve the velocity of quality delivery, and enable Agile and DevOps frameworks and AI (AI). One such strategy that incorporates AI and ML (Machine Learning) into automation is autonomous testing.

Best Read: Testing a bank application: A Success Story

Autonomous Testing: The Next Big Thing in Automation

Many sectors, including software, strive for flawless user experiences. However, to deliver swiftly, organisations must reduce delivery cycles from months or weeks to hours or less. This is where automation comes in. Before automation, test cycles were frequently lengthy. Handwritten tests have a set of stages and checks that require involvement. 

When the first wave of test automation arrived, it completely transformed the test execution process. Automation allowed tests to run more quickly and constantly. However, the disadvantages of manual triggers and maintenance issues quickly became apparent. The CI/CD methods further blurred the line between developers and testers. At this moment, the automation sector appears to be at a standstill.

Challenges of Automation Testing

Some experts argue that automation testing, when faced with fast-paced new-age practices, failed to fully deliver on its promises. Some of the main challenges with automation are listed below:

Challenge to build

A high degree of skill and domain expertise is essential to creating high-quality automation scripts. These include proficiency in programming languages, mastery of the tools, and thoroughness with client requirements. Manual creation of tests remains the major bottleneck for automation.

Costly debugging and maintenance

It would not be incorrect to say that automation is a maintenance nightmare in and of itself. It incurs high costs and is time-consuming. Sometimes a minor update to the GUI would result in re-recording an entire script. The fact that debugging a script is equally troublesome adds to the difficulty.

Stability concerns 

Considering the most widely used Selenium automation tool, most scripts rely on XPath. Any change in it, and the scripts become useless. Also, the scripts don’t consider that the infrastructure is inherently unstable, so tests “fail” when they time out or crash.

Automation vs. Autonomous Testing

We read out both to test in automation, considering all forms of testing as verification of interaction. What if we could automate dictation entirely? To do this, autonomous testing makes use of AI and ML.

Autonomous testing is the automated creation and maintenance of tests and the analysis of results using an application blueprint. Using machine learning algorithms, these tools cut through redundant data in testing and spot irregularities.

Benefits of Autonomous Testing

• Early bug detection through impact and root-cause analysis lowers the risk of failure.
• Autonomy in test infrastructure facilitates faster test cycles. 
• Ability to self-heal the scripts while encountering unexpected application changes.
• Reduction in maintenance costs through automatic identification of scenarios and test data
• Better allocation of resource productivity by eliminating human dependencies and skillset requirements 
• Increased productivity with faster and improved release cycles

The transition from testing automation to fully autonomous testing is in and of itself a challenge. Most businesses’ current automation approaches can be summarized as follows:

• Assisted Automation: Using an automation tool, QA creates and executes tests, but manually manages and maintains scripts.
• Partial Automation: This tool helps create test cases and execute them, and the tester is responsible for test monitoring.
• Accelerated Automation and Unassisted Automation: The tool manages every component of the test, reducing the tester’s responsibility to assess the impact of a change.
• Autonomous Testing: The automation tool is redefined as autonomous and learns from failed tests to make the best judgments. Human-machine collaboration via seamless AI-powered systems is critical to embracing total autonomy. 

Chaos Testing: A Method for TiP 

While autonomous testing seeks to reduce test cycle times, chaos testing employs “Testing in Production” (TiP) to increase the fault tolerance of a built system.

What is TiP? 

Testing in production is based on the premise that only real-time input can successfully supplement the quality assurance process. Every version release is, in theory, a production test with high value-to-cost risk.

Principles and Steps in Chaos Testing 

Chaos testing assesses the resilience of a system to unexpected events by inducing failure. Experts explain this by stating that it is preferable to break a system intentionally at the optimal time than to have it crash unexpectedly. With the rising importance of “mean time to repair” (MTTR), it is vital to minimise the recovery time of a business.

Chaos testing is based on the following established plans and experiments

• Create a hypothesis: Here, we define the steady state, which is the system response to unstable variables. Metrics such as the total throughput, error rates, and system latency help make assumptions about its status. 
• Design and run experiments: Identify, design, and run all probable failure scenarios in a controlled manner. Doing this ensures you have a backup plan for even the worst-case scenario. if the backup strategy is unclear, the system recovery pathways are documented.
• Analyse the impact: Compare and contrast the outcomes with the initial hypothesis. The error difference in the metrics aids in deciding whether to improve or limit the blast radius and reveals additional issues. 

Benefits of Chaos Testing 

• Prepare for the unforeseen: By simulating all potential failures, you get the information necessary to fortify the system against surprises. 
• Uncover the unknowns: It illustrates the system failure reaction and facilitates the development of associated recovery routes by utilizing several experiment-specific indicators. 
• Reduce system downtime: With a good grasp of the common failures encountered by a system, we can reduce the MTTR by making continuous fault injections and regularly targeted improvements. 
• Improve customer satisfaction: Chaos testing empowers organizations to gain confidence in their systems’ resilience and increase customer satisfaction.

Chaos Test Tools 

The origin of chaos testing dates to when Netflix transitioned to the cloud via AWS. The unexpected AWS outage caused the Netflix platform to be unavailable for several hours, inducing them to implement the concept of chaos engineering. Today, many tools are available to perform Chaos testing, such as: 

• Chaos Monkey: The first open-source tool via which Netflix assembles an entire suite of fault injection tools called Simian Army. It attacks by discontinuing the VM instances. 
• Chaos Blade: Built for Alibaba, to support wide range of platforms and has dozens of attack modes.
• Chaos Mesh: This tool possesses a fully featured web UI encompassing 17 unique attacks.
• Litmus: This is a Kubernetes-native tool that utilizes Litmus probes to monitor the system’s health statistics. 
• AWS-FIS: This works only with AWS services through a fault injection simulator. 
• Gremlin: Offered as a SaaS technology, Gremlin can be integrated with CI/CD to test entire systems. 
• ChaosNative: It is a SaaS platform that hosts LitmusChaos to perform end-to-end chaos testing. 

Chaos testing is thus a proactive approach to testing, which can begin with small trials in controlled environments. It can then be effectively implemented and scaled up through proper rollback planning and appropriate tools. 

Indium’s Approach to Autonomy in Testing 

Through the uphoriX Test-Accelerator, Indium has taken a step towards autonomy by providing low code automation, intelligent scripting, Script once NFT, a native test data generator, and swift compatibility. Indium considers smart test automation as an emerging vertical that ‘addresses one of the fundamental aspects of cloud-native initiatives – the velocity of code deployment’ and a prerequisite to the ongoing digital transformation.