To Shift Left or to Shift Right? Continuous Testing is Everyone's Responsibility
By Jeff Scheaffer, General Manager, Continuous Delivery, CA Technologies
However, it could be argued that the industry hasn’t closed the circle yet when it comes to realizing a full continuous delivery process. The reason? Testing challenges throughout the application lifecycle.
Today’s traditional and sometimes manual, testing methods inhibit development speed and impact quality; legacy test methods just can’t keep pace with a continuous application delivery model. Testing teams, no longer have weeks to test, but days, and in some cases, only hours. Yet in most cases, their current testing processes are ad hoc, with test cases and test data being manually created. This becomes a tedious process that can introduce a lot of errors and confusion.
Despite evolution in other areas of the software lifecycle, testing techniques, tools, requirements, test cases and test data are still largely being designed and created the same way they have been for the past 30 years. Requirements are still specified through written natural language, which leads to ambiguity and poor testability. Test cases are being manually designed and built on incomplete requirements. Test automation requires a human being to manually create the automation scripts first, which then must be manually maintained sprint after sprint. A process that most would agree to be ‘extremely inefficient’.
Because testing efficiency and effectiveness are paramount in today’s DevOps world, a new comprehensive testing methodology or approach, combined with advanced technology is vital. In many ways, testing is the final frontier of DevOps and continuous delivery. To release the high-quality code that enables businesses to stay ahead of their competition faster, organizations must adopt next-generation testing practices that enable them to test early, often, automatically and continuously.
The Simple Truth: To achieve continuous delivery, you need to adopt continuous testing
There is one process that binds all application delivery stages together into one well-orchestrated machine—continuous testing. The goal of continuous testing is to shift testing left to find defects earlier in the software development lifecycle, enabling the release high quality software, faster.
To do this, testing must become automated. It has to be continuous. This means environments, systems, APIs and microservices need to be simulated, or virtualized, to give developers instant access to tests against these virtual services, day or night. This means being able to mask or better yet, instantly create test data and offer it on demand to whoever needs it. Test cases are created right from the requirements phase to ensure the right test coverage, and are able to adapt automatically to changing business requirements.
With continuous testing, test scripts are run automatically. Developers and testing teams are armed with testing tools that are easy to use, even SaaS-based that allow tests to be constantly run at any phase in the lifecycle. And it means having an orchestrated release engine that promotes code as soon as all the tests pass. This is testing unleashed. Continuous testing goes hand in hand with agile development to accelerate good, quality code into production.
In reality, continuous testing is the conveyer belt within the software factory between business ideas and production stages. Artifacts are moved from one stage to the next only after they have been tested and approved to continue. New code submitted to the repository is tested upon commit.
The Simple Truth: To achieve continuous delivery, you need to adopt continuous testing
When tests fail, the code is rejected. Subsystems are approved as part of periodic builds on their way to the delivery stage, where the system is being tested as production ready. The release process stops when tests fail.
The key to success is to shift test creation and execution to the left, rather than creating tests after development is complete. As code is committed and promoted, all tests run in the background nearly instantaneously, there is no longer time or need for human intervention in a continuous deployment cycle.
Shift left. Shift right. The testing two-step
We’ve established that delivering quality apps faster is not something to be left to chance. To deliver at high velocity, teams must collaborate in an environment where developers are integrated into testing from the beginning stages. When testing is conducted earlier, the entire team is focused on quality, and a culture of “getting code done right the first time” saves time and reduces the number of iterations teams need to perform for a particular code.
A shift-left approach in continuous testing leads to identifying performance failures in advance, finding new design alternatives that work better for the user, and glitches fixed early on.
But is testing for quality at the early stages of the application lifecycle good enough? Without also shifting to the right, does it fall short? What tested well in the lab, could still fail in production. Gathering continuous feedback from users later in the lifecycle is just as important as testing early and testing often. For instance, understanding how users actually engage with the application, knowing the user journey, and tracking performance and availability are essential for your application to be a success for the end user.
Shifting-right in continuous testing means arming application teams with SaaS-based real-user monitoring tools that observe performance data about actual live user transactions within applications. It can also include re-use of performance tests to exercise critical steps within an application at continuous or timed intervals. The goal is to continuously validate that performance patterns meet expectations. By shifting to the right, user issues are gathered, isolated and the feedback used to enhance and improve future iterations.
The Solution: continuous testing strategy built on end-to-end integrations and open source
To achieve a successful continuous testing strategy, you need integrated solutions that enable: test environment simulation; automatic test case creation, even from requirements; on demand test data management; always available SaaS-based performance testing; open source integrations with tools like JMeter, Jenkins, Selenium, Appium; and automated application performance monitoring and post-production performance testing to achieve continuous feedback.
A successful approach to continuous testing allows DevOps teams to:
• Automatically convert user stories from the planning stages into the smallest set of test cases to ensure the right test coverage and adapt automatically to changing business requirements.
• Request, reserve, and create test data at any time, even from requirements phase.
• Create simulated, production-like testing environments to provide total access to third party systems and APIs, mainframe environments, and more.
• Feed test data directly and securely to virtual services.
• Track multi-application release content through the cycle and accelerate feedback loops.
• Implement and orchestrate a release engine that promotes code as soon as all the tests pass.
• Virtualize APIs for better testing, while managing API access to developers.
• Democratize testing with SaaS-based performance, functional and load testing available early in the development cycle.
• Use open-source testing tools like JMeter, Taurus, Jenkins, Selenium, Appium, Gatling, Locust, etc.
• Track application performance and availability in production, monitoring both real and synthetic user journeys to ensure continuous feedback.
Continuous testing strategy at work
I’ve worked with engineering teams that have applied testing automation tools to address issues around regression test suite creation and maintenance. These teams transformed a process that once took 40 days to manually create test cases, and reduced the planning/requirements phase to completion in just 3 days – a 93 percent time savings improvement.
Another instrument we’ve seen used quite commonly as part of a successful continuous testing strategy are release automation and orchestration tools that automate the deployments of test environments. With these tools, DevOps teams are able to replace manual processes and reduce time to deploy by 80 percent.
Service virtualization tools are also used to configure, load and provision “clean” labs in a consistent and repeatable manner to enable software and QA engineers to test earlier and more often. Seeing teams reduce lab provisioning time from 3 hours to 17 minutes, and platform testing from 25 days to 1 day is not uncommon.
By adopting these next-generation continuous testing practices you too can have your QA teams and developers test early, often, automatically, and more importantly continuously so your organization can take an idea from design to deployment at pace—without compromising quality.