CIOApplications
static-image
  • Home
  • Applications
      • Auction Software
      • Audit Management
      • Bioinformatics
      • Business Continuity
      • Business Intelligence
      • Chatbot
      • Club Management
      • Collaboration
      • CPQ
      • Custom Software Development
      • CRM
      • Data Platform
      • Digital Experience
      • e-Discovery
      • E-Signature
      • Employee Engagement
      • EAM
      • Enterprise Communications
      • Enterprise Mobility
      • Enterprise Reporting Software
      • Enterprise Search
      • Forestry
      • GIS
      • GRC
      • Human Resource
      • Indoor Positioning
      • Innovation Management
      • InsurTech
      • Intellectual Property
      • IT Infrastructure
      • IT Operations Management
      • IT Service Management
      • License Management
      • Low Code Platform
      • Maintenance Management
      • Managed IT Services
      • MarTech
      • Master Data Management
      • Order Management
      • Parking Management
      • Personalization
      • Portal Software
      • Procurement Tech
      • Publishing Software
      • Quality Management System
      • Remote Monitoring
      • Remote Support
      • RFID
      • Sales Tech
      • Software Asset Management
      • Software Testing
      • Task Management
      • Unified Communications
      • Visitor Management
      • Voice Recognition
      • Web Design and Development
      • Workflow
      • Workplace Management
  • Verticals
      • Automotive
      • Banking
      • BioTechnology
      • Casino
      • Construction
      • CPG
      • Contact Center
      • Education
      • Energy
      • Field Service
      • Fintech
      • Food and Beverages
      • Government
      • Healthcare
      • Legal
      • Life Sciences
      • Logistics
      • Manufacturing
      • Media and Entertainment
      • Retail
      • Travel and Hospitality
      • Utilities
  • Technologies
      • 3D Scanning
      • Agile
      • API
      • Artificial Intelligence
      • Augmented Reality
      • Blockchain
      • CAD/CAM
      • Cloud
      • Container Management
      • Cyber Security
      • Data Analytics
      • Data Center
      • DevOps
      • Drone
      • Graphics
      • Hyper Converged Infrastructure
      • Image Recognition
      • IoT
      • Lidar
      • Machine Learning
      • Machine Vision and Imaging
      • Nano Tech
      • Networking
      • Predictive Analytics
      • PropTech
      • RF and Microwave
      • Robotic Process Automation
      • Robotics
      • Security
      • Simulation
      • Smart Labelling
      • Space Tech
      • Telematics
      • Virtual Assistant
  • Company Eco System
      • Adobe
      • Amazon
      • Avaya
      • ESRI
      • Google
      • IBM
      • Infor
      • Microsoft
      • Mitel Partners
      • National Instruments
      • NetSuite
      • Oracle
      • Progress
      • Qlik Partner
      • Riverbed
      • Salesforce
      • SAP
      • ServiceNow
      • SiteCore
      • Splunk Partner
      • VMware
  • News
  • conferences
  • Newsletter
  • Whitepaper
  • About us
  • Specials

  • Menu
      • Amazon
      • Artificial Intelligence
      • Auction Software
      • Blockchain
      • CAD/CAM
      • Club Management
      • CPQ
      • Digital Experience
      • E-Signature
      • Education
      • Energy
      • Enterprise Mobility
      • Food and Beverages
      • GRC
      • Healthcare
      • Image Recognition
      • Indoor Positioning
      • Infor
      • Intellectual Property
      • IoT
      • Legal
      • Lidar
      • Logistics
      • NetSuite
      • Personalization
      • Predictive Analytics
      • Quality Management System
      • Remote Monitoring
      • Retail
      • RF and Microwave
      • Riverbed
      • Simulation
      • Task Management
      • Utilities
      • VMware
      • Workflow
  • VMware
  • Amazon
  • Artificial Intelligence
  • Auction Software
  • Blockchain
  • CAD/CAM
  • Club Management
  • CPQ
Specials
  • Specials

  • Amazon
  • Artificial Intelligence
  • Auction Software
  • Blockchain
  • CAD/CAM
  • Club Management
  • CPQ
  • Digital Experience
  • E-Signature
  • Education
  • Energy
  • Enterprise Mobility
  • Food and Beverages
  • GRC
  • Healthcare
  • Image Recognition
  • Indoor Positioning
  • Infor
  • Intellectual Property
  • IoT
  • Legal
  • Lidar
  • Logistics
  • NetSuite
  • Personalization
  • Predictive Analytics
  • Quality Management System
  • Remote Monitoring
  • Retail
  • RF and Microwave
  • Riverbed
  • Simulation
  • Task Management
  • Utilities
  • VMware
  • Workflow
×
#

CIO Applications Weekly Brief

Be first to read the latest tech news, Industry Leader's Insights, and CIO interviews of medium and large enterprises exclusively from CIO Applications

Subscribe

loading
  • Home
  • Testing
Editor's Pick (1 - 4)
left
It is Probably your Communication

It is Probably your Communication
Greg Smiley, CIO, Florida Department of Transportation

Automation Helps IT be a Valuable Business Partner

Automation Helps IT be a Valuable Business Partner
Sandy Figurski, CIO, Horace Mann

Transforming Software Testing Landscape

Transforming Software Testing Landscape
Sachin Mulik, VP & Chief Solutions Architect, Amdocs Testing Services

Top Reasons to Adopt Five Automated Testing

Top Reasons to Adopt Five Automated Testing
Mick Whittemore, VP of IT Enterprise Operations, Paychex [NASDAQ:PAYX]

Extending Automated Testing

Extending Automated Testing
Steve Berczuk, Software Developer and Team Lead, Fitbit

To Shift Left or to Shift Right? Continuous Testing is Everyone's Responsibility

To Shift Left or to Shift Right? Continuous Testing is Everyone's...
Jeff Scheaffer, General Manager, Continuous Delivery, CA Technologies

Transforming Testing: A Recipe for Quality Engineering

Transforming Testing: A Recipe for Quality Engineering
Tariq King, Ph.D., Director of Quality Engineering, Ultimate Software

right

Thank you for Subscribing to CIO Applications Weekly Brief

Has Your Test Automation Become A Big Ball Of Mess?

By Mary Thorn, Director, Ipreo

Tweet
content-image

Mary Thorn, Director, Ipreo

Over the past 10 years automated testing and continuous integration have become the way of deploying faster in an agile software development world. The problem is, what team members automate is all over the place. There is no maintained strategy to automating the right thing at the right level and automation suites are becoming massive, costly to maintain, and not-valuable and they don’t find the defects that the customers seem to be finding.

There are several reasons why this happens in my experience. It usually starts when the automation framework is created and the first tests are written. The easiest tests to automate usually are at the user interface therefore usually slower and brittle. The testers who write these tests are also typically inexperienced in writing the tests therefore the tests end up inefficient and now the foundation of your framework is not solid or scalable. Some automation strategies believe when you start automating to just let the team start writing tests to get used to how to write them. However, in my opinion you need to have a strategy and doggedly pursue it. Mike Cohn’s automation pyramid strategy is my prefered strategy. At the base of the pyramid you have unit automated tests where the majority of your tests should be written. In the middle of the pyramid you have your API/services tests that have quite a few but not as many as unit. Lastly you have UI tests which only make up about 10 percent of your automation.

While implementing this strategy, first, train the whole team on the automation framework, so the testers and developers know how to write tests in the framework. Secondly, decide upon basic coding standards. Just like code for the application should be treated at production level, your test automation code should be the same. The same type of coding standards should be defined so that when you do code reviews you have standards to review against. Lastly, discuss at what level you should write the automation test, at the UI, API or Unit test level.

Maintaining the test automation strategy has been one of the biggest challenges to date in agile software development


For every sprint there should be a conversation that confirms which tests will be written at what level that the whole team agrees to and the tests are not duplicated in the framework.

Another reason the test automation becomes a big ball of mess is because teams never get into the “Stop the Line” mindset when a test fails. Without this culture, more tests get put into an “ignored” state and or are just plain ignored. This accrues debt as team members have to spend time analyzing these failed tests instead of fixing tests or fixing code. I have seen instances where a test automation run takes two weeks to analyze all the failures. In that case just throw out the tests and start over. If your automation is in this state today you can first move all your “unstable” tests out of your regression run and put them in their own automation run. Then negotiate with the business time to re-factor those tests, if you find that they still have value. The goal at the end of this is that once you put focus to get these tests “passing” the team moves to a culture that if the test fails you stop everything and fix it. Second option is to throw them out completely, which from my experience is just an equal option.

Lastly and the main reason tests become inefficient is that we don’t spend time in refactoring them. Tests like code can become brittle, obsolete, slow, and not valuable. You should put effort in quarterly to review your slowest test steps, slowest tests cases, and slowest test suites. Spend time refactoring them to speed them up or delete the tests, yes, I just said delete the tests. If a test has never failed in five years of running then maybe it is not a good test. Spend time reviewing the tests that have never failed, make sure that they are good tests, or maybe see if this test is duplicated at a lower level of automation therefore being found before your test runs.

Maintaining the test automation strategy has been one of the biggest challenges to date in agile software development. This is mainly due to the fact that most frameworks are maintained by a few people and those people leave after a few years. They attributed their leaving was due to “maintaining” old tests versus “creating” new tests. One team member even went on to say he was on the maintenance team as the test automation was so complex and robust that the only thing to do was to maintain them and modify a few scenarios here and there when new functionality was being added.

Here is where the cyclical problem occurs and why your test automation can get out of control. When a new team member comes on-board if we don’t review the automation strategy, we don’t train them on the framework, we don’t focus on the importance of the tests passing, we don’t emphasize the importance of refactoring their tests, we end up in this big ball of mess. What’s interesting is that we usually will put that level of diligence of onboarding for our production code. Just imagine if we treated our test code to the same level of craftsmanship as our production code how we might never have gotten in this mess in the first place?

See Also:

Top software Testing solution Companies 
Top Software Testing Consulting Companies

Read Also

Extending Automated Testing

Extending Automated Testing

Steve Berczuk, Software Developer and Team Lead, Fitbit
To Shift Left or to Shift Right? Continuous Testing is Everyone's Responsibility

To Shift Left or to Shift Right? Continuous Testing is Everyone's Responsibility

Jeff Scheaffer, General Manager, Continuous Delivery, CA Technologies
Transforming Testing: A Recipe for Quality Engineering

Transforming Testing: A Recipe for Quality Engineering

Tariq King, Ph.D., Director of Quality Engineering, Ultimate Software

Weekly Brief

loading
ON THE DECK

Web Design and Development 2019

Top Vendors

Enterprise Asset Management 2019

Top Vendors

Digital Experience 2019

Top Vendors

Adobe 2019

Top Vendors

Previous Next

Copyright © 2019 CIOApplications. All rights reserved. Registration on or use of this site constitutes acceptance of our Terms of Use and Privacy Policy |  Sitemap  |  Subscribe

follow on linkedin follow on twitter follow on rss
This content is copyright protected close

However, if you would like to share the information in this article, you may use the link below:

https://www.cioapplications.com/cxoinsights/has-your-test-automation-become-a-big-ball-of-mess-nid-1556.html