By Jim Azar, Orasi Senior VP and Chief Technology Officer
Courtesy of App Developer Magazine
As most organizational leaders recognize at some level, the increasing dominance of web and mobile applications has completely turned the software world on its ear. The number of critical business functions that are processed via a browser or mobile device is escalating, and inaccurate results, aberrant behaviors, and security flaws can all be absurdly costly.
Virtually no company expects to produce perfect software. Management decides that a certain level of defects is acceptable, and teams strive to get to or below that level. Despite this fact, many organizations experience detrimental events that make them wish they had focused more on quality. By this, I mean producing software to meet the targeted quality level, on time and on budget, with no critical defects escaping into production. Project managers and the managers above them would be delighted if critical defects were always found sooner, rather than later.
Yet, critical flaws continue to plague companies large and small. With so much at stake, how is this possible? The reality is that current practices – even “modern” ones – often prevent firms from discovering defects as early as they might, increasing the challenge of producing high-quality software. Fortunately, there are practical solutions, one of which can be implemented very easily.
Flawed by Design
Most firms currently developing web and mobile apps have incorporated some elements of agile development and/or continuous delivery, and many have adopted them completely. With the speed of development and the number of app updates that take place, older, more cautious approaches such as waterfall have fallen out of favor. This is understandable, but in the race for speed, regression testing is being given short shrift, and quality suffers.
While it’s true that agile and other modern methodologies can help teams catch code defects more quickly, the accelerated pace of update cycles often means important business processes go untested until late in the process. For “minor” updates, the system in its entirety may not be tested at all. To meet release deadlines, companies adopt an attitude of “deploy and fix,” leading to faster, more frequent updates that should be tested just as quickly and often.
Let’s consider the scenario for a critical defect that escapes detection in a release update. How does this happen? In a common scenario for a web app – let’s assume an updated web ordering system – several developers write minimum viable product code for various functions—address fields, zip code fields, and coupon fields, for example. Each then writes and runs a unit test and checks in the code.
In many situations, the quality assurance (QA) team won’t enter the picture to regression test functionality until all these bits of code have been integrated. Only then would they discover that one didn’t function within the system as intended.
When this happens, presumably the project manager would assign the defect to one of the team members, who would then debug code until he finds the problem. The release date might slip, and the company would lose money (and possibly customers).
An equally likely and potentially more dangerous outcome is that management might proceed with the launch without debugging code, planning to fix the defect in production and hoping only a few users would be affected. In the most potentially disastrous scenario, regression testing might have been skipped altogether, enabling a major functional defect to escape into production.
Finding a Better Solution
This scenario for an updated ordering system is just one of many examples where a “modern” approach can have a negative impact. I am not suggesting that agile and continuous delivery are bad methods. They are practical solutions to the real problem of building complex software solutions.
Nevertheless, the reality is that goal achievement with these methodologies is often in direct conflict with quality assurance. In most agile methodologies, testing is embedded within the development cycles. The agile team is focused on delivering bursts of tested features and functions that they have committed to provide on schedule. Their attention is focused on meeting their short-term commitments.
However, even though the new functionality is tested during the dev cycle, the entire system is often not tested for each deployment. Dev/test/deploy cycles are happening too fast to support the use of traditional testing methods. Manual testing can’t keep up, and fast, frequent, and effective automation becomes the key to achieving quality across the entire system. Yet, many organizations are so deeply focused on meeting the timelines of dev/test/deploy cycles that they can’t allocate the resources (or hire the experts) to develop and deploy automation successfully.
For web and mobile applications in particular, the optimal in-house solution is not only to implement automation but also to embrace newer techniques that ensure no code enters production until it passes QA.
Continuous deployment is one approach that achieves this goal, as is DevOps, sometimes referred to as “shift left,” whereby QA and its user-based tests are integrated into the development process. In doing so, system-level problems are more likely to be discovered before they result in serious problems.
However, such changes require realignment of not only processes but also mindsets. As with test automation, most organizations experience false starts and missteps as they attempt a complete overhaul of deeply entrenched methodologies. Until they expand their mindsets—and their resource pools, either in-house or out—teams can’t avoid the missteps that prevent them from realizing high-quality results at an accelerated pace.
A Viable Alternative: Testing as a Service (TaaS)
An emerging alternative that is gaining tremendous ground is cloud-based TaaS. Whether as an interim approach until an organization can modernize its methodologies, or as a permanent solution due to its efficiency, even very large organizations with significant resources are embracing TaaS. In doing so, they eliminate resource fluctuations, speed release cycles, and ensure software quality.
Organizations for which TaaS adoption is most imperative are those for which the pace of technology, paired with the need for continual app and system updates, is preventing in-house teams from staying abreast of testing advances. Yet, management has recognized that system glitches and downtime are too costly to bear.
Orasi recently helped one of its clients adopt TaaS to address significant infrastructure and resource concerns, all of which are compelling arguments for using this approach. This multinational organization runs a customized product ordering system on its public-facing website. Daily updates require some 2,300 tests to be run, per night, on the platforms and devices that customers use on the open Internet. Considerations that favored TaaS adoption included:
- The firm had no existing lab or processes to perform sufficiently broad validation of the public-facing app’s behaviors across all platforms and devices.
- Management felt it was inefficient to purchase, deploy, and maintain/update the necessary automation technologies and environments to keep pace with constantly changing requirements.
- In-house teams lacked the management and process skills to achieve goals, and adding/managing more highly skilled resources was cost prohibitive.
After considering all the skills and infrastructure needed, company leadership determined the most economical solution that would ensure quality and accelerate updates was TaaS.
To reap the most benefit, especially when QA is a fundamental challenge, firms considering TaaS should look for offerings that include a comprehensive palette of best practices QA and testing services, as opposed to siloed assistance. Services should include not only setting up and maintaining a dynamically scalable test environment but also creating and updating scripts, running manual and automated tests, and performing other functions common to fully developed in-house testing and QA efforts.
Final Thoughts
By all estimates, the pace of application updates is only going to accelerate, making it even harder for in-house teams to stay abreast of QA and testing. At the same time, QA is invaluable to achieving app success, fostering user loyalty, and reducing the potential for crippling business risk.
Not only are customers, brand reputations, and sales volumes in jeopardy, but we have passed the point where code defects are innocuous. Most cybercriminals now use sophisticated technology to troll the Internet for vulnerabilities. The aberrant system behavior that results from software defects is like a beacon in the night. With so much at stake, and with proven solutions available, the only logical decision is to recognize when help is needed, and seek it.