Enterprise Bug Busting: From Testing through CI/CD to Deliver Business Results

Enterprise Bug Busting

From Testing through CI/CD to Deliver Business Results

©2021, Rosalind Radcliffe

All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the publisher except for the use of brief quotations in a book review.

Published by Accelerated Strategies Press

acceleratedstrategiespress.com

ISBN: 978-1-09838-149-3

ISBN eBook: 978-1-09838-150-9

There is no longer any question whether modern DevOps practices can be used to accelerate and improve quality for legacy systems. Rosalind describes DevOps testing concepts in language that makes sense for large enterprise systems and describes large system challenges in a way that makes sense for DevOps testing. If you were looking for a primer of what you need to know and understand before you apply continuous testing in a complex system, you found it. There are two modes for using this book: 1) With a highlighter in hand to mark the ideas you want to apply as you bring enterprise DevOps testing to life; and 2) Hand this book to a peer or leader in your organization and tell them to read it so that you can talk about what to do next. I recommend both modes.

Mark Moncelle

Enterprise DevOps Practitioner

Enterprise Bug Busting hits home on the all important topic of testing in the enterprise. It defines a roadmap for implementing a continuous integration practice in your organization, and Rosalind Radcliffe, drawing on a lifetime of DevOps experience, demonstrates how application development leaders can improve software quality while building hybrid cloud applications that are delivered continuously.

Sherri Hanna

Senior Offering Manager, DevOps for IBM Z Hybrid Cloud

Mainframes are still a critical part of IT infrastructures, and Rosalind Radcliffe is an expert in bringing modern development technologies to trans- form these systems beyond their traditional role as systems of record.

Charlene O’Hanlon

Chief Content Officer, MediaOps

In my years in the DevOps community, Rosalind Radcliffe has been a shining, true North Star pointing the way for enterprises to adopt the latest technologies, while leveraging their existing investments to do more, better and faster. This book is a continuation of this common sense, best practices approach to success.

Alan Shimel

CEO and Founder, MediaOps

Software quality is an immutable theme throughout Rosalind’s remarkable career. She artfully imparts her masters-level knowledge. Enterprise software teams will immediately benefit from this book and alter course towards improved software quality in this age of digital transformation.

Mitch Ashley

CEO and Managing Analyst, Accelerated Strategies Group

This book showcases Rosalind’s unique ability at providing common sense transformational guidance to enterprises with mainframe and complex environments.

Jayne Groll

CEO, DevOps Institute

To my husband Bob who has always been there to support me, and for my children so they can understand more of what I do.

Foreword

Do you work in a large organization that has a variety of applications including mainframe systems running your core business? Do you struggle to deliver improvements with the speed the business requires and the quality the customer expects? When working with the mainframe teams, are you confused as to why everything tends to work differently than the rest of the applications? If so, then this book is for you, and Rosalind is the best guide to help you navigate all of these challenges and understand how to create quality systems for complex environments.

I met Rosalind when she presented at the 2015 DevOps Enterprise Summit. It was in the early days at the conference when most of what was being presented was about improvements on small applications that were not core to the business. Everyone was recommending small-isolated teams working with front-end code. When Rosalind stepped onto the stage to talk about mainframes I thought, Well, this is going to be interesting.

She started clearly describing the differences between the stability provided by the redundancy of distributed systems vs. the stability provided by mainframes that support the core business of most large enterprises. Rosalind stepped through each and every DevOps practice being discussed at the conference and showed how each and every one of the practices can be implemented on mainframes. Rosalind closed by stating that there isn’t anything fundamentally wrong with mainframes, and in fact they run most of the core businesses in the industry. We just shouldn’t be developing on them the same way we did 30 years ago. It was one of the best presentations I have ever seen, and I decided I had to meet her.

As I have gotten to know her better over the years, I have learned she has the unique ability and skill to speak strategically with executives and technically with senior engineers. She can go into an organization and help the executives understand the changes required. When the senior technical leads that have been doing the same thing the same way for 30 years say it won’t work, she can dive into the technical details to address their concerns and help them understand how it actually can and should work. Rosalind is one of the most impressive people I have met in the industry.

She has a passion for quality and 30 years of experience driving improvements both inside IBM and with its customers. Through this experience she has realized quality is not something that can be delegated to the quality assurance organization. It needs to be everyone’s job. It is not just a function of testing. It involves everything from the idea for a new capability to ensuring it meets the needs of the business and the expectations of the customer. An effective quality system involves all the steps in between.

Rosalind starts with this broader view of quality and shows how to provide an end-to-end system. She demonstrates her knowledge of how to apply these concepts to a broad range of applications. More importantly she shows how to apply these concepts to mainframes, which she is uniquely qualified to do. She helps people understand how mainframes are different from the other applications.

She deep dives on how to create effective quality systems for the mainframe with customer examples and key learnings. Rosalind also covers the latest capabilities of the IBM Z that can support these more modern development approaches that are valuable but not broadly used. She shows how and why organizations that haven’t modified their development approaches as much as they should have over the last 30 years should be leveraging these new capabilities.

If you work in a large enterprise with lots of different applications, including the mainframe, this is a must read. It will help you understand how to create a truly effective quality system and explain how and why the mainframe is different. It will also help you understand the more modern approaches to development that your mainframe teams need to start adopting.

Gary Gruver

President of Gruver Consulting

Table of Contents

Who Should Read This Book

Introduction

Section 1: Introduction

What Software Quality Is

Metrics

Story

Types of Metrics

Importance of Testing

What Is Testing and Who Does It?

What Is Continuous Integration?

Key Parts of Continuous Integration

Single source code manager (SCM)

Fully automated build process

Build runs on a build machine

Fix broken builds right away

Keep the build fast

Make the builds and tests visible

Make sure the output is published to easily get it

Why Continuous Integration

How Does Continuous Integration Support Building Quality Apps?

Key Insight: Continuous Integration

Continuous Delivery, Continuous Deployment

Key insight: Automated Deployment

IBM Z: A High-Level View

IBM Z overview:

Definitions of key terms:

Traditional z/OS development:

Background for Enterprises

How Organizational Structure Impacts Software Quality

Story

Types of Customers

Types of Application Architectures

Summary

Section 2:Essential Components

Pipeline

Key insights: Pipeline

Individual Development Environment (IDE) Code section

Source Code Manager (SCM)

Build

Key insights: Static Analysis

Artifact Repository

Provision and Deploy and Test

Story: Deployment errors

Key insight: Manual Changes

Monitoring

Planning

Analyze

Provisioning vs. Deployment

Key insights: Application Infrastructure

Environments for Testing

Isolated and Shared Environments

Examples of shared environments:

Story

Key Insights: Isolated test environments

Provisioning Environments

Story

Key insight: Shared test environments

Containers

Test Data Management

Key insights: Test data creation

Key insights: Test data

Stable Quality Signal

Story: Unreliable Tests

Key Insight: Stable quality signal

Get Clean, Stay Clean

Summary

Section 3: Types of Testing

Scope of Testing

Unit Testing

Key Insight: Automated Unit testing

Beyond Unit, But Still in the Build

Component Testing

Applications

End-to-End Testing

Key Insight: End-to-End Testing

Types of Testing

Functional Verification Testing

API Testing

Key Insight: API Testing

Sniff Testing, Smoke Testing

Story: The 99 pencil test

Integration/System Testing

Function testing

Integration testing/system testing

User Testing

User experience testing

Key Insight: User Experience Testing

Freeform testing

Malicious testing

User acceptance testing

Story

Regression Testing

Performance/Scalability Testing

Key Insights: Performance Testing

Infrastructure Testing

Chaos or Fault Injection Testing

Additional mainframe background:

Production Verification Test

A/B or Market Testing

Story

TDD/BDD Testing

Frameworks for testing

Summary

Section 4: Putting It All Together

Why include IBM Z in this change?

Story

Key Insight: Cultural Change

Cultural Change

Bringing the Parts Together

Story: Bringing it all together

Getting Started

Story: When it does not work

Key Insight: Willingness to change

Story

Conclusion

Story

Acknowledgments

Who Should

Read This Book

This book is targeted at large enterprise organizations that have systems including IBM Z (the mainframe as many may consider it), or anyone interested in learning more about software quality including IBM Z. This book is targeted to executives who are in an organization with these large complex systems, as well as engineers working to improve the overall process. This book does not assume any specific background knowledge, and therefore, includes definitions and terms so those with multiple/various backgrounds can gain an understanding of software quality for the large enterprise.

Introduction

No matter where you look today, you will find technology everywhere. This includes software, which plays a significant part in our lives in some way every day. I’ve witnessed this evolution to software everywhere over the course of my lifetime, going from a day when the only software directly affecting my life was in the telephone system, though not in the telephone attached to the wall. A time when cars were mechanical machines without software guiding, helping and entertaining, and when watches were about fine Swiss movement. Today, we can’t get away from software. Software is in hospitals, cars, it controls the electric grid, it’s used daily in our phones and our homes. Any financial transaction you make involves software as well, even if cash is used, the processing in the cash register is all software.

With software everywhere, the quality of that software and supporting hardware can affect our lives in many varying ways. If an internet search renders a ‘page not found’ message it’s a minor inconvenience. But when software fails in a car or in an airplane, it’s life threatening. The combination of software and hardware controls so many facets of our lives in ways many people don’t even recognize. The importance of this must be recognized and acknowledged by those of us in the IT industry.

As noted, there are instances where the quality of software can determine life and death. While most of the time it’s not that critical, it is important to understand that

software in any form does impact our lives. This understanding is why it is important to improve the overall quality of the systems being built.

The goal of this book is to provide insights that will help enterprises improve the overall quality of software being created today. This overall quality is achieved throughout the entire development lifecycle based on the inclusion or omission of activities, automation and organizational strategy. I will provide definitions for areas that drive software quality, key insights from years working with various organizations, and stories to explain what others have done. Some stories will be positive and some not so positive. Sharing both the good and the bad is intended to help others learn from and steer clear of the mistakes made by those before them. We all learn through experimentation, I hope to help readers avoid some trouble, and offer insights into new options to improve the overall process. I will share stories from my 30+ years of experience working with various companies, across the globe, large, small and everywhere in-between, as well as my work within IBM.

One important note is that this book is primarily focused on the existing enterprises with existing applications and code bases. Organizations that have been around for a while vs. the general startup organization who’s entire code base may be less than a few years old. Though there are lessons for everyone, the examples provided and discussions will be based on large long-term enterprises. Startup environments will have a different view as they will generally have started more with modern DevOps practices, or cloud native development practices. Common attributes of large enterprises are:

Widely diverse languages, coding styles and application architectures.

Testing and deployment practices are often well established and require a culture change.

Up or downstream systems that just work and are seen to be too expensive or risky to change.

Complexity of the system is too large for any one team to understand, develop, or manage. This requires multiple layers, which adds additional complexity when testing the entire system.

The scale, complexity and reliability requirements, drive the requirements that automation has to be designed from the start for production standards.

Return on investment needs to be considered in all aspects of change, change does not happen just because something new comes along, but it has to provide business value. An example would be the focus on automated testing for a part of the solution that does not change often.

Why is it so important to consider the overall software quality of large complex systems? Large complex systems including IBM Z, host more than 70% of the total structured data. IBM Z is used by most of the world’s top banks, insurance companies and retailers, it is heavily used in travel and transportation, not to mention the various government systems. These systems provide the business