Agile From First Principles

PREFACE

A quick search will reveal that there are already many books on Agile, covering a wide range of topics. Many focus on Agile methods and approaches; others on specific topics or techniques. Some books expect little prior knowledge; others expect at least an appreciation or understanding of the basic concepts.

In this book we have attempted to live up to the title Agile From First Principles and cater both for those new to Agile and more experienced practitioners who would benefit from a refresher. We incorporate both theoretical and practical application of Agile, combining our experience and stories with external references and guidance.

In this preface we provide a brief overview of Agile, provide indications about who this book is for, introduce you to our case study and provide a short synopsis of what to expect of the rest of the book.

WHAT IS AGILE?

Before we start, it is worth considering what ‘Agile’ actually is. At its simplest, it is an approach to developing products, often but not always software products. It applies an iterative and incremental approach, favouring the release of versions of the product throughout the development process, seeking feedback on them and incorporating that feedback into the development process.

This means that the development cannot be planned far in advance, as we can’t second guess the feedback. Therefore, Agile teams must be able to plan for short horizons and be able to respond to change. Agile teams typically work in short cycles, a few days to a few weeks, before pausing to reflect on how they are doing and replan for the next cycle.

Agile teams use certain frameworks and practices to help them plan and deliver the product. They form teams and assign roles in a particular way, adopt high levels of collaboration and tend to operate relatively autonomously.

Already, we can see this is different from how many products are developed. We are used to projects deciding the product specification at the start and releasing the product, when it is complete, at the end.

Agile emerged in the early 2000s and initially had quite a niche following. However, over the intervening years it has grown in popularity and influence. As we write in 2022 it is extremely popular, well known and widespread – not only for software products, but for solutions and projects of all shapes, sizes and types.

IF IT’S SO POPULAR, WHY DO WE NEED THIS BOOK?

You may be wondering why, if Agile is so widespread, this book is necessary? As you will discover, despite Agile being simple to understand, in practice it is very hard to do well.

Over the years, we have witnessed many teams struggle to apply Agile concepts consistently and successfully even with experienced teams. One of the reasons for this is that many people focus on the mechanics of Agile methods and don’t give enough consideration to the underlying principles and values. The problem is that it’s easy to apply the mechanical elements in ways that don’t embody those values, and can even undermine them.

That’s why we have called this book Agile From First Principles – we believe successful Agile adoption starts with a solid understanding of the fundamentals, and only once these are mastered can you expect to apply any Agile method or framework successfully.

WHO IS THIS BOOK FOR?

The short answer is that this book is for anybody who wants to understand Agile better. That includes those who are brand new to the subject. It also includes coaches like us who value a single book that reminds us of what is important and is a handy reference to a broad range of Agile concepts. This is the sort of book we would buy and use ourselves.

More specifically, there are certain groups of people to whom we particularly recommend this book.

Those new to Agile

This book provides a comprehensive introduction to Agile with a base of knowledge that makes learning and applying any Agile approach easier and more likely to succeed. The important elements are highlighted and reinforced together with examples and stories that help bring it to life.

Leaders and managers

Even if you are not involved in the day-to-day work of Agile teams, leaders and managers have an important role to play. Bringing out the best of Agile teams requires different sorts of leadership from what you may be used to. This type of leadership will feel uncomfortable and perhaps counter-intuitive. This book will help you to understand why it is important to lead differently and help you work out how. Of course, it will also help you to understand what’s going on in Agile teams, demystify some of their language and behaviour, and help you to lead and manage them better.

Business stakeholders

Agile teams exist to provide value to customers. In order to provide the maximum value, those customers and other stakeholders need to understand how Agile teams work. Whether you are representing a direct user, or are a conduit to external customers and users, this book will help you to understand why Agile teams can be more effective and what you need to do to help them. This will usually involve some changes in behaviour and perspective. This book will help you with ideas and tools that can help you to achieve that.

Project managers

Being thrust into Agile can be confusing and bewildering for project managers, particularly those who are experienced and skilled. The Agile ceremonies, language and assumptions can be very alien when compared to best practice in project and programme management.

This book will help you to understand how Agile teams work and why they do some of the things they do. It will help you to understand how project managers can help Agile teams succeed, and why some things that project managers can expect may not be helpful for Agile teams. If you are taking on an Agile role, this book will help you to understand what is expected of you and give you the knowledge and understanding to perform it well.

Students and trainers of the BCS Foundation Certificate in Agile

The BCS Foundation Certificate in Agile is an excellent certification that gives a broad grounding in Agile concepts and practices without focusing on a particular method or practice. This book covers the whole syllabus, although it isn’t written in quite the same order.

The 2020 version of the syllabus was written by one of this book’s authors and one of its reviewers, so the perspective of the book represents what the syllabus intends to convey.

Agile practitioners and coaches

It may sound odd that a book that addresses first principles, and is suitable for novices, should also be valuable to experienced practitioners. However, there are two good reasons why this is the case and why this is exactly the sort of book we would buy for ourselves.

First, this book covers a lot of ground to explain the principles and practices of Agile. There are dozens of techniques, ideas, methods and practices described; many in brief detail with references to more detailed descriptions. As such, this one volume contains just enough information to be useful but references a huge body of knowledge that is invaluable to any Agile practitioner.

Secondly, even experts in Agile practices are not immune to falling into the traps and anti-patterns that we discuss. Having a reminder of the core principles of Agile to hand helps to keep us grounded. The discussions on the values and principles will be particularly useful to help Agile teams improve.

WHAT TO EXPECT?

This book is a mix of fact, opinion, guidance and stories. Agile emerged as a short website in 2001 with just 68 words describing its values and a further 186 covering the principles.

We help you to understand those values and principles from our experience as Agile practitioners and coaches with many years of experience. We have tested our opinions with reviewers, colleagues, peers and others. We think that understanding what’s behind Agile and what it means in practice is important. That’s why we have devoted so much space and effort to explaining what we think they mean. We have illustrated this with examples of anti-patterns and problems we have experienced and use them to show not only why it’s so easy to get Agile wrong, but ways that you can avoid making those same mistakes yourself.

We wanted reading this book to feel like a conversation with an experienced Agile coach. One who can explain new things to you, share their experiences, point you towards some good books or references, suggest new ideas and, most importantly, challenge you and make you think. You may not agree with all of our opinions, and you may speak to other Agile coaches who have different ideas. That’s fine. Just treat this as new information to add to your existing knowledge and opinions and perhaps you will formulate a slightly new and evolved opinion. As you will read, we call this inspection and adaptation, and iterative development. It can apply just as much to ideas and beliefs as it can to products or processes.

It is worth mentioning that this book is unashamedly about Agile – the word is used over 900 times. This book is definitely about Agile delivery, Agile teams and for people interested in Agile. However, we are not trying to tell you that Agile must be the approach to everything, or that it will be the best choice for every endeavour. Agile is an excellent choice for many types of project and deliveries. It is the best choice for some; one of several good choices for others; and a poor choice for some. There are many situations where a well-executed alternative approach will be better than a poorly executed Agile approach. There are many scenarios where either a traditional or Agile approach can work well, provided they are executed well. Poor application of any approach will lead to poor results, even if that choice of method is ideal for the situation.

This book isn’t about encouraging you to use Agile when it isn’t appropriate. But any time you do choose Agile, this book will help you to execute it well and maximise the value you will achieve. By understanding the principles of Agile, you will also be able to judge when Agile will be appropriate to use and when it is not.

HOW TO READ THIS BOOK

We have written the book to flow logically: it starts with the background and history of the Agile movement, describes the main concepts, then goes into more detail on some specific, important areas in the later chapters. Thus, you can start at the beginning and the book will make sense.

However, you can also jump into a chapter that interests you and it should still make sense without having read the previous chapters. For example, much of the leadership chapter stands alone, as does the chapter on managing the product. This does mean there is a little repetition, particularly in the chapter on Agile practices. This is because we feel it is useful to have a summary or introduction to a practice even when we later expand on it in more detail.

Later in this preface, we provide a summary of each chapter to help you work out where to start. The table of contents and index will also help. However, we would urge you to spend some time on the values and principles chapters. Even if you think you know the Agile Manifesto – and, perhaps, especially if you think you know the Agile Manifesto – these chapters contain a wealth of experience and discussion on how the values and principles can be applied well and, importantly, how they are commonly misapplied. We are confident even experienced Agile practitioners will find something of value.

AN INTRODUCTION TO AG STORES

Having an example always makes things easier to understand. In this book, we have tied most of the examples to a fictional product that we hope you will be able to relate to.

AG Stores is a chain of large superstores that sell a variety of products, including food, hardware and home decoration products. Having relied for years on a loyal customer base, AG Stores are experiencing increased competition from other shops and also from online competitors. They believe that introducing a loyalty programme will allow them to reward their existing customers, dissuade the customers from shopping elsewhere and provide a rich set of data that they can use to improve marketing, store layout and stock control.

To cater for some customers, they will need physical cards and telephone-based account management, but they also want an electronic solution that can be more tightly integrated to their other IT programmes, allowing real time data analytics and personalised content such as offers or adverts. The proposed solution will include:

a mobile app for customers to manage their account, view offers and discounts and present at the till;

a website for managing accounts and viewing discounts and offers;

integration between the loyalty programme and the till structure so that points can be accrued and spent;

data analysis components that can generate insights into customer spending habits, steer marketing campaigns, assist with financial analysis and help with product placement;

offline support for customers who don’t want to use the app, including physical membership cards.

This scenario allows us to explore many aspects of product delivery, including how non-IT elements (such as marketing) can be included. There are many ways to deliver such a solution, and many ways to split up the work. We don’t pretend our examples are the perfect answer (if such a thing even exists) and there are many alternative ways we could have chosen. We hope that the examples will help bring some of the concepts to life.

A BRIEF WALKTHROUGH

In this section we provide a brief walkthrough of the chapters in this book to help you decide where to start.

Chapter 1 – The importance of Agile today

This chapter explains why Agile approaches are still important today – despite emerging over 20 years ago, the conditions that led to Agile are not only still present but have intensified. Agile has matured beyond its origins in software to many other fields, and modern leadership theories are converging with the types of leadership advocated by Agile approaches. Coupled with Agile’s growing dominance in the technology sector, it is clear that Agile is still relevant and the ability to apply Agile well has never been more important.

Chapter 2 – The origins of Agile

Here we discuss the theories and practices that have influenced Agile, some of which date back centuries. We discuss how non-Agile teams plan and manage their work and tell the story of how the Agile Manifesto came to be created in a ski lodge in Utah in 2001.

Chapter 3 – Projects and products in Agile

One of the ways that Agile bucks the trends set by traditional development approaches is that it has a product focus rather than a project focus. In fact, not only is the term ‘project’ not even mentioned in many Agile approaches, we suggest its use can actually be harmful – encouraging an unhelpful focus on tasks and milestones and minimising the importance of responding to change. We explain how Agile delivery is incremental and iterative and this is how we can deliver more value. The ‘Product Backlog’ is introduced as an important artefact.

Chapter 4 – Understanding the Manifesto for Agile

This chapter, along with Chapter 5, is one of the two cornerstone chapters in the book. The Agile Manifesto defines what we mean by ‘Agile’ so, to apply Agile well, it is critical that we understand what is meant by the Agile values to understand how we can apply and reinforce them.

A recurring theme throughout the book is that Agile is difficult to do well and we have seen many teams struggle. Even though the values look like common sense, they are not always commonly applied. Therefore, we look at common challenges and anti-patterns we have seen through the lens of each value, showing how they can be reinforced or undermined. There are plenty of references and further reading offered here too.

Chapter 5 – Understanding the Agile principles

Similar to Chapter 4, here we explain and examine each of the Agile principles in turn. We share our perspectives on what they mean and why each is important. We focus on particular words and phrases and drill down to consider them more deeply. This allows a far greater insight into what Agile means and how we can apply it ourselves and with our teams.

Additionally, we unpick the intent behind each of the 12 Agile principles and explore anti-patterns that may prevent you from applying Agile as it was intended.

Chapter 6 – Fundamental concepts in Agile

There are several common themes and concepts that run through the values, principles, core behaviours and mindset that underpin any Agile approach. This chapter introduces and discusses them. They are: empiricism; the three pillars of transparency, inspection and adaptation; lean approaches and wastes; incremental and iterative delivery; customer centricity; and self-organising, empowered teams. The chapter explains with some examples how these reinforce the values and principles.

Chapter 7 – Agile delivery

Agile teams need some defined approach to help them apply the values, principles and fundamental concepts to deliver value to their customers. There are several common patterns and techniques that are present in most Agile methods and frameworks. This chapter shows the delivery approach in a generic way, agnostic of any specific approach.

We discuss the overall life cycle; how to get started; how Agile delivery differs from a traditional approach; how to measure progress; getting to the first delivery of value; how to include continuous improvement; how iteration works; and how the team behaves and is organised. We introduce some of the other roles commonly found, including some that are specific to a particular method or framework.

Chapter 8 – Agile practices

Here we discuss and briefly describe a range of practices, techniques, methods, approaches and other ways of working that Agile teams commonly use. They are grouped into categories covering the following areas: leadership; ways of working; requirements, estimation; prioritisation; software development; and measuring success. We briefly describe two of the most common Agile approaches used by teams: Scrum and the Kanban Method.

Chapter 9 – Agile leadership

The easiest way to make an Agile team fail is to surround them with poor leadership and management that don’t understand Agile. This chapter will help to address that by helping leaders and teams develop the kind of leadership that will help Agile teams to deliver the best results.

We describe the evolution of leadership and how many management techniques that evolved in early 20th-century manufacturing are still prevalent today, despite the conditions that Agile teams operate in today being vastly different from those factories or mines. Agile leadership needs to be different; ironically, we look to a German field marshall from the 1800s for a description of what it should be.

For many leaders, effectively leading Agile teams requires a shift in mindset, and we identify and discuss our six key leadership outcomes that leaders should strive for: servant leadership; psychological safety; intrinsic motivation; trusted, empowered people; self-organising teams; and creating strategic alignment. Many of these are not unique to Agile leadership, and we share stories and references that show how these leadership outcomes bring value to many different types of teams. We talk about the importance of creating the right culture and why Agile coaching is a powerful path to success.

Chapter 10 – Managing the product

One of the biggest differences between traditional and Agile delivery is how the product requirements are understood and managed, in particular how the customer is involved and how teams cope with changes to the customer needs. This chapter looks in detail at how Agile teams understand their customers and manage their product and their stakeholders using techniques such as roadmaps, backlogs and story mapping. Some of these techniques have entire books to describe them, so there are lots of examples and references to help bring the ideas to life and point you towards further information.

Chapter 11 – Beyond the basics

Finally, in this chapter we introduce scaling. We have consciously left it until last because we think it is important to properly understand the basics before trying to scale.

It is also important to only introduce scaling when you need it, so we devote the bulk of the scaling discussion to ways that you can avoid the need to scale, even when you have large products or lots of teams. We show ways you can reduce the drivers for scaling by reducing complexity, decoupling everything, managing the product differently, splitting goals, refactoring, using feature teams and shortening the cycle time. If you really, really want to scale, we also cover that briefly.

A second practice that you should only embark upon when you understand the basics is tailoring a process. An aspect of tailoring is the use of specialist roles or teams that can sometimes be sensible to use.

We finish by describing in brief some of the more commonly used Agile methods and frameworks, including references to their source to allow you to find the current versions.

LET’S GET STARTED

We set out to write this book with the expectation that it would mostly involve writing down what we had learned and experienced through our years working with organisations, teams and individuals. In fact, it’s been much more than that. Thinking through the Manifesto values and principles in detail, and spending time reflecting on ways that we have seen them reinforced and undermined, has improved our own understanding of them immensely. Similarly, researching more in-depth leadership and managing the product chapters has improved our understanding and knowledge of those important areas. As a result, this book is far more than an expression of our experience – it is a book that we will refer to, recommend and use in our day-to-day work with Agile teams and others.

We hope you will find it as useful and as illuminating as we have.

1THE IMPORTANCE OF AGILE TODAY

It has never been more important for teams and leaders to understand Agile and how to apply it well

This chapter explains why Agile approaches are not only still important today but are more important than ever. We will explore why:

Despite emerging over 20 years ago, the conditions that brought about the movement towards Agile development are still present and have intensified.

Agile approaches are being applied in a growing range of areas and becoming commonplace, although they are also hard to do well and often applied poorly.

Agile approaches, which began with software projects, are being applied to a wider range of business problems and are well matched to them.

The secret to applying Agile well is to focus on acquiring a deep understanding of the underpinning fundamentals of Agile rather than just learning a framework.

WHY ARE WE HERE?

Before we start to consider the importance of Agile today, let’s look back at why Agile emerged.

In the late 1990s, development of new IT systems was characterised by large, expensive projects that seemingly took a long time to deliver. Teams would often attempt to understand the totality of the solution before any of the ideas had been tested. The result of this was lots of time spent ensuring a full set of requirements were documented up front before a line of code was written.

This approach is reasonable when there is little change and when you can be confident that your requirements are right. This may have been true for some of the systems and projects in the late 1990s, such as tax systems or banking systems that had a lifetime measured in years, but even then it was still a bit of a gamble. It is much less true now. Today, even a tax system requires regular changes to keep pace with the changing world that it supports.

In the mid-1990s huge organisations such as Lockheed, Chrysler, Xerox, Honda and Borland were trying things differently and getting better results. As we will explore in Chapter 2, those pioneers and innovators joined forces with others, also doing things differently, to create the paradigm shift of Agile development.

At the heart of this paradigm shift was the belief that reducing lead times, embracing changing requirements and improving customer buy-in was possible and would provide the ability for technology to create solutions faster than had been possible in the past. These core factors are not only still true today but have become orders of magnitude more significant.

We live in an increasingly VUCA world

The term VUCA emerged from the US War College in the wake of the collapse of the USSR and the end of the cold war. It was inspired by the leadership theories in the 1985 book Leaders: Strategies for Taking Charge by Warren G. Bennis and Burt Nanus.¹ They recognised that what had been a relatively stable and predictable state was suddenly anything but that; things had become volatile, uncertain, complex and ambiguous. This predicated an entirely different military approach.

What is meant by a VUCA world? It’s one that has the characteristics set out in Table 1.1.

Table 1.1 VUCA

The COVID-19 pandemic that began in 2020 is the perfect example of a VUCA situation. The situation was complex and constantly changing; large amounts of data were available, but they were contradictory and incomplete. The political challenge of balancing economic impact, protecting society, preserving medical capacity and managing infection rates was an impossible task. Any solution that helped in one dimension caused harm in another. As more data emerged, previous decisions needed to be revised.

It is easy to find other examples of a VUCA world, particularly with knowledge work (i.e. work that is non-routine and requires divergent and convergent thinking to solve problems). We have learned to construct complicated physical solutions such as bridges and electronics dependably, yet even the most experienced and skilled teams struggle to deliver IT or business change projects without exception reports, changes to requirements and rework. This isn’t a failure of their management approach or poor quality implementations – this is because the problems being solved exhibit VUCA.

We find it hard to predict the future at the best of times. That’s why IT projects are so difficult. They generally depend on user requirements being an accurate prediction of what users will need – yet, people are very bad at accurately predicting this. The problem is exacerbated when the data that are informing the prediction keep changing. This is true of non-IT projects, such as business change or service delivery projects, too.

When a problem is a VUCA problem, you cannot follow the current plan or analyse your way to a solution using traditional methods. When problems happen – the requirements change or the solutions don’t work – it isn’t generally a failure in planning or implementation; instead, it is often an acknowledgement that the problem wasn’t capable of being analysed in advance. More or better analysis probably wouldn’t have resulted in a successful delivery – it would likely have resulted in different things going wrong.

When we are confronted with VUCA problems, we need to solve them in different ways, and that’s where Agile comes in.

Converging leadership approaches

The problems of a VUCA world are not restricted to areas where Agile is traditionally used, so it should be no surprise that the direction of travel of Agile leadership and more general management theories are converging. Leadership experts and gurus who have never even heard of the Agile Manifesto are coming to conclusions about how best to lead teams that are very similar to those of Agile thought leaders.

Joost Minnaar and Pim De Morree quit their corporate jobs and embarked on a mission to meet the people behind the world’s most inspiring organisations and understand what makes them think differently. They documented their findings on their website² and in their inspiring book, Corporate Rebels.³

The organisations that inspire them include clothing companies, bus manufacturers, public sector organisations and banks. Few would describe themselves as an ‘Agile organisation’, yet the practices, behaviours and cultures they have created resonate strongly with those described in the Agile