मुख्य Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

,

Getting software released to users is often a painful, risky, and time-consuming process.

This groundbreaking new book sets out the principles and technical practices that enable

rapid, incremental delivery of high quality, valuable new functionality to users. Through

automation of the build, deployment, and testing process, and improved collaboration between

developers, testers, and operations, delivery teams can get changes released in a matter of hours—

sometimes even minutes–no matter what the size of a project or the complexity of its code base.

 

Jez Humble and David Farley begin by presenting the foundations of a rapid, reliable, low-risk

delivery process. Next, they introduce the “deployment pipeline,” an automated process for

managing all changes, from check-in to release. Finally, they discuss the “ecosystem” needed to

support continuous delivery, from infrastructure, data and configuration management to governance.

 

The authors introduce state-of-the-art techniques, including automated infrastructure management

and data migration, and the use of virtualization. For each, they review key issues, identify best

practices, and demonstrate how to mitigate risks. Coverage includes

 

• Automating all facets of building, integrating, testing, and deploying software

• Implementing deployment pipelines at team and organizational levels

• Improving collaboration between developers, testers, and operations

• Developing features incrementally on large and distributed teams

• Implementing an effective configuration management strategy

• Automating acceptance testing, from analysis to implementation

• Testing capacity and other non-functional requirements

• Implementing continuous deployment and zero-downtime releases

• Managing infrastructure, data, components and dependencies

• Navigating risk management, compliance, and auditing

 

Whether you’re a developer, systems administrator, tester, or manager, this book will help your

organization move from idea to release faster than ever—so you can deliver value to your business

rapidly and reliably.

 

वर्ष:
2010
संस्करण:
1
प्रकाशक:
Addison-Wesley Professional
भाषा:
english
पन्ने:
497
ISBN 10:
0321601912
ISBN 13:
9780321601919
Series:
Addison-Wesley Signature Series Fowler
File:
PDF, 13.78 MB
डाउनलोड (pdf, 13.78 MB)

You may be interested in

 

Most frequently terms

 
 
Jack Pham
there is no link to download, could you guys please check it
18 December 2015 (18:08) 
You can write a book review and share your experiences. Other readers will always be interested in your opinion of the books you've read. Whether you've loved the book or not, if you give your honest and detailed thoughts then people will find new books that are right for them.
Praise for Continuous Delivery
“If you need to deploy software more frequently, this book is for you. Applying it will help you reduce
risk, eliminate tedious work, and increase confidence. I’ll be using the principles and practices here
on all my current projects.”
—Kent Beck, Three Rivers Institute
“Whether or not your software development team already understands that continuous integration
is every bit as necessary as source code control, this is required reading. This book is unique in tying
the whole development and delivery process together, providing a philosophy and principles, not
just techniques and tools. The authors make topics from test automation to automated deployment
accessible to a wide audience. Everyone on a development team, including programmers, testers,
system administrators, DBAs, and managers, needs to read this book.”
—Lisa Crispin, co-author of Agile Testing
“For many organizations Continuous Delivery isn’t just a deployment methodology, it’s critical to
doing business. This book shows you how to make Continuous Delivery an effective reality in your
environment.”
—James Turnbull, author of Pulling Strings with Puppet
“A clear, precise, well-written book that gives readers an idea of what to expect for the release process.
The authors give a step-by-step account of expectations and hurdles for software deployment. This
book is a necessity for any software engineer’s library.”
—Leyna Cotran, Institute for Software Research, University of California, Irvine
“Humble and Farley illustrates what makes fast-growing web applications successful. Continuous
deployment and delivery has gone from controversial to commonplace and this book covers it excellently. It’s truly the intersection of development and operations on many levels, and these guys
nailed it.”
—John Allspaw, VP Technical Operations, Etsy.com and author of
The Art of Capacity Planning and Web Operations
“If you are in the business of building and delivering a software-based service, you would be well
served to internalize th; e concepts that are so clearly explained in Continuous Delivery. But going
beyond just the concepts, Humble and Farley provide an excellent playbook for rapidly and reliably
delivering change.”
—Damon Edwards, President of DTO Solutions and co-editor of dev2ops.org
“I believe that anyone who deals with software releases would be able to pick up this book, go to
any chapter and quickly get valuable information; or read the book from cover to cover and be able
to streamline their build and deploy process in a way that makes sense for their organization. In my
opinion, this is an essential handbook for building, deploying, testing, and releasing software.”
—Sarah Edrie, Director of Quality Engineering, Harvard Business School
“Continuous Delivery is the logical next step after Continuous Integration for any modern software
team. This book takes the admittedly ambitous goal of constantly delivering valuable software to
customers, and makes it achievable through a set of clear, effective principles and practices.”
—Rob Sanheim, Principal at Relevance, Inc.

This page intentionally left blank

Continuous
Delivery

Continuous
Delivery

Jez Humble and David Farley

Upper Saddle River, NJ • Boston • Indianapolis • San Francisco
New York • Toronto • Montreal • London • Munich • Paris • Madrid
Cape Town • Sydney • Tokyo • Singapore • Mexico City

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim,
the designations have been printed with initial capital letters or in all capitals.
The authors and publisher have taken care in the preparation of this book, but make no expressed or implied
warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental
or consequential damages in connection with or arising out of the use of the information or programs contained
herein.
The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special
sales, which may include electronic versions and/or custom covers and content particular to your business,
training goals, marketing focus, and branding interests. For more information, please contact:
U.S. Corporate and Government Sales
(800) 382–3419
corpsales@pearsontechgroup.com
For sales outside the United States please contact:
International Sales
international@pearson.com
Visit us on the Web: informit.com/aw
Library of Congress Cataloging-in-Publication Data:

Humble, Jez.
Continuous delivery : reliable software releases through build, test, and deployment automation
/ Jez Humble, David Farley.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-321-60191-9 (hardback : alk. paper) 1. Computer software--Development.
2. Computer software--Reliability. 3. Computer software--Testing. I. Farley, David, 1959II. Title.
QA76.76.D47H843 2010
005.1--dc22
2010022186
Copyright © 2011 Pearson Education, Inc.
All rights reserved. Printed in the United States of America. This publication is protected by copyright, and
permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval
system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or
likewise. For information regarding permissions, write to:
Pearson Education, Inc
Rights and Contracts Department
501 Boylston Street, Suite 900
Boston, MA 02116
Fax (617) 671 3447
ISBN-13: 978–0–321–60191–9
ISBN-10: 0–321–60191–2
Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana.
First printing August 2010

This book is dedicated to my dad, who has always given me his unconditional
love and support.
—Jez

This book is dedicated to my dad, who always showed me the right direction.
—Dave

This page intentionally left blank

Contents

Foreword . ................................................................................................. xxi
Preface . ................................................................................................... xxiii
Acknowledgments . .................................................................................. xxxi
About the Authors . ............................................................................... xxxiii
Part I: Foundations . ..................................................................................

1

Chapter 1: The Problem of Delivering Software . .................................
Introduction . ................................................................................
Some Common Release Antipatterns . ..........................................
Antipattern: Deploying Software Manually . ....................................

3
3
4

Antipattern: Deploying to a Production-like Environment Only after
Development Is Complete .............................................................

5
7

Antipattern: Manual Configuration Management of Production
Environments . .............................................................................

9

Can We Do Better? . ...................................................................

10

How Do We Achieve Our Goal? . .................................................
Does
This
Process
Scale?
.Receive
............................................................
The
Delivery
Team
Must
and Then
Act on It . ......
Feedback
Must
BeTrigger
Received
Soon as Process
Possible
......................
Every
Change
Should
theasFeedback
.........................
What Are the Benefits? . ................................................................
Empowering Teams . ...................................................................
Reducing Errors . ........................................................................
Lowering Stress . ........................................................................
Deployment Flexibility . ...............................................................
Practice Makes Perfect ..................................................................

ix

11
16
15
14
13

17
17
18
20
21
22

x

Contents

Contents

The Release Candidate . ................................................................ 22
Principles
of Software
. ...................................................
Every Check-in
Leads Delivery
to a Potential
Release . ................................. 24
23
Automate
Almost Everything
. ......................................................
Create a Repeatable,
Reliable Process
for Releasing Software . .......... 25
24
Keep Everything in Version Control . ............................................ 26
Build
Quality
........................................................................
If It Hurts,
DoInIt. More
Frequently, and Bring the Pain Forward . ..... 27
26
Done Means Released . ................................................................ 27
Continuous
...........................................................
Everybody IsImprovement
Responsible .for
the Delivery Process . ........................ 28
Summary . .................................................................................... 29
Chapter 2: Configuration Management . ..............................................
Introduction . ................................................................................
Using Version Control . ................................................................
Check
In Regularly
to Trunkin. Version
......................................................
Keep Absolutely
Everything
Control . .............................
Use Meaningful Commit Messages . ..............................................
Managing Dependencies . .............................................................
Managing External Libraries . ......................................................
Managing Components . ..............................................................
Managing Software Configuration . ..............................................
Configuration and Flexibility . ......................................................
Types of Configuration . ..............................................................
Managing Application Configuration . ...........................................
Managing
Environments
....................................................
Principles
of
Managing
Application
Configuration
. .........................
ManagingYour
Configuration
across. Applications
. ................................
Tools to Manage Environments . ...................................................
Managing the Change Process . .....................................................
Summary . ....................................................................................
Chapter 3: Continuous Integration . .....................................................
Introduction . ................................................................................
Implementing Continuous Integration . ........................................
What You Need Before You Start . ................................................
A Basic Continuous Integration System ...........................................

31
31
32
35
33
37

38
38
39

39
40
41
43

49
47
53
53

54
55
55
56
56
57

Contents

Contents

Prerequisites for Continuous Integration . .....................................
Check In Regularly . ....................................................................
Keep
Build and Test Process
Short
. .........................................
Createthe
a Comprehensive
Automated
Test
Suite . .............................
Managing Your Development Workspace . .....................................
Using Continuous Integration Software . ......................................
Basic Operation . ........................................................................
Bells and Whistles . .....................................................................
Essential Practices . .......................................................................
Don’t Check In on a Broken Build . ...............................................

59
59
60
62

63
63
63

66
66

Always Run All Commit Tests Locally before Committing, or Get Your 66
CI Server to Do It for You .............................................................
Wait for Commit Tests to Pass before Moving On . .........................

67

Never Go Home on a Broken Build . .............................................

68

Time-Box
before
Reverting
. ...............................................
Always Be Fixing
Prepared
to Revert
to the
Previous Revision . ..................

70
69

Don’t Comment Out Failing Tests . ...............................................

70

Take Responsibility for All Breakages That Result from Your Changes . 70
Test-Driven Development . ..........................................................
Suggested Practices . .....................................................................
Failing
Build
Slow
Tests
.Breaches
...................................................
Build
forfor
Architectural
. Practices
.....................................
Extremeathe
Programming
(XP)
Development
. ......................
Distributed
. ......................................................................
Failing the Teams
Build for
Warnings and Code Style Breaches . .................
The Impact on Process . ...............................................................
Centralized Continuous Integration . .............................................
Technical Issues . ........................................................................
Alternative Approaches . ..............................................................
Distributed Version Control Systems . ..........................................
Summary . ....................................................................................

Chapter 4: Implementing a Testing Strategy . .......................................
Introduction . ................................................................................
Types of Tests . .............................................................................
Business-Facing
Tests
That
Critique
the the
Project
. ..............................
Technology-Facing
Tests
That
Support
Development
Process
. ....
Support
Development
Process
. .........

71

71
73
72
71

75
73
75
76
76
77

79
82
83
83
84
89
85

xi

xii

Contents

Contents
Technology-Facing Tests That Critique the Project . ........................

91

Test Doubles . ............................................................................

91

Real-Life Situations and Strategies . ..............................................
New Projects . ............................................................................
Midproject . ...............................................................................
Legacy Systems . .........................................................................
Integration Testing . ....................................................................
Process . ........................................................................................
Managing Defect Backlogs . .........................................................
Summary . ....................................................................................

92
92
94
95
96

99
100

101

Part II: The Deployment Pipeline . ............................................................. 103
Chapter 5: Anatomy of the Deployment Pipeline . ............................... 105
Introduction . ................................................................................ 105
What Is a Deployment Pipeline? . ................................................. 106
A Basic Deployment Pipeline . ...................................................... 111
Deployment Pipeline Practices . .................................................... 113
Only Build Your Binaries Once . ................................................... 113
Smoke-Test
Your Way
Deployments
. ...................................................
Deploy the Same
to Every Environment
. ................................ 117
115
Deploy into a Copy of Production . ............................................... 117
TheIf
Commit
Stage
. ......................................................................
AnyChange
Part of
the Pipeline
Fails,through
Stop thethe
Line
. ..............................
119
Each
Should
Propagate
Pipeline
Instantly . ........ 120
118
Commit Stage Best Practices . ....................................................... 121
The Automated Acceptance Test Gate . ........................................ 122
Subsequent
Stages .Test
...............................................................
AutomatedTest
Acceptance
Best Practices . ................................... 126
124
Manual Testing . ......................................................................... 128
Nonfunctional Testing . ............................................................... 128
Preparing to Release . ................................................................... 128
Automating Deployment and Release . .......................................... 129
Backing Out Changes . ................................................................ 131
Building on Success . ................................................................... 132
Implementing a Deployment Pipeline . .......................................... 133
Automating
the Value
Build Stream
and Deployment
Process
................................
Modeling Your
and Creating
a Walking
Skeleton . ....... 134
133

Contents

Contents

Automating the Unit Tests and Code Analysis . ............................... 135
Automating Acceptance Tests . ..................................................... 136
Evolving Your Pipeline . .............................................................. 136

Metrics . ....................................................................................... 137
Summary . .................................................................................... 140
Chapter 6: Build and Deployment Scripting . ....................................... 143
Introduction . ................................................................................ 143
An Overview of Build Tools . ....................................................... 144
Make . ....................................................................................... 146
Ant . .......................................................................................... 147
NAnt and MSBuild . .................................................................... 148
Maven . ..................................................................................... 149
Rake . ........................................................................................ 150
Buildr . ...................................................................................... 151
Psake . ....................................................................................... 151
Principles and Practices of Build and Deployment Scripting . ........ 152
Project
Structure
forEach
Applications
That
Target
the
JVM
. .............
Evolve
Your
Deployment
System
Incrementally
............................
157
Ensure
the
Deployment
Process
Is
Idempotent
.................................
155
Use
Your
Operating
System’s
Packaging
Tools
.............................
154
the
Same
Scripts
to Deploy
Every
Environment
. ...................
153
an
Technology
to
Deploy
Your
Application
.......... 157
152
Create
aAppropriate
Script
for
Stage
into
Your
Deployment
Pipeline
.............
Project Layout . .......................................................................... 157
Deployment Scripting . ................................................................. 160
Deploying and Testing Layers . ..................................................... 162
Testing Your Environment’s Configuration . ................................... 163
Tips and Tricks . ........................................................................... 164
Always Use Relative Paths . .......................................................... 164
Eliminate Manual Steps . .............................................................. 165
Test
Should
the Build
. .........................................
Don’t
Check
BinariesNot
intoFail
Version
Control
as Part
of Your
Build . .... 166
BuildTargets
In
Traceability
from
Binaries
to Version
Control
. ..................
165
.NET
TipsYour
and Tricks
. ................................................................
Constrain
Application
with Integrated Smoke Tests . ............... 167
Summary . ..................................................................................... 168

xiii

xiv

Contents

Contents

Chapter 7: The Commit Stage .............................................................. 169
Introduction . ................................................................................ 169
Commit Stage Principles and Practices . ........................................ 170
Provide Fast, Useful Feedback . ..................................................... 171
What Should Break the Commit Stage? . ........................................ 172
Tend the Commit Stage Carefully . ................................................ 172
Give Developers Ownership . ....................................................... 173
Use a Build Master for Very Large Teams . .................................... 174
The Results of the Commit Stage . ................................................ 174
The Artifact Repository . .............................................................. 175
Commit Test Suite Principles and Practices . ................................. 177
Avoid the User Interface . ............................................................ 178
Use Dependency Injection . .......................................................... 179
Avoid the Database . ................................................................... 179
Avoid Asynchrony in Unit Tests . .................................................. 180
Using Test Doubles . ................................................................... 180
Minimizing State in Tests . ........................................................... 183
Faking Time . ............................................................................. 184
Brute Force . .............................................................................. 185
Summary . .................................................................................... 185
Chapter 8: Automated Acceptance Testing . ........................................ 187
Introduction . ................................................................................ 187
Why Is Automated Acceptance Testing Essential? . ....................... 188
Testing
the GUI . .............................................................
How to against
Create Maintainable
Acceptance Test Suites . ...................... 192
190
Creating Acceptance Tests . .......................................................... 193
The Role of Analysts and Testers . ................................................. 193
Analysis on Iterative Projects . ...................................................... 193
TheAcceptance
Application
Driver
Layer . .....................................................
Criteria
as Executable
Specifications . ........................... 198
195
How to Express Your Acceptance Criteria . .................................... 200
Implementing
Tests
. ..................................................
The WindowAcceptance
Driver Pattern:
Decoupling
the Tests from the GUI . .... 204
201
State in Acceptance Tests . ............................................................ 204
Managing
Asynchrony
and Timeoutsand
. ...........................................
Process Boundaries,
Encapsulation,
Testing . ............................ 207
206
Using Test Doubles ...................................................................... 210

Contents

Contents

The Acceptance Test Stage . ..........................................................
Keeping Acceptance Tests Green . .................................................
Deployment Tests . .....................................................................
Acceptance Test Performance . .....................................................
Refactor Common Tasks . ............................................................
Share Expensive Resources . .........................................................
Parallel Testing . .........................................................................
Using Compute Grids . ................................................................
Summary . ....................................................................................

213
214
217

218
219
219
220
220

222

Chapter 9: Testing Nonfunctional Requirements . ............................... 225
Introduction . ................................................................................ 225
Managing Nonfunctional Requirements . ..................................... 226
Analyzing Nonfunctional Requirements . ....................................... 227
Programming for Capacity . .......................................................... 228
Measuring Capacity . .................................................................... 231
TheHow
Capacity-Testing
Environment
. .............................................
Should Success and
Failure Be Defined
for Capacity Tests? . ..... 234
232
Automating Capacity Testing . ..................................................... 238
Capacity Testing via the User Interface . ........................................ 240
Using
Recorded
Interaction
Templates
.........................................
Recording
Interactions
against
a Service. or
Public API . ................... 241
Adding
TestsStubs
to the
Deployment
. ..................... 244
UsingCapacity
Capacity Test
to Develop
Tests Pipeline
. ..................................
244
Additional Benefits of a Capacity Test System . ............................ 247
Summary . .................................................................................... 248
Chapter 10: Deploying and Releasing Applications . ............................ 249
Introduction . ................................................................................ 249
Creating a Release Strategy . ......................................................... 250
The Release Plan . ....................................................................... 251
Releasing Products . .................................................................... 252
Deploying and Promoting Your Application . ............................... 253
The First Deployment . ................................................................ 253
Promoting
Configuration
. ...........................................................
Modeling Your
Release Process
and Promoting Builds . ................... 257
254
Orchestration . ........................................................................... 258
Deployments to Staging Environments ............................................ 258

xv

xvi

Contents

Contents

Rolling Back Deployments and Zero-Downtime Releases . ........... 259
Zero-Downtime
Releases . ...........................................................
Rolling Back by Redeploying
the Previous Good Version . ............... 260
Blue-Green Deployments . ........................................................... 261
Canary Releasing . ...................................................................... 263
Emergency Fixes . ......................................................................... 265
Continuous Deployment . ............................................................. 266
Tips
and TricksReleasing
. ............................................................................
Continuously
User-Installed Software . ............................ 270
267
270
The People Who Do the Deployment Should Be Involved in Creating
the Deployment Process ................................................................
Log Deployment Activities . ......................................................... 271
Don’t Delete the Old Files, Move Them . ....................................... 271
Fail Fast
. ...................................................................................
Have
aApplications
Warm-Up
a New
Deployment
.......................... 273
272
Server
Shouldfor
Not
Have
GUIs . ..................................
271
Deployment
Is the Period
Whole
Team’s
Responsibility
.............................
Summary
. ....................................................................................
Don’t Make
Changes Directly on the Production Environment . ....... 273
273

Part III: The Delivery Ecosystem . .............................................................. 275
Chapter 11: Managing Infrastructure and Environments . ................... 277
Introduction . ................................................................................ 277
Understanding the Needs of the Operations Team . ...................... 279
Documentation and Auditing . ...................................................... 280
Alerts for Abnormal Events . ........................................................ 281
IT Service Continuity Planning . ................................................... 282
Modeling
and Managing
Infrastructure
......................................
Use the Technology
the Operations
Team. Is
Familiar With . ............ 283
282
Making
Changes
toto
Infrastructure
. ...............................................
Controlling
Access
Your Infrastructure
. .................................... 287
285
Managing Server Provisioning and Configuration . ....................... 288
Provisioning Servers . .................................................................. 288
Ongoing Management of Servers . ................................................. 290
Managing the Configuration of Middleware . ............................... 295
Managing Configuration . ............................................................ 296
Research the Product . ................................................................. 298
Examine How Your Middleware Handles State ................................ 298

Contents

Contents

Look for a Configuration API . ..................................................... 299
Use a Better Technology . ............................................................ 299

Managing Infrastructure Services . ................................................
Multihomed Systems . ..................................................................
Virtualization . ..............................................................................
Managing Virtual Environments . .................................................
Cloud
Computing
. .......................................................................
Highly
Parallel
Testing
with
Environments
Virtual
Environments
and
theVirtual
Deployment
Pipeline..........................
.......................
Infrastructure in the Cloud . .........................................................
Platforms in the Cloud . ...............................................................
One Size Doesn’t Have to Fit All . .................................................
Criticisms of Cloud Computing . ...................................................
Monitoring Infrastructure and Applications . ...............................
Collecting Data . .........................................................................
Logging . ...................................................................................
Creating Dashboards . .................................................................
Behavior-Driven Monitoring . ......................................................
Summary . ....................................................................................

300
301

303
305

312
310
308
313
314
315
316

317
318
320
321
323

323

Chapter 12: Managing Data . ............................................................... 325
Introduction . ................................................................................ 325
Database Scripting . ...................................................................... 326
Initializing Databases . ................................................................ 327
Incremental Change . .................................................................... 327
Versioning Your Database . .......................................................... 328
Managing Orchestrated Changes . ................................................. 329
Rolling Back Databases and Zero-Downtime Releases . ............... 331
Rolling Back without Losing Data . .............................................. 331
Managing
Test
Data . ...................................................................
Decoupling
Application
Deployment from Database Migration . ...... 334
333
Faking the Database for Unit Tests . .............................................. 335
Test
Isolation
............................................................................
Managing
the .Coupling
between Tests and Data . ........................... 337
336
Setup and Tear Down . ................................................................ 337
Coherent Test Scenarios . ............................................................. 337
Data Management and the Deployment Pipeline . ........................ 338
Data in Commit Stage Tests .......................................................... 338

xvii

xviii

Contents

Contents
Data in Acceptance Tests . ........................................................... 339
Data in Capacity Tests . ............................................................... 341
Data in Other Test Stages . .......................................................... 342

Summary . .................................................................................... 343
Chapter 13: Managing Components and Dependencies . ..................... 345
Introduction . ................................................................................ 345
Keeping Your Application Releasable . ......................................... 346
Make
All Changes
Incrementally
Hide New
Functionality
Until It Is. .................................................
Finished . .................................. 349
347
Branch by Abstraction . ............................................................... 349
Dependencies . .............................................................................. 351
Dependency Hell . ....................................................................... 352
Managing Libraries . ................................................................... 354
Components . ............................................................................... 356
Pipelining
Components
. ..............................................................
How to Divide
a Codebase
into Components . ................................ 360
356
The Integration Pipeline . ............................................................. 361
Managing Dependency Graphs . ................................................... 363
Building Dependency Graphs . ..................................................... 363
Pipelining Dependency Graphs . ................................................... 365
When Should We Trigger Builds? . ................................................ 369
Cautious Optimism . ................................................................... 370
Circular Dependencies . ............................................................... 372
Managing Binaries . ...................................................................... 373
How an Artifact Repository Should Work . ..................................... 373
374
How Your Deployment Pipeline Should Interact with the Artifact
Repository ..................................................................................

Managing Dependencies with Maven . ......................................... 375
Maven Dependency Refactorings . ................................................. 377
Summary . .................................................................................... 379
Chapter 14: Advanced Version Control . ............................................. 381
Introduction . ................................................................................ 381
A Brief History of Revision Control . ............................................ 382
CVS . ........................................................................................ 382
Subversion . ............................................................................... 383
Commercial Version Control Systems . .......................................... 385
Switch Off Pessimistic Locking ...................................................... 386

Contents

Contents

Branching and Merging . .............................................................. 388
Merging . ................................................................................... 389
Distributed
SystemsIntegration
. ..........................................
Branches, Version
Streams, Control
and Continuous
. ............................ 393
390
399
Stream-Based
Version
Control
Systems
.
.......................................
Using
Distributed
Version
Control
Systems
.
..................................
Distributed
Version
Control
Systems
in
Corporate
Environments
.
....
396
A
Brief
History
of
Distributed
Version
Control
Systems
.
.................
395
What Is a Distributed Version Control System? . ............................. 397
393
Development
Models with
Streams
. ..............................................
What Is a Stream-Based
Version
Control
System? . .......................... 400
399
Static and Dynamic Views . .......................................................... 403
Continuous Integration with Stream-Based Version Control Systems ... 403
Develop on Mainline . .................................................................. 405
Branch
forComplex
Release Changes
. ......................................................................
Making
without Branching . .............................. 408
406
Branch by Feature . ....................................................................... 410
Branch by Team . .......................................................................... 412
Summary . .................................................................................... 415
Chapter 15: Managing Continuous Delivery . ...................................... 417
Introduction . ................................................................................ 417
A Maturity Model for Configuration and Release Management .... 419
How to Use the Maturity Model . ................................................. 419
Project Lifecycle . .......................................................................... 421
Identification . ............................................................................ 422
Inception . ................................................................................. 423
Initiation . .................................................................................. 424
Develop and Release . .................................................................. 425
Operation . ................................................................................ 428
A Risk Management Process . ....................................................... 429
Risk Management 101 . ............................................................... 429
Risk Management Timeline . ........................................................ 430
How to Do a Risk-Management Exercise . ..................................... 431
Common Delivery Problems—Their Symptoms and Causes . ....... 432
Infrequent or Buggy Deployments . ............................................... 433
Poor Application Quality . ........................................................... 434
Poorly Managed Continuous Integration Process . ........................... 435

xix

xx

Contents

Contents

Poor Configuration Management . ................................................
Compliance and Auditing . ...........................................................
Automation over Documentation . ................................................
Enforcing Traceability . ...............................................................
Working in Silos . .......................................................................
Change Management . .................................................................
Summary . ....................................................................................

436

436
437
438
439
440

442

Bibliography . ............................................................................................ 443
Index .......................................................................................................... 445

Foreword by Martin Fowler

In the late 90s, I paid a visit to Kent Beck, then working in Switzerland for an
insurance company. He showed me around his project, and one of the interesting
aspects of his highly disciplined team was the fact that they deployed their software into production every night. This regular deployment gave them many advantages: Written software wasn’t waiting uselessly until it was deployed, they
could respond quickly to problems and opportunities, and the rapid turnaround
led to a much deeper relationship between them, their business customer, and
their final customers.
In the last decade I’ve worked at ThoughtWorks, and a common theme of our
projects has been reducing the cycle time between an idea and usable software.
I see plenty of project stories, and almost all involve a determined shortening of
that cycle. While we don’t usually do daily deliveries into production, it’s now
common to see teams doing bi-weekly releases.
Dave and Jez have been part of that sea change, actively involved in projects
that have built a culture of frequent, reliable deliveries. They and our colleagues
have taken organizations that struggled to deploy software once a year into the
world of Continuous Delivery, where releasing becomes routine.
The foundation for the approach, at least for the development team, is Continuous Integration (CI). CI keeps the entire development team in sync, removing
the delays due to integration issues. A couple of years ago, Paul Duvall wrote a
book on CI in this series. But CI is just the first step. Software that’s been successfully integrated into a mainline code stream still isn’t software that’s out in production doing its job. Dave and Jez’s book pick up the story from CI to deal with
that “last mile,” describing how to build the deployment pipeline that turns
integrated code into production software.
This kind of delivery thinking has long been a forgotten corner of software
development, falling into a hole between developers and operations teams. So
it’s no surprise that the techniques in this book rest upon bringing these teams
together—a harbinger of the nascent but growing DevOps movement. This process
also involves testers, as testing is a key element of ensuring error-free releases.

xxi

xxii

Foreword
Threading through all this is a high degree of automation, so things can be done
quickly and without error.
Getting all this working takes effort, but benefits are profound. Long, highintensity releases become a thing of the past. Customers of software see ideas
rapidly turn into working code that they can use every day. Perhaps most
importantly, we remove one of the biggest sources of baleful stress in software
development. Nobody likes those tense weekends trying to get a system upgrade
released before Monday dawns.
It seems to me that a book that can show you how to deliver your software
frequently and without the usual stresses is a no-brainer to read. For your team’s
sake, I hope you agree.

Preface

Introduction
Yesterday your boss asked you to demonstrate the great new features of your
system to a customer, but you can’t show them anything. All your developers
are halfway through developing new features and none of them can run the application right now. You have code, it compiles, and all the unit tests pass on
your continuous integration server, but it takes a couple of days to release the
new version into the publicly accessible UAT environment. Isn’t it unreasonable
to expect the demo at such short notice?
You have a critical bug in production. It is losing money for your business
every day. You know what the fix is: A one-liner in a library that is used in all
three layers of your three-tier system, and a corresponding change to one database
table. But the last time you released a new version of your software to production
it took a weekend of working until 3 A.M., and the person who did the deployment
quit in disgust shortly afterward. You know the next release is going to overrun
the weekend, which means the application will be down for a period during the
business week. If only the business understood our problems.
These problems, although all too common, are not an inevitable outcome of
the software development process: They are an indication that something is
wrong. Software release should be a fast, repeatable process. These days, many
companies are putting out multiple releases in a day. This is possible even with
large projects with complex codebases. In this book, we will show you how this
is done.
Mary and Tom Poppendieck asked, “How long would it take your organization
to deploy a change that involves just one single line of code? Do you do this on
a repeatable, reliable basis?”1 The time from deciding that you need to make a
change to having it in production is known as the cycle time, and it is a vital
metric for any project.

1.

Implementing Lean Software Development, p. 59.

xxiii

xxiv

Preface
In many organizations, cycle time is measured in weeks or months, and the
release process is certainly not repeatable or reliable. It is manual and often requires a team of people to deploy the software even into a testing or staging environment, let alone into production. However, we have come across equally
complex projects which started out like this but where, after extensive reengineering, teams were able to achieve a cycle time of hours or even minutes for a critical
fix. This was possible because a fully automated, repeatable, reliable process was
created for taking changes through the various stages of the build, deploy, test,
and release process. Automation is the key. It allows all of the common tasks
involved in the creation and deployment of software to be performed by
developers, testers, and operations personnel, at the push of a button.
This book describes how to revolutionize software delivery by making the path
from idea to realized business value—the cycle time—shorter and safer.
Software delivers no revenue until it is in the hands of its users. This is obvious,
but in most organizations the release of software into production is a manually
intensive, error-prone, and risky process. While a cycle time measured in months
is common, many companies do much worse than this: Release cycles of more
than a year are not unknown. For large companies every week of delay between
having an idea and releasing the code that implements it can represent millions
of dollars in opportunity costs—and yet these are often the ones with the longest
cycle times.
Despite all this, the mechanisms and processes that allow for low-risk delivery
of software have not become part of the fabric in most of today’s software
development projects.
Our aim is to make the delivery of software from the hands of developers into
production a reliable, predictable, visible, and largely automated process with
well-understood, quantifiable risks. Using the approach that we describe in this
book, it is possible to go from having an idea to delivering working code that
implements it into production in a matter of minutes or hours, while at the same
time improving the quality of the software thus delivered.
The vast majority of the cost associated with delivering successful software is
incurred after the first release. This is the cost of support, maintenance, adding
new features, and fixing defects. This is especially true of software delivered via
iterative processes, where the first release contains the minimum amount of
functionality providing value to the customer. Hence the title of this book,
Continuous Delivery, which is taken from the first principle of the Agile Manifesto: “Our highest priority is to satisfy the customer through early and continuous
delivery of valuable software” [bibNp0]. This reflects the reality: For successful
software, the first release is just the beginning of the delivery process.
All the techniques we describe in this book reduce the time and risks associated
with delivering new versions of your software to users. They do this by increasing
feedback and improving collaboration between the development, testing, and
operations personnel responsible for delivery. These techniques ensure that when
you need to modify applications, either to fix bugs or deliver new features, the

Preface
time between making modifications and having the results deployed and in use
is as low as possible, problems are found early when they are easy to fix, and
associated risks are well understood.

Who Is This Book for, and What Does It Cover?
One of the major aims of this book is to improve collaboration between the
people responsible for delivering software. In particular, we have in mind developers, testers, systems and database administrators, and managers.
We cover topics from traditional configuration management, source code
control, release planning, auditing, compliance, and integration to the automation
of your building, testing, and deployment processes. We also describe techniques
such as automated acceptance testing, dependency management, database migration, and the creation and management of testing and production environments.
Many people involved in creating software consider these activities secondary
to writing code. However, in our experience they take up a great deal of time
and effort, and are critical to successful software delivery. When the risks surrounding these activities are not managed adequately, they can end up costing a
lot of money, often more than the cost of building the software in the first place.
This book provides the information that you need to understand these risks and,
more importantly, describes strategies to mitigate them.
This is an ambitious aim, and of course we can’t cover all these topics in detail
in one book. Indeed we run the risk of alienating each of our target audiences:
developers, by failing to treat topics such as architecture, behavior-driven development, and refactoring in depth; testers, by not spending sufficient time on exploratory testing and test management strategies; operations personnel, by not
paying due attention to capacity planning, database migration, and production
monitoring.
However, books exist that address each of these topics in detail. What we
think is lacking in the literature is a book that discusses how all the moving parts
fit together: configuration management, automated testing, continuous integration
and deployment, data management, environment management, and release
management. One of the things that the lean software development movement
teaches is that it is important to optimize the whole. In order to do this, a holistic
approach is necessary that ties together every part of the delivery process and
everybody involved in it. Only when you have control over the progression of
every change from introduction to release can you begin to optimize and improve
the quality and speed of software delivery.
Our aim is to present a holistic approach, as well as the principles involved in
this approach. We will provide you with the information that you will need to
decide how to apply these practices in your own projects. We do not believe that
there is a “one size fits all” approach to any aspect of software development, let
alone a subject area as large as the configuration management and operational
control of an enterprise system. However, the fundamentals that we describe in

xxv

xxvi

Preface
this book are widely applicable to all sorts of different software projects—big,
small, highly technical or short sprints to early value.
As you begin to put these principles into practice, you will discover the areas
where more detail is required for your particular situation. There is a bibliography
at the end of this book, as well as pointers to other resources online where you
can find more information on each of the topics that we cover.
This book consists of three parts. The first part presents the principles behind
continuous delivery and the practices necessary to support it. Part two describes
the central paradigm of the book—a pattern we call the deployment pipeline.
The third part goes into more detail on the ecosystem that supports the deployment pipeline—techniques to enable incremental development; advanced version
control patterns; infrastructure, environment and data management; and
governance.
Many of these techniques may appear to apply only to large-scale applications.
While it is true that much of our experience is with large applications, we believe
that even the smallest projects can benefit from a thorough grounding in these
techniques, for the simple reason that projects grow. The decisions that you make
when starting a small project will have an inevitable impact on its evolution, and
by starting off in the right way, you will save yourself (or those who come after
you) a great deal of pain further down the line.
Your authors share a background in lean and iterative software development
philosophies. By this we mean that we aim to deliver valuable, working software
to users rapidly and iteratively, working continuously to remove waste from the
delivery process. Many of the principles and techniques that we describe were
first developed in the context of large agile projects. However, the techniques
that we present in this book are of general applicability. Much of our focus is
on improving collaboration through better visibility and faster feedback. This
will have a positive impact on every project, whether or not it uses iterative
software development processes.
We have tried to ensure that chapters and even sections can be read in isolation.
At the very least, we hope that anything you need to know, as well as references
to further information, are clearly sign-posted and accessible so that you can use
this book as a reference.
We should mention that we don’t aim for academic rigor in our treatment of
the subjects covered. There are plenty of more theoretical books on the market,
many of which provide interesting reading and insights. In particular, we will
not spend much time on standards, concentrating instead on battle-tested skills
and techniques every person working on a software project will find useful, and
explaining them clearly and simply so that they can be used every day in the real
world. Where appropriate, we will provide some war stories illustrating these
techniques to help place them in context.

Preface

Conspectus
We recognize that not everyone will want to read this book from end to end. We
have written it so that once you have covered the introduction, you can attack
it in several different ways. This has involved a certain amount of repetition,
but hopefully not at a level that becomes tedious if you do decide to read it
cover-to-cover.
This book consists of three parts. The first part, Chapters 1 to 4, takes you
through the basic principles of regular, repeatable, low-risk releases and the
practices that support them. Part two, Chapters 5 through 10, describe the deployment pipeline. From Chapter 11 we dive into the ecosystem that supports
continuous delivery.
We recommend that everybody read Chapter 1. We believe that people who
are new to the process of releasing software, even experienced developers, will
find plenty of material challenging their view of what it means to do professional
software development. The rest of the book can be dipped into either at your
leisure—or when in a panic.

Part I—Foundations
Part I describes the prerequisites for understanding the deployment pipeline. Each
chapter builds upon the last.
Chapter 1, “The Problem of Delivering Software,” starts by describing some
common antipatterns that we see in many software development teams, and
moves on to describe our goal and how to realize it. We conclude by setting out
the principles of software delivery upon which the rest of the book is based.
Chapter 2, “Configuration Management,” sets out how to manage everything
required to build, deploy, test, and release your application, from source code
and build scripts to your environment and application configuration.
Chapter 3, “Continuous Integration,” covers the practice of building and
running automated tests against every change you make to your application so
you can ensure that your software is always in a working state.
Chapter 4, “Implementing a Testing Strategy,” introduces the various kinds
of manual and automated testing that form an integral part of every project, and
discusses how to decide which strategy is appropriate for your project.

Part II—The Deployment Pipeline
The second part of the book covers the deployment pipeline in detail, including
how to implement the various stages in the pipeline.
Chapter 5, “Anatomy of the Deployment Pipeline,” discusses the pattern that
forms the core of this book—an automated process for taking every change from
check-in to release. We also discuss how to implement pipelines at both the team
and organizational levels.

xxvii

xxviii

Preface
Chapter 6, “Build and Deployment Scripting,” discusses scripting technologies
that can be used for creating automated build and deployment processes, and
the best practices for using them.
Chapter 7, “The Commit Stage,” covers the first stage of the pipeline, a set of
automated processes that should be triggered the moment any change is introduced
into your application. We also discuss how to create a fast, effective commit test
suite.
Chapter 8, “Automated Acceptance Testing,” presents automated acceptance
testing, from analysis to implementation. We discuss why acceptance tests are
essential to continuous delivery, and how to create a cost-effective acceptance
test suite that will protect your application’s valuable functionality.
Chapter 9, “Testing Nonfunctional Requirements,” discusses nonfunctional
requirements, with an emphasis on capacity testing. We describe how to create
capacity tests, and how to set up a capacity testing environment.
Chapter 10, “Deploying and Releasing Applications,” covers what happens
after automated testing: push-button promotion of release candidates to manual
testing environments, UAT, staging, and finally release, taking in essential topics
such as continuous deployment, roll backs, and zero-downtime releases.

Part III—The Delivery Ecosystem
The final part of the book discusses crosscutting practices and techniques that
support the deployment pipeline.
Chapter 11, “Managing Infrastructure and Environments,” covers the automated creation, management, and monitoring of environments, including the use
of virtualization and cloud computing.
Chapter 12, “Managing Data,” shows how to create and migrate testing and
production data through the lifecycle of your application.
Chapter 13, “Managing Components and Dependencies,” starts with a discussion of how to keep your application in a releasable state at all times without
branching. We then describe how to organize your application as a collection of
components, and how to manage building and testing them.
Chapter 14, “Advanced Version Control,” gives an overview of the most
popular tools, and goes into detail on the various patterns for using version
control.
Chapter 15, “Managing Continuous Delivery,” sets out approaches to risk
management and compliance, and provides a maturity model for configuration
and release management. Along the way, we discuss the value of continuous
delivery to the business, and the lifecycle of iterative projects that deliver
incrementally.

Preface

Web Links in This Book
Rather than putting in complete links to external websites, we have shortened
them and put in the key in this format: [bibNp0]. You can go to the link in one
of two ways. Either use bit.ly, in which case the url for the example key would
be http://bit.ly/bibNp0. Alternatively, you can use a url shortening service
we’ve installed at http://continuousdelivery.com/go/ which uses the same keys—so
the url for the example key is http://continuousdelivery.com/go/bibNp0. The
idea is that if for some reason bit.ly goes under, the links are preserved. If
the web pages change address, we’ll try to keep the shortening service at
http://continuousdelivery.com/go/ up-to-date, so try that if the links don’t work
at bit.ly.

About the Cover
All books in Martin Fowler’s Signature Series have a bridge on the cover. We’d
originally planned to use a photo of the Iron Bridge, but it had already been
chosen for another book in the series. So instead, we chose another British bridge:
the Forth Railway Bridge, captured here in a stunning photo by Stewart Hardy.
The Forth Railway Bridge was the first bridge in the UK constructed using
steel, manufactured using the new Siemens-Martin open-hearth process, and delivered from two steel works in Scotland and one in Wales. The steel was delivered
in the form of manufactured tubular trusses—the first time a bridge in the UK
used mass-produced parts. Unlike earlier bridges, the designers, Sir John Fowler,
Sir Benjamin Baker, and Allan Stewart, made calculations for incidence of erection
stresses, provisions for reducing future maintenance costs, and calculations for
wind pressures and the effect of temperature stresses on the structure—much like
the functional and nonfunctional requirements we make in software. They also
supervised the construction of the bridge to ensure these requirements were met.
The bridge’s construction involved more than 4,600 workers, of whom tragically around one hundred died and hundreds more were crippled. However, the
end result is one of the marvels of the industrial revolution: At the time of completion in 1890 it was the longest bridge in the world, and at the start of the 21st
century it remains the world’s second longest cantilever bridge. Like a long-lived
software project, the bridge needs constant maintenance. This was planned for
as part of the design, with ancillary works for the bridge including not only a
maintenance workshop and yard but a railway “colony” of some fifty houses at
Dalmeny Station. The remaining working life of the bridge is estimated at over
100 years.

xxix

xxx

Preface

Colophon
This book was written directly in DocBook. Dave edited the text in TextMate,
and Jez used Aquamacs Emacs. The diagrams were created with OmniGraffle.
Dave and Jez were usually not in the same part of the world, and collaborated
by having everything checked in to Subversion. We also employed continuous
integration, using a CruiseControl.rb server that ran dblatex to produce a PDF
of the book every time one of us committed a change.
A month before the book went to print, Dmitry Kirsanov and Alina Kirsanova
started the production work, collaborating with the authors through their
Subversion repository, email, and a shared Google Docs table for coordination.
Dmitry worked on copyediting of the DocBook source in XEmacs, and Alina
did everything else: typesetting the pages using a custom XSLT stylesheet and an
XSL-FO formatter, compiling and editing the Index from the author’s indexing
tags in the source, and final proofreading of the book.

Acknowledgments

Many people have contributed to this book. In particular, we’d like to thank our
reviewers: David Clack, Leyna Cotran, Lisa Crispin, Sarah Edrie, Damon Edwards, Martin Fowler, James Kovacs, Bob Maksimchuk, Elliotte Rusty Harold,
Rob Sanheim, and Chris Smith. We’d also like to extend special thanks to our
editorial and production team at Addison-Wesley: Chris Guzikowski, Raina
Chrobak, Susan Zahn, Kristy Hart, and Andy Beaster. Dmitry Kirsanov and
Alina Kirsanova did a fantastic job of copyediting and proofreading the book,
and typesetting it using their fully automated system.
Many of our colleagues have been instrumental in developing the ideas in this
book, including (in no particular order) Chris Read, Sam Newman, Dan North,
Dan Worthington-Bodart, Manish Kumar, Kraig Parkinson, Julian Simpson,
Paul Julius, Marco Jansen, Jeffrey Fredrick, Ajey Gore, Chris Turner, Paul
Hammant, Hu Kai, Qiao Yandong, Qiao Liang, Derek Yang, Julias Shaw,
Deepthi, Mark Chang, Dante Briones, Li Guanglei, Erik Doernenburg, Kraig
Parkinson, Ram Narayanan, Mark Rickmeier, Chris Stevenson, Jay Flowers,
Jason Sankey, Daniel Ostermeier, Rolf Russell, Jon Tirsen, Timothy Reaves, Ben
Wyeth, Tim Harding, Tim Brown, Pavan Kadambi Sudarshan, Stephen Foreshew,
Yogi Kulkarni, David Rice, Chad Wathington, Jonny LeRoy, and Chris
Briesemeister.
Jez would like to thank his wife, Rani, for being the most loving partner he
could wish for, and for cheering him up when he was grumpy during the writing
of this book. He also thanks his daughter, Amrita, for her babbling, cuddles,
and big gummy smiles. He is also profoundly grateful to his colleagues at
ThoughtWorks for making it such an inspiring place to work, and to Cyndi
Mitchell and Martin Fowler for their support of this book. Finally, a big shout
out to Jeffrey Fredrick and Paul Julius for creating CITCON, and to the people
he met there for many great conversations.
Dave would like to thank his wife Kate, and children Tom and Ben, for their
unfailing support at every point, in this project and in many others. He would
also like to make a special mention of ThoughtWorks, who, although no longer
his employer, provided an environment of enlightenment and encouragement for

xxxi

xxxii

Acknowledgments
the people that worked there, thus fostering a creative approach to finding solutions, many of which populate the pages of this book. In addition, he would like
to thank his current employer, LMAX, with a special mention for Martin
Thompson, for their support, trust, and willing adoption of the techniques
described in this book in an intensely challenging technical environment of
world-class high-performance computing.

About the Authors

Jez Humble has been fascinated by computers and electronics since getting his
first ZX Spectrum at age 11, and spent several years hacking on Acorn machines
in 6502 and ARM assembler and BASIC until he was old enough to get a proper
job. He got into IT in 2000, just in time for the dot-com bust. Since then he has
worked as a developer, system administrator, trainer, consultant, manager, and
speaker. He has worked with a variety of platforms and technologies, consulting
for nonprofits, telecoms, financial services, and online retail companies. Since
2004 he has worked for ThoughtWorks and ThoughtWorks Studios in Beijing,
Bangalore, London, and San Francisco. He holds a BA in Physics and Philosophy
from Oxford University and an MMus in Ethnomusicology from the School of
Oriental and African Studies, University of London. He is presently living in San
Francisco with his wife and daughter.
Dave Farley has been having fun with computers for nearly 30 years. Over
that period he has worked on most types of software—from firmware, through
tinkering with operating systems and device drivers, to writing games and commercial applications of all shapes and sizes. He started working in large-scale
distributed systems about twenty years ago, doing research into the development
of loose-coupled, message-based systems—a forerunner of SOA. He has a wide
range of experience leading the development of complex software in teams, both
large and small, in the UK and USA. Dave was an early adopter of agile development techniques, employing iterative development, continuous integration, and
significant levels of automated testing on commercial projects from the early
1990s. He honed his approach to agile development during his four-and-a-halfyear stint at ThoughtWorks where he was a technical principal working on some
of their biggest and most challenging projects. Dave is currently working for the
London Multi-Asset Exchange (LMAX), an organization that is building one of
the highest-performance financial exchanges in the world, where they rely upon
all of the major techniques described in this book.

xxxiii

This page intentionally left blank

Part I

Foundations

This page intentionally left blank

Chapter 1

The Problem of Delivering
Software
Introduction
The most important problem that we face as software professionals is this: If
somebody thinks of a good idea, how do we deliver it to users as quickly as
possible? This book shows how to solve this problem.
We focus on the build, deploy, test, and release process, about which relatively
little has been written. This is not because we think that software development
approaches are not important; rather, that without a focus on the other aspects
of the software lifecycle—aspects that are all too commonly treated as peripheral
to the overall problem—it is impossible to achieve reliable, rapid, low-risk software releases that get the fruits of our labors into the hands of our users in an
efficient manner.
There are many software development methodologies, but they focus primarily
on requirement management and its impact on the development effort. There
are many excellent books that cover in detail different approaches to software
design, development, and testing; but these, too, cover only a fragment of the
value stream that delivers value to the people and organizations that sponsor our
efforts.
What happens once requirements are identified, solutions designed, developed,
and tested? How are these activities joined together and coordinated to make
the process as efficient and reliable as we can make it? How do we enable
developers, testers, build and operations personnel to work together effectively?
This book describes an effective pattern for getting software from development
to release. We describe techniques and best practices that help to implement this
pattern and show how this approach interfaces with other aspects of software
delivery.
The pattern that is central to this book is the deployment pipeline. A deployment pipeline is, in essence, an automated implementation of your application’s
build, deploy, test, and release process. Every organization will have differences
in the implementation of their deployment pipelines, depending on their value

3

4

Chapter 1

The Problem of Delivering Software

stream for releasing software, but the principles that govern them do not vary.
An example of a deployment pipeline is given in Figure 1.1.

Commit stage
Compile
Unit test
Analysis
Build installers

Automated
acceptance
testing

Automated
capacity
testing

Manual testing
Showcases
Exploratory
testing

Release

Figure 1.1 The deployment pipeline

The way the deployment pipeline works, in a paragraph, is as follows. Every
change that is made to an application’s configuration, source code, environment,
or data, triggers the creation of a new instance of the pipeline. One of the first
steps in the pipeline is to create binaries and installers. The rest of the pipeline
runs a series of tests on the binaries to prove that they can be released. Each test
that the release candidate passes gives us more confidence that this particular
combination of binary code, configuration information, environment, and data
will work. If the release candidate passes all the tests, it can be released.
The deployment pipeline has its foundations in the process of continuous
integration and is in essence the principle of continuous integration taken to its
logical conclusion.
The aim of the deployment pipeline is threefold. First, it makes every part of
the process of building, deploying, testing, and releasing software visible to
everybody involved, aiding collaboration. Second, it improves feedback so that
problems are identified, and so resolved, as early in the process as possible. Finally,
it enables teams to deploy and release any version of their software to any
environment at will through a fully automated process.

Some Common Release Antipatterns
The day of a software release tends to be a tense one. Why should this be the
case? For most projects, it is the degree of risk associated with the process that
makes release a scary time.
In many software projects, release is a manually intensive process. The environments that host the software are often crafted individually, usually by an operations or IS team. Third-party software that the application relies on is installed.
The software artifacts of the application itself are copied to the production host
environments. Configuration information is copied or created through the admin
consoles of web servers, applications servers, or other third-party components
of the system. Reference data is copied, and finally the application is started,
piece by piece if it is a distributed or service-oriented application.
The reason for the nervousness should be clear: There is quite a lot to go wrong
in this process. If any step is not perfectly executed, the application won’t run

Some Common Release Antipatterns
properly. At this point it may not be at all clear where the error is, or which step
went wrong.
The rest of this book discusses how to avoid these risks—how to reduce the
stress on release days, and how to ensure that each release is predictably reliable.
Before that, let’s be clear about the kinds of process failures that we are trying
to avoid. Here are a few common antipatterns that prevent a reliable release
process, but nevertheless are so common as to be the norm in our industry.

Antipattern: Deploying Software Manually
Most modern applications of any size are complex to deploy, involving many
moving parts. Many organizations release software manually. By this we mean
that the steps required to deploy such an application are treated as separate and
atomic, each performed by an individual or team. Judgments must be made
within these steps, leaving them prone to human error. Even if this is not the
case, differences in the ordering and timing of these steps can lead to different
outcomes. These differences are rarely good.
The signs of this antipattern are:
• The production of extensive, detailed documentation that describes the
steps to be taken and the ways in which the steps may go wrong
• Reliance on manual testing to confirm that the application is running
correctly
• Frequent calls to the development team to explain why a deployment is
going wrong on a release day
• Frequent corrections to the release process during the course of a release
• Environments in a cluster that differ in their configuration, for example
application servers with different connection pool settings, filesystems with
different layouts, etc.
• Releases that take more than a few minutes to perform
• Releases that are unpredictable in their outcome, that often have to be
rolled back or run into unforeseen problems
• Sitting bleary-eyed in front of a monitor at 2 A.M. the day after the release
day, trying to figure out how to make it work
Instead . . .

Over time, deployments should tend towards being fully automated. There should
be two tasks for a human being to perform to deploy software into a development,

5

6

Chapter 1

The Problem of Delivering Software

test, or production environment: to pick the version and environment and to
press the “deploy” button. Releasing packaged software should involve a single
automated process that creates the installer.
We discuss automation a lot in the course of this book, and we know that
some people aren’t totally sold on the idea. Let us explain why we see automated
deployment as an indispensable goal.
• When deployments aren’t fully automated, errors will occur every time they
are performed. The only question is whether or not the errors are significant.
Even with excellent deployment tests, bugs can be hard to track down.
• When the deployment process is not automated, it is not repeatable or
reliable, leading to time wasted on debugging deployment errors.
• A manual deployment process has to be documented. Maintaining the
documentation is a complex and time-consuming task involving collaboration between several people, so the documentation is generally incomplete
or out-of-date at any given time. A set of automated deployment scripts
serves as documentation, and it will always be up-to-date and complete,
or the deployment will not work.
• Automated deployments encourage collaboration, because everything is
explicit in a script. Documentation has to make assumptions about the
level of knowledge of the reader and in reality is usually written as an aidememoire for the person performing the deployment, making it opaque to
others.
• A corollary of the above: Manual deployments depend on the deployment
expert. If he or she is on vacation or quits work, you are in trouble.
• Performing manual deployments is boring and repetitive and yet needs
significant degree of expertise. Asking experts to do boring and repetitive,
and yet technically demanding tasks is the most certain way of ensuring
human error that we can think of, short of sleep deprivation, or inebriation.
Automating deployments frees your expensive, highly skilled, overworked
staff to work on higher-value activities.
• The only way to test a manual deployment process is to do it. This is often
time-consuming and expensive. An automated deployment process is cheap
and easy to test.
• We have heard it said that a manual process is more auditable than an
automated one. We are completely baffled by this statement. With a manual
process, there is no guarantee that the documentation has been followed.
Only an automated process is fully auditable. What is more auditable than
a working deployment script?

Some Common Release Antipatterns
The automated deployment process must be used by everybody, and it should
be the only way in which the software is ever deployed. This discipline ensures
that the deployment script will work when it is needed. One of the principles that
we describe in this book is to use the same script to deploy to every environment.
If you use the same script to deploy to every environment, then the deploymentto-production path will have been tested hundreds or even thousands of times
before it is needed on release day. If any problems occur upon release, you can
be certain they are problems with environment-specific configuration, not your
scripts.
We are certain that, occasionally, manually intensive releases work smoothly.
We may well have been unlucky in having mostly seen the bad ones. However,
if this is not recognized as a potentially error-prone step in the process of software
production, why is it attended by such ceremony? Why all the process and documentation? Why are the teams of people brought in during weekends? Why have
people waiting on standby in case things go less than well?

Antipattern: Deploying to a Production-like Environment Only
after Development Is Complete
In this pattern, the first time the software is deployed to a production-like environment (for example, staging) is once most of the development work is done—at
least, “done” as defined by the development team.
The pattern looks a bit like this.
• If testers have been involved in the process up to this point, they have tested
the system on development machines.
• Releasing into staging is the first time that operations people interact with
the new release. In some organizations, separate operations teams are used
to deploy the software into staging and production. In this case, the first
time an operations person sees the software is the day it is released into
production.
• Either a production-like environment is expensive enough that access to it
is strictly controlled, or it is not in place on time, or nobody bothered to
create one.
• The development team assembles the correct installers, configuration files,
database migrations, and deployment documentation to pass to the people
who perform the actual deployment—all of it untested in an environment
that looks like production or staging.
• There is little, if any, collaboration between the development team and the
people who actually perform deployments to create this collateral.

7

8

Chapter 1

The Problem of Delivering Software

When the deployment to staging occurs, a team is assembled to perform it.
Sometimes this team has all the necessary skills, but often in very large organizations the responsibilities for deployment are divided between several groups.
DBAs, middleware teams, web teams, and others all take a hand in deploying
the latest version of the application. Since the various steps have never been
tested in staging, they often have errors. The documentation misses important
steps. The documentation and scripts make assumptions about the version
or configuration of the target environment that are wrong, causing the
deployment to fail. The deployment team has to guess at the intentions of
the development team.
Often the poor collaboration that causes so many problems in deployment to
staging is shored up with ad-hoc telephone calls, emails, and quick fixes. A very
disciplined team will incorporate all of this communication into the deployment
plan—but it is rare for this process to be effective. As pressure increases, the
defined process for collaboration between the development and deployment teams
is subverted, in order to get the deployment done within the time allocated to
the deployment team.
In the process of performing the deployment, it is not uncommon to find that
incorrect assumptions about the production environment have been baked into
the design of the system. For example, one application we had a hand in deploying
used the filesystem to cache data. This worked fine on a developer workstation,
but less well in a clustered environment. Solving problems like this one can take
a long time, and the application cannot be said to have been deployed until they
are resolved.
Once the application is deployed into staging, it is common for new bugs to
be found. Unfortunately, there is often no time to fix them all because the deadline
is fast approaching and, at this stage of the project, deferring the release date is
unacceptable. So the most critical bugs are hurriedly patched up, and a list of
known defects is stored by the project manager for safekeeping, to be deprioritized
when work begins on the next release.
Sometimes it can be even worse than this. Here are a few things that can
exacerbate the problems associated with a release.
• When working on a new application, the first deployment to staging is
likely to be the most troublesome.
• The longer the release cycle, the longer the development team has to make
incorrect assumptions before the deployment occurs, and the longer it will
take to fix them.
• In large organizations where the delivery process is divided between different
groups such as development, DBA, operations, testing, etc., the cost of
coordination between these silos can be enormous, sometimes stalling the
release process in ticketing hell. In this scenario, developers, testers, and
operations personnel are constantly raising tickets (or sending emails) to

Some Common Release Antipatterns
each other to perform any given deployment—and worse, to resolve
problems that arise during deployment.
• The bigger the difference between development and production environments, the less realistic are the assumptions that have to be made during
development. This can be difficult to quantify, but it’s a good bet that
if you’re developing on a Windows machine and deploying to a Solaris
cluster, you are in for some surprises.
• If your application is installed by users or contains components that are,
you may not have much control over their environments, especially outside
of a corporate setting. In this case, a great deal of extra testing will be
required.
Instead . . .

The remedy is to integrate the testing, deployment, and release activities into the
development process. Make them a normal and ongoing part of development so
that by the time you are ready to release your system into production there is
little to no risk, because you have rehearsed it on many different occasions in a
progressively more production-like sequence of test environments. Make sure
everybody involved in the software delivery process, from the build and release
team to testers to developers, work together from the start of the project.
We are test addicts, and the extensive use of continuous integration and continuous deployment, as a means of testing both our software and our deployment
process, is a cornerstone of the approach that we describe.

Antipattern: Manual Configuration Management of Production
Environments
Many organizations manage the configuration of their production environments
through a team of operations people. If a change is needed, such as a change to
database connection setting or an increase in the number of threads in a thread
pool on an application server, then it is carried out manually on the production
servers. If a record is kept of such a change, it is probably an entry in a change
management database.
Signs of this antipattern are:
• Having deployed successfully many times to staging, the deployment into
production fails.
• Different members of a cluster behave differently—for example, one node
sustaining less load or taking longer to process requests than another.
• The operations team take a long time to prepare an environment for a
release.

9

10

Chapter 1

The Problem of Delivering Software

• You cannot step back to an earlier configuration of your system, which
may include operating system, application server, web server, RDBMS, or
other infrastructural settings.
• Servers in clusters have, unintentionally, different versions of operating
systems, third-party infrastructure, libraries, or patch levels.
• Configuration of the system is carried out by modifying the configuration
directly on production systems.
Instead . . .

All aspects of each of your testing, staging, and production environments,
specifically the configuration of any third-party elements of your system, should
be applied from version control through an automated process.
One of the key practices that we describe in this book is configuration management, part of which means being able to repeatably re-create every piece of infrastructure used by your application. That means operating systems, patch levels,
OS configuration, your application stack, its configuration, infrastructure
configuration, and so forth should all be managed. You should be able to recreate your production environment exactly, preferably in an automated fashion.
Virtualization can help you get started with this.
You should know exactly what is in production. That means that every change
made to production should be recorded and auditable. Often, deployments fail
because somebody patched the production environment last time they deployed,
but the change was not recorded. Indeed it should not be possible to make
manual changes to testing, staging, and production environments. The only way
to make changes to these environments should be through an automated process.
Applications often depend on other applications. It should be possible to see
at a glance exactly what the currently released version of every piece of software is.
While releases can be exhilarating, they can also be exhausting and depressing.
Almost every release involves last-minute changes, such as fixing the database
login details or updating the URL for an external service. There should be a way
of introducing such changes so that they are both recorded and tested. Again,
automation is essential. Changes should be made in version control and then
propagated to production through an automated process.
It should be possible to use the same automated process to roll back to a
previous version of production if the deployment goes wrong.

Can We Do Better?
You bet, and the goal of this book is to describe how. The principles, practices,
and techniques we describe are aimed at making releases boring, even in complex
“enterprise” environments. Software release can—and should—be a low-risk,
frequent, cheap, rapid, and predictable process. These practices have been
developed over the last few years, and we have seen them make a huge difference

How Do We Achieve Our Goal?
in many projects. All of the practices in this book have been tested in large enterprise projects with distributed teams as well as in small development groups. We
know that they work, and we know that they scale to large projects.

The Power of Automated Deployment
One of our clients used to have a large team of people dedicated to each release.
The team worked together for seven days, including the entire weekend, to get
the application into production. Their success rate was poor, with many releases
introducing errors or requiring high levels of intervention on the day of release as
well as, often, patches and fixes on subsequent days to correct errors introduced
with the release or caused by human errors in configuring the new software.
We helped them to implement a sophisticated automated build, deploy, test, and
release system and to introduce the development practices and techniques necessary to support it. The last release we saw took seven seconds to deploy the
application into production. No one noticed anything had happened, except of
course that the new behaviors that the release implemented suddenly became
available. Had the successful deployment of the system behind this major website
failed for any reason, we could have backed out the change in the same amount
of time.

Our goal is to describe the use of deployment pipelines, combined with high
levels of automation of both testing and deployment and comprehensive
configuration management to deliver push-button software releases. That is,
push-button software releases to any deployment target—development, test, or
production.
Along the way we will describe the pattern itself and the techniques that
you will need to adopt to make it work. We will provide advice on different
approaches to solving some of the problems that you will face. We have found
that the advantages of such an approach vastly outweigh the costs of achieving it.
None of this is outside the reach of any project team. It does not require rigid
process, significant documentation, or lots of people. By the end of this chapter,
we hope that you will understand the principles behind this approach.

How Do We Achieve Our Goal?
As we said, our goal as software professionals is to deliver useful, working
software to users as quickly as possible.
Speed is essential because there is an opportunity cost associated with not delivering software. You can only start to get a return on your investment once
your software is released. So, one of our two overriding goals in this book is to
find ways to reduce cycle time, the time it takes from deciding to make a change,
whether a bugfix or a feature, to having it available to users.

11

12

Chapter 1

The Problem of Delivering Software

Delivering fast is also important because it allows you to verify whether your
features and bugfixes really are useful. The decision maker behind the creation
of an application, who we’ll call the customer, makes hypotheses about which
features and bugfixes will be useful to users. However, until they are in the hands
of users who vote by choosing to use the software, they remain hypotheses. It is
therefore vital to minimize cycle time so that an effective feedback loop can be
established.
An important part of usefulness is quality. Our software should be fit for its
purpose. Quality does not equal perfection—as Voltaire said, “The perfect is the
enemy of the good,”—but our goal should always be to deliver software of
sufficient quality to bring value to its users. So while it is important to deliver
our software as quickly as possible, it is essential to maintain an appropriate
level of quality.
So, to slightly refine our goal, we want to find ways to deliver high-quality,
valuable software in an efficient, fast, and reliable manner.
We, and our fellow practitioners, have discovered that in order to achieve these
goals—low cycle time and high quality—we need to make frequent, automated
releases of our software. Why is this?
• Automated. If the build, deploy, test, and release process is not automated,
it is not repeatable. Every time it is done, it will be different, because of
changes in the software, the configuration of the system, the environments,
and the release process. Since the steps are manual, they are error-prone,
and there is no way to review exactly what was done. This means there is
no way to gain control over the release process, and hence to ensure high
quality. Releasing software is too often an art; it should be an engineering
discipline.
• Frequent. If releases are frequent, the delta between releases will be
small. This significantly reduces the risk associated with releasing and
makes it much easier to roll back. Frequent releases also lead to faster
feedback—indeed, they require it. Much of this book concentrates on
getting feedback on changes to your application and its associated
configuration (including its environment, deployment process, and data)
as quickly as possible.
Feedback is essential to frequent, automated releases. There are three criteria
for feedback to be useful.
• Any change, of whatever kind, needs to trigger the feedback process.
• The feedback must be delivered as soon as possible.
• The delivery team must receive feedback and then act on it.

How Do We Achieve Our Goal?
Let’s examine these three criteria in detail and consider how we can
achieve them.

Every Change Should Trigger the Feedback Process
A working software application can be usefully decomposed into four components:
executable code, configuration, host environment, and data. If any of them
changes, it can lead to a change in the behavior of the application. Therefore we
need to keep all four of these components under control and ensure that a change
in any one of them is verified.
Executable code changes when a change is made to the source code. Every
time a change is made to the source code, the resulting binary must be built and
tested. In order to gain control over this process, building and testing the binary
should be automated. The practice of building and testing your application on
every check-in is known as continuous integration; we describe it in detail in
Chapter 3.
This executable code should be the same executable code that is deployed into
every environment, whether it is a testing environment or a production environment. If your system uses a compiled language, you should ensure that the binary
output of your build process—the executable code—is reused everywhere it is
needed and never rebuilt.
Anything that changes between environments should be captured as configuration information. Any change to an application’s configuration, in whichever
environment, should be tested. If the software is to be installed by the users,
the possible configuration options should be tested across a representative range
of example systems. Configuration management is discussed in Chapter 2.
If the environments the application is to be deployed into change, the whole
system should be tested with the changes to the environment. This includes
changes in the operating system configuration, the software stack that supports
the application, the network configuration, and any infrastructure and external
systems. Chapter 11 deals with managing infrastructure and environments, including automation of the creation and maintenance of testing and production
environments.
Finally, if the structure of the data changes, this change must also be tested.
We discuss data management in Chapter 12.
What is the feedback process? It involves testing every change in a fully automated fashion, as far as possible. The tests will vary depending on the system,
but they will usually include at least the following checks.
• The process of creating the executable code must work. This verifies that
the syntax of your source code is valid.
• The software’s unit tests must pass. This checks that your application’s
code behaves as expected.

13

14

Chapter 1

The Problem of Delivering Software

• The software should fulfill certain quality criteria such as test coverage and
other technology-specific metrics.
• The software’s functional acceptance tests must pass. This checks that your
application conforms to its business acceptance criteria—that it delivers
the business value that was intended.
• The software’s nonfunctional tests must pass. This checks that the application performs sufficiently well in terms of capacity, availability, security,
and so on to meet its users’ needs.
• The software must go through exploratory testing and a demonstration to
the customer and a selection of users. This is typically done from a manual
testing environment. In this part of the process, the product owner might
decide that there are missing features, or we might find bugs that require
fixing and automated tests that need creating to prevent regressions.
The environments these tests run in must be as similar as possible to production,
to verify that any changes to our environments have not affected the application’s
ability to work.

The Feedback Must Be Received as Soon as Possible
The key to fast feedback is automation. With fully automated processes, your
only constraint is the amount of hardware that you are able to throw at the
problem. If you have manual processes, you are dependent on people to get
the job done. People take longer, they introduce errors, and they are not auditable.
Moreover, performing manual build, test, and deployment processes is boring
and repetitive—far from the best use of people. People are expensive and
valuable, and they should be focused on producing software that delights its
users and then delivering those delights as fast as possible—not on boring, errorprone tasks like regression testing, virtual server provisioning, and deployment,
which are best done by machines.
However, implementing a deployment pipeline is resource-intensive, especially
once you have a comprehensive automated test suite. One of its key objectives
is to optimize for human resource usage: We want to free people to do the
interesting work and leave repetition to machines.
We can characterize the tests in the commit stage of the pipeline (Figure 1.1)
as follows.
• They run fast.
• They are as comprehensive as possible—that is to say, they cover more than
75% or so of the codebase, so that when they pass, we have a good level
of confidence that the application works.

How Do We Achieve Our Goal?
• If any of them fails, it means our application has a critical fault and should
not be released under any circumstances. That means that a test to check
the color of a UI element should not be included in this set of tests.
• They are as environment-neutral as possible—that is, the environment does
not have to be an exact replica of production, which means it can be simpler
and cheaper.
On the other hand, the tests in the later stages have the following general
characteristics.
• They run more slowly and therefore are candidates for parallelization.
• Some of them may fail, and we may still choose to release the application
under some circumstances (perhaps there is a critical fix in the release
candidate that causes the performance to drop below a predefined
threshold—but we might make the decision to release anyway).
• They should run on an environment that is as similar as possible to production, so in addition to the direct focus of the test they also test the
deployment process and any changes to the production environment.
This organization of the testing process means that we have a high level of
confidence in the software after the first set of tests, which run fastest on the
cheapest hardware. If these tests fail, the release candidate does not progress to
later stages. This ensures optimal use of resources. There is much more on
pipelining in Chapter 5, “Anatomy of the Deployment Pipeline,” and the later
Chapters 7, 8, and 9 which describe the commit testing stage, automated
acceptance testing, and testing nonfunctional requirements.
One of the fundamentals of our approach is the need for fast feedback. Ensuring
fast feedback on changes requires us to pay attention to the process of developing
software—in particular, to how we use version control and how we organize our
code. Developers should commit changes to their version control system frequently, and split code into separate components as a way of managing large or distributed teams. Branching should, in most circumstances, be avoided. We discuss
incremental delivery and the use of components in Chapter 13, “Managing
Components and Dependencies,” and branching and merging in Chapter 14,
“Advanced Version Control.”

The Delivery Team Must Receive Feedback and Then Act on It
It is essential that everybody involved in the process of delivering software is involved in