The iteration is a cornerstone of agile development. It provides a heartbeat for the team and its stakeholders, and a structure for various routine activities that help keep development work aligned with what the customer needs. However, the way many teams run their iterations creates serious pitfalls which can keep them from delivering software as effectively as they could.
The orthodox approach to the iteration is to treat it as a timebox for delivering a batch of stories, which is the approach most Scrum teams take with sprints (the Scrum term for an iteration). In recent years many teams have scrapped this approach, either using iterations more as a checkpoint, as many ThoughtWorks teams do, or scrapping them entirely with Kanban and Lean software development.
For the purpose of this post, I will refer to these two general approaches to running iterations as the "orthodox" or "timeboxed-batch" iteration model on the one hand, and the "continuous development" model on the other hand. Although orthodox iterations have value, certainly over more old-school waterfall project management approaches, continuous development approaches which do away with timeboxing and avoid managing stories in batches allow teams to more effectively deliver higher quality software.
Orthodox iteration model: (or "timeboxed-batch" model). Each iteration works on a fixed batch of stories, all of which must be started and finished within a single iteration.
Continuous development model: Stories are developed in a continuous flow, avoiding the need to stop development in order to consolidate a build containing only fully complete stories.
The anatomy of the orthodox iteration
In the classically run iteration or sprint, the Product Owner (PO) and team choose a set of stories that it commits to deliver at the end of the iteration. All of the stories in this batch should be sufficiently prepared before the iteration begins. The level and type of preparation varies between teams, but usually includes a level of analysis including the definition of acceptance criteria. This analysis should have been reviewed by the PO and developers to ensure there is a common understanding of the story. The PO should understand what they can expect to have when the story is implemented, and the technical team should have enough of an understanding of the story to estimate it and identify potential risks.
The iteration begins with an iteration kickoff meeting (IKO) where the team reviews the stories and confirms their confidence that they can deliver the stories within the iteration. The developers then choose stories to work on, discussing each story with the PO, Business Analyst (BA), and/or Quality Analyst (QA) as appropriate, then breaking it down into implementation tasks. Implementation takes place with continual reviews, or even pairing with these other non-developers, helping to keep implementation on track, and minimizing the amount of rework needed when the story is provisionally finished and goes into testing.
The QA and BA/PO then test and review each story as its implementation is completed. This is in addition to the automated testing which has been written and repeatedly running following TDD and CI practices. Only once the story is signed off do the developers move on to another one of the stories in the iteration's committed batch.
As the end of the iteration approaches, developers and QAs should be wrapping up the last stories and preparing a releasable build for the showcase, which is typically held on the final day of the iteration. In the showcase, the team demonstrates the functionality of the completed stories to the PO and other stakeholders, the stories are signed off. The team holds a retrospective to consider how they can work better, then on the next working day they hold the IKO to start the following iteration.
When the iteration ends the team has a complete, fully tested and releasable build of the application, regardless of whether the software actually will go into production at this point.
The start and end dates of the iteration are firmly fixed. If there are stories (or defect fixes) which aren't quite ready at the end of the iteration, the iteration end date is never slipped. Instead, the story is not counted as completed, so must be carried over to the next iteration.
The benefits of the orthodox iteration
This style of iteration offers many benefits over traditional waterfall methodologies. A short, rigid cycle for producing completely tested and releasable code forces discipline on the team, keeping the code in a near-releasable state throughout the project, and avoiding the temptation to leave work (e.g. testing) for "later", building up unmanageable burdens of work, stress, and defects to be dealt with under the pressure of the final release phase.
The timeboxed iteration also forces the team to learn how to define stories of a manageable size. If stories are routinely too big to complete in one iteration this is a clear sign that the team needs improve the way it defines and prepares stories.
This demonstrates another benefit of the iteration, which is frequent feedback. The team is able to evaluate not only the quality of their code and its relevance to the business by getting feedback quickly, they are also able to evaluate how effectively they are working, and try out ideas for improving continually throughout the project.
Fundamental problems with the orthodox itertation
The timeboxed-batch approach to iterations has value, particularly for teams inexperienced with agile. However, it has fundamental problems. At core, this approach is waterfall written small, with many of the same flaws, albeit with a small enough cycle that issues can be dealt with more quickly than with a full waterfall project.
To understand why this is so, let's flesh out the idealized anatomy of the iteration from above with some of the things which often happen in practice.
- At the start of the iteration, no development is taking place, because everyone is working on preparing the new batch of stories. The BA's are extremely busy now, because they have a full working set of stories to hand over to developers (i.e. however many stories the team can work on at once, that's how many stories the BA's must hand over all at the same time). The QA's are less busy now, although they may be helping the BA's out, and planning their testing for the iteration's stories.
- Actually, I lied. Development is taking place, and QA's are extremely busy. Testing and bugfixing stories left over from the previous iteration is still going on. See points below to understand why. As a result, preparation and starting of the stories for this iteration is sluggish because of work carried over from the previous iteration. This actually isn't necessarily bad, since it helps to stagger the story preparation work, preventing the BA's from becoming a bottleneck. However, depending on whether carryover work was factored into the number of stories chosen for the current iteration (business pressures often make this difficult), this may mean the team is already at risk of failing to meet its commitment.
- Once the previous iteration's work has settled down, QA's have little to do until the end of the iteration approaches, at which point they come under enormous pressure. Developers are humping to get their stories done in time, leaving QA's with a pile of stories to be tested in time for the showcase. Any defects they find increase this pressure even more, with very little time to get the fixes in and then re-tested (maybe needing even more fixing and re-testing!)
- If developers complete a story with a bit of time left in the iteration, they aren't able to start new stories because the stories for the following iteration won't be ready to work on until the IKO.
- In the end, some stories don't get fully tested during the iteration. They may be tested in the following iteration, after having already been signed off as "complete" by the unsuspecting Product Owner. If so, developers need to be pulled away to fix the defects found, or else the defects are added to a backlog to be fixed "later" (also known as "probably never"). Other developed code is left completely untested or under-tested, with the vague hope that any defects will be found in later testing phases, or that maybe there aren't any important bugs anyway. In fact, these defects will be found, but not at a time more convenient to the team.
- If any serious issues are raised by stakeholders during the showcase there is not time to fix them until the next iteration, which means it will take the full length of an iteration before a truly releasable build is created.
The root problem of the orthodox iteration
At the end of the day, the orthodox iteration suffers from two problems which are inherent in its very definition: it organizes work into batches, and it enforces a timebox.
Batching work is the antithesis to flow. The Lean approach to working aims to maximize the flow of work for the members of a team, which in software development translates to getting stories flowing easily through creation, analysis, implementation, validation, and release. When a developer finishes one story and it is signed off, she should have another story ready for her to pick up and start on. This shouldn't need to wait on an arbitrary ceremony, and certainly shouldn't have to wait for everyone else on the team to finish their stories and get them all signed off.
The batching focus of orthodox iterations doesn't only cause developers to block, it also turns BA's, QA's, and the PO into bottlenecks. As described above, the start and end of the iteration each put a full working set of stories in the same state at once, all needing the same activity carried out on them at once.
Imagine an assembly line which starts up to assemble twenty cars, then stops while they are all inspected at once. Only once all of the cars are inspected and their defects fixed does the line start up again to begin assembling another twenty cars.
Timeboxing is also a source of problems for iterations. The main problem is the arbitrary deadline creates pressure to get stories "over the line" so they can count towards the velocity for the iteration. Unless management is enlightened (or uninterested) enough to avoid focusing on fluctuations of velocity from iteration to iteration (and even the most enlightened managers I've worked with do get worked up over velocity) this leads to the temptation to rush and cut corners, or to play games with stories.
Rushing obviously endangers the quality of the code, which almost certainly leads to delays down the line when the defects surface. Playing games, such as closing unfinished stories and opening defects to complete the work, or counting some points towards an unfinished story, undermines the team's ability to measure and predict its work honestly. These bad habits will catch up one way or another.
Expecting code to be complete at the end of the iteration, fully tested, fixed, and ready for deployment, is unrealistic unless the iteration is structured with significant padding at the end. This padding must come after all reviews, including the stakeholder showcase, to allow time to make corrections, unless those reviews are mere rubber stamp sessions, with no genuine feedback permitted. This then means the team will be underutilized during the padding time. Otherwise, if there is so much rework done during this period that the entire team is fully engaged, then the risk of introducing new defects is too high to be confident in stable code by the end.
The alternative is to break the strict timeboxed-batched iteration model by interleaving work on the next iteration with the cleanup work from the previous iteration. This turns out to not be such a bad idea, and leads to evolving away from the timeboxed-batch iteration model towards the continuous development model.
The continuous development model
The continuous development model may be purely iteration-less, e.g. Kanban, or it may still retain the iteration as a period for measuring progress and for scheduling activities such as showcases. Once development is underway stories are prepared, developed, and tested using a "pull" approach, being worked on as team members become available, so that stories are constantly flowing, and everyone is constantly working on the highest value work available at the moment. This requires some different approaches to managing work flow than are used with other approaches. For more information, look into Kanban and Lean software development.
Since joining a year or so ago I've found that although no two ThoughtWorks projects run in exactly the same way, there is a strong tendency to use iterations which look a lot like Kanban, but retaining a one or two week iteration. Iterations are used to report progress (including velocity), and to schedule showcases and other regular meetings, but stories are not moved through the process in batches. Teams don't start and stop work as a whole other than the start and end of a release. If the showcase is two days away, nothing stops a developer pair from starting on a new story knowing full well it will be incomplete when the codebase is demoed to the stakeholders, and possibly even deployed to later stage environments.
Although we do make projections and aim to have certain stories done by the next showcase, the team doesn't promise to deliver a specific batch of stories. If it makes sense, stories can be dropped, added, or swapped as needed. This gives the business more flexibility to adapt their vision of the software as it is developed. It also reduces the pressure to mark a given story as "done" by a hard deadline, since there is no disruption from letting work carry on over the end of an iteration.
I've seen a Scrum team become ornery and rebellious when a PO made a habit of asking to swap stories after a sprint had started, even though work hadn't been started on the particular stories involved. This was made worse because bugfixes were scheduled into sprints alongside stories, meaning that any serious defect found in production completely disrupted the team.
Another factor that aggravated the situation was that the stories for each sprint were agreed before the end of the previous iteration. So if the showcase raised ideas for improvements to the functionality completed in iteration N, new stories could only be started in iteration N + 2 at the soonest. This hardly created a situation where the PO or the business felt the development team was responsive to business needs.
Also see Oren Teich's post Go slow to go fast, which points out the problems with deadlines, and that iterations are simply a shorter deadline.
Challenges and rewards of continuous development
There are certainly challenges in moving to continuous development over the timeboxed-batch model. There is more risk of stories dragging on across multiple iterations. This can be mitigated by monitoring cycle time and keeping things visible, so that the team can discuss the issue and make changes to their processes if it becomes a problem.
For teams which are new to agile and still struggle to create appropriately sized stories, the timeboxed model may be more helpful to build the discipline and experience needed before being able to move to a continuous model. However, for experienced teams, timeboxing and batching stories simply has too many negative effects.
Continuous development, with a looser approach to iterations, maximizes the productivity of the team, avoids pitfalls that put quality at risk, and offers the business and the team more flexibility.
