Is Predictability Really What We Want?

Predictability in software system delivery is as close to a Holy Grail as it comes in the IT industry. I’ve heard many people stress being able to have a predictable delivery cadence as something valuable to them. As recently as today I saw a reference to “predictability over commitment” on Twitter! But why is predictability so important to so many people?

The people who pay for the software we create certainly want to get away from systems that cost more than was originally expected and take longer to deliver than thought. They also want fewer surprises from defects that show up only when a system is in production use, which require more time and money to fix.

The people who manage software delivery groups certainly want to be able to know how much their teams can deliver over a certain time period so they can work proactively to deal with issues such as training and career development.

And, of course, the people who actually build the systems would like predictability not only of their own work but also that of the people who will be consuming their software. “If the business would stop changing their mind, we could get something finished!”, is a complaint I’ve heard many times (and probably made myself in earlier years).

So, predictability sure looks like it would be a great thing! When I learned about Agile processes like XP and Scrum the concept of velocity was introduced, which was intended as a measure of value delivered over some time period. A team’s average velocity was put forth as their capacity, i.e. a predictable measure of how much a team could complete in that time. Teams were (and still are) encouraged to find ways to minimize the variance in velocity from iteration to iteration. They are, in effect, being encouraged to be predictable.

But what’s really at the root of this need for predictability? Why is it so important and garner so much attention? Why do we go to so much effort to be predictable?

The Risk

Early in my career I was asked to estimate how long it would take to build a system. I reviewed the existing system that it would replace and came up with an estimate of 90 days. The manager to whom I gave this estimate said, “No, the business people will never go for that.” I responded that I didn’t believe that all of the work could be completed in any less time than that. He told me that he believed me, but the business people would balk at the cost. He asked me to “get the estimate down to 70–75 days”, and he’d take care of getting them to agree to the larger amount later.

I wasn’t terribly comfortable with the idea of lying to the business people, but what I didn’t know at the time was that a previous attempt to do the same work had gone poorly and cost the business a significant amount of money for absolutely nothing delivered. They had been burned and would be hesitant at best to commit to another large development effort. So, I went with the manager’s plan and sure enough at about the 50 day mark he was able to get them to commit to supporting the full 90 days (I believe the actual number of days to complete the initial work was 92!).

During the development process, I worked very closely with the main business person. I was seated nearby, and could show her and her colleagues the work proceeding and get their feedback right on the spot. We delivered that system successfully and started to look at the next stage of its development.

By this time, my contract had been extended to cover the new work. I wasn’t asked for the number of days they thought it would take, but rather an rough idea in months. The knew that an estimate at that level was good enough for their needs.

I started work on that system in April 1992, working on it constantly until 1996. I also did more work on it part time until early 1998. After the initial development stage, I never had to provide a detailed estimate of the work over the next 5 years, just a gut feel idea of the size.

Why? Because the business and IT management trusted me!

Was I predictable? Yes, to an extent. There were times that I needed to introduce new technologies and new approaches in order to meet new business needs. Some of that work was exploratory and I didn’t know if it would take 2 hours, 2 days or 2 weeks! What was predictable was that I would work as closely as possible with the business people to deliver systems that met their needs. What was also predictable was that I’d do my level best to build systems of the highest quality and long-term maintainability… because I would probably be the person maintaining them!

In the end, though, what replaced predictability at the work level was trust. I did have to earn that trust through my initial work, but that represented less than 20% of the overall time I spent at that client.

Think about this, then. If you already have a good trust relationship between your development teams and the business they support, do you really need predictability? Does it really matter if a team’s velocity fluctuates from iteration to iteration? If you have that trust and frequent collaboration, do you even need things like burndown/burnup charts? After all, one of the core values of Agile is "Customer collaboration over contract negotiation". Predictability with respect to a team's velocity and sweating over charts sounds to me a lot like meeting the commitments in a contract.

From the team's perspective, predictability on behalf of the business people sounds completely counter to the Agile value of responding to change. Similarly, it would seem to run counter to the principle of welcoming changing requirements in order to maximize the competitive advantage of the business.

Think about situations where there are demands for predictable, level velocity. There mustn’t be much trust in those cases. Ask yourself why would that be? Are you promising or committing to delivering the whole world and falling short? Is what you’re delivering buggy and increasingly hard to maintain due to technical debt? Those are weeds in the garden of trust! If I hadn't shown concrete progress on a working system in 1992, I probably wouldn't have had the first contract extended let alone be with that client for nearly 6 years!

If you were able to simply say, “We’re going to break down this work as small as we can so we minimize surprises, build it as well as we possibly can so that we don’t have defects showing up in production, and work directly with you as often as we can so that we’re building what you actually need”, would that start creating trust? Possibly, but you also have to walk that walk and deliver on the promise.

The Reward

What does trust give you over predictability? When was the last time you heard about some ground-breaking innovation coming from a predictable team? When a group is trusted to deliver, they have the latitude to experiment with new approaches and technologies that can provide immense benefits. If business people are trusted by the development teams to know what's best, everyone can benefit from a sudden departure from the plan in order to seize a market opportunity.

If you look at 3M's 15% time, Atlassian's ShipIt Days and Shopify's Hack Days, there's a substantial investment in time that isn't necessarily related to the current work of the people involved. However, numerous innovations and even products emerged from those. The ubiquitous Post-it Notes originated as a 15% time project. Numerous internal tools and aspects of the core product at Shopify began life as Hack Days projects.

If those companies and many like them didn't trust their people, then there would have been a concern that the lazy employees would simply slack off. My experience is the complete opposite - people work harder during these sessions because they're following their passion! This is straight out of Dan Pink's work on motivation - trust enables autonomy and provides the time to achieve mastery.

Motivated people do better work, have more ideas and can actually deliver on these innovations. That's rather difficult to plan, schedule and track in a predictable way; you can't say, "OK, next Tuesday we're going to be innovative!"

Conclusion

Finally, if you embrace and build trust versus being predictable, your life and that of your colleagues can become much easier. You aren’t worried about an iteration that doesn’t go well for some reason. You aren’t worried when the business people change their minds about something because you trust that they know what they’re doing and they trust that you can adjust to the change. The business people aren't worried that you'll ship another steaming pile of software later than you promised.

All that everyone is worried about is doing the best possible work that you and your colleagues can do under the current circumstances.

If that sounds familiar, it should. It’s from Norm Kerth’s Retrospectives Prime Directive. It’s also the very basis for trust in the workplace.