Waterfall vs. RAD vs. RUP vs. Agile (Scrum, XP, etc.)

When I broke into this industry in the mid to late 1980's, it was pretty well a Waterfall world. By that I mean the minimal definition of waterfall from the top of page 2 of Winston Royce's seminal 1970 paper Managing the Development of Large Computer Systems that showed the flowing steps of Requirements, Analysis, Design, Coding, Testing, etc., and not the more comprehensive definition that followed in the rest of that paper. I was a good soldier, defining and designing everything up front... well, OK, sometimes I cheated and did some coding work because I was impatient or bored and needed to get moving forward, but generally I followed those steps.

In the early 1990's, along came this cool concept called Rapid Application Development (RAD). The consumers of the systems we were building were also impatient - they wanted value NOW, and were willing to have corners cut with the design and documentation. So, RAD espoused getting something - anything - into production RIGHT NOW!! That worked fine, except that you didn't spend much time on any of the activities that promoted sustainable development.

So, we went from too much process to too little.

At the same time, Object Oriented languages were coming into prominence and the development models created by Grady Booch, Ivar Jacobsen and James Rumbaugh (with contributions from Philippe Kruchten) were combined into a single process know as the Rational Unified Process. The RUP dealt with many of the shortcomings of both Waterfall and RAD - it was iterative allowing the constant feedback missing in Waterfall, and was heavily weighted towards architecture and risk management which was missing in RAD. Cool, now the industry could move forward with a standard process.

But there was a problem. It wasn't so much the process contained in the RUP but how it was packaged. The RUP was put together as a process framework, which was not unlike a large pantry containing all the ingredients you would need to make any number of meals. Unfortunately, people didn't seem (or didn't want?) to realize that you didn't need all of the ingredients for every project. The older Waterfall mentality was used when attempting to tailor the process to local needs, and when faced with paring down a large process people seem to err on the side of not paring enough.

What resulted was many projects using way too much process, which in turn resulted in more documentation, more design and less software. This was not what the RUP's creators had intended, not unlike what happened to Royce's definition of Waterfall.

So, in the late 90's, people tired of the heavyweight processes started rebelling against them. Along came what eventually became known as the Agile processes such as Extreme Programming, Scrum, Crystal, Feature Driven Development, etc. They focused on delivering value, and value to them was running software. Many of the Agile processes also added technical practices intended to provide the sustainability missing from RAD. They also overtly identified the people as being as important as the process itself.

Success stories abounded, not unlike what happened with each of the prior waves of processes. There were also, though, some failures. Those have, in my observation, increased over time as more teams started to use Agile processes. When looking at the causes of these failures, they often show that teams weren't using all of the practices in a given process, or they didn't use good technical practices at all and just adopted the management practices.

So now, is the pendulum swinging back towards more process?

There was an interesting post from David Anderson on the KanbanDev Yahoo Group this morning. In that post he states,

At this point, it is impossible to take the 'David' or 'Jeff" or 'Israel' factor out of the achievement of the high maturity.

This is in reference to increasing the maturity level of organizations beyond just "Agile", and to do so he believes that we must remove the human element not from the process itself, but from the stewardship of the process.

I think this is the correct direction - acknowledge, retain and leverage the human factors of executing the process, while at the same time removing the human element from managing it. Any team could be successful using XP if Kent Beck, Ron Jeffries, Jim Shore, etc. were running it. What we want to achieve, though, is a point where any team could be successful with (insert process name here) regardless of who is running it.

To answer the question, then, yes I believe that some more process is required. However, I think the amplitude if the pendulum swings is decreasing and eventually we're going to find a happy medium that's sustainable both within an enterprise setting, and over a long term.