Waterfall vs. RAD vs. RUP vs. Agile (Scrum, XP, etc.)

When I broke into this industry in the mid to late 1980's, it was pretty well a Waterfall world. By that I mean the minimal definition of waterfall from the top of page 2 of Winston Royce's seminal 1970 paper Managing the Development of Large Computer Systems that showed the flowing steps of Requirements, Analysis, Design, Coding, Testing, etc., and not the more comprehensive definition that followed in the rest of that paper. I was a good soldier, defining and designing everything up front... well, OK, sometimes I cheated and did some coding work because I was impatient or bored and needed to get moving forward, but generally I followed those steps.

In the early 1990's, along came this cool concept called Rapid Application Development (RAD). The consumers of the systems we were building were also impatient - they wanted value NOW, and were willing to have corners cut with the design and documentation. So, RAD espoused getting something - anything - into production RIGHT NOW!! That worked fine, except that you didn't spend much time on any of the activities that promoted sustainable development.

So, we went from too much process to too little.

At the same time, Object Oriented languages were coming into prominence and the development models created by Grady Booch, Ivar Jacobsen and James Rumbaugh (with contributions from Philippe Kruchten) were combined into a single process know as the Rational Unified Process. The RUP dealt with many of the shortcomings of both Waterfall and RAD - it was iterative allowing the constant feedback missing in Waterfall, and was heavily weighted towards architecture and risk management which was missing in RAD. Cool, now the industry could move forward with a standard process.

But there was a problem. It wasn't so much the process contained in the RUP but how it was packaged. The RUP was put together as a process framework, which was not unlike a large pantry containing all the ingredients you would need to make any number of meals. Unfortunately, people didn't seem (or didn't want?) to realize that you didn't need all of the ingredients for every project. The older Waterfall mentality was used when attempting to tailor the process to local needs, and when faced with paring down a large process people seem to err on the side of not paring enough.

What resulted was many projects using way too much process, which in turn resulted in more documentation, more design and less software. This was not what the RUP's creators had intended, not unlike what happened to Royce's definition of Waterfall.

So, in the late 90's, people tired of the heavyweight processes started rebelling against them. Along came what eventually became known as the Agile processes such as Extreme Programming, Scrum, Crystal, Feature Driven Development, etc. They focused on delivering value, and value to them was running software. Many of the Agile processes also added technical practices intended to provide the sustainability missing from RAD. They also overtly identified the people as being as important as the process itself.

Success stories abounded, not unlike what happened with each of the prior waves of processes. There were also, though, some failures. Those have, in my observation, increased over time as more teams started to use Agile processes. When looking at the causes of these failures, they often show that teams weren't using all of the practices in a given process, or they didn't use good technical practices at all and just adopted the management practices.

So now, is the pendulum swinging back towards more process?

There was an interesting post from David Anderson on the KanbanDev Yahoo Group this morning. In that post he states,
At this point, it is impossible to take the 'David' or 'Jeff" or 'Israel' factor out of the achievement of the high maturity.
This is in reference to increasing the maturity level of organizations beyond just "Agile", and to do so he believes that we must remove the human element not from the process itself, but from the stewardship of the process.

I think this is the correct direction - acknowledge, retain and leverage the human factors of executing the process, while at the same time removing the human element from managing it. Any team could be successful using XP if Kent Beck, Ron Jeffries, Jim Shore, etc. were running it. What we want to achieve, though, is a point where any team could be successful with (insert process name here) regardless of who is running it.

To answer the question, then, yes I believe that some more process is required. However, I think the amplitude if the pendulum swings is decreasing and eventually we're going to find a happy medium that's sustainable both within an enterprise setting, and over a long term.

Comments

Anonymous said…
What I love about Royce's article is that he completely discredits the Waterfall approach ("In my experience, however, the simpler
method (ie. waterfall) has never worked on large software development efforts ...").

That this was written in August 1970 and we are only in recent years really grappling with the best way to develop software is the staggering truth that sends a shiver down my spine!
Dave Rooney said…
In August 1970, I was just about to enter Kindergarten. I'm now a "grizzled veteran" of over 20 years in the industry, and we're still dealing with the issues caused by people stopping after the top of page 2 in Royce's paper!!

I'm coaching at a client right now where they overtly use Waterfall as what they consider to be a viable approach to delivering software, and they're feeling immense pain (hence my coaching engagement!).

"Agile" as we know it now will continue to evolve, and like OO and RDBMD it will take almost a full generation to become the norm rather than the exception.
Anonymous said…
The odd thing is that it is never about the process or the methodology ... its the people that make or break the project. I could use a waterfall and have a project success, I could use Agile and have a failure.
Unknown said…
Good article, interesting read.

What I have observed over time now is that no one process/framework applies anymore and you alluded to that in your article. Today's fast-paced times call for more of a blended-approach as opposed to a single framework adoption. One should try to evaluate and pick the best of all the different framework and adopt its use...but more importantly constantly re-evaluate and revise the approach/paradigm for it s applicability otherwise in spite of best intentions, desired results would not be achieved.

Thanks again!
Syth said…
It is nothing but the time of development is the matter in concern. Waterfall model is good and RAD is good as well. If I need to develop an iphone application , if I selected water fall model, and an implementation time of two months, on the very next week of development, there may another trend come in developing the application due to the pace of technology. Those who started development on facebook app or mozilla plugin, they already changed the style of implementation in a three months interval time. So, if any development which is technology oriented, rapidness and agility is needed anyway.
Anonymous said…
"Waterfall model is good and RAD is good as well". I do agree, but if I were to choose among the various well applied approaches, I would choose (in most cases) agile.
I have applied agility successfully in a couple of projects. Notice I didn't mention I applied Scrum, XP, Crystal or other agile methodology. What I did, it was to adapt the Unified Process according to the needs I foresaw, adapt, and embrace some agile practices (mainly from XP, Scrum, and Craig Larman's books). That is why I prefer saying I run agile projects instead of saying I run Scrum or XP or any other fancy thing. Many "agile adopters" even forget or don't know of the existence of the Agile Manifiesto, which should be the core of any so called agile methodology. I believe the problem with agile processes is when are applied because "adopters" think it is the current desirable fashion and trend in the software industry, or any other misleading issue. For being an agile adopter, the mindset of people in the team (and in the project) has to change, that's the hardest part.
Of course there are other variables involved in delivering a successful project. But in most my projects, Waterfall usually works for larger project with big budget and ample timeline. Otherwise, be Agile!
Royston said…
Thanks for this article, and good replies too!
Casey Dale said…
An agile process tends to focus on iterations, and client feedback, to allow for the inevitabilty of changing requirements whereas a waterfall process tries to define all requirements up front, and tends to be inflexible to changing requirements. You can learn more about agile and scrum by referring to some free resouces (http://www.scrumstudy.com/free-resources.asp) provided by scrumstudy or by attending any agile scrum certification courses. I would personally suggest Agile Expert Certified course or Scrum Master Certification to you.