Why "Waterfall" Stinks

I never claimed to be the smartest guy, and the older I get, the more I realize how much I don’t know. There are a lot of things I can’t seem to understand. I don’t understand how there are over 300 million Americans, some of them brilliant, and yet the best Presidential candidates we could come up with are “The Donald” and “Crooked Hillary”. I don’t understand Super PACs, I don’t understand chemistry, I don’t understand how to file taxes, and I don’t understand why enterprises still embrace the Waterfall Methodology.

The Waterfall Methodology has been around for a long time, dating back to the 70’s in fact. For most of the App Dev Era, Waterfall was the only known way to deliver. So is it fair to make a correlation between Waterfall Development and the statistic that nearly 70% of software projects fail? I’ll go out on a limb and say, “Heck yea!” Call me “Captain Obvious” (I’ve been called worse), but technology changes at such a rapid rate, how can one spend months, maybe even a year, writing monolithic Requirements, give those Requirements to a Dev Team, the Dev Team disappears for a year (with exception to the occasional Status Meeting where the same Dev is STILL working on a small feature for the third month in a row), and believe a successful product will be launched? Well 70% of the time, the launch is indeed unsuccessful, or worse, the Dev Team comes back with a product the business didn’t even want, or no longer needs after the 2 year SDLC.

All of the advice I can find online about Blogging tells me to not write about myself, so this next blurb is about the experience, not the person. I spent several years in the Government Contracting world, working with the major Integrators, supporting a considerable Command in Tampa, FL. I learned a thing or two. Both Contractors and Military alike have their frustrations about the current RFP (Request for Proposal…go ahead, call me “Captain Obvious” again.) process. Here is a little story: A General has a bright idea to purchase a technology that will help his Warfighter “down range”. The General pings the Feds in D.C. for budget approval (which depending on where the U.S. is in the fiscal year, can take awhile). If approval is given, the General has his team write the RFP Requirements, which also can take quite some time. Then the Requirements are released in the RFP to Industry, there are several months given to the Contractors to respond, and then the RFP’s are due. The Proposals are then painstakingly reviewed by Procurement and Buyers. Then a contract is awarded, the Vendor/Contractor disappears in to their “Development Hole” for a year, then brings back technology that is far from what the General wanted two years ago. Sounds a lot like Waterfall, doesn’t it? The General’s hands are tied, as legislation dictates there has to be a competitive bid, but for commercial and private organizations, who are free to make buying decisions at will, why on earth would you want to follow suit?

Agile is the way to go. I coined a phrase many years ago (or at least I think I own the copyright), “The only two things in life that do not lie are photographs and numbers”. A 70% failure rate speaks volumes, and arguing against a statistic becomes a moot point quickly. I also have heard that “trying the same thing over and over, expecting a different result is called insanity”. Continuing down the Waterfall path, without exploring Agile options sounds insane.

Agile is iterative, adaptive, and organic. Rather than having Business Analysts spend months gathering requirements, Agile Story Mapping will uncover a high-level overview of the application within 2 hours to a few days. Yes, I said “days”. Requirements gathering is then done, and we’re ready to go to work. User Stories (or “app features” …Captain Obvious is at it again) are entered in to a Backlog, the Stakeholders/Business collaborate with the Dev Team, prioritize items in that Backlog, and the Devs are free to start Dev’ing. Sprints allow small chunks of work to be sliced off at a time, and that work reaching full completion by the end of 2 weeks (or 3 if you’re a 3-week Sprint shop). That’s right, “completion”, as in pushed to Production, live and working software.

At the end of each Sprint is a demo, where Stakeholders actually get to see, use, and give feedback on live software. This method allows quick turns towards success, rather than steering the Waterfall battleship destined for doom. We then go back in to another Sprint Planning session where Stakeholders and the Dev Team prioritize User Stories for the next Sprint…rinse, wash, repeat. With this process, there is NO MORE, “I wanted “A”, and you deliver me “P”, two years later, and millions of dollars wasted!?” Those days are done, if one is willing to make a change.

So why is there a school of thought opposed to Agile? Well, no matter our title, what Fortune 500 company we work for, and how many years of experience/expertise we have; we are all still human…and to humans, change is scary. Change also means learning something new, and learning curves can be disruptive and take away from productivity. However, the old “risk/reward proposition” can alleviate this anxiety. The other major motivators for Agile resistance are: the necessity to surrender power, company culture, and accountability.

Surrendering power is contrary to all human nature. The Moors, The Persians, The Greeks, The Romans, The British, The Americans, and others were all thirsty for control, and so are some IT Managers. Telling a QA Manager that they no longer “own” all of the QA’s in the company is a painful blow. Agile teams are self-organizing, cross-functional, interdisciplinary, and destroy all department siloes (i.e. Development, QA, PMO, BA’s, etc., all segregated). Clearly Agile has engaged on many Agile Training/Coaching endeavors at enterprises across the country, and resistance from middle-management on giving up control, and having teams cross-pollenate, is fairly typical. And in defense of those managers who subscribe to team empowerment (giving their crew the freedom to experiment, grow, and even fail), sometimes it is the overall company culture (maybe dictated from executives) of micro-management, and endless processes to manage other processes, that inhibit team freedom and an innovative mindset/culture.

 Not all, but some Devs, coming from a Waterfall environment, despise Agile for the accountability that comes along with the process. In the “old days” (or current for some orgs), a 591 page Requirements document was handed out, responsibilities delegated across the Dev Team, and with the exception of the occasional “Status Meeting”, no one really held the Devs accountable for what was getting done. It was very easy to “hide” in a cubicle, and work on a small effort for months. There was nothing relative to compare up against, dictating on how long a task should take. In Agile, everything is relevant (I wrote a term paper in college before receiving my Philosophy Minor, titled, “Everything is Relevant” … just thought I’d throw that out there). Stories/Backlog Items/App Features are scored by Story Points in Sprint Planning, and given a weighted measurement using the Fibonacci Sequence (1, 2, 3, 5, 8, 13, 20, 40, 100). Each Story is scored based on the relevancy on how long a similar effort took in the past.  A Dev in an Agile environment will only take on enough Stories/Backlog Items that he/she knows they can complete in one day’s time. The next morning, in the Stand Up (a hated term by many Waterfallists), the Dev will explain if they completed the allotted tasks attached to a Story, and if not, what the impediment was (in which the ScrumMaster will remove that impediment). This type of accountability is very scary to one who has not had to explain their efforts from the day before.

There is no “one thing” in life that is for everyone, and Agile certainly isn’t for every organization. Scrum is not a “one size fits all”, however, if we all take a hard look where we came from, how we have been doing things (the Waterfall way), and where we have fallen short; then we all would be insane to not admit that there is an issue with failures in software development. I work for an Agile company (uh oh, Captain Obvious peeks his head out again…duh, the company name is “Clearly Agile”), live, eat, and breathe the Scrum Process every day, and despite noise from Agile critics, we are focusing on Working Software over Documentation, Teams and Individuals over Bureaucratic Processes and Tools, Customer Collaboration over Contracts and Negotiation, and Responding to Change over Following a “Plan” (a few abbreviated points from the Agile Manifesto).

Moral of the story: give Agile a try.

I appreciate you reading, hope you enjoyed, and encourage you to take a ride on the Agile Transformation bus.


Captain Obvious