From Traditional Project Development to Agile Project Management

SA,

The Story:

I've been working on a Middle-ware integration project using Microsoft BizTalk Server 2009 in a bank in Qatar, I've developed the project plan using my favorite iterative waterfall approach which worked fine with Software Development as I know the weakness of the theoretical waterfall model, but this what fits the market here best in the Middle-East as most of the clients want to know how much the full project will take time and cost them

The nature of this project was new to me, I a pproached with the same approach which pr oved that it is not working during the project implementation, I've met many times with my team leader to figure out how to manage this project the best, and we concluded that we should follow the agile methodology (which I realized that it is the most efficient way to manage integration project) and we finally agreed with the customer (after analyzing the project progress, costing, estimates vs. actual durati ons and delays) that he will be paying us by services, this way we can group tasks related to implement each services together as a use-case or a user story (according to the Scrum terminology) so we can have control over the implementation of services independently

Our development environment helped us as we where using Visual Studio Team System with Team Foundation Server that has a projec t template for Scrum

The Approach:

I did my research in Agile Scrum methodology and I ran into this nice PDF, "The Scrum Primer", and I read a very interesting paragraph that I wanted to share

Here we go:

Traditional Software Development

The traditional way to build software, used by companies big and small, is commonly known as “The Waterfall”. There are many variants, but it typically begins with a detailed planning phase, where the end product is carefully thought through, designed, and documented in great detail. The tasks necessary to execute the design are determined, and the work is planned using tools like Gantt charts and programs like Microsoft Project. The team arrives at an estimate of how long the project will take by adding up detailed estimates of the individual steps involved.

Once stakeholders have thoroughly reviewed the plan and provided their approvals, the team starts to build. Team members complete their specialized portion of the work, and then hand it off to others in production-line fashion. Once the work is complete, it is delivered to a Quality Assurance organization, which completes testing prior to the product reaching the customer.

Throughout the process, strict controls are placed on deviations from the plan, to ensure that what is produced is actually what was designed. This approach has strengths and weaknesses. Its great strength is that it is supremely logical: think before you build, write it all down, follow a plan, and keep everything as organized as possible. It has just one great weakness: humans are involved.
For example: this approach requires that the good ideas all come at the beginning of the development cycle, where they can be incorporated into the plan. But as we all know, good ideas appear spontaneously throughout the process – in the beginning, the middle, and sometimes even the day before launch, and a process that doesn’t permit change will stifle this innovation. With the Waterfall approach, a great idea late in the development cycle is not a gift, it’s a threat.

The Waterfall approach also places a great emphasis on writing things down as a primary method for communicating critical information. The very reasonable assumption is that if I can write down on paper as much as possible of what’s in my head, it will more reliably make it into the head of everyone else on the team; plus, if it’s on paper, there is tangible proof that I’ve done my job. The reality, though, is that most of the time, these highly detailed 50-page requirements documents just don’t get read. And that’s probably just as well, because when they do get read, the misunderstandings are often compounded. A written document is an incomplete abstraction of a picture I have in my head; when you read that document, you create yet another abstraction, which is now two steps away from what I’m really thinking of. It should come as no surprise that serious misunderstandings would occur.

Something else that happens when you have humans involved is the hands-on “aha” moment – the first time that you actually use the working product, and you immediately think of 20 ways you could have made it better. Unfortunately, these very valuable insights often come at the end of the development cycle, when changes are most difficult and disruptive – in other words, when doing the right thing is most expensive.

Humans also have a poor ability to predict the future. For example, the competition makes an announcement that wasn’t expected. Unanticipated technical problems crop up that force a change in direction. Furthermore, people tend to be particularly bad at planning things far into the future – guessing today how you’ll be spending your week eight months from now is something of a fallacy, and it’s been the downfall of many a Gantt chart.

In addition, the Waterfall also tends to foster an adversarial relationship between the team-members that are handing work off from one to the next. “He’s asking me to build something that’s not in the spec.” “She’s changing her mind about what she wants.” “I can’t be held responsible for something I don’t control.” And this gets us to another observation about the Waterfall – it’s not that much fun to work within. In fact, we’d go a step further and say that the Waterfall is a cause of great misery for the people who build products, and the resulting products fall well short of expressing the creativity, skill, and passion of their creators.

People aren’t robots, and a process that requires them to act like robots often results in unhappy people. A rigid, change-resistant process will also tend to produce mediocre products. Customers may get what they first ask for, but is it what they really want once they see the product begin to emerge? By gathering all the requirements up front and having them set in stone with little chance of change, the product is condemned to be only as good as the initial idea, instead of being the best it could be once the team knows more about the possibilities.

Many users of the Waterfall experience these shortcomings again and again, but it seems like such a logical approach, the natural reaction is to turn the blame inward: “If only we did it better, it would work” – if we just planned more, documented more, resisted change more, everything would work smoothly. Unfortunately, many teams find just the opposite: the harder they try, the worse it gets!

The Conclusion

I think this is piratical world experience, I feel the author pain as I'm facing the same pain everyday, but unfortunately most of the clients require the vendor to submit a proposal for the whole project with the estimated time and total cost in advance, they don't want to pay per feature as they need to have more control over the project due date and costing (they think this way will prevent the spiraling costs of the projects)

To be honest I know it is hard to convince the Agile Development approach to clients, which suits best for the projects that has frequent requirements changes (which happens most of the time), but hopefully cleints will change thier mindset regarding the classic project management by time

Till the next post, Leave you in peace

References and Further readings:
Scrum Primer
Agile Principles and Values, by Jeff Sutherland
MSF for Agile Software Development v5.0