It's not all Coding, What's the Process?: Waterfall, Agile, Lean, Design...

I have been reading a book recently called "The Lean Startup". When I first started reading it, I realized they were taking Test Driven Development processes and applying them at a higher level to actual product development. The book also described using agile approaches to figuring out what an overall startup should be doing. There are number of cases described in the book that cover aspects of different businesses. From how a startup can use the test, build, learn model to quickly evolve a product or entire business. It also describes large companies building departments that act like a startup with an entrepreneurial management team. I found these descriptions interesting because of how directly they correlated to modern software development paradigms. 

I have worked in IT for over 30 years, and in that time, the majority of projects I have worked on followed a waterfall approach. A business analyst or IT manager will talk to the business unit and put together a set of requirements. The developer and the analyst would work on putting together a functional spec and a technical spec. There would be some back and forth between IT and the business, and eventually there would a be a kickoff meeting, and the project would get started. The analyst would act as the go between to the business unit, and there would be periodic update meetings. Then finally you would have the roll out.

I have done a number of projects this way, and some of them have been successes, and others have been abysmal failures. The successes are great, but often they would still need some tweaking to be truly successful. The failures are extremely frustrating. I have written a number of applications that were well designed and met the requirements laid out in the specs perfectly, but when they get released the end user finds no value in the product and refuses to use it. The most common cause of this is in the level of abstraction and not understanding the business process or the end user.

Often times the business analyst will work with the customer, and get an idea in their head of how the product should work to meet their needs that isn't really what the customer wants. The customer may not understand the solution being proposed well enough to spot that it isn't going to do what they want in the requirements phase and they sign off anyway. Developers can also have similar blinders thinking they know better how their application should work, but they usually get more information from the analyst than they do the customer. In the end, the product gets delivered and the customer looks at it and says that it isn't what they wanted at all. Months of development can be thrown down the drain. 

If you apply another layer of abstraction to the development process, things can get even more out of hand. In many of my projects, the analyst would talk to stakeholders in the business and define the spec, and at the end of the development cycle the stakeholders would be really happy with the product and we would roll it out. At this point, the real users of the application would complain that we didn't understand their process at all, and that the application is worthless to them. This can be really embarrassing for the managers in the business unit, and it is really frustrating from the IT side. This can be a really bad situation for an internal product, but what about when it is for an external customer?

I have never worked for a company where software was the primary product, but I have developed applications that are used by the customers of our products. When you ad another layer of abstraction and develop a product that a customer of the business is going to use, it can really hurt the reputation of the company when you run into issues where the business doesn't understand the customers needs. There have been so many times that I would roll out an application and end up talking to the end user where they would ask, why we didn't talk to them while building the software? They would have been more than happy to tell us what they needed and how their process works. 

I haven't actually worked on a team large enough to follow a formal agile process like SCRUM. Usually I am the sole developer or on a team of two developers, a business analyst, and a manager. In a small group like this, the methodology of SCRUM or KANBAN don't work, but that doesn't mean you can follow an agile process. Those methodologies are used to push larger teams in an agile direction and keep them managed. Being Agile at the core means you are working directly with the end user, and you are making corrections to your assumptions along the development cycle instead of at the end. 

In the "Lean Startup" book, the author is constantly referring to creating an MVP, minimum viable product. The idea is simply this, you have a hypothesis of how the product should work, you design a test for the basic functionality, you build the smallest solution to satisfy the test, you give it to the customer, and you learn from the feedback. Then you repeat the process over and over until you have a usable product. The MVP doesn't even have to have any code or automation. It could just be a mock up of a web form or a PowerPoint presentation. The point is to get something tangible in front of the user as quickly as possible and adjust the project based on their feedback. 

As developers, we can look at a spec and visualize what the software will look like. The end users can't do that. They don't need a polished product to understand the vision, but they do need something they can see and touch. I am often amazed at how I will describe a particularly elegant piece of code, and how it took me weeks to come up with a great solution to a logic problem, just to receive blank stares. Then a minute later I will show how I changed the color of a drop down list to match the design better and the customer will be amazed at my programming talent. The end user doesn't understand the developer and the developer doesn't get the end user, but by incrementally testing and building, you can find the common ground that ends up in an accepted and usable product. 

One of the common complaints I hear managers make is that Agile processes lead to scope creep. Given that you don't have as rigid of a scope as you do in a waterfall project this is a reasonable concern. It is hard to budget time and money for a project whose direction can change every week. Just because you are following an agile method, it doesn't mean you aren't managing the project. The project management still has to be done. You have to make sure you have the right stakeholders using the product and giving the feedback. As the project is going, you need to test, build, and learn from the process itself. In the end, chances are pretty good that the project will be better accepted by the end user.