I have helped small startups and enterprises such as AWS, Brighton Healthplan solutions, Gigaspaces etc. I had built my consulting business on an agile mindset. I have failed, succeeded and learnt my own lessons. In my experience, not many understand the agile mindset inherently, nor are ready organizationally for agile development.. However, here are my thoughts on how you could look at agile development, whether your organization is ready for it, what you need to do to be prepared etc.

Who should read this

I never cease to be amazed – even after all these years of agile being around in one form or another.. Enterprises still consult me on what exactly agile development is, how exactly they could implement agile methodologies within their organizations, how scaled agile can solve all their problems, how they would deliver software reliably on time and on budget using agile etc..

I actually didn’t think I would ever have to write a whole post about this.

This is mostly a post for managers were trying to get a better handle on Agile development Technologies and how their team could get better at development and delivery of software using agile methodologies.

From what I have noticed most of the managers are, as expected, a little out of touch from daily development.  They have been there, done that, have delivered software reliably in their past without even using agile methodologies;  so what exactly does this newfangled way help them and their team? More importantly how would it help them deliver on time and on budget?

I am not going to attempt to write a book on Agile development as there are WAY too many already.

Rather this is just a quick read for managers, product owners, non technical entrepreneurs etc (i.e people that really hire me to work for them) to understand what it is, how to use it, whether agile would benefit them, how it will benefit them (if at all) and most importantly what the risk profile of using agile development is.

Basic stuff first – What Is Agile Software Development?

It’s not a silver bullet, a specific formula, some specific technique or anything of that sort. It is simply an umbrella term that is used by various software development methodologies.. (various methodologies to develop software in an iterative and incremental manner).

You’ve probably already heard some of these – e.g Extreme Programming (XP), Scrum, Crystal, Lean Development, DSDM, and Feature-Driven Development etc.

These are all talking about the same thing – agile software development.

At some point in time, google “Agile manifesto” and read up on it.. It’s a quick read and here’s the basic stuff you need to understand about that manifesto

“Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan”

You can choose any of the agile methodologies that you want – the underlying concept and principles remain the same… developing software collaboratively, responding to and embracing change + frequent feedback, preferring working software over boat loads of documentation.

Note that agile development is an approach that’s nowadays used even beyond software development..

How’s that possible?

Because agile development is more about a mindset and approach to developing “stuff” – whether it is software, a market, a product etc..

Just let that sink in first.. Then proceed.

Remembering waterfall development methodology

So, as an old timer, you are surely used to waterfall methodologies. That’s what we used to do before…

We’d get marching orders to develop *something*.. We would get together and start the process to uncover and gather all the possible requirements.

This would give us reams of project requirements documentation (about what needs to be done).

Then we would go ahead with the analysis – therefore producing our models, actors, schemas, business rules etc.

Then we would also start putting the design together. We would break this into high level and low level designs and would also get the system architecture.

Finally we would start development of the product – i.e the actual coding.

Then we would go through a boat load of testing and validation..

We would produce reams of product documentation and support the system etc..

That worked fine for us.. That worked well for the market back then. Even with inherent issues and the massive project failure rates, we didn’t know any better.

Then the business would see our work, validate that the product meets all the requirements we had been given, then go to market with what they intended to… 6-12 months ago (sometimes years ago). More often than not, the market didn’t stay where it was when the requirements were written.. The market had already moved on.. Potential customers of the product now wanted something more or something different..

And the “success” of the project was.. Well.. dubious at best.

That worked.. Eh.. fine.. So why change now?

What are the benefits of agile development

Markets and opportunities are coming up fast, and competitors are coming up faster..

These days it has become a “you snooze, you lose” game. Customers have too many options available to them.. There are too many competitors and even small, agile product companies are giving the big guys a run for their money.

The challenge back in the day was that the software development team was almost never in direct touch with the business team. The business team was always in touch with the end customer, but the dev team not being in touch with the business, was, in effect, never really building a system / product directly for the customer.

So, let’s think about this.. The product management team might have been talking to customers that would “some day, potentially” buy a product that they envisioned. They spoke to these potential customers, pitched them their product idea, got an agreement and then the software development process took over.. Using waterfall…

In other words.. Now the product management folks didn’t have any real good reason to get back in touch with or continue to stay in daily/weekly touch with the end customers.. Why? Cos they have nothing to show..

Nothing to collaborate on..  Nothing to get feedback on.. The only feedback they could get from the customer would be .. well, 6-12 months away.

Even if the business was in regular touch with the customers, and found that the requirements changed due to some reason (whatever be that reason).. Getting those changes done in the middle of a large project was.. Well.. arduous.

That’s the problem – customer development was hard and only kept getting harder. You couldn’t go to a set of ideal customers, show and tell what your idea was, get their nods and get back to them quickly with some progress.

Meanwhile, the customer either moved on or put that project in the back of their minds.

That had to change.. And so it did.

Why do we use agile methodology

Experienced software development professionals saw the issues in this kind of development approach. So did the business.

What if there was a way to accelerate the delivery of initial business value?

What if there was a way to show constant progress & get feedback on that progress?

We all know that as soon as a customer sees their vision come to life in any shape or form (i.e. tangible), their minds start working faster.. They start thinking about how else the system could solve their problems.. What would be a better way to solve their problem etc.. So, what if there was a way to get feedback and be able to incorporate those changes immediately instead of having to put a huge risk burden?

What if we didn’t have to collect reams of requirements, analysis and design documentation for months before starting the project? What if we could talk about a product at a high level, figure out the bare minimum to get started and show value in that week itself?

Of course, we don’t want to throw planning out the window, but what if planning was done in an iterative way because we got feedback on business value delivered iteratively and continuously?

Would that result in a software system that much better addresses the business and customer needs?

This is not to say that it was a brand new concept.. Even back then, extreme programming was already into the picture. Gosh, I remember those days when 12 of us were locked up in a room working on the E*Trade project.. or those days of 7 stinky folks working in a teeny, tiny room in the LIC Citibank building working on their Citi Private Banking Portal.

That’s how agile was dreamt of and started being adopted.

OK, So what is the difference between agile and waterfall again?

If nothing, keep at least these few things in mind

Agile has releases and iterations.

Each release usually has many iterations.. And each iteration itself is like a micro project.

So, what you did was to take a giant waterfall project, break it down into many, many smaller chunks of features and releases.. Then you took each release and broke it down into many iterations..

You put work items, enhancement requests, bugs, features etc into an iteration. After an iteration, you get feedback, collect those and prioritize for the next iteration(s).

And most importantly – each such iteration produced tested, working software.

Is this cruel to the development team? A little 🙂 yes.. But it’s in the best interest of the overall team (ah.. That includes business as well, in case you are wondering)

So, advantages of agile methodology over waterfall model?

Quite a few actually ! You can google for this and get a ton of information.. But keep these few things in mind.. (and don’t get carried away with the purists’ propaganda)

Working, tested software each time

That’s pretty much a hallmark of agile methodology. A giant project broken down into baby steps. Success shown in baby steps.

Working, tested software in each such baby step.

Think about your risks – you just brought it down to a single iteration.

Does this mean that there are no bugs or errors in each iteration/release? Sure there is.. But instead of discovering a giant bug at the end of 12 months that makes the development team do heart surgery to fix it..

You find those bugs in smaller chunks, address it immediately, in smaller chunks.

It delivers value, constantly

Why? Because your development team is focused on delivering a feature (or features) to you in every iteration and release.

You see tangible business value in every release.

You measure progress and assess its value in every iteration or release.

That’s the point of it.. Break your risks down to smallest chunks possible..

It helps you plan, constantly

As you are seeing incremental progress and are validating your idea(s), this allows you to plan iteratively.

It allows your development team to plan iteratively instead of having to sit through all the planning way up front in the project.

It allows the business to plan iteratively about their activities and customer development. Is the customer asking for something else or something different now? Well, no problem. In waterfall, you’d find out the changed customer needs at the end of 6-12 months.. In agile, you find that out in 2-3 weeks.

And you incorporate those changes in the next releases.

See the difference?

Discovering, learning, collaborating

Developers develop for end users, customers. In waterfall, we had customers in one building, business (product management, marketing etc) in another and developers in yet another building.

If you really think about it – solving that business problem with the help of software was the goal for the *entire* team. So why were each sitting in their own silos?

That’s what agile software development allows you to break down. Breaking down those walls and really allows you to collaborate across teams.

To achieve the same goal.. While collaborating.. As a team.

Instead of the customer having the entire onus of figuring out everything up front, this allows the broader team to discover features and functionality together… as a team.

The development team can come up with various ideas of solving a problem because they have worked across verticals and business problems.

The business can advise the development team that the feature that they are saying will cost $5,000 to build might actually not be such an important one.

The development team can constantly provide feedback to the business with the costs associated with their choices..

And so on..

Real collaboration to produce business value within a reasonable budget.

Disadvantages of agile methodology? Risks?

Everything has risks. Agile development does as well.

Experts say the core elements of agile development such as frequent customer interaction to refine scope, frequent delivery incremental software capabilities, product backlog can be effective in some software projects but can be ineffective or suboptimal in others.

If your team or your organization is not ready for agile development, then you are taking on a risk.

If you are going the route of “agile development” cos that’s the latest fad.. You are at risk

If you have been tasked by your boss to become an “agile development” enterprise.. Well.. depending on how you responded to your boss.. You *might* be at risk.

In other words, your mileage may vary (YMMV).

The good thing is … you can actually evaluate whether you are actually ready for agile or not.

And hint.. You are not ready if you just had your entire team sit through some darn scaled agile course.. No, sorry to say that you wasted your money.

Agile needs commitment and it needs a cultural change..

And that change needs a strong sponsor – usually at the top of an organization. If you don’t have sponsorship, don’t try to boil the ocean. Start with a small committed team on a small, low risk project… show progress, get buy in for the next bigger project.. And that’s how you bring about that change in your organization.

How to decide if your team is ready for agile development?

There are several considerations when deciding which methodology is best for a specific project and a specific organization.

First thing to ask yourself would be, how does my team communicate?

Does your team really have open and face-to-face communications?

Even if it is not face-to-face communications (i.e. remote, VOIP based communications), does your team discuss pros, cons and everything else in between, openly?

You might be tempted to answer this question flippantly by saying “yes of course!”.

Stop right there and ask yourself again – does your team really do this?

If your team is not the best at communication then agile is not for you.

Keep in mind that agile (e.g. Scrum) has only 3 roles – product owner, scrum master, development team member.

This means that the overall team is going to communicate everyday, multiple times a day. They are going to discuss issues and development approaches, pros and cons etc.. each day. They are going to white board whenever necessary.

Be prepared to break down the barriers of formal communication structures. Be prepared to forgo giant documents to communicate between team members. That’s an anti-pattern and is simply not going to work in agile.

You can be a distributed agile development team (if that’s your only option) but that doesn’t negate the need for openly flowing communications, the need for stand up meetings (short), the need for discussions on a daily basis, the need to collaborate between development, support, marketing, product management etc.

Next, how is your team organized and structured?

Most enterprises I have worked with, are organized in a very top-down way. It’s not a bad thing – a top-down approach has its own benefits as well.

But don’t try to fit agile within a “top-down heavy” kind of organization.

Agile development needs cross-functional teams.

Understand that.

Being agile means that you are going to need a team of developers, testers, scrum managers, business folks, user-experience developers, user interface developers & hopefully even marketing people all together in one room.

Can you do that? Can you truly get all these folks in one room to discuss and develop a product iteratively? If you cannot, agile development is going to be really painful for you.

Look at your team’s composition. Most organizations have a pyramid structure with senior and junior team members. Keep that in mind and understand that very well.

As I mentioned, agile teams will have just *developers*. So, if you have senior people mixed with very junior people, you are going to face problems. The team will only be as fast as the slowest developer. I believe in the book “Mythical man month” it is reported that the productivity difference between best and worst programmers is to the tune of 10:1.

This doesn’t mean that junior developers are a no-no in an agile development team. It just means that you need to be cognizant of your team mix.

Again, go back to the 3 roles in scrum/agile. If your organization cannot thrive in such a flat structure, you are going to fail. The managers are no longer managers, the lead developers are no longer lead developers, the rockstars are no longer rockstars.

One of the benefits of agile development teams is that silos of information and silos of skill sets are reduced and bus factors are increased. Your frontend dev will learn from your backend dev and vice versa. You need to ensure that your organization can thrive in those situations.

Can you really handle frequent delivery of working software?

No, it’s not a trick question.

In my experience with most organizations, especially larger ones, I have found that many (if not all) are not ready nor have the bandwidth to accept frequent delivery of working, tested software.

Why? Simply because they don’t have the bandwidth nor the team needed on their end to validate what we have delivered.

This is demotivating for the agile team. If you are not truly ready to work hand in hand with your agile development team – don’t take the agile route.

Agile works best when feedback is frequent. If business and possibly customers cannot provide feedback on a frequent basis, there’s no point in doing agile development.

Think about it. Basically when you are developing a product, there are 3 main functions – developing it, operating it, selling it.

What good is it if these three sets of people are not on the same page?

What good would it do anyone if development produces a release and marketing cannot give feedback – “hey, this is going to help us sell” or “hey, our customers won’t understand this/that”?

What good would it do anyone if operations folks cannot give feedback like “wait a minute, if I get a call from a customer that X or Y is not working, where do I go look for it?” or “How do I answer question A from a customer when they call me?”

You get the picture?

Are you ready to measure progress by features and working software rather than phases?

In my experience, even if an organization starts off by saying that they are ready to work in an agile manner, they truly are not.

Most organizations (larger ones) need reams of documents, progress reports, requirements traceability matrices, etc to see and report progress.

Agile development works around features and delivery of working features.

Can you live with that? If you cannot, don’t go into agile development.

I’ll tell you this. Many developers initially look at agile development as a way for managers to “micromanage”. They see it as a way for managers to say “what are you doing on a daily basis?”

Why? Because they are used to meeting with managers, probably, once a week to discuss progress. In agile, they will be meeting with them on a daily basis.

This is also where managers need to have a fundamental change in their behavior and something where I myself have committed this crime. Instead of barraging the team about why a feature took longer to implement, a manager needs to work towards removing obstacles from the team’s path. The manager would need to spend this time with the developers understanding why the delay occurred.. So that they can address removing those obstacles for the next iteration.

Your organization needs to be able to measure progress in terms of feature development, not in terms of gantt charts.

Do you need to the ability to change directions during the project?

If you really don’t need the flexibility of changing directions of a project while the development is underway, don’t necessarily jump into the agile bandwagon.

If the product that you are developing has set features, set functionality, will not change (yes, there are MANY such projects) then don’t bother with agile. Waterfall will work just fine and will will actually be easier.

Are you comfortable with “just enough”, iterative and continuous planning?

As I mentioned before – that’s a hallmark of agile development. If you cannot handle the fact that the entire product and project will not be planned for up front, then don’t do agile development.

If you are not comfortable proceeding with short, iterative planning, intermediate steps of code hardening, adjusting plans based on prior learnings.. Then don’t go into agile development.

This is where upper management has most of their issues.

When can we say that the project is “done”? When can we promise what to customers? I need a better handle on the budget.. How do I do that?

Yes, that’s understandable as well. It’s hard to get upper management comfortable when you tell them that you are going to keep iterating on the project as long as the customer continues to identify high priority/high value work.

You are going to have to come up with some ways to allow project tracking and project reporting. Your development team is going to have to come up with a general idea of a timeframe that they would need to implement the product. That could very well be a range between 6-9 months.. (or whatever they feel comfortable with).

At this point, you can ask management to budget for at least 6 months and get an agreement that at the end of these 6 months, you could be asking for more budget if you have been demonstrating quantifiable value.

Allow upper management the satisfaction that in agile development, they can also yank the budget much easier if customer stops seeing the initial perceived value in the project or product. Tell them that instead of canning a project and its budget AFTER completing it in 6-12 months, they have the ability to plan iteratively and based on customer feedback, have the ability to can it in the 2nd month itself.

Give management that – they deserve to define their own boundaries of risk and be comfortable with it.

Do keep in mind that unless your management believes that your organization’s main goal is to delight customers (rather than increase shareholder value), it is, ultimately, a difficult sale.

Of course, your mileage may vary 🙂

Good luck !


Your thoughts?