Japanesse companies, Toyota in particular, took hold of American management thought and applied it to their manufacturing processes better than we did. The wisest method was the concept of "continuous improvement", which is to say they constantly are going back and refactoring portions of their manufacturing process to reduce waste, increase throughput, and so forth.
It's very similar to how you might optimize a program. You find a hotspot you think might need improvement and you spend a portion of time improving it. Measure results before and after. Rinse and repeat.
I'm sure there are a lot of small things they learned along the way but the simple idea of continuously improving their processes has kept them on top.
Truly enlightened Agile development is not doing Scrum, XP, etc., it is being wise enough to look at your process and see how it can be improved. Pick and choose different ideas that will fix the parts that need to be fixed, but never be fully satisfied with what you are doing.
The real failure in any project management philosophy is believing that it is the "One True Way" that all things must be done for all teams everywhere all the time.
I have to disagree here, as Japanese culture is generally quite strict about rules and processes. An average japanese company is very far from what you would even dream to call agile. Rules and conventions about face-time, not being able to disagree with your superiors (basically anyone who entered the company before you) and valuing facade over results seriously hinders efficiency and trying out new and brave things.
However, I'm constantly surprised by the great ideas the Japanese get despite all that stiffness in the society.
The rules may be strict; my point was, they don't do things just because they did it that way all the time. If they see something better, they bow before the change.
If it comes from the management, I'm impressed even more.
This does not apply in general. A lot of things in the Japanese society are done in old, even arcane ways just because they always have been done like that. The society is not built on bowing before change but on to reach some kind of harmony. Changes might be accepted eventually, but getting there is the hard part.
Very good points. At the MagicRuby conference this year, @pragdave said "Agile is not a noun." He went on to say something like "Agile is not what you do, it's how you do it." (I probably mis-quoted that.)
It made a big impact on me and made me realize why I had been so disturbed by how the company (that I worked for then) was handling project management. Everything was locked down. Everything had deadlines. Every process was locked in stone and decided on by the manager. Heck, they even said "This is your deadline. Tell me how many people you need to make it happen." ... And when I told them, they failed to get those people on time and still demanded on the deadline. (I left the company about halfway to the deadline as I had found a better position elsewhere, so I have no idea how they are doing on that deadline, but I can't imagine they are going to make it.)
Ironically, Ken Schwaber has made a lot of money off of "Agile" the product. Becoming a "Certified Scrum Master" will cost you about $1k. You get that by taking a class with a "Certified Scrum Trainer" who spends quite a bit per year (payable to The Scrum Alliance) to keep that certification. On my planet, we call this a "pyramid scheme". I understand that this is an ad hominem attack; nevertheless, when talking "agile" (either big A or little a) with the Scrum dudes, one would do well to consider the source.
A pyramid scheme is a non-sustainable business model that involves promising
participants payment, services or ideals, primarily for enrolling other people
into the scheme or training them to take part, rather than supplying any real
investment or sale of products or services to the public.
It's absolutely true that Ken makes money recruiting people to use Scrum, and he has in turn anointed other trainers to teach Scrum and they make their money recruiting people to use Scrum. However, the vast, vast majority of the people who have taken Scrum courses use them to ship software, either directly as employees or indirectly as consultants. Scrum is not a pyramid scheme.
Now let's talk about whether Ken sells a product. Of course he does!!!! Ken sells training, and it is perfectly valid to say that his training is a product as are his books. But that is not the same thing as saying that Scrum itself is a product. You can't just buy the training and expect results. Scrum Training != Scrum.
This is trivially true. Let's compare to programming. I have a Java programming team. I can "buy" the Ruby product, but to experience change, I have to embrace the Ruby Way, and that goes beyond simply installing the Ruby interpreter. Ruby the interpreter is a product, the Ruby Way is not a product (love it or hate it, I'm sure you agree).
What sounds fishy is that the Scrum Trainer has to keep paying money to keep his certification.
Certifications are a joke when it comes to programming anyway, much less when it comes to programming methodologies. What if someone came up to you and told you that they are a certified Ruby Way Master?
Yet it seems like certification matters to some people (otherwise there would be no point in paying someone money to retain it). And it's obviously in the interest of the Scrum Alliance to make people believe that it matters because it lets them sell it over and over again to the Scrum Trainers. So it seems like they build their hierarchy to put themselves in a position where they profit from the lie that certification matters.
Moreover, they also added another couple of layers of people who also want people to believe that certification matters (Trainers and Masters because they want to make back the money they paid for their cert). While this is not exactly a pyramid scheme, there are some similarities in that money gets passed up and people are trying to convince everyone of a lie.
I think certifications might make sense for things where it is impossible for other people to tell whether someone knows their stuff. I don't think Agile is that hard conceptually.
I also think re-certification makes sense in an area where the state of the art is constantly advancing, so that if someone is certified you know that they are really up to date with the latest developments. I don't think Agile changes that much.
I'm troubled by certification as well, although that is somewhat orthogonal to whether I value Ken's training... I personally took the training to improve myself. The "certification" didn't mean much to me one way or the other.
In fact, I would tie it back to what I said above and suggets that the certificate is a product just as a the training is a product, but the certificate is not the training and the training is not the practice. If the training is one step removed from Scrum, the certificate is two steps removed from Scrum!
This. Pyramid schemes are necessarily unsustainable, because everyone who gets involved is solely relying on a windfall for getting still more people involved, and while the world is overbreeding, it's not at the rate they need. As soon as you find end users who will see some kind of net value even after growth stops, you have something that can sustain itself indefinitely. At worst it could be a bubble of investment in training capacity, if they overshot the market size.
If you have to have everything on your requirements list, you can't be agile
If you need to plan beyond the next sprint with any degree of accuracy, you're not agile
If you think you can "fix" an iteration by adding more people, you're not agile
If you are prepared to change the end date of an iteration to "fit something in", you're not agile
If you have to give a fixed date for delivery - it's very, very difficult to be agile.
90% of companies that are committing to Agile need to read this list, be honest with themselves, and choose a different process because there is something on this list they are not ready to give up.
project management is about getting business results, not sticking to a certain process.
we can't wave a magic wand and get a project done with features, quality, and budget at a certain date that we choose. we can use project management to understand what we can do realistically.
"agile vs. Agile" you mix the cases in this :-)
I wasn't "there" in 2001, but it seems to me that Agile (the proper noun) is nothing more or less than :
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Where do sprints and iterations fit into this?
I think way too much has been made from the Agile ideal. Many things try to codify Agile into practice, Iterations are great, but why timebox them? Is timeboxing an Agile pattern? Then does a process like Scrumban fail out of being Agile? Does this differentiation help us as practitioners?
IMHO: an Agile organization is one where all the members agree to the Agile Manifesto and use it to help guide their methodology and process selection and optimization.
I use big-A Agile to talk about the codified and sclerotic form of Agile taught by cookie-cutter consultants. Somewhere there is a sacred text, an Agile Leviticus, containing ceremonial forms that people adhere to with religious faith. Even if the forms fail and are adjusted or abandoned at one company, they will be revived in their original form at the next company, citing success at the first company as proof of their power. Thus Agile never changes no matter how often it fails, which is why it deserves to be a capital-A proper noun.
Little-a agile is a characteristic of software development organizations that programmers and managers strive to achieve, pragmatically adopting the processes that help them achieve it.
The Agile Manifesto can be used in service of either agile software development or Agile the consulting religion.
I think this applies to any methodology, including waterfall. Another way to phrase this is "never stop observing, never stop acting on observations". Acting does not equate doing something, however. Even if you do nothing, and let the wagon roll on, you must know why you are doing nothing.
"if you have to give a fixed date for delivery - it's very, very difficult to be agile"
An agile project delivers continuously. So you can deliver every time, especially at a fixed date. The fact is, that if you have to be feature complete at a fixed date, you can't be agile. Hence, you can't be feature complete at a fixed date in an agile project.
"fixed date" in this case means "We have the following list of features we want in the product, to be launched as our big update. How many months will it take? (and we'll hold you to that exact date)"
The further ahead you must forecast, the less accurate your estimate will be. In many organizations, the hapless architect is asked to give a solid date for a laundry list of features, many of which he can't research very well up front. Many technologies exist to solve a number of the problems, but the experienced architect KNOWS that each and every one of those technologies will fail to work, be poorly documented, be poorly supported, contain subtle bugs that affect nobody else but him, be incompatible with existing infrastructure, etc.
There will be a great number of dead ends, which can only be discovered once the team has spent a good amount of time learning the technologies via documentation, web searches, object code disassembly, threatening vendors' families and pets, and so on.
And fixing these emerging problems will involve other technologies, each with their own problems, resulting in a factorial solution set.
This is why the traditional method of estimation has been to estimate how long you expect it to take with a lot of things going wrong, and then triple it (you triple it so that you can be "talked down" to double, even though that increases the risk of going over).
In short, long term project completion forecasts are a fairy tale, and so anyone who expects you to adhere to them makes agility near impossible.
1. developers find a new way to do things efficiently
2. its good
3. word spreads in the community. Method gets a nice name
4. consulting companies smell money. Develop training for big $. Win managements
5. management wants to be modern, but doesn't want to change key aspects, like number of developers, communication, carrier paths, hierarchies etc.
6. management requires: everybody has to be able to use this method. Name of the method is filled with different contents: easy to understand, fits well into every companies profile, works for every dumbass
7. method stops to work
8. consultants still sell it. Everybody has to do the sh....
I share your cynicism, but you're incorrect. The original XP crew was very much at steps 1 - 3. But when you try moving that stuff into a huge corporate space, there are big forces at work that (sadly) make steps 4+ almost inevitable.
This is so true. When I first read about Agile it was like an extraordinary breath of fresh air compared to the stifling spec-driven CYA corporate environment I was working in at the time. Now 95% of what is termed "Agile" seems to be as much pointless drivel as I ever saw in that environment.
It never ceases to amaze me how we take good ideas and screw them up. It also amazes me how many people are pissed about agile. I blogged about this a year or two ago, and got something like 50K visits (shameless plug for those who haven't read the article: http://www.whattofix.com/blog/archives/2010/09/agile-ruined-...? )
Just to be a bit of a contrarian, there's a difference between what something means on paper, how it's practiced, and what people think it means. We get into trouble when we confuse this.
A good article but I don't believe that the points presented in the article support the conclusion "why it never really works". "An alarming number of people... believe that Agile is a project management methodology, and that Agile really means SCRUM, XP, Kanban..." Why is it alarming for anyone to believe this? "I was once told by an Agile Trainer (LOL) that the correct way to phrase the requirement..." OK - so that doesn't make much sense to me. "Needless to say his company lost a $m project on the back of such BS" Companies can fail for lots of different reasons - and perhaps this particular Agile Trainer contributed and then again maybe not. That particular Agile Trainer may have been incompetent or misrepresented. The company may have failed for any number of different reasons. So I don't think that this example supports that conclusion. "Agile (big 'A' again) has become a bit of an albatross - it doesn't really work, it doesn't deliver the benefits it promised" I found that the article moved too quickly to this conclusion.
You could argue that that wasn't the main point of the article - but if that is the case then I would suggest that the title is a little too provoctive and the first few paragraphs are irrelevant. I enjoyed the article but I think that it would have been better if it simply didn't try to suggest that Agile never really works and just put forward what agile means to you. If the article was called "What agile means to me" and started from the line "agility [...] is a wonderful thing" it would have been great.
See my comment above - I didn't actually think that much about the article before submitting it - my fault. Re. your comment that the post moved too quickly to the conclusion that Agile doesn't work - you're right, I don't really provide much evidence of that - but I do stand by the statement, and the tile of the article - in my experience (most of which is not in this post) it doesn't - at least not in the manner in which it was sold in to the software community.
Your piece nailed it. I'm not sympathetic to mmeadows' caveats. They ignore the fact that Agile (aka XP, Scrum) has sold itself as a Magic Bullet (per Brooks) for a long time. Big-A Agile proponents need to chose between big claims and plausible excuses -- they can't have both.
True - I wrote the article more as a brain dump of a bunch of stuff I was thinking about, and submitted to HN myself following a discussion with someone else, and did it only to see what would happen - to be honest I didn't think anyone would read it.
Indeed, the submission title unfortunately omits the distinction between "Big-A Agile" and being agile. It's an omission that almost caused me not to click.
In my own experience, SCRUM is only as efficient as the scrum master and product owner. For the scrum master, it takes leadership to motivate the rest of the team and provide help when there are blockers. For the product owner, it takes a high level of organization to clearly define the sprints' tasks and meaning behind the tasks.
I think a good elaboration would be that for a project to build an Address Book is still a project to deliver an Address Book regardless of Agile or Waterfall or Scrumfall. You still have the same exact steps to execute, Agile just organizes the work differently than a Waterfall project, biting off a small piece of requirements, building the "vertical" of say what a contact form looks like. Waterfall would define the entire app first, hand off to development, etc. Both can fail for the same reasons though, poorly defined requirements, lack of funding, personnel issues, etc.
Not sure this is a good elaboration - if you know exactly what you want to build, then you're not agile. The point of being agile is that you start off building towards an address book, but along the way you discover that actually what your customers want is something slightly different - so you change the direction you're going in.
Agility allows you to follow your customers. If you have a fixed requirements - "we're building a product that does A,B,C and it looks like this" - then you're not agile, and as I point out in the post, agile is NOT a project management methodology. You can use SCRUM, XP, whatever - doesn't make you agile. (And it won't make delivery any faster, just more annoying, as everyone around you will assume it should be faster, and will constantly bang on about why nothing's happening yet.)
Hard to agree here. In my experience waterfall and agile projects fail for completely different reasons.
Waterfall projects fail when they are too rigid and try to define everything beforehand without leaving any space for iteration. The reality checks come too late when it's not easy/possible to change direction any more.
Agile projects fail when they are too loose and no one is actively taking responsibility for the bigger picture.
you need to produce requirements, make a design, create plans, implement code, test, deploy and all that -- it's the same if you're doing waterfall or agile, you're just sequencing these activities in time differently.
if you look at the project management body of knowledge (PMBOK) you find a comprehensive list of things that have to be done to make a project happen and this list isn't any different for agile or waterfall... it's just the answers for how certain things are done (communications in the team) for instance are different.
One advantage of agile is that all phases of project management are going on in each cycle, so you don't run into the problem that now you're testing after five years of doing everything else and now you don't have any idea how to do it.
On the other hand, you can screw everything up in an agile project the same way you can in a waterfall, although the symptoms show up differently. Since you're testing every phase of the process, problems will show up earlier. However, it's easy for people to miss them.
In an agile process, for instance, you can plan a sprint and accomplish the goals you set for the sprint and feel like you're doing a great job. However, the sprints might never end and the product never gets delivered. All your PM tools tell you're winning all the battles but you're still losing the war.
The key difference, IMO, is that true agile doesn't require overt project management or more importantly, project managers. the usual response to the "wining battles, losing war" is to introduce someone who's keeping track. unless that someone is "everyone on the team", you tend to move away from agile because:
a)its not everyone's problem, so its not going to be fixed, or
b)the pm guy becomes the fountain head of knowledge from the pmbok that guide the "one true way"
if a project goes beyond a certain size, somebody has to be in charge. you could build, say, the Dallas Cowboys stadium, without a project manager who heads a project management office.
if your project is small enough that "everybody is in charge" and you've got the consensus to work that way you're not talking agile, you're talking magic... and you're going to be 3-10x as productive as a conventional "team"
"Measure as much as you can - no feedback == no direction"
Can't agree more. We have built some internal applications where everything was implemented, the acceptance criterias were met and everyone was satisfied until they actually started using the system and said, ".. wait a minute, this doesnt help us as much as we had liked". But, we had already moved on to the next big thing.
"If I’d asked people what they wanted, they would have asked for a faster horse" - Henry Ford, http://bit.ly/kmLhvZ
Had a tdd openspace at codestock this weekend that morphed in to an agile discussion. Someone asked "what's the one key thing I need as a foundation to become Agile?" (paraphrasing that).
4 people all said "communication" at the same time.
Communication has always been the backbone of successful projects, in my view. In 16+ years of software dev, I've only had 2 situations where there were technical knowledge issues that were a huge barrier - issues like "how do I connect to a database?" being something someone on a team had massive trouble with.
Outside of those outliers, pretty much all other issues have come down to a communication issue of some type - not identifying things clearly enough, not raising the alarm when a roadblock was hit, not alerting a client in time, not getting adequate feedback from a client, not getting things in writing, etc. To the extent that "Agile" practices help encourage regular communication to avoid those issues, it's great. When "Agile" gets formalized in to such a state that it becomes a roadblock itself, that's where the problems start. Since "Agile" has been productized and sold over the last decade, I suspect it's becoming a stumbling block in many orgs, and we'll hopefully see a new iteration/generation of Agile practices which free up people again. Perhaps a stronger embrace of mobile tech in Agile practices would help?
Not to take all the good points away from this article, but I found these two statements contradictory:
1. If you do have a fixed time, then scope is variable
2. If you have to give a fixed date for delivery - it's very, very difficult to be agile.
I've found that #2 is possible as long as you allow for #1. When you have a deadline, you actually become more agile - decisions on what should and shouldn't be done are quicker and the cut off line between "must have"s and "nice to have"s is more obvious - simply because decisions have to be made. Think of moving out of an apartment: are you surprised how productive you get in those last few days?
I have also see misuse of #2: teams take #2 as carte blanche to hold the business at ransom to the "it'll be done when its done" mantra.
It almost seems like there's three kinds of agile nowadays:
1. Big A - the agile that the manifesto became for whatever reasons
2. Small a - the agile that the purists conjure up in reaction to Big A
3. Reality - where when it happens, you can look at each other and say "We were really agile right there - and shipped something right!"
In my opinion, we developers are trying too hard to come up with rules on project management, and it just doesn't work.
The reason agile works is because capable developers are given freedom to make good decisions based on the situation; and it fails when decisions are made by someone else -- be it incapable developers, sales person, management or even another capable developer who is not quite involved in the project.
Making up rules just gives a false sense of confidence to those "someone else".
Strikes me that many sorts of methodologies and approaches need a reminder about this. The point of agility or any other engineering methodology is to get better at what you're doing, not a slavish adherence to a set of Ten Commandments without thinking about what they really mean for you and the folks with whom you work.
I'll leave the obvious religious corollaries as an exercise for the reader.
The key tenant to agile is built in 'tolerance' for change in resource and scope with emphasis on pipeline management instead of scheduling.
The reason why classic forms of development fail is because these issues (changes) were the exception instead of the rule in methodologies leading up to the 'agile' generation
From my (little) experience and reading I've done so far, if I had to sum up what 'agile' is, I'd say: 'feedback'.
I'm not surprised that 'waterfall' methodologies fail - they seem to have little to no feedback built in. And for what I remember from Control Theory course, feedback is essential to control a system we don't have full knowledge about. I do believe that it's a more general principle than just about differential equations and integration.
>And for what I remember from Control Theory course, feedback is essential to control a system we don't have full knowledge about.
it works both ways - incorrectly designed feedback control loop (remember the classical example of overly sensitive steam engine control?) may make the system less stable
> The best way to achieve your goal is to start with the right people
That is pie in the sky. Everyone says it and it's bogus for most people/companies/projects. 90% of the time you have the people you're gonna have. You need to know and make the best of them.
This is so true. Doesn't mean the idea is not important, just that, like everything, there are compromises.
More generally, I find that an uncharacteristic proportion of blog entries are written from the perspective of "change the world" efforts. Not everything is like that. I agree with you njharman, most of the time, there are certain things, like the team, that you can't do a whole lot about. The real skill is getting the most out of those situations. If it is a true "change the world" effort then sure, before you join the project demand that everyone else be let go and you hire the team from scratch. <tongue in cheek>
You make an important point. However I think there is value in thinking about the "right people" also in fixed organizations, especially larger ones.
If you have tens or hundreds of developers, you will have various kinds of tasks. Some people feel more committed to certain kinds of tasks, so matching these will certainly make your organization more efficient. Sadly, this fact is often overlooked by work-dispatcher who might be under the illusion that for developers it does not matter what they are working with or how long they are committed to a certain area.
no doubt, agile is not the "answer" to traditional workflow models, but my (admittedly somewhat limited) experience with it has been pretty good. it's really dependent on the people you work with (something the author states).
coding with people you know, and understand (and perhaps hence predict) really makes production fast. however, the "tight-nit"-ness of a team like that, means that it doesn't scale so well in ways that businesses like, i.e, throw more money/resources at it, and it gets better/faster.
mind you, i think the real lesson to be taken away, is that, when there is a problem with something like this, there is no "correct" solution, which works in all cases, for everyone.
I've had a lot of trouble with agile projects, though I do think that when it works, it's by far the best way to create working software.
My biggest trouble with agile happened when I was dealing with a client who clearly intended to hold my feet to the fire where it came to promised functionality and deadlines, but was absolutely unwilling to consider an alternative approach to a feature or removing functionality to meet an iteration.
"Customer collaboration over contract negotiation" is the best way to write good software, but if your client is going to go to your boss and complain that say "hey you promised functionality X by date Y!", you better be careful negotiating that contract!
"Responding to change over following a plan" is the best way to write software, but if your client is going to rake you over the coals if you don't meet the deadline that was in. the. plan., well then you're probably better off just following the plan.
Really, agile is a highly effective way of working that requires a tremendous amount of skill and mutual respect (and trust) between all members of a team.
Another problem with agile is that just so much has been built up around it. I'm not saying that these things are bad, but they have diluted the original message.
I remember once a consultant ambushed me in a meeting and asked "what are your developer velocities?" I didn't know what that meant, and he said (I think he was setting this up) "well, if you don't know your velocities, you aren't agile."
For the record, a developer velocity is (as this dude explained it) a way of measuring whether a developer is struggling based on how much of a user "story" is being developed over a particular period of time. It's not a bad way to measure things, but this is a process, a tool, a procedure. As the agile manifesto says, we value these things, but it seems like "agile" has less and less to do with the simple and profound (I mean that without any irony) statements and is starting to become a big mess of consultant and techno babble.
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
To me, this is the best way to write software, but it isn't something you can just decide to do. It requires a really major mind shift from everyone. And if it turns out your client values the things on the things on the right more than the things on the left after all,you can find yourself in a very precarious situation as a dev.
You list the following as your "biggest trouble" with agile-
"hey you promised functionality X by date Y"
But that's not an agile contract! So, your biggest trouble with agile is being contractually prohibited from doing it, which is more a problem with the contract than with agile.
Also, the consultant dude explained velocity wrong. Velocity is how you measure the amount of work your team got done this time so you can plan how much you are going to do next time...
It's very similar to how you might optimize a program. You find a hotspot you think might need improvement and you spend a portion of time improving it. Measure results before and after. Rinse and repeat.
I'm sure there are a lot of small things they learned along the way but the simple idea of continuously improving their processes has kept them on top.
Truly enlightened Agile development is not doing Scrum, XP, etc., it is being wise enough to look at your process and see how it can be improved. Pick and choose different ideas that will fix the parts that need to be fixed, but never be fully satisfied with what you are doing.
The real failure in any project management philosophy is believing that it is the "One True Way" that all things must be done for all teams everywhere all the time.