I've never experienced estimation improving over time in an Agile process. In fact, since short estimation cycles are just as susceptible to political games as longer cycles, and this dominates the whole process in any organization, the length of the estimation cycle really has nothing to do with the estimation issue.
But, what shorter estimation cycles are responsible for is a decoupling of the overall big picture progress from the immediate tasks being prioritized. This often leads to sprints that are well scoped, and everything in the sprint gets completed close to on time (at least as close as in longer work cycles, but not usually any closer), but, crucially all of the work has to be scrapped because the whole sprint, conceptually, was not right. In my experience, this happens maybe 1/5 of the time in Agile exactly in ways that would be prevented with longer-term planning.
It's very similar to the classical fractal effects of measuring the coast of Britain. By refining your ruler, you merely think you're being more accurate, and a lot of people are doing a lot of performative managerial crap for the sake of the Scrum performance, like a ritual. It feels like you are estimating better, but really because the whole meta-level concept you are working on isn't guided by longer-term thinking, you end up having so much round off error, added up over more and more cycles, that the overall waste is colossally greater than systems that involve some more significant longer-term pre-planning and which work on variable cycle lengths instead of a cookie-cutter, one-size-fits-all approach (which is the core philosophy of Agile, though people try to obfuscate this fact by No True Scotsman-ing it with the Agile Principles).
All excellent points. But having done big waterfall, bad corporate agile, and good agile, the problem of work failing conceptually is common to all of them. The question is, how do you recover from it? I think good agile recovers more gracefully from conceptual failures than either waterfall (which tends to throw good money after bad, because admitting failure is not an option), or corporate agile (because corporate agile is usually just buzzword-compliant waterfall).
I'm not saying waterfall is a good alternative. Waterfall is just as shitty as Agile, but one benefit is that wastes less of everyone's time with meetings and busywork. But waterfall is equally guilty of a one-size-fits-all mentality, from the opposite view of Agile, and this cookie-cutter property is what makes both methods attractive to bureaucracies.
I'm a fan of common sense. If it's clear in a given team and project context that two week sprints, story points, and frequent meetings are really going to help, then just do it. And when that stops working, just switch and do something else. When architects or researchers are telling you a given project direction needs to slow down for some more intensive piloting research, do it and don't hesitate to violate sacred Scrum principles.
This stuff should be figured out organically, based on the given project and given personnel. Never with a fixed mandate to a single prescribed method or time frame.
Continuous improvement of the process itself is part of any good agile process. Corporate buzzword compliance agile is often guilty of not doing that - the "cookie-cutter property".
But importantly, I've found that a refined agile process saves more work than it costs, by figuring out what doesn't need done before doing it, and prioritizing immediate value over future value. It's very important to be able to say "This is good enough for now, but we know it's not good enough for the future". It keeps the perfect from being the enemy of the good. Likewise, it's important to be able to say "Well, I guess that didn't work", and have it be an integral part of the process. Without process, you have no guidance over what wrong turns you took and what those wrong turns cost.
I dunno, what you say just reads like a bunch of buzzwords that Agile, even supposedly "refined" or "good" versions of Agile, never prioritizes or delivers. More often, when quality work is produced it's produced in spite of Agile, not because of it, which is a shame because then people falsely attribute that success back to Agile.
I admit though that it's hard to know exactly what you mean just exchanging textual comments like this, so I'm happy to leave it at that.
But, what shorter estimation cycles are responsible for is a decoupling of the overall big picture progress from the immediate tasks being prioritized. This often leads to sprints that are well scoped, and everything in the sprint gets completed close to on time (at least as close as in longer work cycles, but not usually any closer), but, crucially all of the work has to be scrapped because the whole sprint, conceptually, was not right. In my experience, this happens maybe 1/5 of the time in Agile exactly in ways that would be prevented with longer-term planning.
It's very similar to the classical fractal effects of measuring the coast of Britain. By refining your ruler, you merely think you're being more accurate, and a lot of people are doing a lot of performative managerial crap for the sake of the Scrum performance, like a ritual. It feels like you are estimating better, but really because the whole meta-level concept you are working on isn't guided by longer-term thinking, you end up having so much round off error, added up over more and more cycles, that the overall waste is colossally greater than systems that involve some more significant longer-term pre-planning and which work on variable cycle lengths instead of a cookie-cutter, one-size-fits-all approach (which is the core philosophy of Agile, though people try to obfuscate this fact by No True Scotsman-ing it with the Agile Principles).
I also can't miss the opportunity to point out this great article on the political games we play during estimation: < http://research.cs.queensu.ca/home/ahmed/home/teaching/CISC3... >