Unfortunately, that doesn't change the fact that customers want estimates, and the more complex the project the more they need it for budgeting and scheduling purposes.
In the end, you have a few options for estimating:
1. Estimate the most optimistic scenario, knowing you'll end up using change requests and extensions to deliver (one of the worst options, but if you're responding to a proposal that's what the opposition is assuredly going to be doing);
2. Propose an initial phase to nail down requirements and deliver a schedule using traditional estimation tools (an easier sell if you're an internal team, tough if you're a vendor);
3. Use experience with similar projects to estimate, and then refine your assumptions with the client, using change orders to account for unexpected problems (roughly accurate to the degree of imported ambiguity, but now you have to risk going to war to get your change orders signed off on);
4. Use a top-down estimation model such as function points-based estimation to get in the ballpark, and then add buffer (and pray you aren't going to either lose money or blow the top off the client's budget levels -- or both);
5. Delphi the problem and get a bunch of developers and PMs to estimate the effort, then refine from there (useful for sanity checks, not useful as a real estimation process, yet one of the more common ways small shops build their numbers).
As you say, you can't make risk go away, you can only try to segregate it within your estimation. In my experience, clients really need stability in proposal numbers -- they're often willing to accept a significant embedded risk buffer in a proposal if it means they can limit unexpected costs. Conversely, you should also have a rough idea going into a project of what functionality can be cut and when so you can go/nogo those pieces when necessary.
> ...they're often willing to accept a significant embedded risk buffer in a proposal if it means they can limit unexpected costs.
So if they want 95% confidence in estimates, you need to pad the estimate a lot. It doesn't make risk go away. It just offloads risk onto the contractor.
That just means the contractor is assuming the risk of missing the 95% estimate. It also means the head of the contracting operation needs to be diligent about backing up the estimates of his employees. Or mitigating risk some other way (making sure each contract is a small fraction of the business, having enough cash to absorb risk, owning lots of other businesses, etc.).
Now, realistically, the risk is also absorbed by the developers in the form of unpaid overtime, stress, lost bonuses, and in other ways. That's just all the more reason for individual contributors to insist on honest estimates. Or find bosses that understand business models better and don't offload risk onto employees.
[] I understand that this is complicated and bosses don't typically mean to make employees miss bonuses and have terrible work/life balance. But to some degree it doesn't matter. Intentions don't make down payments on houses, make sure kids do their homework, or get us a full night of sleep.
In the end, you have a few options for estimating:
1. Estimate the most optimistic scenario, knowing you'll end up using change requests and extensions to deliver (one of the worst options, but if you're responding to a proposal that's what the opposition is assuredly going to be doing);
2. Propose an initial phase to nail down requirements and deliver a schedule using traditional estimation tools (an easier sell if you're an internal team, tough if you're a vendor);
3. Use experience with similar projects to estimate, and then refine your assumptions with the client, using change orders to account for unexpected problems (roughly accurate to the degree of imported ambiguity, but now you have to risk going to war to get your change orders signed off on);
4. Use a top-down estimation model such as function points-based estimation to get in the ballpark, and then add buffer (and pray you aren't going to either lose money or blow the top off the client's budget levels -- or both);
5. Delphi the problem and get a bunch of developers and PMs to estimate the effort, then refine from there (useful for sanity checks, not useful as a real estimation process, yet one of the more common ways small shops build their numbers).
As you say, you can't make risk go away, you can only try to segregate it within your estimation. In my experience, clients really need stability in proposal numbers -- they're often willing to accept a significant embedded risk buffer in a proposal if it means they can limit unexpected costs. Conversely, you should also have a rough idea going into a project of what functionality can be cut and when so you can go/nogo those pieces when necessary.