> The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it.
That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be able to deliver an estimate, and the best approach to take is to provide an estimate with Quantified Uncertainty. The phrase is important, because the estimate is for managers and managers are all about quantifying risks, and uncertainty in a development estimate is definitely a risk.
When you say "I don't know how long this will take, because I don't know how much I don't know about the solution" you're giving an estimate with uncertainty, but you're not quantifying the uncertainty. It's got no bounds, it's limitless. Managers can't make any decisions with that, so it's useless information. But if you say "This will take 4 to 8 effort weeks, and I'll be able to give a more precise estimate after 2 effort weeks of investigation" that gives your manager something to work with. You're still uncertain, but you've quantified the bounds on the uncertainty, and decisions can be made. For example, solving the problem might not be worthwhile unless it happens within the next two weeks (eg: to meet a release deadline) so the manager can just delay the work until later.
The trick with this approach is setting the bounds. The man who taught me this style of estimating called it a Surprise Range. The lower bound should be an estimate such that you'd be surprised if it took less effort than that, and the upper bound should be an estimate such that you'd be surprised if it took more effort than that. Given whatever information you have about the requirements, you just keep pushing your bounds until you can honestly say "I really don't think it would take less/more effort than that."
As you get more experienced, you'll naturally start accounting for the unknown unknowns, because your past experiences with them will be nagging in the back of your mind while you're trying to come up with an estimate for some new task. This is especially true if you're estimating work on a project you've worked on before; you'll have a feel for where the trouble spots are and whether or not the task in question is going to get into those spots.
The other trick with this approach is the "more precise estimate after some investigation" bit of the statement. You need to provide some estimate now, buy some investigation time, and commit to another estimate later. Most of the time the second estimate will fall somewhere within the bounds of the first one, but not always, and your manager should be aware of that possibility. But with experience, you'll usually wind up within the original bounds and narrower than the original bounds too, which is quantifiably less uncertainty than before so you can show progress.
Company wanted to port an OS to 32-bit (yes I know that dates me). I spent a day, reviewed the code (500 modules), wrote a report - two manyears. Project deferred (ashcanned).
A month later my peer, a senior Engineer with the Company, thought "We should run 32-bit! How long can it take?", checked out the source and started changing files. Gradually people noticed what he was doing, resources got added under the table, finally a year later he had something working. Lots of kudos, smiles, what a hero!
I showed him the report I filed a year ago - and he said "yeah, that's about what I had to do. I'm glad I didn't see that to begin with, I would never have started".
Two man-years can easily become one calendar year, if 2-3 people are working fulltime on the project.
Also, "two man-years" isn't a complete estimate using my approach. "One to two man-years" would be a complete estimate, providing a range. Sometimes you really do come in near the bottom of the range, and maybe a month into it if you re-estimated you would have come up with a lower range.
This is a reasonable method, but I think it's worth remembering that you have not actually "quantified the bounds on the uncertainty." What you have done is quantified what you think are the bounds on the uncertainty.
Of course you're right, but part of the nature of uncertainty is that you can't know the bounds for sure. That's why it's an estimate rather than a commitment.
This brings up another point of contention between development and project management. When developers say "estimate", they mean "best guess in the face of uncertainty", but project managers tend think of developer estimates the same as plumber, electrician, and car mechanic estimates. Those guys are saying "I'm not sure exactly how much work I'm going to have to do, but if you pay me $X I'll do it." They inflate $X so that most of the time they wind up overcharging in order to cover the occasions when they wind up doing much more work than expected. The $X is a commitment, and the estimate is how much work is needed.
It doesn't work this way in software development, because there are factors out of the developers control. The developer can say how much effort a task is likely to take (the ranged estimate) but not how much time (the commitment.) The time depends on other things the developer has to work on, both expected and unexpected, dependencies on other developers and tasks, holidays and days off for the developer and others the developer is dependent upon, etc. That's all stuff the project manager is supposed to keep track of, so the developer doesn't have a full picture. Unfortunately, most project managers don't recognize this, so they treat the effort estimate as a time commitment, set an arbitrary start date, and expect the developer to be done X days later.
That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be able to deliver an estimate, and the best approach to take is to provide an estimate with Quantified Uncertainty. The phrase is important, because the estimate is for managers and managers are all about quantifying risks, and uncertainty in a development estimate is definitely a risk.
When you say "I don't know how long this will take, because I don't know how much I don't know about the solution" you're giving an estimate with uncertainty, but you're not quantifying the uncertainty. It's got no bounds, it's limitless. Managers can't make any decisions with that, so it's useless information. But if you say "This will take 4 to 8 effort weeks, and I'll be able to give a more precise estimate after 2 effort weeks of investigation" that gives your manager something to work with. You're still uncertain, but you've quantified the bounds on the uncertainty, and decisions can be made. For example, solving the problem might not be worthwhile unless it happens within the next two weeks (eg: to meet a release deadline) so the manager can just delay the work until later.
The trick with this approach is setting the bounds. The man who taught me this style of estimating called it a Surprise Range. The lower bound should be an estimate such that you'd be surprised if it took less effort than that, and the upper bound should be an estimate such that you'd be surprised if it took more effort than that. Given whatever information you have about the requirements, you just keep pushing your bounds until you can honestly say "I really don't think it would take less/more effort than that."
As you get more experienced, you'll naturally start accounting for the unknown unknowns, because your past experiences with them will be nagging in the back of your mind while you're trying to come up with an estimate for some new task. This is especially true if you're estimating work on a project you've worked on before; you'll have a feel for where the trouble spots are and whether or not the task in question is going to get into those spots.
The other trick with this approach is the "more precise estimate after some investigation" bit of the statement. You need to provide some estimate now, buy some investigation time, and commit to another estimate later. Most of the time the second estimate will fall somewhere within the bounds of the first one, but not always, and your manager should be aware of that possibility. But with experience, you'll usually wind up within the original bounds and narrower than the original bounds too, which is quantifiably less uncertainty than before so you can show progress.